From peridot.faceted at gmail.com  Sun Jun  1 03:37:29 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sun, 1 Jun 2008 03:37:29 -0400
Subject: [Numpy-discussion] Slice assignment and overlapping views (was:
	Strange behavior in setting masked array values in Numpy 1.1.0)
In-Reply-To: <1212271760.8410.45.camel@localhost.localdomain>
References: <6F7AB06B-DB3A-4DFE-80F3-0BF6C32F75E2@gmail.com>
	<1212271760.8410.45.camel@localhost.localdomain>
Message-ID: <ce557a360806010037k242cd040rd6c370e70eac8a35@mail.gmail.com>

2008/5/31 Pauli Virtanen <pav at iki.fi>:

> The reason for the strange behavior of slice assignment is that when the
> left and right sides in a slice assignment are overlapping views of the
> same array, the result is currently effectively undefined. Same is true
> for ndarrays:
>
>>>> import numpy
>>>> a = numpy.array([1, 2, 3, 4, 5])
>>>> a[::-1]
> array([5, 4, 3, 2, 1])
>>>> a[:] = a[::-1]
>>>> a
> array([5, 4, 3, 4, 5])

I think that the current rule is, slices are walked from low index to
high index. This doesn't help with multidimensional arrays, where the
order of the axes is (and should be) determined by efficiency
considerations.

Unfortunately there's really no good way to warn about overlapping
copies. Remember that this is a frequent operation, so it has to be
fast for small arrays.

I think changing base so that it points to the real base and not the
parent would help (and clear up a memory leak: try "while True: A =
A[::-1]" some time) eliminate some cases where overlap cannot occur,
but what about the following cases?

A[:5] = A[-5:]
A[::2] = A[1::2]
A[1:] = A[-1:]

The last is actually fairly common (I've needed it), and relies on
numpy's ordering of copies. The middle one is very common, and the
first one would be a royal pain to code around if the slices were not
allowed to overlap.

I think I at one point wrote an algorithm that would detect many cases
where overlap could not occur (maybe in the smarter reshape code?) but
I came to the conclusion that detecting all the cases was infeasible.
It's a number-theoretic problem - "can you write the same number as
the sum of multiples of these two lists of numbers with nonnegative
integer coefficients less than these other two lists of numbers?" -
and I suspect it's NP-complete. (Ah, yes, you can make a knapsack out
of it - take an array with strides a_1, ... a_n and shape (2,...,2),
and a single-element array starting at position N.) In any case it's
infeasible to solve it every time somebody tries to do a slice
assignment.

In any case, many users need nearly-overlapping slices, and some need
really-overlapping slices. Preventing problems is going to have to
happen at a higher level.

Anne


From pav at iki.fi  Sun Jun  1 04:29:15 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 01 Jun 2008 11:29:15 +0300
Subject: [Numpy-discussion] Slice assignment and overlapping views	(was:
	Strange behavior in setting masked array values in Numpy 1.1.0)
In-Reply-To: <ce557a360806010037k242cd040rd6c370e70eac8a35@mail.gmail.com>
References: <6F7AB06B-DB3A-4DFE-80F3-0BF6C32F75E2@gmail.com>
	<1212271760.8410.45.camel@localhost.localdomain>
	<ce557a360806010037k242cd040rd6c370e70eac8a35@mail.gmail.com>
Message-ID: <1212308955.8078.25.camel@localhost.localdomain>

su, 2008-06-01 kello 03:37 -0400, Anne Archibald kirjoitti:
> 2008/5/31 Pauli Virtanen <pav at iki.fi>:
> 
> > The reason for the strange behavior of slice assignment is that when
> the
> > left and right sides in a slice assignment are overlapping views of
> the
> > same array, the result is currently effectively undefined. Same is
> true
> > for ndarrays:
> >
> >>>> import numpy
> >>>> a = numpy.array([1, 2, 3, 4, 5])
> >>>> a[::-1]
> > array([5, 4, 3, 2, 1])
> >>>> a[:] = a[::-1]
> >>>> a
> > array([5, 4, 3, 4, 5])
> 
> I think that the current rule is, slices are walked from low index to
> high index. This doesn't help with multidimensional arrays, where the
> order of the axes is (and should be) determined by efficiency
> considerations.

Yes, I figured that it is not possible to do what the user usually means
in all cases without using at least one scalar temporary.

[clip]
> Unfortunately there's really no good way to warn about overlapping
> copies. Remember that this is a frequent operation, so it has to be
> fast for small arrays.

?I didn't actually mean detecting overlaps, but detecting when assigning
to a view:

        while (a->base) a = a->base;
        while (b->base) b = b->base;
        if (a == b) {
            raise warning, but continue
        }

The leading loops can be removed if the bases are walked upwards on
array creation, and in this case an added single branch cannot be so bad
of an performance hit. If the branch is taken, then there's a
performance hit of course.

> I think changing base so that it points to the real base and not the
> parent would help (and clear up a memory leak: try "while True: A =
> A[::-1]" some time) eliminate some cases where overlap cannot occur,
> but what about the following cases?
> 
> A[:5] = A[-5:]
> A[::2] = A[1::2]
> A[1:] = A[-1:]
> 
> The last is actually fairly common (I've needed it), and relies on
> numpy's ordering of copies. The middle one is very common, and the
> first one would be a royal pain to code around if the slices were not
> allowed to overlap.

?But maybe the warning is still too much hand-holding, and we can just
add a warning about undefined behavior in the documentation and leave it
at that. I can't argue much against your examples.

If the behavior in 1d for overlapping parts is well-defined and relied
upon, we should add a unit test that ensures it, if there isn't one yet
(didn't check yet).

[clip]
> In any case, many users need nearly-overlapping slices, and some need
> really-overlapping slices. Preventing problems is going to have to
> happen at a higher level.

"Higher level" means "Documentation" here, right?

	Pauli




From robert.kern at gmail.com  Sun Jun  1 04:49:54 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 1 Jun 2008 03:49:54 -0500
Subject: [Numpy-discussion] What does "Ignoring attempt to set 'name'
	(from ... " mean ?
In-Reply-To: <483E9A7B.7060405@ar.media.kyoto-u.ac.jp>
References: <483D227B.9030902@ar.media.kyoto-u.ac.jp>
	<3d375d730805281202v1dcbaef5ocb4f1c45f3ea5fde@mail.gmail.com>
	<5b8d13220805281803n1212f8a4hd8d5e540c9ef9e0e@mail.gmail.com>
	<3d375d730805281816j56ca4500v85ece93c501db218@mail.gmail.com>
	<483E9A7B.7060405@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730806010149u36a79988w4e1328f187a00f8c@mail.gmail.com>

On Thu, May 29, 2008 at 6:58 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Robert Kern wrote:
>>
>> They're fine. Ignore them. They are silenced from the main setup.py with
>>
>>     config.set_options(quiet=True)
>>
> What are the cases where those message are meaningful ? I did not
> understand from the distutils code what kind of issues were related to
> this message,

There probably aren't any significant ones in production. It was
probably useful during development of the recursive setup.py feature.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From gav451 at gmail.com  Sun Jun  1 08:11:58 2008
From: gav451 at gmail.com (Gerard Vermeulen)
Date: Sun, 1 Jun 2008 14:11:58 +0200
Subject: [Numpy-discussion] ANN: PyQwt-5.1.0 released
Message-ID: <20080601141158.72b4e916@jupiter.rozan.fr>

What is PyQwt ( http://pyqwt.sourceforge.net ) ?

- it is a set of Python bindings for the Qwt C++ class library which
  extends the Qt framework with widgets for scientific and engineering
  applications.   It provides a 2-dimensional plotting widget and
  various widgets to display and control bounded or unbounded floating
  point values.

- it requires and extends PyQt, a set of Python bindings for Qt.

- it supports the use of PyQt, Qt, Qwt, and optionally NumPy or SciPy
  in a GUI Python application or in an interactive Python session.

- it runs on POSIX, Mac OS X and Windows platforms (practically any
  platform supported by Qt and Python).

- it plots fast: fairly good hardware allows a rate of 100,000
points/second. (PyQwt with Qt-3 is faster than with Qt-4).

- it is licensed under the GPL with an exception to allow dynamic
linking with non-free releases of Qt and PyQt. 


The most important new features of PyQwt-5.1.0 are:
- support for Qwt-5.1.0.
- support for PyQt-4.4.2 and SIP-4.7.6.
- support for Qt-4.4.
- the CartesianDemo.py, MaskedDataDemo.py, PickerDemo.py examples.

The most important bug fixes in PyQwt-5.1.0 are:
- fixed QwtPicker::stateMachine() to allow for subclassing of QwtPicker
in Python.

PyQwt-5.1.0 supports:
1. Python-2.5, or -2.4. 
2. PyQt-3.17.
3. PyQt-4.4, PyQt-4.3, or PyQt-4.2.
3  SIP-4.7, or SIP-4.6.
4. Qt-3.3, or Qt-3.2.
5. Qt-4.4, Qt-4.3, or Qt-4.2.
6. Qwt-5.1, or Qwt-5.0.
7. Recent versions of NumPy, numarray, and/or Numeric.


Enjoy -- Gerard Vermeulen


From charlesr.harris at gmail.com  Sun Jun  1 11:08:06 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 1 Jun 2008 09:08:06 -0600
Subject: [Numpy-discussion] Slice assignment and overlapping views (was:
	Strange behavior in setting masked array values in Numpy 1.1.0)
In-Reply-To: <ce557a360806010037k242cd040rd6c370e70eac8a35@mail.gmail.com>
References: <6F7AB06B-DB3A-4DFE-80F3-0BF6C32F75E2@gmail.com>
	<1212271760.8410.45.camel@localhost.localdomain>
	<ce557a360806010037k242cd040rd6c370e70eac8a35@mail.gmail.com>
Message-ID: <e06186140806010808x1b3b66e2x1c96d8554ac1637e@mail.gmail.com>

On Sun, Jun 1, 2008 at 1:37 AM, Anne Archibald <peridot.faceted at gmail.com>
wrote:

> 2008/5/31 Pauli Virtanen <pav at iki.fi>:
>
> > The reason for the strange behavior of slice assignment is that when the
> > left and right sides in a slice assignment are overlapping views of the
> > same array, the result is currently effectively undefined. Same is true
> > for ndarrays:
> >
> >>>> import numpy
> >>>> a = numpy.array([1, 2, 3, 4, 5])
> >>>> a[::-1]
> > array([5, 4, 3, 2, 1])
> >>>> a[:] = a[::-1]
> >>>> a
> > array([5, 4, 3, 4, 5])
>
> I think that the current rule is, slices are walked from low index to
> high index. This doesn't help with multidimensional arrays, where the
> order of the axes is (and should be) determined by efficiency
> considerations.
>
> Unfortunately there's really no good way to warn about overlapping
> copies. Remember that this is a frequent operation, so it has to be
> fast for small arrays.
>
> I think changing base so that it points to the real base and not the
> parent would help (and clear up a memory leak: try "while True: A =
> A[::-1]" some time) eliminate some cases where overlap cannot occur,
> but what about the following cases?
>
> A[:5] = A[-5:]
> A[::2] = A[1::2]
> A[1:] = A[-1:]
>
> The last is actually fairly common (I've needed it), and relies on
> numpy's ordering of copies. The middle one is very common, and the
> first one would be a royal pain to code around if the slices were not
> allowed to overlap.
>
> I think I at one point wrote an algorithm that would detect many cases
> where overlap could not occur (maybe in the smarter reshape code?) but
> I came to the conclusion that detecting all the cases was infeasible.
> It's a number-theoretic problem - "can you write the same number as
> the sum of multiples of these two lists of numbers with nonnegative
> integer coefficients less than these other two lists of numbers?" -
> and I suspect it's NP-complete. (Ah, yes, you can make a knapsack out
> of it - take an array with strides a_1, ... a_n and shape (2,...,2),
> and a single-element array starting at position N.) In any case it's
> infeasible to solve it every time somebody tries to do a slice
> assignment.
>

Since one could compute all the addresses and check for duplicates, I would
guess it is O(n), maybe O(n*log(n)) worst case, but I can't convince myself
that it would help much to know about overlaps. As you point out,  a lot of
times one needs overlapping ranges, it is when the user has designed an
algorithm without taking the possibility into account that problems arise.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080601/2765c6cc/attachment.html>

From tsyu80 at gmail.com  Sun Jun  1 11:22:05 2008
From: tsyu80 at gmail.com (Tony Yu)
Date: Sun, 1 Jun 2008 11:22:05 -0400
Subject: [Numpy-discussion] array += masked_array gives a normal array,
	not masked array
Message-ID: <57363A4A-EF8E-4748-8C9E-D8A497D359E7@gmail.com>

Ok, so you guys shot down my last attempt at finding a bug :). Here's  
another attempt.

      array + masked_array

outputs a masked array

      array += masked_array

outputs an array.

I'm actually not sure if this is a bug (works the same for both the  
old and new masked arrays), but I thought this was unexpected.

*However*, if the right side were an old masked array, the masked  
values would not be added to the output array. With the new masked  
arrays, the data from the masked array is added regardless of whether  
the value is masked (example, below).

-Tony

Example:
=======

In [1]: import numpy

In [2]: normal = numpy.array([1, 1])

In [3]: masked = numpy.ma.array([1, 1], mask=[True, False])

In [4]: new_masked = normal + masked

In [5]: new_masked

Out[5]:
masked_array(data = [-- 2],
       mask = [ True False],
       fill_value=999999)


In [6]: normal += masked

In [7]: normal

Out[7]: array([2, 2])

# If used old masked arrays in the above, the final output would be:  
array([1, 2])



From pgmdevlist at gmail.com  Sun Jun  1 21:28:04 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 1 Jun 2008 21:28:04 -0400
Subject: [Numpy-discussion] array += masked_array gives a normal array,
	not masked array
In-Reply-To: <57363A4A-EF8E-4748-8C9E-D8A497D359E7@gmail.com>
References: <57363A4A-EF8E-4748-8C9E-D8A497D359E7@gmail.com>
Message-ID: <200806012128.05765.pgmdevlist@gmail.com>

On Sunday 01 June 2008 11:22:05 Tony Yu wrote:
> array + masked_array
> outputs a masked array

That's expected: you create a new object from two objects (array and 
masked_array), the latter has a higher priority than the former, you end up 
with a masked array

> array += masked_array
> outputs an array.

That's expected as well: you modify the values of array in place, but don't 
modify its type. Similarly,
masked_array += array
gives a masked array.

> *However*, if the right side were an old masked array, the masked
> values would not be added to the output array. With the new masked
> arrays, the data from the masked array is added regardless of whether
> the value is masked (example, below).

With oldnumeric.ma, masked_array has to be converted to a ndarray. During the 
conversion, the masked data are filled with a value that depends on the 
context of the call: here, the function called is add, and the corresponding 
filling value is 0. Your line 
normal += masked_array
is then equivalent to
normal += [0,1]

With numpy.ma, masked_array is already recognized as a ndarray, and no 
conversion takes place. In particular, the values of masked_array are not 
filled.

You should use
normal += masked_array.filled(0)



From bala.biophysics at gmail.com  Mon Jun  2 01:19:28 2008
From: bala.biophysics at gmail.com (Bala subramanian)
Date: Mon, 2 Jun 2008 10:49:28 +0530
Subject: [Numpy-discussion] numpy import problem
In-Reply-To: <1d36917a0805310815o695605aepcbbbedd8ebcec383@mail.gmail.com>
References: <288df32a0805310752y76cee520oa1d5379e854d9c94@mail.gmail.com>
	<1d36917a0805310815o695605aepcbbbedd8ebcec383@mail.gmail.com>
Message-ID: <288df32a0806012219h2c474ae2sf79a1c4991fc2c9b@mail.gmail.com>

Dear Alan,
I am using python 2.4.3 that came pre-installed with my linux. I dnt have
any other version. But i noticed another problem. In fact no other modules
are getting imported now. I have pasted the error below. Suggest me what i
should do now.

$ python
'import site' failed; use -v for traceback
Python 2.4.3 (#1, Aug  8 2006, 18:51:20)
[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-20)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
ImportError: No module named numpy
>>> import string
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
ImportError: No module named string



On Sat, May 31, 2008 at 8:45 PM, Alan McIntyre <alan.mcintyre at gmail.com>
wrote:

> Bala,
>
> One thing I can think of is that you might have multiple versions of
> Python installed.  For example, I have Python 2.4 and 2.5 on my
> machine, but numpy is only installed for 2.5.   Since just running
> "python" brings up 2.4, sometimes I find myself in the wrong
> interpreter typing "import numpy", which fails in the way you
> described.
>
> Cheers,
> Alan
>
> On Sat, May 31, 2008 at 10:52 AM, Bala subramanian
> <bala.biophysics at gmail.com> wrote:
> > Dear friends,
> >
> > I installed numpy in a 32-bit machines running with RHEL3. The
> installation
> > was successful. I tested the installtion by importint numpy inside python
> > interpreter.
> > By once i shutdown the system and restart, and try the same, it says
> > ImportError: No module named numpy.
> >
> > What could be the problem i dnt get. Kindly someone write me on the same.
> >
> > Thanks,
> > Bala
> >
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080602/e15cc8b7/attachment.html>

From cournapeau at cslab.kecl.ntt.co.jp  Mon Jun  2 01:34:12 2008
From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau)
Date: Mon, 02 Jun 2008 14:34:12 +0900
Subject: [Numpy-discussion] numpy import problem
In-Reply-To: <288df32a0806012219h2c474ae2sf79a1c4991fc2c9b@mail.gmail.com>
References: <288df32a0805310752y76cee520oa1d5379e854d9c94@mail.gmail.com>
	<1d36917a0805310815o695605aepcbbbedd8ebcec383@mail.gmail.com>
	<288df32a0806012219h2c474ae2sf79a1c4991fc2c9b@mail.gmail.com>
Message-ID: <1212384852.32675.14.camel@bbc8>

On Mon, 2008-06-02 at 10:49 +0530, Bala subramanian wrote:
> Dear Alan,
> I am using python 2.4.3 that came pre-installed with my linux. I dnt
> have any other version. But i noticed another problem. In fact no
> other modules are getting imported now. I have pasted the error below.
> Suggest me what i should do now.
> 

That's strange, you definitely have something wrong with your python.
Could you try this: python -c "import sys; print sys.path"
And also tell us what your environment look like ? (just dump it using
the env command if you are using bash).

cheers,

David



From bala.biophysics at gmail.com  Mon Jun  2 03:03:03 2008
From: bala.biophysics at gmail.com (Bala subramanian)
Date: Mon, 2 Jun 2008 12:33:03 +0530
Subject: [Numpy-discussion] numpy import problem
In-Reply-To: <1212384852.32675.14.camel@bbc8>
References: <288df32a0805310752y76cee520oa1d5379e854d9c94@mail.gmail.com>
	<1d36917a0805310815o695605aepcbbbedd8ebcec383@mail.gmail.com>
	<288df32a0806012219h2c474ae2sf79a1c4991fc2c9b@mail.gmail.com>
	<1212384852.32675.14.camel@bbc8>
Message-ID: <288df32a0806020003q67e3d58bvc6a737d53cbbce1b@mail.gmail.com>

Dear David,

I gave the following and pressed TAB, it seems that i have two python
installation in my linux.

$ python

python python2 python2.2 python2.4 pythonsh

When i give python, python 2.4.3 interpreter opens
When i give python2, python 2.2.3 interpreter opens

The default is python 2.4.3.

Now when i use 2.4.3, i get the import error but when i use 2.2.3 i dnt have
the import problem.
When i give *ls -d /usr/lib/py** ( i get only the lib for pythong2.2)

*/usr/lib/python2.2*

I dnt understand why python 2.4 dosent contain seperate libraries installed
in /usr/lib?

The following is the result of the command - python -c "import sys; print
sys.path"

['', '/usr/local/lib/python24.zip', '/usr/local/lib/python2.4',
'/usr/local/lib/python2.4/plat-linux2', '/usr/local/lib/python2.4/lib-tk',
'/usr/local/lib/python2.4/lib-dynload',
'/usr/local/lib/python2.4/site-packages']

-----------------------------------------------------------------------------------------------------------------------------------

OUTPUT OF env command

MANPATH=/usr/local/ifc/man:/usr/local/ifc/man:/usr/local/ifc/man:/usr/kerberos/man:/usr/share/man/en:/usr/share/man:/usr/X11R6/man:/usr/local/man

SSH_AGENT_PID=1970

HOSTNAME=bala_lin

PVM_RSH=/usr/bin/rsh

INTEL_LICENSE_FILE=/usr/local/ifc/licenses:/opt/intel/licenses:/home/souvik/intel/licenses:/usr/local/ifc/licenses:/opt/intel/licenses:/home/souvik/intel/licenses

DESKTOP_STARTUP_ID=

TERM=xterm

SHELL=/bin/bash

HISTSIZE=1000

GTK_RC_FILES=/etc/gtk/gtkrc:/home/souvik/.gtkrc-1.2-gnome2

WINDOWID=25165899

QTDIR=/usr/lib/qt-3.1

USER=souvik

LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:

LD_LIBRARY_PATH=/usr/local/ifc/lib:/usr/local/ifc/lib

SSH_AUTH_SOCK=/tmp/ssh-zHIO1917/agent.1917

PVM_ROOT=/usr/share/pvm3

SESSION_MANAGER=local/bala_lin:/tmp/.ICE-unix/1917

PATH=/usr/local/autodock/share/bin:/usr/local/autodock/i86Linux2/bin:/usr/local/ifc/bin:/usr/local/autodock/share/bin:/usr/local/autodock/i86Linux2/bin:/usr/local/ifc/bin:/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/local/amber9/exe:/usr/local/ifc/bin/:/usr/local/gromacs/bin:/usr/local/fftw/bin:/usr/local/vmd-1.8.6/lib:/usr/local/vmd-1.8.6/bin:/usr/local/pass:/usr/local/nab-5.0/bin:/usr/local/RealPlayer/plugins:/usr/local/mmtsb_toolset/perl:/usr/local/mmtsb_toolset/bin:/home/souvik/X3DNA/BASEPARS:/home/souvik/X3DNA/bin:/usr/local/modeller-9v2/bin:/home/souvik/bin:/usr/local/amber9/exe:/usr/local/ifc/bin/:/usr/local/gromacs/bin:/usr/local/fftw/bin:/usr/local/vmd-1.8.6/lib:/usr/local/vmd-1.8.6/bin:/usr/local/pass:/usr/local/nab-5.0/bin:/usr/local/RealPlayer/plugins:/usr/local/mmtsb_toolset/perl:/usr/local/mmtsb_toolset/bin:/home/souvik/X3DNA/BASEPARS:/home/souvik/X3DNA/bin:/usr/local/modeller-9v2/bin

MAIL=/var/spool/mail/souvik

PWD=/home/souvik

INPUTRC=/etc/inputrc

XMODIFIERS=@im=none

LANG=en_US.UTF-8

LAMHELPFILE=/etc/lam/lam-helpfile

PYTHONHOME=/usr/local/lib_LINUX-rh9/

GDMSESSION=Default

SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass

SHLVL=2

HOME=/home/souvik

GNOME_DESKTOP_SESSION_ID=Default

LOGNAME=souvik

LESSOPEN=|/usr/bin/lesspipe.sh %s

DISPLAY=:0.0

GRACE_HOME=/usr/local/grace

G_BROKEN_FILENAMES=1

COLORTERM=gnome-terminal

XAUTHORITY=/home/souvik/.Xauthority

_=/usr/bin/env
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080602/df0b9f88/attachment.html>

From matthieu.brucher at gmail.com  Mon Jun  2 03:15:21 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 2 Jun 2008 09:15:21 +0200
Subject: [Numpy-discussion] numpy import problem
In-Reply-To: <288df32a0806020003q67e3d58bvc6a737d53cbbce1b@mail.gmail.com>
References: <288df32a0805310752y76cee520oa1d5379e854d9c94@mail.gmail.com>
	<1d36917a0805310815o695605aepcbbbedd8ebcec383@mail.gmail.com>
	<288df32a0806012219h2c474ae2sf79a1c4991fc2c9b@mail.gmail.com>
	<1212384852.32675.14.camel@bbc8>
	<288df32a0806020003q67e3d58bvc6a737d53cbbce1b@mail.gmail.com>
Message-ID: <e76aa17f0806020015h1f862a25p26d1ded2feffcc6c@mail.gmail.com>

Hi,

Where is numpy installation located ? (you should find it with locate numpy)

Matthieu

2008/6/2 Bala subramanian <bala.biophysics at gmail.com>:

> Dear David,
>
> I gave the following and pressed TAB, it seems that i have two python
> installation in my linux.
>
> $ python
>
> python python2 python2.2 python2.4 pythonsh
>
> When i give python, python 2.4.3 interpreter opens
> When i give python2, python 2.2.3 interpreter opens
>
> The default is python 2.4.3.
>
> Now when i use 2.4.3, i get the import error but when i use 2.2.3 i dnt
> have the import problem.
> When i give *ls -d /usr/lib/py** ( i get only the lib for pythong2.2)
>
> */usr/lib/python2.2*
>
> I dnt understand why python 2.4 dosent contain seperate libraries installed
> in /usr/lib?
>
> The following is the result of the command - python -c "import sys; print
> sys.path"
>
> ['', '/usr/local/lib/python24.zip', '/usr/local/lib/python2.4',
> '/usr/local/lib/python2.4/plat-linux2', '/usr/local/lib/python2.4/lib-tk',
> '/usr/local/lib/python2.4/lib-dynload',
> '/usr/local/lib/python2.4/site-packages']
>
>
> -----------------------------------------------------------------------------------------------------------------------------------
>
> OUTPUT OF env command
>
>
> MANPATH=/usr/local/ifc/man:/usr/local/ifc/man:/usr/local/ifc/man:/usr/kerberos/man:/usr/share/man/en:/usr/share/man:/usr/X11R6/man:/usr/local/man
>
> SSH_AGENT_PID=1970
>
> HOSTNAME=bala_lin
>
> PVM_RSH=/usr/bin/rsh
>
>
> INTEL_LICENSE_FILE=/usr/local/ifc/licenses:/opt/intel/licenses:/home/souvik/intel/licenses:/usr/local/ifc/licenses:/opt/intel/licenses:/home/souvik/intel/licenses
>
> DESKTOP_STARTUP_ID=
>
> TERM=xterm
>
> SHELL=/bin/bash
>
> HISTSIZE=1000
>
> GTK_RC_FILES=/etc/gtk/gtkrc:/home/souvik/.gtkrc-1.2-gnome2
>
> WINDOWID=25165899
>
> QTDIR=/usr/lib/qt-3.1
>
> USER=souvik
>
>
> LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:
>
> LD_LIBRARY_PATH=/usr/local/ifc/lib:/usr/local/ifc/lib
>
> SSH_AUTH_SOCK=/tmp/ssh-zHIO1917/agent.1917
>
> PVM_ROOT=/usr/share/pvm3
>
> SESSION_MANAGER=local/bala_lin:/tmp/.ICE-unix/1917
>
>
> PATH=/usr/local/autodock/share/bin:/usr/local/autodock/i86Linux2/bin:/usr/local/ifc/bin:/usr/local/autodock/share/bin:/usr/local/autodock/i86Linux2/bin:/usr/local/ifc/bin:/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/local/amber9/exe:/usr/local/ifc/bin/:/usr/local/gromacs/bin:/usr/local/fftw/bin:/usr/local/vmd-1.8.6/lib:/usr/local/vmd-1.8.6/bin:/usr/local/pass:/usr/local/nab-5.0/bin:/usr/local/RealPlayer/plugins:/usr/local/mmtsb_toolset/perl:/usr/local/mmtsb_toolset/bin:/home/souvik/X3DNA/BASEPARS:/home/souvik/X3DNA/bin:/usr/local/modeller-9v2/bin:/home/souvik/bin:/usr/local/amber9/exe:/usr/local/ifc/bin/:/usr/local/gromacs/bin:/usr/local/fftw/bin:/usr/local/vmd-1.8.6/lib:/usr/local/vmd-1.8.6/bin:/usr/local/pass:/usr/local/nab-5.0/bin:/usr/local/RealPlayer/plugins:/usr/local/mmtsb_toolset/perl:/usr/local/mmtsb_toolset/bin:/home/souvik/X3DNA/BASEPARS:/home/souvik/X3DNA/bin:/usr/local/modeller-9v2/bin
>
> MAIL=/var/spool/mail/souvik
>
> PWD=/home/souvik
>
> INPUTRC=/etc/inputrc
>
> XMODIFIERS=@im=none
>
> LANG=en_US.UTF-8
>
> LAMHELPFILE=/etc/lam/lam-helpfile
>
> PYTHONHOME=/usr/local/lib_LINUX-rh9/
>
> GDMSESSION=Default
>
> SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
>
> SHLVL=2
>
> HOME=/home/souvik
>
> GNOME_DESKTOP_SESSION_ID=Default
>
> LOGNAME=souvik
>
> LESSOPEN=|/usr/bin/lesspipe.sh %s
>
> DISPLAY=:0.0
>
> GRACE_HOME=/usr/local/grace
>
> G_BROKEN_FILENAMES=1
>
> COLORTERM=gnome-terminal
>
> XAUTHORITY=/home/souvik/.Xauthority
>
> _=/usr/bin/env
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080602/db20d29b/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Mon Jun  2 03:07:32 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 02 Jun 2008 16:07:32 +0900
Subject: [Numpy-discussion] numpy import problem
In-Reply-To: <288df32a0806020003q67e3d58bvc6a737d53cbbce1b@mail.gmail.com>
References: <288df32a0805310752y76cee520oa1d5379e854d9c94@mail.gmail.com>	<1d36917a0805310815o695605aepcbbbedd8ebcec383@mail.gmail.com>	<288df32a0806012219h2c474ae2sf79a1c4991fc2c9b@mail.gmail.com>	<1212384852.32675.14.camel@bbc8>
	<288df32a0806020003q67e3d58bvc6a737d53cbbce1b@mail.gmail.com>
Message-ID: <48439C34.3070608@ar.media.kyoto-u.ac.jp>

Bala subramanian wrote:
>
> I dnt understand why python 2.4 dosent contain seperate libraries 
> installed in /usr/lib?
>

Because your python 2.4 was installed in /usr/local, I guess:

> The following is the result of the command - python -c "import sys; 
> print sys.path"
>
> ['', '/usr/local/lib/python24.zip', '/usr/local/lib/python2.4', 
> '/usr/local/lib/python2.4/plat-linux2', 
> '/usr/local/lib/python2.4/lib-tk', 
> '/usr/local/lib/python2.4/lib-dynload', 
> '/usr/local/lib/python2.4/site-packages']
>

It is still not clear to me why you cannot import string in python 2.4: 
python automatically add $PREFIX/lib/python-$VERSION/site-packages to 
the path list it is looking at for import.

Did you install python 2.4 from sources ? Or is it installed by your 
package manager (rpm, since it looks like you are using Red Hat).

cheers,

David


From hanni.ali at gmail.com  Mon Jun  2 04:41:09 2008
From: hanni.ali at gmail.com (Hanni Ali)
Date: Mon, 2 Jun 2008 09:41:09 +0100
Subject: [Numpy-discussion] Installation info
In-Reply-To: <5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>
	<6be8b94a0805300431p14eb1ae0pc5fd111db9b0503@mail.gmail.com>
	<789d27b10805300435t361d12a8j59f69aadefd3f9e6@mail.gmail.com>
	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>
Message-ID: <789d27b10806020141m43d8d699u8d9dbe8d593d8e3e@mail.gmail.com>

Hi David,


> Unfortunately, this is difficult: windows 64 is not commonly available
> (I don't have any access to it personally, for example), and mingw is
> not available yet for windows 64 either.


I had managed to compile and install 1.04 with vs 2005 64 bit with a bit of
hacking, however it had failed one of the regression tests. I will try
numpy-1.1.0, has anyone used vs 2008 to compile it for 64 bit windows?

Cheers,

Hanni
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080602/30732ccd/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Mon Jun  2 04:51:59 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 02 Jun 2008 17:51:59 +0900
Subject: [Numpy-discussion] Installation info
In-Reply-To: <789d27b10806020141m43d8d699u8d9dbe8d593d8e3e@mail.gmail.com>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>	<6be8b94a0805300431p14eb1ae0pc5fd111db9b0503@mail.gmail.com>	<789d27b10805300435t361d12a8j59f69aadefd3f9e6@mail.gmail.com>	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>
	<789d27b10806020141m43d8d699u8d9dbe8d593d8e3e@mail.gmail.com>
Message-ID: <4843B4AF.4070605@ar.media.kyoto-u.ac.jp>

Hanni Ali wrote:
>  
> I had managed to compile and install 1.04 with vs 2005 64 bit with a 
> bit of hacking,

I can't build official binaries with VS: I don't have VS. Also, there is 
the problem of building the fortran dependencies (BLAS/LAPACK). I don't 
know if it is even possible to build ATLAS on windows x64. Even assuming 
we only use netlib BLAS/LAPACK, I still need a fortran compiler (gfortran).

> however it had failed one of the regression tests. I will try 
> numpy-1.1.0, has anyone used vs 2008 to compile it for 64 bit windows?

You need to use the same VS as python: this means VS 2003 for official 
python 2.5 (you can build python yourself with VS 2005 or 2008, though, 
but for distribution, this is not useful).

cheers,

David


From hanni.ali at gmail.com  Mon Jun  2 05:30:59 2008
From: hanni.ali at gmail.com (Hanni Ali)
Date: Mon, 2 Jun 2008 10:30:59 +0100
Subject: [Numpy-discussion] Installation info
In-Reply-To: <4843B4AF.4070605@ar.media.kyoto-u.ac.jp>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>
	<6be8b94a0805300431p14eb1ae0pc5fd111db9b0503@mail.gmail.com>
	<789d27b10805300435t361d12a8j59f69aadefd3f9e6@mail.gmail.com>
	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>
	<789d27b10806020141m43d8d699u8d9dbe8d593d8e3e@mail.gmail.com>
	<4843B4AF.4070605@ar.media.kyoto-u.ac.jp>
Message-ID: <789d27b10806020230l1d88f270l52a62ec90954aeb7@mail.gmail.com>

  Hi David,


> I can't build official binaries with VS: I don't have VS. Also, there is
> the problem of building the fortran dependencies (BLAS/LAPACK). I don't
> know if it is even possible to build ATLAS on windows x64. Even assuming
> we only use netlib BLAS/LAPACK, I still need a fortran compiler (gfortran).
>

Yes I had used the internal versions in the mean time, but I do want to try
to use the intel fortran compiler in all likelyhood.

Python 2.6 is compiled with vs 2008 is is important that numpy is also
compiled with the same compiler or not. The fact that a fortran compiler is
necessary makes me think no?


>
> > however it had failed one of the regression tests. I will try
> > numpy-1.1.0, has anyone used vs 2008 to compile it for 64 bit windows?
>

It looks like I'm going to look at 2.6 now due to dependencies on pywin32 as
well.

Also is it important that BLAS/LAPACK are compiled with the same compiler as
python or not? (Obviously just the C++ parts)

Thanks,

Hanni
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080602/de1bf60f/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Mon Jun  2 05:30:58 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 02 Jun 2008 18:30:58 +0900
Subject: [Numpy-discussion] Installation info
In-Reply-To: <789d27b10806020230l1d88f270l52a62ec90954aeb7@mail.gmail.com>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>	<6be8b94a0805300431p14eb1ae0pc5fd111db9b0503@mail.gmail.com>	<789d27b10805300435t361d12a8j59f69aadefd3f9e6@mail.gmail.com>	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>	<789d27b10806020141m43d8d699u8d9dbe8d593d8e3e@mail.gmail.com>	<4843B4AF.4070605@ar.media.kyoto-u.ac.jp>
	<789d27b10806020230l1d88f270l52a62ec90954aeb7@mail.gmail.com>
Message-ID: <4843BDD2.8060002@ar.media.kyoto-u.ac.jp>

Hanni Ali wrote:
>  
> Yes I had used the internal versions in the mean time, but I do want 
> to try to use the intel fortran compiler in all likelyhood.

Yes, people can try to build as they want, that's the beauty of open 
source :) But for official distribution, I don't want to depend on non 
free software (outside windows, obviously). I need a process to build 
numpy in a reproducible and in a hassle way (well, as much as possible, 
at least).

>  
> Python 2.6 is compiled with vs 2008 is is important that numpy is also 
> compiled with the same compiler or not. The fact that a fortran 
> compiler is necessary makes me think no?

This has nothing to do with fortran per se, but with the fact that MS 
keeps breaking the standard C runtime. Hopefully, with python 3, this 
may not be an issue anymore (python on windows won't use the standard C 
API, but windows API instead, because MS guarantees ABI in this case, at 
least that's what I understood).

The fortran compiler has to be the same to build everything, but python 
does not use any fortran, so you can choose whatever you want as long as 
you use it for everything (BLAS, LAPACK and numpy).

>  
> It looks like I'm going to look at 2.6 now due to dependencies on 
> pywin32 as well.
>  
> Also is it important that BLAS/LAPACK are compiled with the same 
> compiler as python or not?

BLAS/LAPACK is built with fortran, and python with C compiler, so no :) 
What may be important is the C++ compiler to build (more exactly to 
link) python if you build python by yourself, but I don't know how this 
works on windows. I just build softwares for windows, I don't use it.

cheers,

David


From klemm at phys.ethz.ch  Mon Jun  2 08:03:15 2008
From: klemm at phys.ethz.ch (Hanno Klemm)
Date: Mon, 02 Jun 2008 14:03:15 +0200
Subject: [Numpy-discussion] PyTables, array and recarray
In-Reply-To: <200805302033.26525.falted@pytables.org>
References: <twig.1212160638.71062@phys.ethz.ch>,
	<twig.1212160638.71062@phys.ethz.ch>
Message-ID: <twig.1212408194.26644@phys.ethz.ch>


Francesc, Pierre,

thanks for your replies, the view works fine. The reason why I like
recarrays is that you can address your fields simply with a.Name, when
a is your array. I find that highly convenient when working with data
interactively.

As far as I know there is no similar possbility to do that with
ndarrays, or is there?

Cheers,
Hanno

Francesc Alted <falted at pytables.org> said:

> A Friday 30 May 2008, Hanno Klemm escrigu?:
> > Hi,
> >
> > I try to save the contents of a numpy recarray with PyTables into a
> > file. That works well, however, if I then try to retrieve the data, I
> > get back an array with matching dtypes rather than a recarray.
> >
> > What is the best way to get a recarray back, or if that's not
> > possible what's the most efficient way to convert an array to a
> > recarray?
> >
> > This is what I am doing at the moment:
> >
> > Python 2.5 (r25:51908, Oct  4 2006, 17:28:51)
> > [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-52)] on linux2
> > Type "help", "copyright", "credits" or "license" for more
> > information.
> >
> > >>> import numpy as N
> > >>> import tables as t
> > >>> num = 2
> > >>> a = N.recarray(num, formats='i4,f8,f8',names='id,x,y')
> > >>> a['id'] = [3,4]
> > >>> a['x'] = [3.4,4.5]
> > >>> a['y'] = [4.6,4.5]
> > >>> a
> >
> > recarray([(3, 3.3999999999999999, 4.5999999999999996), (4, 4.5,
> > 4.5)], dtype=[('id', '<i4'), ('x', '<f8'), ('y', '<f8')])
> >
> > >>> f = t.openFile('test.h5', 'w')
> > >>> f.createTable('/', 'test', a)
> >
> > /test (Table(2L,)) ''
> >   description := {
> >   "id": Int32Col(shape=(), dflt=0, pos=0),
> >   "x": Float64Col(shape=(), dflt=0.0, pos=1),
> >   "y": Float64Col(shape=(), dflt=0.0, pos=2)}
> >   byteorder := 'little'
> >   chunkshape := (409,)
> >
> > >>> f.close()
> > >>> f = t.openFile('test.h5', 'r')
> > >>> b = f.root.test[:]
> > >>> b
> >
> > array([(3, 3.3999999999999999, 4.5999999999999996), (4, 4.5, 4.5)],
> >       dtype=[('id', '<i4'), ('x', '<f8'), ('y', '<f8')])
> >
> > >>> type(b)
> >
> > <type 'numpy.ndarray'>
> >
> > >>> type(a)
> >
> > <class 'numpy.core.records.recarray'>
> 
> Yeah, this is on purpose because ndarray classes are actually C 
> extensions and they are generally more efficient than Python classes.  
> 
> If what you want is a recarray class (Python class), you can always 
> create a view as Pierre suggested.  However, this is rarely needed 
> because you can access most of the recarray functionality out of the 
> ndarray class.
> 
> Cheers,
> 
> -- 
> Francesc Alted
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



-- 
Hanno Klemm
klemm at phys.ethz.ch




From lranderson at pppl.gov  Mon Jun  2 08:16:30 2008
From: lranderson at pppl.gov (Lewis E. Randerson)
Date: Mon, 2 Jun 2008 08:16:30 -0400
Subject: [Numpy-discussion] numpy 1.1.0 build can not find lahey (lf95)
	fortran compiler (short version)
In-Reply-To: <128818DD3B256A41A94F4F2564E2FABF054DF5B4@MAIL-STORE-2.pppl.gov>
References: <128818DD3B256A41A94F4F2564E2FABF054DF5B4@MAIL-STORE-2.pppl.gov>
Message-ID: <128818DD3B256A41A94F4F2564E2FABF054DF5B5@MAIL-STORE-2.pppl.gov>

Hi,

When trying to build numpy 1.1.0 with python 2.5.2 on an RHEL3 32-bit
system, using command:
  python setup.py config_fc --fcompiler=lahey install &> install_LEW.log
I am seeing these warnings:
  warning: build_ext: f77_compiler=lahey is not available.
      and
  warning: build_ext: extension 'numpy.linalg.lapack_lite' has Fortran \
    libraries but no Fortran linker found, using default linker

Then lapack_lite.so is linked incorrectly with gcc instead of lf95.  File
distutils/fcompiler/lahey.py is being read but the information inside is
not being used.

--Lew
----------------------------------------------------------------------
Lewis E. Randerson
DOE Princeton University Plasma Physics Laboratory,
Princeton University, James Forrestal Campus
100 Stellarator Road, Princeton, NJ 08543
Work: 609/243-3134, Fax: 609/243-3086, PPPL Web: http://www.pppl.gov 


From hanni.ali at gmail.com  Mon Jun  2 08:26:19 2008
From: hanni.ali at gmail.com (Hanni Ali)
Date: Mon, 2 Jun 2008 13:26:19 +0100
Subject: [Numpy-discussion] Installation info
In-Reply-To: <4843BDD2.8060002@ar.media.kyoto-u.ac.jp>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>
	<6be8b94a0805300431p14eb1ae0pc5fd111db9b0503@mail.gmail.com>
	<789d27b10805300435t361d12a8j59f69aadefd3f9e6@mail.gmail.com>
	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>
	<789d27b10806020141m43d8d699u8d9dbe8d593d8e3e@mail.gmail.com>
	<4843B4AF.4070605@ar.media.kyoto-u.ac.jp>
	<789d27b10806020230l1d88f270l52a62ec90954aeb7@mail.gmail.com>
	<4843BDD2.8060002@ar.media.kyoto-u.ac.jp>
Message-ID: <789d27b10806020526h47d08418v4f0e0752bbcabb1a@mail.gmail.com>

Excellenm, thanks for clearing all that up.

How about numpy with 2.6, any issues?

Hanni

2008/6/2 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:

> Hanni Ali wrote:
> >
> > Yes I had used the internal versions in the mean time, but I do want
> > to try to use the intel fortran compiler in all likelyhood.
>
> Yes, people can try to build as they want, that's the beauty of open
> source :) But for official distribution, I don't want to depend on non
> free software (outside windows, obviously). I need a process to build
> numpy in a reproducible and in a hassle way (well, as much as possible,
> at least).
>
> >
> > Python 2.6 is compiled with vs 2008 is is important that numpy is also
> > compiled with the same compiler or not. The fact that a fortran
> > compiler is necessary makes me think no?
>
> This has nothing to do with fortran per se, but with the fact that MS
> keeps breaking the standard C runtime. Hopefully, with python 3, this
> may not be an issue anymore (python on windows won't use the standard C
> API, but windows API instead, because MS guarantees ABI in this case, at
> least that's what I understood).
>
> The fortran compiler has to be the same to build everything, but python
> does not use any fortran, so you can choose whatever you want as long as
> you use it for everything (BLAS, LAPACK and numpy).
>
> >
> > It looks like I'm going to look at 2.6 now due to dependencies on
> > pywin32 as well.
> >
> > Also is it important that BLAS/LAPACK are compiled with the same
> > compiler as python or not?
>
> BLAS/LAPACK is built with fortran, and python with C compiler, so no :)
> What may be important is the C++ compiler to build (more exactly to
> link) python if you build python by yourself, but I don't know how this
> works on windows. I just build softwares for windows, I don't use it.
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080602/9c324a75/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Mon Jun  2 08:22:38 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 02 Jun 2008 21:22:38 +0900
Subject: [Numpy-discussion] numpy 1.1.0 build can not find lahey (lf95)
 fortran compiler (short version)
In-Reply-To: <128818DD3B256A41A94F4F2564E2FABF054DF5B5@MAIL-STORE-2.pppl.gov>
References: <128818DD3B256A41A94F4F2564E2FABF054DF5B4@MAIL-STORE-2.pppl.gov>
	<128818DD3B256A41A94F4F2564E2FABF054DF5B5@MAIL-STORE-2.pppl.gov>
Message-ID: <4843E60E.30707@ar.media.kyoto-u.ac.jp>

Lewis E. Randerson wrote:
> I am seeing these warnings:
>   warning: build_ext: f77_compiler=lahey is not available.
>   
>   

Hi Lewis,

    What happens when you execute directly the file 
numpy/distutils/fcompiler/lahey.py ?

    cheers,

    David


From pgmdevlist at gmail.com  Mon Jun  2 09:08:53 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 2 Jun 2008 09:08:53 -0400
Subject: [Numpy-discussion] PyTables, array and recarray
In-Reply-To: <twig.1212408194.26644@phys.ethz.ch>
References: <twig.1212160638.71062@phys.ethz.ch>
	<twig.1212408194.26644@phys.ethz.ch>
Message-ID: <200806020908.54206.pgmdevlist@gmail.com>

On Monday 02 June 2008 08:03:15 Hanno Klemm wrote:
> thanks for your replies, the view works fine. The reason why I like
> recarrays is that you can address your fields simply with a.Name, when
> a is your array. I find that highly convenient when working with data
> interactively.
>
> As far as I know there is no similar possbility to do that with
> ndarrays, or is there?

No, but you can access the field Name of your ndarray a with a['Name'], which 
is just as convenient. An issue with recarrays is that the __getattribute__ 
and __setattr__ methods are overwritten, which tends to reduce performance.


From lranderson at pppl.gov  Mon Jun  2 09:39:06 2008
From: lranderson at pppl.gov (Lewis E. Randerson)
Date: Mon, 2 Jun 2008 09:39:06 -0400
Subject: [Numpy-discussion] numpy 1.1.0 build can not find lahey (lf95)
	fortran compiler (short version)
In-Reply-To: <4843E60E.30707@ar.media.kyoto-u.ac.jp>
References: <128818DD3B256A41A94F4F2564E2FABF054DF5B4@MAIL-STORE-2.pppl.gov><128818DD3B256A41A94F4F2564E2FABF054DF5B5@MAIL-STORE-2.pppl.gov>
	<4843E60E.30707@ar.media.kyoto-u.ac.jp>
Message-ID: <128818DD3B256A41A94F4F2564E2FABF054DF5B6@MAIL-STORE-2.pppl.gov>

Hi,

When executing numpy/distutils/fcompiler/lahey.py,
  I get "none"

Here is what I get from the following command. Note the 
Lahey entry under: 
 Compilers available for this platform, but not found:

============================ available compilers ==========
python setup.py config_fc --help-fcompiler Running from numpy source
directory.
GnuFCompiler instance properties:
  archiver        = ['/usr/bin/g77', '-cr']
  compile_switch  = '-c'
  compiler_f77    = ['/usr/bin/g77', '-g', '-Wall', '-fno-second-
                    underscore', '-fPIC', '-O2', '-funroll-loops', '-mmmx',
'-
                    m3dnow', '-msse2', '-msse']
  compiler_f90    = None
  compiler_fix    = None
  libraries       = ['g2c']
  library_dirs    = []
  linker_exe      = ['/usr/bin/g77', '-g', '-Wall', '-g', '-Wall']
  linker_so       = ['/usr/bin/g77', '-g', '-Wall', '-g', '-Wall', '-
                    shared']
  object_switch   = '-o '
  ranlib          = ['/usr/bin/g77']
  version         = LooseVersion ('3.2.3')
  version_cmd     = ['/usr/bin/g77', '--version']
Fortran compilers found:
  --fcompiler=gnu  GNU Fortran 77 compiler (3.2.3)
Compilers available for this platform, but not found:
  --fcompiler=absoft   Absoft Corp Fortran Compiler
  --fcompiler=compaq   Compaq Fortran Compiler
  --fcompiler=g95      G95 Fortran Compiler
  --fcompiler=gnu95    GNU Fortran 95 compiler
  --fcompiler=intel    Intel Fortran Compiler for 32-bit apps
  --fcompiler=intele   Intel Fortran Compiler for Itanium apps
  --fcompiler=intelem  Intel Fortran Compiler for EM64T-based apps
  --fcompiler=lahey    Lahey/Fujitsu Fortran 95 Compiler
  --fcompiler=nag      NAGWare Fortran 95 Compiler
  --fcompiler=pg       Portland Group Fortran Compiler
  --fcompiler=vast     Pacific-Sierra Research Fortran 90 Compiler
Compilers not available on this platform:
  --fcompiler=hpux     HP Fortran 90 Compiler
  --fcompiler=ibm      IBM XL Fortran Compiler
  --fcompiler=intelev  Intel Visual Fortran Compiler for Itanium apps
  --fcompiler=intelv   Intel Visual Fortran Compiler for 32-bit apps
  --fcompiler=mips     MIPSpro Fortran Compiler
  --fcompiler=none     Fake Fortran compiler
  --fcompiler=sun      Sun or Forte Fortran 95 Compiler
For compiler details, run 'config_fc --verbose' setup command.

--Lew 

-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of David Cournapeau
Sent: Monday, June 02, 2008 8:23 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] numpy 1.1.0 build can not find lahey (lf95)
fortran compiler (short version)

Lewis E. Randerson wrote:
> I am seeing these warnings:
>   warning: build_ext: f77_compiler=lahey is not available.
>   
>   

Hi Lewis,

    What happens when you execute directly the file 
numpy/distutils/fcompiler/lahey.py ?

    cheers,

    David
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion


From lranderson at pppl.gov  Mon Jun  2 09:48:22 2008
From: lranderson at pppl.gov (Lewis E. Randerson)
Date: Mon, 2 Jun 2008 09:48:22 -0400
Subject: [Numpy-discussion] numpy 1.1.0 build can not find lahey (lf95)
	fortran compiler (short version)
In-Reply-To: <128818DD3B256A41A94F4F2564E2FABF054DF5B6@MAIL-STORE-2.pppl.gov>
References: <128818DD3B256A41A94F4F2564E2FABF054DF5B4@MAIL-STORE-2.pppl.gov><128818DD3B256A41A94F4F2564E2FABF054DF5B5@MAIL-STORE-2.pppl.gov>
	<4843E60E.30707@ar.media.kyoto-u.ac.jp>
	<128818DD3B256A41A94F4F2564E2FABF054DF5B6@MAIL-STORE-2.pppl.gov>
Message-ID: <128818DD3B256A41A94F4F2564E2FABF054DF5B7@MAIL-STORE-2.pppl.gov>

Here's fuller output from the lahey.py script.

python
/usr/pppl/python/2.5.2/lib/python2.5/site-packages/numpy/distutils/fcompiler/
lahey.py
customize LaheyFCompiler
find_executable('lf95')
Found executable /usr/pppl/lff95/6.20c/bin/lf95
exec_command(['/usr/pppl/lff95/6.20c/bin/lf95', '--version'],)
Retaining cwd:
/local/randerso/2008-06-01/numpy-1.1.0/numpy/distutils/fcompiler
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( /usr/pppl/lff95/6.20c/bin/lf95 --version ; echo $? >
/tmp/tmp5WTGB6/Miygel ) > /tmp/tmp5WTGB6/yKqO37 2>&1')
_update_environment(...)
None
removing /tmp/tmp5WTGB6

Hmmm.  Maybe it doesn't like the version?.
How can I find that out? 

--Lew 

-----Original Message-----
From: Lewis E. Randerson 
Sent: Monday, June 02, 2008 9:39 AM
To: 'Discussion of Numerical Python'
Subject: RE: [Numpy-discussion] numpy 1.1.0 build can not find lahey (lf95)
fortran compiler (short version)

Hi,

When executing numpy/distutils/fcompiler/lahey.py,
  I get "none"

Here is what I get from the following command. Note the 
Lahey entry under: 
 Compilers available for this platform, but not found:

============================ available compilers ==========
python setup.py config_fc --help-fcompiler Running from numpy source
directory.
GnuFCompiler instance properties:
  archiver        = ['/usr/bin/g77', '-cr']
  compile_switch  = '-c'
  compiler_f77    = ['/usr/bin/g77', '-g', '-Wall', '-fno-second-
                    underscore', '-fPIC', '-O2', '-funroll-loops', '-mmmx',
'-
                    m3dnow', '-msse2', '-msse']
  compiler_f90    = None
  compiler_fix    = None
  libraries       = ['g2c']
  library_dirs    = []
  linker_exe      = ['/usr/bin/g77', '-g', '-Wall', '-g', '-Wall']
  linker_so       = ['/usr/bin/g77', '-g', '-Wall', '-g', '-Wall', '-
                    shared']
  object_switch   = '-o '
  ranlib          = ['/usr/bin/g77']
  version         = LooseVersion ('3.2.3')
  version_cmd     = ['/usr/bin/g77', '--version']
Fortran compilers found:
  --fcompiler=gnu  GNU Fortran 77 compiler (3.2.3)
Compilers available for this platform, but not found:
  --fcompiler=absoft   Absoft Corp Fortran Compiler
  --fcompiler=compaq   Compaq Fortran Compiler
  --fcompiler=g95      G95 Fortran Compiler
  --fcompiler=gnu95    GNU Fortran 95 compiler
  --fcompiler=intel    Intel Fortran Compiler for 32-bit apps
  --fcompiler=intele   Intel Fortran Compiler for Itanium apps
  --fcompiler=intelem  Intel Fortran Compiler for EM64T-based apps
  --fcompiler=lahey    Lahey/Fujitsu Fortran 95 Compiler
  --fcompiler=nag      NAGWare Fortran 95 Compiler
  --fcompiler=pg       Portland Group Fortran Compiler
  --fcompiler=vast     Pacific-Sierra Research Fortran 90 Compiler
Compilers not available on this platform:
  --fcompiler=hpux     HP Fortran 90 Compiler
  --fcompiler=ibm      IBM XL Fortran Compiler
  --fcompiler=intelev  Intel Visual Fortran Compiler for Itanium apps
  --fcompiler=intelv   Intel Visual Fortran Compiler for 32-bit apps
  --fcompiler=mips     MIPSpro Fortran Compiler
  --fcompiler=none     Fake Fortran compiler
  --fcompiler=sun      Sun or Forte Fortran 95 Compiler
For compiler details, run 'config_fc --verbose' setup command.

--Lew 

-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of David Cournapeau
Sent: Monday, June 02, 2008 8:23 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] numpy 1.1.0 build can not find lahey (lf95)
fortran compiler (short version)

Lewis E. Randerson wrote:
> I am seeing these warnings:
>   warning: build_ext: f77_compiler=lahey is not available.
>   
>   

Hi Lewis,

    What happens when you execute directly the file 
numpy/distutils/fcompiler/lahey.py ?

    cheers,

    David
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion


From lranderson at pppl.gov  Mon Jun  2 11:03:58 2008
From: lranderson at pppl.gov (Lewis E. Randerson)
Date: Mon, 2 Jun 2008 11:03:58 -0400
Subject: [Numpy-discussion] numpy 1.1.0 build can not find lahey (lf95)
	fortran compiler (short version)
In-Reply-To: <4843E60E.30707@ar.media.kyoto-u.ac.jp>
References: <128818DD3B256A41A94F4F2564E2FABF054DF5B4@MAIL-STORE-2.pppl.gov><128818DD3B256A41A94F4F2564E2FABF054DF5B5@MAIL-STORE-2.pppl.gov>
	<4843E60E.30707@ar.media.kyoto-u.ac.jp>
Message-ID: <128818DD3B256A41A94F4F2564E2FABF054DF5B8@MAIL-STORE-2.pppl.gov>

Found the problem.

The pattern in lahey.py differs from the version of Lahey Fujitsu
compiler I have.  Once corrected, the compiler is now found. 

P.S.  One of the libraries defined in the file is also incorrect.
Once I have gotten all the way through the SciPy build and
the users have said all works, I'll send in the version of
the successful lahey.py. 

Thanks for the help.
Also thanks to the original contributor of lahey.py.  It saved
me a lot of time!

--Lew

-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of David Cournapeau
Sent: Monday, June 02, 2008 8:23 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] numpy 1.1.0 build can not find lahey (lf95)
fortran compiler (short version)

Lewis E. Randerson wrote:
> I am seeing these warnings:
>   warning: build_ext: f77_compiler=lahey is not available.
>   
>   

Hi Lewis,

    What happens when you execute directly the file 
numpy/distutils/fcompiler/lahey.py ?

    cheers,

    David
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion


From david.huard at gmail.com  Mon Jun  2 11:20:36 2008
From: david.huard at gmail.com (David Huard)
Date: Mon, 2 Jun 2008 11:20:36 -0400
Subject: [Numpy-discussion] NumpyTest problem
Message-ID: <91cf711d0806020820o17343fdfua8fa2ee9ed91f081@mail.gmail.com>

Hi,

There are 2 problems with NumpyTest

1. It fails if the command is given the file name only (without a directory
structure)

E.g.:

huardda at angus:~/repos/numpy/numpy/tests$ python test_ctypeslib.py
Traceback (most recent call last):
  File "test_ctypeslib.py", line 87, in <module>
    NumpyTest().run()
  File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py", line
655, in run
    testcase_pattern=options.testcase_pattern)
  File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py", line
575, in test
    level, verbosity)
  File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py", line
453, in _test_suite_from_all_tests
    importall(this_package)
  File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py", line
681, in importall
    for subpackage_name in os.listdir(package_dir):
OSError: [Errno 2] No such file or directory: ''
huardda at angus:~/repos/numpy/numpy/tests$



2. It doesn't find tests it used to find:

huardda at angus:~/repos/numpy/numpy$ python tests/test_ctypeslib.py

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
huardda at angus:~/repos/numpy/numpy$

Cheers,

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080602/ac976b86/attachment.html>

From charlesr.harris at gmail.com  Mon Jun  2 11:42:14 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 2 Jun 2008 09:42:14 -0600
Subject: [Numpy-discussion] NumpyTest problem
In-Reply-To: <91cf711d0806020820o17343fdfua8fa2ee9ed91f081@mail.gmail.com>
References: <91cf711d0806020820o17343fdfua8fa2ee9ed91f081@mail.gmail.com>
Message-ID: <e06186140806020842o31fcd333nefa8fbf39f92cfa3@mail.gmail.com>

On Mon, Jun 2, 2008 at 9:20 AM, David Huard <david.huard at gmail.com> wrote:

> Hi,
>
> There are 2 problems with NumpyTest
>
> 1. It fails if the command is given the file name only (without a directory
> structure)
>
> E.g.:
>
> huardda at angus:~/repos/numpy/numpy/tests$ python test_ctypeslib.py
> Traceback (most recent call last):
>   File "test_ctypeslib.py", line 87, in <module>
>     NumpyTest().run()
>   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py",
> line 655, in run
>     testcase_pattern=options.testcase_pattern)
>   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py",
> line 575, in test
>     level, verbosity)
>   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py",
> line 453, in _test_suite_from_all_tests
>     importall(this_package)
>   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py",
> line 681, in importall
>     for subpackage_name in os.listdir(package_dir):
> OSError: [Errno 2] No such file or directory: ''
> huardda at angus:~/repos/numpy/numpy/tests$
>
>
>
> 2. It doesn't find tests it used to find:
>
> huardda at angus:~/repos/numpy/numpy$ python tests/test_ctypeslib.py
>


There haven't been many changes to the tests.  Could you fool with
numpy.test(level=10,all=0) and such to see what happens? All=1 is now the
default.

I've also seen test run some tests twice. I don't know what was up with
that.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080602/7cd1d35e/attachment.html>

From alan.mcintyre at gmail.com  Mon Jun  2 11:45:31 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 2 Jun 2008 11:45:31 -0400
Subject: [Numpy-discussion] NumpyTest problem
In-Reply-To: <91cf711d0806020820o17343fdfua8fa2ee9ed91f081@mail.gmail.com>
References: <91cf711d0806020820o17343fdfua8fa2ee9ed91f081@mail.gmail.com>
Message-ID: <1d36917a0806020845k55805582j848a9d746e08e587@mail.gmail.com>

David,

We're in the process of switching to nose
(http://www.somethingaboutorange.com/mrl/projects/nose/) as the test
framework for 1.2; I'll try to keep an eye on stuff like that and make
it work properly if I can.

Alan

On Mon, Jun 2, 2008 at 11:20 AM, David Huard <david.huard at gmail.com> wrote:
> Hi,
>
> There are 2 problems with NumpyTest
>
> 1. It fails if the command is given the file name only (without a directory
> structure)
>
> E.g.:
>
> huardda at angus:~/repos/numpy/numpy/tests$ python test_ctypeslib.py
> Traceback (most recent call last):
>   File "test_ctypeslib.py", line 87, in <module>
>     NumpyTest().run()
>   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py", line
> 655, in run
>     testcase_pattern=options.testcase_pattern)
>   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py", line
> 575, in test
>     level, verbosity)
>   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py", line
> 453, in _test_suite_from_all_tests
>     importall(this_package)
>   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py", line
> 681, in importall
>     for subpackage_name in os.listdir(package_dir):
> OSError: [Errno 2] No such file or directory: ''
> huardda at angus:~/repos/numpy/numpy/tests$
>
>
>
> 2. It doesn't find tests it used to find:
>
> huardda at angus:~/repos/numpy/numpy$ python tests/test_ctypeslib.py
>
> ----------------------------------------------------------------------
> Ran 0 tests in 0.000s
>
> OK
> huardda at angus:~/repos/numpy/numpy$
>
> Cheers,
>
> David
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From david.huard at gmail.com  Mon Jun  2 13:09:45 2008
From: david.huard at gmail.com (David Huard)
Date: Mon, 2 Jun 2008 13:09:45 -0400
Subject: [Numpy-discussion] NumpyTest problem
In-Reply-To: <e06186140806020842o31fcd333nefa8fbf39f92cfa3@mail.gmail.com>
References: <91cf711d0806020820o17343fdfua8fa2ee9ed91f081@mail.gmail.com>
	<e06186140806020842o31fcd333nefa8fbf39f92cfa3@mail.gmail.com>
Message-ID: <91cf711d0806021009h3444992n63b2dc8a13d93b89@mail.gmail.com>

numpy.test(level=10,all=0) seems to work fine.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080602/fb04ef40/attachment.html>

From david.huard at gmail.com  Mon Jun  2 13:18:45 2008
From: david.huard at gmail.com (David Huard)
Date: Mon, 2 Jun 2008 13:18:45 -0400
Subject: [Numpy-discussion] NumpyTest problem
In-Reply-To: <1d36917a0806020845k55805582j848a9d746e08e587@mail.gmail.com>
References: <91cf711d0806020820o17343fdfua8fa2ee9ed91f081@mail.gmail.com>
	<1d36917a0806020845k55805582j848a9d746e08e587@mail.gmail.com>
Message-ID: <91cf711d0806021018j514f878bj78aff5d7d3e92dc5@mail.gmail.com>

Hi Alan,

Thanks for looking into it.

David


2008/6/2 Alan McIntyre <alan.mcintyre at gmail.com>:

> David,
>
> We're in the process of switching to nose
> (http://www.somethingaboutorange.com/mrl/projects/nose/) as the test
> framework for 1.2; I'll try to keep an eye on stuff like that and make
> it work properly if I can.
>
> Alan
>
> On Mon, Jun 2, 2008 at 11:20 AM, David Huard <david.huard at gmail.com>
> wrote:
> > Hi,
> >
> > There are 2 problems with NumpyTest
> >
> > 1. It fails if the command is given the file name only (without a
> directory
> > structure)
> >
> > E.g.:
> >
> > huardda at angus:~/repos/numpy/numpy/tests$ python test_ctypeslib.py
> > Traceback (most recent call last):
> >   File "test_ctypeslib.py", line 87, in <module>
> >     NumpyTest().run()
> >   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py",
> line
> > 655, in run
> >     testcase_pattern=options.testcase_pattern)
> >   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py",
> line
> > 575, in test
> >     level, verbosity)
> >   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py",
> line
> > 453, in _test_suite_from_all_tests
> >     importall(this_package)
> >   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py",
> line
> > 681, in importall
> >     for subpackage_name in os.listdir(package_dir):
> > OSError: [Errno 2] No such file or directory: ''
> > huardda at angus:~/repos/numpy/numpy/tests$
> >
> >
> >
> > 2. It doesn't find tests it used to find:
> >
> > huardda at angus:~/repos/numpy/numpy$ python tests/test_ctypeslib.py
> >
> > ----------------------------------------------------------------------
> > Ran 0 tests in 0.000s
> >
> > OK
> > huardda at angus:~/repos/numpy/numpy$
> >
> > Cheers,
> >
> > David
> >
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080602/778d018b/attachment.html>

From Chris.Barker at noaa.gov  Mon Jun  2 13:46:42 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 02 Jun 2008 10:46:42 -0700
Subject: [Numpy-discussion] Test Faliures on OS-X
In-Reply-To: <91cf711d0806021009h3444992n63b2dc8a13d93b89@mail.gmail.com>
References: <91cf711d0806020820o17343fdfua8fa2ee9ed91f081@mail.gmail.com>
	<e06186140806020842o31fcd333nefa8fbf39f92cfa3@mail.gmail.com>
	<91cf711d0806021009h3444992n63b2dc8a13d93b89@mail.gmail.com>
Message-ID: <48443202.5050804@noaa.gov>

HI all,

I missed a bit of the release process, but I thought all the test 
failures had been squashed. However, I just installed the OS-X 1.1.0 
binary, and got:

 >>> import numpy
 >>> numpy.__version__
'1.1.0'
 >>>
 >>> from numpy import test
 >>> test(all=True)
Numpy is installed in 
/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy
Numpy version 1.1.0
Python version 2.5.1 (r251:54869, Apr 18 2007, 22:08:04) [GCC 4.0.1 
(Apple Computer, Inc. build 5367)]
...


...................................................../Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/core/ma.py:608: 
UserWarning: Cannot automatically convert masked array to numeric 
because data
     is masked in one or more locations.
   warnings.warn("Cannot automatically convert masked array to "\
F.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
......................................................................................................................................................................................................................................................................................
======================================================================
FAIL: check_testUfuncRegression (numpy.core.tests.test_ma.test_ufuncs)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/core/tests/test_ma.py", 
line 692, in check_testUfuncRegression
     self.failUnless(eqmask(ur.mask, mr.mask))
AssertionError

----------------------------------------------------------------------
Ran 1321 tests in 4.608s

FAILED (failures=1)
<unittest._TextTestResult run=1321 errors=0 failures=1>
 >>>


Is that expected?

OS-X 10.4.11, Dual G5 PPC, Python2.5.1 python.org build.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From pgmdevlist at gmail.com  Mon Jun  2 13:54:49 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 2 Jun 2008 13:54:49 -0400
Subject: [Numpy-discussion] Test Faliures on OS-X
In-Reply-To: <48443202.5050804@noaa.gov>
References: <91cf711d0806020820o17343fdfua8fa2ee9ed91f081@mail.gmail.com>
	<91cf711d0806021009h3444992n63b2dc8a13d93b89@mail.gmail.com>
	<48443202.5050804@noaa.gov>
Message-ID: <200806021354.49233.pgmdevlist@gmail.com>

On Monday 02 June 2008 13:46:42 Christopher Barker wrote:
> HI all,
>
> I missed a bit of the release process, but I thought all the test
> failures had been squashed. However, I just installed the OS-X 1.1.0
> binary, and got:
>  >>> import numpy
>  >>> numpy.__version__

Mmh, there's something wrong in your installation: there shouldn't be any 
ma.py file in numpy.core, nor test_ma.py in in numpy/core/tests.


From Chris.Barker at noaa.gov  Mon Jun  2 14:11:55 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 02 Jun 2008 11:11:55 -0700
Subject: [Numpy-discussion] Test Faliures on OS-X
In-Reply-To: <200806021354.49233.pgmdevlist@gmail.com>
References: <91cf711d0806020820o17343fdfua8fa2ee9ed91f081@mail.gmail.com>
	<91cf711d0806021009h3444992n63b2dc8a13d93b89@mail.gmail.com>
	<48443202.5050804@noaa.gov> <200806021354.49233.pgmdevlist@gmail.com>
Message-ID: <484437EB.9030507@noaa.gov>

Pierre GM wrote:

> Mmh, there's something wrong in your installation: there shouldn't be any 
> ma.py file in numpy.core, nor test_ma.py in in numpy/core/tests.

OK, I'll do a clean-up and try again. this is probably due to having 
previously installed a release candidate a while back...

Bingo!:

Ran 1283 tests in 3.746s

OK
<unittest._TextTestResult run=1283 errors=0 failures=0>

All is well. Thanks.

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From cournapeau at cslab.kecl.ntt.co.jp  Mon Jun  2 21:16:38 2008
From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau)
Date: Tue, 03 Jun 2008 10:16:38 +0900
Subject: [Numpy-discussion] numpy 1.1.0 build can not find lahey
	(lf95)	fortran compiler (short version)
In-Reply-To: <128818DD3B256A41A94F4F2564E2FABF054DF5B8@MAIL-STORE-2.pppl.gov>
References: <128818DD3B256A41A94F4F2564E2FABF054DF5B4@MAIL-STORE-2.pppl.gov>
	<128818DD3B256A41A94F4F2564E2FABF054DF5B5@MAIL-STORE-2.pppl.gov>
	<4843E60E.30707@ar.media.kyoto-u.ac.jp>
	<128818DD3B256A41A94F4F2564E2FABF054DF5B8@MAIL-STORE-2.pppl.gov>
Message-ID: <1212455798.15771.1.camel@bbc8>

On Mon, 2008-06-02 at 11:03 -0400, Lewis E. Randerson wrote:
> Found the problem.
> 
> The pattern in lahey.py differs from the version of Lahey Fujitsu
> compiler I have.  Once corrected, the compiler is now found. 

Ok, thanks for going through this. Can I ask you one more thing ? Can
you give us the output of the Lahey fortran compiler output ? Such as we
can add one test to the version parsing and avoid getting bitten by this
next time

cheers,

David



From pgmdevlist at gmail.com  Mon Jun  2 23:04:56 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 2 Jun 2008 23:04:56 -0400
Subject: [Numpy-discussion] NumpyTest problem
In-Reply-To: <1d36917a0806020845k55805582j848a9d746e08e587@mail.gmail.com>
References: <91cf711d0806020820o17343fdfua8fa2ee9ed91f081@mail.gmail.com>
	<1d36917a0806020845k55805582j848a9d746e08e587@mail.gmail.com>
Message-ID: <200806022304.57203.pgmdevlist@gmail.com>

On Monday 02 June 2008 11:45:31 Alan McIntyre wrote:
> We're in the process of switching to nose
> (http://www.somethingaboutorange.com/mrl/projects/nose/) as the test
> framework for 1.2; I'll try to keep an eye on stuff like that and make
> it work properly if I can.

Alan,
Thanks a lot again for taking care of that. In the meanwhile:

* is there a way to select only some tests ? It used to be a time not that 
long ago (less than 6 weeks) where I was able to run one specific test of a 
suite (for example, numpy/ma/tests/test_core.py), and that's no longer the 
case. 

* Running the whole test suite with import numpy; numpy.test() works fine, but 
I don't really need to test the rest of numpy when I'm messing with 
numpy.ma...

* Running nosetests on numpy/ma fails (I can't import numpy from the source 
directory), but works partially on numpy/ma/tests: the test_old_ma tests fail 
for asking some arguments when 0 is provided: for example:
================================================
ERROR: test_old_ma.testta
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/nose/case.py", line 203, in runTest
    self.test(*self.arg)
TypeError: testta() takes exactly 2 arguments (0 given)

Thx again


From hanni.ali at gmail.com  Tue Jun  3 04:21:11 2008
From: hanni.ali at gmail.com (Hanni Ali)
Date: Tue, 3 Jun 2008 09:21:11 +0100
Subject: [Numpy-discussion] Installation info
In-Reply-To: <789d27b10806020526h47d08418v4f0e0752bbcabb1a@mail.gmail.com>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>
	<6be8b94a0805300431p14eb1ae0pc5fd111db9b0503@mail.gmail.com>
	<789d27b10805300435t361d12a8j59f69aadefd3f9e6@mail.gmail.com>
	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>
	<789d27b10806020141m43d8d699u8d9dbe8d593d8e3e@mail.gmail.com>
	<4843B4AF.4070605@ar.media.kyoto-u.ac.jp>
	<789d27b10806020230l1d88f270l52a62ec90954aeb7@mail.gmail.com>
	<4843BDD2.8060002@ar.media.kyoto-u.ac.jp>
	<789d27b10806020526h47d08418v4f0e0752bbcabb1a@mail.gmail.com>
Message-ID: <789d27b10806030121s3a3df8a0q3a7bbdd173515ba7@mail.gmail.com>

Hi David,

I compiled numpy with MSVC 9.0 (vs 2008), I am just using the inbuilt LA
libs to minimise complexity.

Although I have hacked it such that I can compile and all but one of the
regression tests passed:

======================================================================
ERROR: Tests reading from a text file.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python26\lib\site-packages\numpy\ma\tests\test_mrecords.py", line
363
, in test_fromtextfile
    fname = 'tmp%s' % datetime.now().strftime("%y%m%d%H%M%S%s")
ValueError: Invalid format string
----------------------------------------------------------------------
Ran 1267 tests in 1.141s
FAILED (errors=1)
<unittest._TextTestResult run=1267 errors=1 failures=0>

This appears to be a problem with the strftime function in test_mrecords.py

The error seems to be created by the millisecond formatting argument %s,
removing this caused the test to pass.

So I think it's all ok really, however in order to get numpy to compile I
have commented out a small part which was causing compilation to fail:

numpy\core\src\umathmodule.c.src(64) : error C2059: syntax error : 'type'
numpy\core\src\umathmodule.c.src(70) : error C2059: syntax error : 'type'

This relates to this section of code:

#ifndef HAVE_FREXPF
static float frexpf(float x, int * i)
{
    return (float)frexp((double)(x), i);
}
#endif
#ifndef HAVE_LDEXPF
static float ldexpf(float x, int i)
{
    return (float)ldexp((double)(x), i);
}
#endif

The compiler directives HAVE_FREXPF and HAVE_LDEXPF do not appear to be
recognised by msvc 9 would you agree with that assessment?

And a redefinition of a function present in the stdc library is occurring.

What do you think? By just commenting out this piece of code numpy compiles
and appears to function.

Hanni




2008/6/2 Hanni Ali <hanni.ali at gmail.com>:

> Excellenm, thanks for clearing all that up.
>
> How about numpy with 2.6, any issues?
>
> Hanni
>
> 2008/6/2 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>
>  Hanni Ali wrote:
>> >
>> > Yes I had used the internal versions in the mean time, but I do want
>> > to try to use the intel fortran compiler in all likelyhood.
>>
>> Yes, people can try to build as they want, that's the beauty of open
>> source :) But for official distribution, I don't want to depend on non
>> free software (outside windows, obviously). I need a process to build
>> numpy in a reproducible and in a hassle way (well, as much as possible,
>> at least).
>>
>> >
>> > Python 2.6 is compiled with vs 2008 is is important that numpy is also
>> > compiled with the same compiler or not. The fact that a fortran
>> > compiler is necessary makes me think no?
>>
>> This has nothing to do with fortran per se, but with the fact that MS
>> keeps breaking the standard C runtime. Hopefully, with python 3, this
>> may not be an issue anymore (python on windows won't use the standard C
>> API, but windows API instead, because MS guarantees ABI in this case, at
>> least that's what I understood).
>>
>> The fortran compiler has to be the same to build everything, but python
>> does not use any fortran, so you can choose whatever you want as long as
>> you use it for everything (BLAS, LAPACK and numpy).
>>
>> >
>> > It looks like I'm going to look at 2.6 now due to dependencies on
>> > pywin32 as well.
>> >
>> > Also is it important that BLAS/LAPACK are compiled with the same
>> > compiler as python or not?
>>
>> BLAS/LAPACK is built with fortran, and python with C compiler, so no :)
>> What may be important is the C++ compiler to build (more exactly to
>> link) python if you build python by yourself, but I don't know how this
>> works on windows. I just build softwares for windows, I don't use it.
>>
>> cheers,
>>
>> David
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080603/8738ed7b/attachment.html>

From maitai02 at excite.com  Tue Jun  3 12:21:56 2008
From: maitai02 at excite.com (Jose Martin)
Date: Tue,  3 Jun 2008 12:21:56 -0400 (EDT)
Subject: [Numpy-discussion] Pointers to arrays in C
Message-ID: <20080603162156.B7B558B35B@xprdmxin.myway.com>


Hi, I read a file with array data (one per line), and send the arrays to a module in C. In the C module, I need to store pointers to the arrays, so I don't have to make a copy for each array it receives from python.

I found that if I reuse the same variable name to create the array, the pointer's values in C program will be lost next time the variable is reassigned. 

For example:
- for line in file:
  - myarray=create an int array from line
  - my_c_module.function(myarray)

'myarray' is reused to create a new array for each line, however I see that the data in myarray in previous loops (which is stored in C program) is destroyed.

To avoid this, I create a python list that appends each 'myarray' instances, and thus are not overwritten. Is there a better approach to avoid this?

Thanks in advance!
 
  




_______________________________________________
Join Excite! - http://www.excite.com
The most personalized portal on the Web!




From ahalda at cs.mcgill.ca  Tue Jun  3 12:29:36 2008
From: ahalda at cs.mcgill.ca (ahalda at cs.mcgill.ca)
Date: Tue, 3 Jun 2008 12:29:36 -0400 (EDT)
Subject: [Numpy-discussion] object-arrays and isnan
Message-ID: <57778.204.52.215.70.1212510576.squirrel@mail.cs.mcgill.ca>

Hi

I'm trying to make a python class to be used in object arrays
(specifically, an mpfr type for multiprecision). Numpy lets me create
member functions like 'cos' which get called elemenwise when I call cos(a)
on an object array a. However, this doesn't work for some functions, like
isnan.

Looking in the numpy code, I see in
numpy/core/code_generators/generate_umath.py that the type description for
isnan does not include objects (type 'M'). Every math ufunc takes objects
in the type description except the following ones:

isnan
isfinite
isinf
signbit
modf

Is there a reason for this? I want to be able to use my mpfr type with
scipy functions, but for example I can't use optimize.fmin_bfgs because it
calls isnan. Other scipy functions work great though,.

If there is a reason, how can I get around this?



In case I was not clear, here is a demonstration:


from scipy import cos, isnan, array

class myNumber:
        def __init__(self, n):
                self.n = n

        def cos(self):
                return cos(self.n)

        def isnan(self):
                return isnan(self.n)

a = array([myNumber(3)])

print "cos: %s"  % str( cos(a)   )
print "isnan %s" % str( isnan(a) )


which prints:


cos: [-0.9899924966]
Traceback (most recent call last):
  File "./test.py", line 18, in <module>
    print "isnan %s" % str( isnan(a) )
TypeError: function not supported for these types, and can't coerce safely
to supported types


Thanks,
Allan





From dalcinl at gmail.com  Tue Jun  3 13:08:56 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Jun 2008 14:08:56 -0300
Subject: [Numpy-discussion] Pointers to arrays in C
In-Reply-To: <20080603162156.B7B558B35B@xprdmxin.myway.com>
References: <20080603162156.B7B558B35B@xprdmxin.myway.com>
Message-ID: <e7ba66e40806031008h4e570f98sa646287c33f20f9e@mail.gmail.com>

I believe in your current setup there is no better way. But you should
seriously consider changing the way of using array data. Storing bare
pointers in the C side and not holding a reference to the object
providing the data in the C side is really error prone.


On 6/3/08, Jose Martin <maitai02 at excite.com> wrote:
>
>  Hi, I read a file with array data (one per line), and send the arrays to a module in C. In the C module, I need to store pointers to the arrays, so I don't have to make a copy for each array it receives from python.
>
>  I found that if I reuse the same variable name to create the array, the pointer's values in C program will be lost next time the variable is reassigned.
>
>  For example:
>  - for line in file:
>   - myarray=create an int array from line
>   - my_c_module.function(myarray)
>
>  'myarray' is reused to create a new array for each line, however I see that the data in myarray in previous loops (which is stored in C program) is destroyed.
>
>  To avoid this, I create a python list that appends each 'myarray' instances, and thus are not overwritten. Is there a better approach to avoid this?
>
>  Thanks in advance!
>
>
>
>
>
>
>  _______________________________________________
>  Join Excite! - http://www.excite.com
>  The most personalized portal on the Web!
>
>
>  _______________________________________________
>  Numpy-discussion mailing list
>  Numpy-discussion at scipy.org
>  http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594


From Chris.Barker at noaa.gov  Tue Jun  3 13:27:09 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 03 Jun 2008 10:27:09 -0700
Subject: [Numpy-discussion] Pointers to arrays in C
In-Reply-To: <e7ba66e40806031008h4e570f98sa646287c33f20f9e@mail.gmail.com>
References: <20080603162156.B7B558B35B@xprdmxin.myway.com>
	<e7ba66e40806031008h4e570f98sa646287c33f20f9e@mail.gmail.com>
Message-ID: <48457EED.2010600@noaa.gov>

Lisandro Dalcin wrote:
> I believe in your current setup there is no better way. But you should
> seriously consider changing the way of using array data. Storing bare
> pointers in the C side and not holding a reference to the object
> providing the data in the C side is really error prone.

exactly.

as you are passing a ndarray into your C code, another option is that 
you could increase the reference count to that array in your C code 
(Py_INCREF). Then it would not be destroyed on the Python side. However, 
you'd have to make sure you Py_DECREF it when you don't need it anymore.

There may be a way to tell the array that is does not "own" the data 
block also, but I don't know the numpy API enough to tell you how to do 
that.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From robert.kern at gmail.com  Tue Jun  3 14:48:35 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 3 Jun 2008 13:48:35 -0500
Subject: [Numpy-discussion] Installation info
In-Reply-To: <789d27b10806030121s3a3df8a0q3a7bbdd173515ba7@mail.gmail.com>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>
	<6be8b94a0805300431p14eb1ae0pc5fd111db9b0503@mail.gmail.com>
	<789d27b10805300435t361d12a8j59f69aadefd3f9e6@mail.gmail.com>
	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>
	<789d27b10806020141m43d8d699u8d9dbe8d593d8e3e@mail.gmail.com>
	<4843B4AF.4070605@ar.media.kyoto-u.ac.jp>
	<789d27b10806020230l1d88f270l52a62ec90954aeb7@mail.gmail.com>
	<4843BDD2.8060002@ar.media.kyoto-u.ac.jp>
	<789d27b10806020526h47d08418v4f0e0752bbcabb1a@mail.gmail.com>
	<789d27b10806030121s3a3df8a0q3a7bbdd173515ba7@mail.gmail.com>
Message-ID: <3d375d730806031148s39f59d4dl9b7bb4b6ce2f611a@mail.gmail.com>

On Tue, Jun 3, 2008 at 3:21 AM, Hanni Ali <hanni.ali at gmail.com> wrote:
> Hi David,
>
> I compiled numpy with MSVC 9.0 (vs 2008), I am just using the inbuilt LA
> libs to minimise complexity.
>
> Although I have hacked it such that I can compile and all but one of the
> regression tests passed:
>
> ======================================================================
> ERROR: Tests reading from a text file.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "C:\Python26\lib\site-packages\numpy\ma\tests\test_mrecords.py", line
> 363
> , in test_fromtextfile
>     fname = 'tmp%s' % datetime.now().strftime("%y%m%d%H%M%S%s")
> ValueError: Invalid format string
> ----------------------------------------------------------------------
> Ran 1267 tests in 1.141s
> FAILED (errors=1)
> <unittest._TextTestResult run=1267 errors=1 failures=0>
>
> This appears to be a problem with the strftime function in test_mrecords.py
>
> The error seems to be created by the millisecond formatting argument %s,
> removing this caused the test to pass.

Well, this should be using tempfile anyways.

> So I think it's all ok really, however in order to get numpy to compile I
> have commented out a small part which was causing compilation to fail:
>
> numpy\core\src\umathmodule.c.src(64) : error C2059: syntax error : 'type'
> numpy\core\src\umathmodule.c.src(70) : error C2059: syntax error : 'type'
>
> This relates to this section of code:
>
> #ifndef HAVE_FREXPF
> static float frexpf(float x, int * i)
> {
>     return (float)frexp((double)(x), i);
> }
> #endif
> #ifndef HAVE_LDEXPF
> static float ldexpf(float x, int i)
> {
>     return (float)ldexp((double)(x), i);
> }
> #endif
>
> The compiler directives HAVE_FREXPF and HAVE_LDEXPF do not appear to be
> recognised by msvc 9 would you agree with that assessment?
>
> And a redefinition of a function present in the stdc library is occurring.
>
> What do you think? By just commenting out this piece of code numpy compiles
> and appears to function.

The presence of these functions should have been detected by the
configuration process of numpy. HAVE_FREXPF and HAVE_LDEXPF would have
been #define'd if we had detected them correctly. It is possible that
our configuration process for this does not work correctly with VS
2008. From a clean checkout, can you please do the build again and
copy-and-paste everything printed to the terminal?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From maitai02 at excite.com  Tue Jun  3 15:25:29 2008
From: maitai02 at excite.com (Jose Martin)
Date: Tue,  3 Jun 2008 15:25:29 -0400 (EDT)
Subject: [Numpy-discussion] Pointers to arrays in C
Message-ID: <20080603192529.526C98B37A@xprdmxin.myway.com>



Thanks for the fast responses!

> data. Storing bare pointers in the C side and not holding a
> reference to the object providing the data in the C side is really 
> error prone.

It's true, I don't do it because I have to process a large number of arrays, and each has thousands of elements; so I tried to avoid making copies because it'd make it slow. 

 
> another option is that you could increase the reference count to 
> that array in your C code (Py_INCREF). Then it would not be 
> destroyed on the Python side. However, you'd have to make sure you
> Py_DECREF it when you don't need it anymore.

Thanks for the suggestion, I didn't know that was possible; I'm new to python.

-Jose  





_______________________________________________
Join Excite! - http://www.excite.com
The most personalized portal on the Web!




From w.gardner at utah.edu  Tue Jun  3 17:19:10 2008
From: w.gardner at utah.edu (Payton Gardner)
Date: Tue, 3 Jun 2008 15:19:10 -0600
Subject: [Numpy-discussion] linked variables?
Message-ID: <25F1CF2C-3012-46C7-8DA9-AAD810AAC915@utah.edu>

Its probably something simple I don't understand but...

I've written a dummy function which takes an array m.  I'd like it to  
return a changed array m_i, and not change the initial array m.  I  
call it with mm = dummy(m);

      3 from numpy import *;
       4 def dummy(m):
       5     m_o = m;
       6     pdb.set_trace();
       7     m_step  = 100;
       8     m_i = m_o;
       9     dummy = m;
      10     dummy2 = m_o;
      11     dummy3 = m_o;
      12     i = 0;
      13     m_i[i] = m_i[i] + m_step;
      14     return(m_i);

But after line 13 m, m_o, dummy, dummy2, and dummy3 are all changed  
by m_step, as well mm which is in a different name space.  All I've  
asked it to do is change m_i.  What's happening?

Thanks,
Payton




Payton Gardner
University of Utah
Department of Geology and Geophysics





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080603/7f539e09/attachment.html>

From matthieu.brucher at gmail.com  Tue Jun  3 17:27:55 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Tue, 3 Jun 2008 23:27:55 +0200
Subject: [Numpy-discussion] linked variables?
In-Reply-To: <25F1CF2C-3012-46C7-8DA9-AAD810AAC915@utah.edu>
References: <25F1CF2C-3012-46C7-8DA9-AAD810AAC915@utah.edu>
Message-ID: <e76aa17f0806031427p2153c03dt9653e8b3f163d9@mail.gmail.com>

Hi,

m is a variable.
m_o refers to m.
m_i refers to m_o which is m.
dummy refers to m.
dummy2 and dummy3 refer to m_o which is m.

So when you modify m_i, you are modifying the variable refered by m_i, m and
also m_o, dummy, dummy2 and dummy3.

It's always the same object, with different names. Contrary to Matlab,
Python does not copy variables, it creates references to them and copies
them only explicitely.

Matthieu

2008/6/3 Payton Gardner <w.gardner at utah.edu>:

> Its probably something simple I don't understand but...
>
> I've written a dummy function which takes an array m.  I'd like it to
> return a changed array m_i, and not change the initial array m.  I call it
> with mm = dummy(m);
>
>      3 from numpy import *;
>       4 def dummy(m):
>       5     m_o = m;
>       6     pdb.set_trace();
>       7     m_step  = 100;
>       8     m_i = m_o;
>       9     dummy = m;
>      10     dummy2 = m_o;
>      11     dummy3 = m_o;
>      12     i = 0;
>      13     m_i[i] = m_i[i] + m_step;
>      14     return(m_i);
>
> But after line 13 m, m_o, dummy, dummy2, and dummy3 are all changed by
> m_step, as well mm which is in a different name space.  All I've asked it to
> do is change m_i.  What's happening?
>
> Thanks,
> Payton
>
>
>
>
> Payton Gardner
> University of Utah
> Department of Geology and Geophysics
>
>
>
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080603/77faf9d4/attachment.html>

From robert.kern at gmail.com  Tue Jun  3 17:28:41 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 3 Jun 2008 16:28:41 -0500
Subject: [Numpy-discussion] linked variables?
In-Reply-To: <25F1CF2C-3012-46C7-8DA9-AAD810AAC915@utah.edu>
References: <25F1CF2C-3012-46C7-8DA9-AAD810AAC915@utah.edu>
Message-ID: <3d375d730806031428i4e480896tdc541cb47bb3ae72@mail.gmail.com>

On Tue, Jun 3, 2008 at 4:19 PM, Payton Gardner <w.gardner at utah.edu> wrote:
> Its probably something simple I don't understand but...
> I've written a dummy function which takes an array m.  I'd like it to return
> a changed array m_i, and not change the initial array m.  I call it with mm
> = dummy(m);
>      3 from numpy import *;
>       4 def dummy(m):
>       5     m_o = m;
>       6     pdb.set_trace();
>       7     m_step  = 100;
>       8     m_i = m_o;
>       9     dummy = m;
>      10     dummy2 = m_o;
>      11     dummy3 = m_o;
>      12     i = 0;
>      13     m_i[i] = m_i[i] + m_step;
>      14     return(m_i);
> But after line 13 m, m_o, dummy, dummy2, and dummy3 are all changed by
> m_step, as well mm which is in a different name space.  All I've asked it to
> do is change m_i.  What's happening?

Python does not copy data when you assign something to a new variable.
Python simply points the new name to the same object. If you modify
the object using the new name, all of the other names pointing to that
object will see the changes. If you want a copy, you will need to
explicitly make one. For numpy arrays, the best way to do this is to
use array().

  m_i = array(m_o)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From j.stark at imperial.ac.uk  Tue Jun  3 17:33:44 2008
From: j.stark at imperial.ac.uk (J. Stark)
Date: Tue, 3 Jun 2008 22:33:44 +0100
Subject: [Numpy-discussion] 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <mailman.32751.1212409596.27196.numpy-discussion@scipy.org>
References: <mailman.32751.1212409596.27196.numpy-discussion@scipy.org>
Message-ID: <p06240800c46b67b9b501@[10.0.0.16]>

I have just tried to run the 1.1.0 OSX installer on a MacBookAir 
running 10.5.3 and the installer fails with

"You cannot install numpy 1.1.0 on this volume. numpy requires System 
Python 2.5 to install."

The system python version reports as

jaroslav$ python
Python 2.5.1 (r251:54863, Apr 15 2008, 22:57:26)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin

which is the same version that Leopard has had all along, as far as I 
am aware. On the the other hand, there have been some reports on 
PythonMac about odd python behaviour following the 10.5.3 upgrade.

Has anybody used this installer successfully under 10.5.3, and/or 
have any idea of what is going on.

Incidentally, this is a new machine with just the default system installation.

Jaroslav


From pgmdevlist at gmail.com  Tue Jun  3 17:20:21 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 3 Jun 2008 17:20:21 -0400
Subject: [Numpy-discussion] linked variables?
In-Reply-To: <e76aa17f0806031427p2153c03dt9653e8b3f163d9@mail.gmail.com>
References: <25F1CF2C-3012-46C7-8DA9-AAD810AAC915@utah.edu>
	<e76aa17f0806031427p2153c03dt9653e8b3f163d9@mail.gmail.com>
Message-ID: <200806031720.22232.pgmdevlist@gmail.com>

Payton,
In your example, use 
>>>m_o = m.copy() 
or
>>>m_o = m + 0 
or
>>>m_o = numpy.array(m, copy=True)


From tgrav at mac.com  Tue Jun  3 17:41:43 2008
From: tgrav at mac.com (Tommy Grav)
Date: Tue, 3 Jun 2008 17:41:43 -0400
Subject: [Numpy-discussion] 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <p06240800c46b67b9b501@[10.0.0.16]>
References: <mailman.32751.1212409596.27196.numpy-discussion@scipy.org>
	<p06240800c46b67b9b501@[10.0.0.16]>
Message-ID: <3D69AB81-8909-436B-9103-96ED2CC49172@mac.com>

Where is your python located> I have installed numpy 1.1.0 using the
binary installer successfully on 10.5.3 but I am using ActiveState
python. I think the problem might be that the installer looks in
/Library/Frameworks/Python.framework/ for python, while the standard
python is located somewhere else.

Cheers
   Tommy

On Jun 3, 2008, at 5:33 PM, J. Stark wrote:

> I have just tried to run the 1.1.0 OSX installer on a MacBookAir
> running 10.5.3 and the installer fails with
>
> "You cannot install numpy 1.1.0 on this volume. numpy requires System
> Python 2.5 to install."
>
> The system python version reports as
>
> jaroslav$ python
> Python 2.5.1 (r251:54863, Apr 15 2008, 22:57:26)
> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
>
> which is the same version that Leopard has had all along, as far as I
> am aware. On the the other hand, there have been some reports on
> PythonMac about odd python behaviour following the 10.5.3 upgrade.
>
> Has anybody used this installer successfully under 10.5.3, and/or
> have any idea of what is going on.
>
> Incidentally, this is a new machine with just the default system  
> installation.
>
> Jaroslav
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From peridot.faceted at gmail.com  Tue Jun  3 19:25:46 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 3 Jun 2008 19:25:46 -0400
Subject: [Numpy-discussion] linked variables?
In-Reply-To: <3d375d730806031428i4e480896tdc541cb47bb3ae72@mail.gmail.com>
References: <25F1CF2C-3012-46C7-8DA9-AAD810AAC915@utah.edu>
	<3d375d730806031428i4e480896tdc541cb47bb3ae72@mail.gmail.com>
Message-ID: <ce557a360806031625t42b4c49es4d62df6314259b3d@mail.gmail.com>

2008/6/3 Robert Kern <robert.kern at gmail.com>:

> Python does not copy data when you assign something to a new variable.
> Python simply points the new name to the same object. If you modify
> the object using the new name, all of the other names pointing to that
> object will see the changes. If you want a copy, you will need to
> explicitly make one. For numpy arrays, the best way to do this is to
> use array().
>
>  m_i = array(m_o)

Is array() really the best way to copy an array? I would have thought
m_i = m_o.copy() would be better - requiring less of array's clever
guessing, and preserving subtypes. On the downside it doesn't work if
you're given a list, but perhaps np.copy() would be better (though it
loses subtype information)? Certainly it's clearer.

Anne


From robert.kern at gmail.com  Tue Jun  3 20:02:54 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 3 Jun 2008 19:02:54 -0500
Subject: [Numpy-discussion] linked variables?
In-Reply-To: <ce557a360806031625t42b4c49es4d62df6314259b3d@mail.gmail.com>
References: <25F1CF2C-3012-46C7-8DA9-AAD810AAC915@utah.edu>
	<3d375d730806031428i4e480896tdc541cb47bb3ae72@mail.gmail.com>
	<ce557a360806031625t42b4c49es4d62df6314259b3d@mail.gmail.com>
Message-ID: <3d375d730806031702k49c8200bt3a6dbbafeeb4c56@mail.gmail.com>

On Tue, Jun 3, 2008 at 6:25 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> 2008/6/3 Robert Kern <robert.kern at gmail.com>:
>
>> Python does not copy data when you assign something to a new variable.
>> Python simply points the new name to the same object. If you modify
>> the object using the new name, all of the other names pointing to that
>> object will see the changes. If you want a copy, you will need to
>> explicitly make one. For numpy arrays, the best way to do this is to
>> use array().
>>
>>  m_i = array(m_o)
>
> Is array() really the best way to copy an array? I would have thought
> m_i = m_o.copy() would be better - requiring less of array's clever
> guessing, and preserving subtypes.

The latter is true, but I didn't want to get into subtypes. In this
case, array's clever guessing involves recognizing that it is an
ndarray and doing the right thing. There is no recursive traversal
like with lists of lists.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Tue Jun  3 22:49:28 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 04 Jun 2008 11:49:28 +0900
Subject: [Numpy-discussion] Installation info
In-Reply-To: <3d375d730806031148s39f59d4dl9b7bb4b6ce2f611a@mail.gmail.com>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>	<6be8b94a0805300431p14eb1ae0pc5fd111db9b0503@mail.gmail.com>	<789d27b10805300435t361d12a8j59f69aadefd3f9e6@mail.gmail.com>	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>	<789d27b10806020141m43d8d699u8d9dbe8d593d8e3e@mail.gmail.com>	<4843B4AF.4070605@ar.media.kyoto-u.ac.jp>	<789d27b10806020230l1d88f270l52a62ec90954aeb7@mail.gmail.com>	<4843BDD2.8060002@ar.media.kyoto-u.ac.jp>	<789d27b10806020526h47d08418v4f0e0752bbcabb1a@mail.gmail.com>	<789d27b10806030121s3a3df8a0q3a7bbdd173515ba7@mail.gmail.com>
	<3d375d730806031148s39f59d4dl9b7bb4b6ce2f611a@mail.gmail.com>
Message-ID: <484602B8.7000906@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
>
> The presence of these functions should have been detected by the
> configuration process of numpy. HAVE_FREXPF and HAVE_LDEXPF would have
> been #define'd if we had detected them correctly. It is possible that
> our configuration process for this does not work correctly with VS
> 2008. From a clean checkout, can you please do the build again and
> copy-and-paste everything printed to the terminal?
>   

As an aside, I don't understand how the configuration is supposed to 
work there: why do we define HAVE_LONGDOUBLE, etc...  for a serie of 
functions ? I don't understand why for example testing for expf is 
supposed to be enough to guarantee the presence of the whole float 
functions (HAVE_FLOAT_FUNCS). What is the rationale for that ? Why not 
testing for every function we are using ?

David


From robert.kern at gmail.com  Tue Jun  3 23:30:43 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 3 Jun 2008 22:30:43 -0500
Subject: [Numpy-discussion] Installation info
In-Reply-To: <484602B8.7000906@ar.media.kyoto-u.ac.jp>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>
	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>
	<789d27b10806020141m43d8d699u8d9dbe8d593d8e3e@mail.gmail.com>
	<4843B4AF.4070605@ar.media.kyoto-u.ac.jp>
	<789d27b10806020230l1d88f270l52a62ec90954aeb7@mail.gmail.com>
	<4843BDD2.8060002@ar.media.kyoto-u.ac.jp>
	<789d27b10806020526h47d08418v4f0e0752bbcabb1a@mail.gmail.com>
	<789d27b10806030121s3a3df8a0q3a7bbdd173515ba7@mail.gmail.com>
	<3d375d730806031148s39f59d4dl9b7bb4b6ce2f611a@mail.gmail.com>
	<484602B8.7000906@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730806032030u5af327bbq6a9a649688e20ae5@mail.gmail.com>

On Tue, Jun 3, 2008 at 9:49 PM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Robert Kern wrote:
>>
>> The presence of these functions should have been detected by the
>> configuration process of numpy. HAVE_FREXPF and HAVE_LDEXPF would have
>> been #define'd if we had detected them correctly. It is possible that
>> our configuration process for this does not work correctly with VS
>> 2008. From a clean checkout, can you please do the build again and
>> copy-and-paste everything printed to the terminal?
>
> As an aside, I don't understand how the configuration is supposed to
> work there: why do we define HAVE_LONGDOUBLE, etc...  for a serie of
> functions ? I don't understand why for example testing for expf is
> supposed to be enough to guarantee the presence of the whole float
> functions (HAVE_FLOAT_FUNCS). What is the rationale for that ?

Someone thought that it was a sufficient condition. If it is not, then
it should be fixed.

> Why not
> testing for every function we are using ?

There are a lot of them. Feel free to add any additional tests you
think are necessary, and we'll see how painful it is at build-time.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From charlesr.harris at gmail.com  Tue Jun  3 23:36:19 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 3 Jun 2008 21:36:19 -0600
Subject: [Numpy-discussion] Installation info
In-Reply-To: <484602B8.7000906@ar.media.kyoto-u.ac.jp>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>
	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>
	<789d27b10806020141m43d8d699u8d9dbe8d593d8e3e@mail.gmail.com>
	<4843B4AF.4070605@ar.media.kyoto-u.ac.jp>
	<789d27b10806020230l1d88f270l52a62ec90954aeb7@mail.gmail.com>
	<4843BDD2.8060002@ar.media.kyoto-u.ac.jp>
	<789d27b10806020526h47d08418v4f0e0752bbcabb1a@mail.gmail.com>
	<789d27b10806030121s3a3df8a0q3a7bbdd173515ba7@mail.gmail.com>
	<3d375d730806031148s39f59d4dl9b7bb4b6ce2f611a@mail.gmail.com>
	<484602B8.7000906@ar.media.kyoto-u.ac.jp>
Message-ID: <e06186140806032036j5b2e292p76133cf6e18434b6@mail.gmail.com>

On Tue, Jun 3, 2008 at 8:49 PM, David Cournapeau <
david at ar.media.kyoto-u.ac.jp> wrote:

> Robert Kern wrote:
> >
> > The presence of these functions should have been detected by the
> > configuration process of numpy. HAVE_FREXPF and HAVE_LDEXPF would have
> > been #define'd if we had detected them correctly. It is possible that
> > our configuration process for this does not work correctly with VS
> > 2008. From a clean checkout, can you please do the build again and
> > copy-and-paste everything printed to the terminal?
> >
>
> As an aside, I don't understand how the configuration is supposed to
> work there: why do we define HAVE_LONGDOUBLE, etc...  for a serie of
> functions ? I don't understand why for example testing for expf is
> supposed to be enough to guarantee the presence of the whole float
> functions (HAVE_FLOAT_FUNCS). What is the rationale for that ? Why not
> testing for every function we are using ?
>

It probably just grew to fix problems as they arose. It should be possible
to test for every function and fall back to the double versions that are
more reliably present. It would be nicer if all compilers tried to conform
to recent standards, i.e., be less than 9 years out of date, but that is a
bit much to ask for.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080603/c9768e12/attachment.html>

From cburns at berkeley.edu  Tue Jun  3 23:38:32 2008
From: cburns at berkeley.edu (Christopher Burns)
Date: Tue, 3 Jun 2008 20:38:32 -0700
Subject: [Numpy-discussion] 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <p06240800c46b67b9b501@10.0.0.16>
References: <mailman.32751.1212409596.27196.numpy-discussion@scipy.org>
	<p06240800c46b67b9b501@10.0.0.16>
Message-ID: <764e38540806032038g11f28ddbm574a4957c5a0e715@mail.gmail.com>

Jaroslav,

The installer works with the MacPython from python.org, not Apple's python
(the one that ships with Leopard).

The MacPython is installed in the /Library/Frameworks... It should work if
your python is here:

cburns$  python -c "import sys; print sys.prefix"
/Library/Frameworks/Python.framework/Versions/2.5

Won't work if your python is here:

cburns$  python-apple -c "import sys; print sys.prefix"
/System/Library/Frameworks/Python.framework/Versions/2.5

Sorry for the confusion,
Chris

On Tue, Jun 3, 2008 at 2:33 PM, J. Stark <j.stark at imperial.ac.uk> wrote:

> I have just tried to run the 1.1.0 OSX installer on a MacBookAir
> running 10.5.3 and the installer fails with
>
> "You cannot install numpy 1.1.0 on this volume. numpy requires System
> Python 2.5 to install."
>
> The system python version reports as
>
> jaroslav$ python
> Python 2.5.1 (r251:54863, Apr 15 2008, 22:57:26)
> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
>
> which is the same version that Leopard has had all along, as far as I
> am aware. On the the other hand, there have been some reports on
> PythonMac about odd python behaviour following the 10.5.3 upgrade.
>
> Has anybody used this installer successfully under 10.5.3, and/or
> have any idea of what is going on.
>
> Incidentally, this is a new machine with just the default system
> installation.
>
> Jaroslav
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Christopher Burns
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080603/5d4447f9/attachment.html>

From peridot.faceted at gmail.com  Tue Jun  3 23:59:28 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 3 Jun 2008 23:59:28 -0400
Subject: [Numpy-discussion] How does Boolean indexing work?
Message-ID: <ce557a360806032059k64a51323w8645fe98d5114a31@mail.gmail.com>

Hi,

I thought it might be useful to summarize the different ways to use
numpy's indexing, slice and fancy. The document so far is here:

http://www.scipy.org/Cookbook/Indexing

While writing it I ran into some puzzling issues. The first of them
is, how is Boolean indexing supposed to work when given more than one
array? What I mean is, suppose we have the following array:

In [3]: A = np.arange(9)

In [4]: B = np.reshape(A,(3,3))

In [5]: B
Out[5]:
array([[0, 1, 2],
       [3, 4, 5],
       [6, 7, 8]])

Then we can use boolean indexing to select some rows or some columns:

In [12]: B[np.array([True,False,True]),:]
Out[12]:
array([[0, 1, 2],
       [6, 7, 8]])

In [13]: B[:,np.array([True,False,True])]
Out[13]:
array([[0, 2],
       [3, 5],
       [6, 8]])

But if you supply Boolean arrays in both row and column places, rather
than select the corresponding rows and columns, you get something
basically mysterious:
In [14]: B[np.array([True,False,True]),np.array([True,True,False])]
Out[14]: array([0, 7])

In [15]: B[np.array([True,False,True]),np.array([True,False,False])]
Out[15]: array([0, 6])

In [16]: B[np.array([True,False,True]),np.array([True,True,True])]
---------------------------------------------------------------------------
<type 'exceptions.ValueError'>            Traceback (most recent call last)

/home/peridot/<ipython console> in <module>()

<type 'exceptions.ValueError'>: shape mismatch: objects cannot be
broadcast to a single shape

What is going on here?

Thanks,
Anne


From cburns at berkeley.edu  Wed Jun  4 00:39:22 2008
From: cburns at berkeley.edu (Christopher Burns)
Date: Tue, 3 Jun 2008 21:39:22 -0700
Subject: [Numpy-discussion] NumpyTest problem
In-Reply-To: <200806022304.57203.pgmdevlist@gmail.com>
References: <91cf711d0806020820o17343fdfua8fa2ee9ed91f081@mail.gmail.com>
	<1d36917a0806020845k55805582j848a9d746e08e587@mail.gmail.com>
	<200806022304.57203.pgmdevlist@gmail.com>
Message-ID: <764e38540806032139k56e40270gbbc548d1a2b63e56@mail.gmail.com>

Pierre,

I believe if you rename your TimingTests they'll work.  Nose looks for
functions starting with "test", and runs those.  So your 'utility' functions
like testta, testtb... should not begin with "test", but the function
calling them, timingTest, should.  Probably want to use more meaningful
names than my hack though.  :)

def test_timingTest():
    for f in [testf, testinplace]:
        for n in [1000,10000,50000]:
            t = func_ta(n, f)
            t1 = testtb(n, f)
            t2 = testtc(n, f)
            print f.test_name
            print """\
n = %7d
numpy time (ms) %6.1f
MA maskless ratio %6.1f
MA masked ratio %6.1f
""" % (n, t*1000.0, t1/t, t2/t)

def func_ta(n, f):
    x=numpy.arange(n) + 1.0
    tn0 = time.time()
    z = f(x)
    return time.time() - tn0

Then to run only test_timingTest:

cburns at tests 21:30:19 $ pwd
/Users/cburns/src/numpy-trunk/numpy/ma/tests

cburns at tests 21:33:13 $ nosetests -sv test_old_ma:test_timingTest
test_old_ma.test_timingTest ... Simple arithmetic
n =    1000
numpy time (ms)    1.9
MA maskless ratio   14.9
MA masked ratio   18.4

[snip]

----------------------------------------------------------------------
Ran 1 test in 2.912s

OK


On Mon, Jun 2, 2008 at 8:04 PM, Pierre GM <pgmdevlist at gmail.com> wrote:

>
> * Running nosetests on numpy/ma fails (I can't import numpy from the source
> directory), but works partially on numpy/ma/tests: the test_old_ma tests
> fail
> for asking some arguments when 0 is provided: for example:
> ================================================
> ERROR: test_old_ma.testta
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/usr/lib64/python2.4/site-packages/nose/case.py", line 203, in
> runTest
>    self.test(*self.arg)
> TypeError: testta() takes exactly 2 arguments (0 given)
>
>
-- 
Christopher Burns
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080603/83c8b722/attachment.html>

From cburns at berkeley.edu  Wed Jun  4 01:16:44 2008
From: cburns at berkeley.edu (Christopher Burns)
Date: Tue, 3 Jun 2008 22:16:44 -0700
Subject: [Numpy-discussion] get range of numpy type
Message-ID: <764e38540806032216o35dfc544gcd4268e504aa22f5@mail.gmail.com>

Is there a way to get the range of a numpy type?  I'd like to clamp a
parameter to be within the range of a numpy type, np.uint8, np.uint32...

Something like:
if x > max_value_of(np.uint8):
   x = max_value_of(np.uint8)


-- 
Christopher Burns
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080603/15b274e5/attachment.html>

From wnbell at gmail.com  Wed Jun  4 02:17:08 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Wed, 4 Jun 2008 01:17:08 -0500
Subject: [Numpy-discussion] get range of numpy type
In-Reply-To: <764e38540806032216o35dfc544gcd4268e504aa22f5@mail.gmail.com>
References: <764e38540806032216o35dfc544gcd4268e504aa22f5@mail.gmail.com>
Message-ID: <d05265cb0806032317y6e4f30fbj911a46ff59c1642@mail.gmail.com>

On Wed, Jun 4, 2008 at 12:16 AM, Christopher Burns <cburns at berkeley.edu> wrote:
> Is there a way to get the range of a numpy type?  I'd like to clamp a
> parameter to be within the range of a numpy type, np.uint8, np.uint32...
>
> Something like:
> if x > max_value_of(np.uint8):
>    x = max_value_of(np.uint8)

That kind of information is available via numpy.finfo() and numpy.iinfo():

In [12]: finfo('d').max
Out[12]: 1.7976931348623157e+308

In [13]: iinfo('i').max
Out[13]: 2147483647

In [14]: iinfo(uint8).max
Out[14]: 255


-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From j.stark at imperial.ac.uk  Wed Jun  4 02:48:26 2008
From: j.stark at imperial.ac.uk (J. Stark)
Date: Wed, 4 Jun 2008 07:48:26 +0100
Subject: [Numpy-discussion] 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <mailman.35258.1212550718.27196.numpy-discussion@scipy.org>
References: <mailman.35258.1212550718.27196.numpy-discussion@scipy.org>
Message-ID: <p06240802c46be8fafa51@[10.0.0.3]>

Chris,

many thanks. Could I suggest that this information be featured 
prominently in the Read Me in the Installer, and perhaps also at 
http://www.scipy.org/Download where this is given as the official 
binary distribution for MacOSX. You might want to change the error 
message too, since I think that some people will interpret "System 
Python" to mean the default Python provided by the standard system 
install. Since this is 2.5.1 on Leopard, the error message could be 
confusing.

On this topic, I would be interested to hear people's advice on using 
the system provided Python v an independent install. In 25 years of 
using Macs I have learned through several painful lessons that its 
wise to customize the system as little as possible: this minimizes 
both conflicts and reduces problems when doing system upgrades. I 
have therefore always used the default Python provided by OSX, so far 
with no obvious disadvantages for the types of scripts I use 
(primarily home written SciPy scientific code). However, I note that 
many people run either the pythomac.org distribution, or the 
ActiveState. What are the advantages to this?

Many thanks

J.

>Jaroslav,
>
>The installer works with the MacPython from python.org, not Apple's python
>(the one that ships with Leopard).
>
>The MacPython is installed in the /Library/Frameworks... It should work if
>your python is here:
>
>cburns$  python -c "import sys; print sys.prefix"
>/Library/Frameworks/Python.framework/Versions/2.5
>
>Won't work if your python is here:
>
>cburns$  python-apple -c "import sys; print sys.prefix"
>/System/Library/Frameworks/Python.framework/Versions/2.5
>
>Sorry for the confusion,
>Chris
>
>On Tue, Jun 3, 2008 at 2:33 PM, J. Stark <j.stark at imperial.ac.uk> wrote:
>
>>  I have just tried to run the 1.1.0 OSX installer on a MacBookAir
>>  running 10.5.3 and the installer fails with
>>
>>  "You cannot install numpy 1.1.0 on this volume. numpy requires System
>>  Python 2.5 to install."
>>
>>  The system python version reports as
>>
>>  jaroslav$ python
>>  Python 2.5.1 (r251:54863, Apr 15 2008, 22:57:26)
>>  [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
>>
>>  which is the same version that Leopard has had all along, as far as I
>>  am aware. On the the other hand, there have been some reports on
>>  PythonMac about odd python behaviour following the 10.5.3 upgrade.
>>
>>  Has anybody used this installer successfully under 10.5.3, and/or
>>  have any idea of what is going on.
>>
>>  Incidentally, this is a new machine with just the default system
>>  installation.
>>
>>  Jaroslav


From robert.kern at gmail.com  Wed Jun  4 02:56:43 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 4 Jun 2008 01:56:43 -0500
Subject: [Numpy-discussion] 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <p06240802c46be8fafa51@10.0.0.3>
References: <mailman.35258.1212550718.27196.numpy-discussion@scipy.org>
	<p06240802c46be8fafa51@10.0.0.3>
Message-ID: <3d375d730806032356n1e46898fi3ffbaa0a50902e95@mail.gmail.com>

On Wed, Jun 4, 2008 at 1:48 AM, J. Stark <j.stark at imperial.ac.uk> wrote:
> On this topic, I would be interested to hear people's advice on using
> the system provided Python v an independent install. In 25 years of
> using Macs I have learned through several painful lessons that its
> wise to customize the system as little as possible: this minimizes
> both conflicts and reduces problems when doing system upgrades. I
> have therefore always used the default Python provided by OSX, so far
> with no obvious disadvantages for the types of scripts I use
> (primarily home written SciPy scientific code). However, I note that
> many people run either the pythomac.org distribution, or the
> ActiveState. What are the advantages to this?

By installing a separate Python, you are actually customizing the
system *less* than if you used the system Python and installed a bunch
of extra packages. Parts of Apple's software uses the system Python.
If you upgrade packages inside there (like numpy!) you might run into
problems.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From eads at soe.ucsc.edu  Wed Jun  4 03:56:30 2008
From: eads at soe.ucsc.edu (Damian Eads)
Date: Wed, 04 Jun 2008 00:56:30 -0700
Subject: [Numpy-discussion] inconsistent behavior in binary_repr
Message-ID: <48464AAE.5090402@soe.ucsc.edu>

Hi,

I noticed some odd behavior in binary_repr when the width parameter is 
used. In most cases it works,

In [23]: numpy.binary_repr(1, width=8)
Out[23]: '00000001'

In [24]: numpy.binary_repr(2, width=8)
Out[24]: '00000010'

In [25]: numpy.binary_repr(3, width=8)
Out[25]: '00000011'

In [26]: numpy.binary_repr(4, width=8)
Out[26]: '00000100'

except when 0 is passed, I get the following

In [27]: numpy.binary_repr(0, width=8)
Out[27]: '0'

Is this what the output is intended to be for the 0 case?

Damian


From robert.kern at gmail.com  Wed Jun  4 03:59:35 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 4 Jun 2008 02:59:35 -0500
Subject: [Numpy-discussion] inconsistent behavior in binary_repr
In-Reply-To: <48464AAE.5090402@soe.ucsc.edu>
References: <48464AAE.5090402@soe.ucsc.edu>
Message-ID: <3d375d730806040059t71437298p3b0c3055d952b6c0@mail.gmail.com>

On Wed, Jun 4, 2008 at 2:56 AM, Damian Eads <eads at soe.ucsc.edu> wrote:
> Hi,
>
> I noticed some odd behavior in binary_repr when the width parameter is
> used. In most cases it works,
>
> In [23]: numpy.binary_repr(1, width=8)
> Out[23]: '00000001'
>
> In [24]: numpy.binary_repr(2, width=8)
> Out[24]: '00000010'
>
> In [25]: numpy.binary_repr(3, width=8)
> Out[25]: '00000011'
>
> In [26]: numpy.binary_repr(4, width=8)
> Out[26]: '00000100'
>
> except when 0 is passed, I get the following
>
> In [27]: numpy.binary_repr(0, width=8)
> Out[27]: '0'
>
> Is this what the output is intended to be for the 0 case?

No. SVN trunk seems to work correctly. What version of numpy do you have?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Wed Jun  4 03:59:57 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 04 Jun 2008 16:59:57 +0900
Subject: [Numpy-discussion] Installation info
In-Reply-To: <3d375d730806032030u5af327bbq6a9a649688e20ae5@mail.gmail.com>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>	<789d27b10806020141m43d8d699u8d9dbe8d593d8e3e@mail.gmail.com>	<4843B4AF.4070605@ar.media.kyoto-u.ac.jp>	<789d27b10806020230l1d88f270l52a62ec90954aeb7@mail.gmail.com>	<4843BDD2.8060002@ar.media.kyoto-u.ac.jp>	<789d27b10806020526h47d08418v4f0e0752bbcabb1a@mail.gmail.com>	<789d27b10806030121s3a3df8a0q3a7bbdd173515ba7@mail.gmail.com>	<3d375d730806031148s39f59d4dl9b7bb4b6ce2f611a@mail.gmail.com>	<484602B8.7000906@ar.media.kyoto-u.ac.jp>
	<3d375d730806032030u5af327bbq6a9a649688e20ae5@mail.gmail.com>
Message-ID: <48464B7D.9070100@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
>
>
> There are a lot of them. Feel free to add any additional tests you
> think are necessary, and we'll see how painful it is at build-time.
>   

What would be acceptable ? I quickly tested on my macbook, on mac os X: 
it takes ~ 2 seconds / 25 functions tests. If speed really is a problem, 
than we can first test them as a group, and then one by one for the 
platforms where it does not work (something like AC_CHECK_FUNCS_ONCE, if 
you are familiar with autotools) ?

It should not be too complicated to add this to distutils, and I believe 
it would make the configuration more robust,

cheers,

David



From david at ar.media.kyoto-u.ac.jp  Wed Jun  4 04:03:24 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 04 Jun 2008 17:03:24 +0900
Subject: [Numpy-discussion] Installation info
In-Reply-To: <e06186140806032036j5b2e292p76133cf6e18434b6@mail.gmail.com>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>	<789d27b10806020141m43d8d699u8d9dbe8d593d8e3e@mail.gmail.com>	<4843B4AF.4070605@ar.media.kyoto-u.ac.jp>	<789d27b10806020230l1d88f270l52a62ec90954aeb7@mail.gmail.com>	<4843BDD2.8060002@ar.media.kyoto-u.ac.jp>	<789d27b10806020526h47d08418v4f0e0752bbcabb1a@mail.gmail.com>	<789d27b10806030121s3a3df8a0q3a7bbdd173515ba7@mail.gmail.com>	<3d375d730806031148s39f59d4dl9b7bb4b6ce2f611a@mail.gmail.com>	<484602B8.7000906@ar.media.kyoto-u.ac.jp>
	<e06186140806032036j5b2e292p76133cf6e18434b6@mail.gmail.com>
Message-ID: <48464C4C.8000308@ar.media.kyoto-u.ac.jp>

Charles R Harris wrote:
>
> It probably just grew to fix problems as they arose. It should be 
> possible to test for every function and fall back to the double 
> versions that are more reliably present. It would be nicer if all 
> compilers tried to conform to recent standards, i.e., be less than 9 
> years out of date, but that is a bit much to ask for.

Most compilers are not compatible (none of them supports C99 at 100 %; 
the better question is which subset is powerfull and implemented 
thouroughly).

I had some surprising problems with numscons and mingw, and obviously MS 
compilers/runtime (each version has a different configuration for the 
part of the code we are talking about).

cheers,

David


From eads at soe.ucsc.edu  Wed Jun  4 04:26:38 2008
From: eads at soe.ucsc.edu (Damian Eads)
Date: Wed, 04 Jun 2008 01:26:38 -0700
Subject: [Numpy-discussion] [numpy-discussion] inconsistent behavior in
	binary_repr
Message-ID: <484651BE.6010509@soe.ucsc.edu>

Whoops. In one xterm, I'm going off the Fedora package and in the other, 
the SVN source tree. SVN seems to work. Sorry for the unnecessary message.

On Wed, Jun 4, 2008 at 2:59 AM, Robert Kern wrote:
 > In [27]: numpy.binary_repr(0, width=8)
 > Out[27]: '0'
 >
 > Is this what the output is intended to be for the 0 case?

No. SVN trunk seems to work correctly. What version of numpy do you have?



From j.stark at imperial.ac.uk  Wed Jun  4 04:25:33 2008
From: j.stark at imperial.ac.uk (J. Stark)
Date: Wed, 4 Jun 2008 09:25:33 +0100
Subject: [Numpy-discussion] Which Python to Use on OSX,
 Was: 1.1.0 OSX Installer Fails Under 10.5.3?
Message-ID: <p06240809c46c00857f66@[10.0.0.3]>

Robert,

I see your point, but why not just install a separate NumPy to run 
with the system Python? That is what I have always done in the past 
without problems.

I guess I always feel a sense of uncertainty with having two separate 
Python installations as to which actually gets used in any particular 
situation. I appreciate that for experts who use Python daily, this 
isn't an issue, but for someone like myself who may have gaps of 
several months between projects that use Python, this is a real issue 
as I forget those kinds of subtleties.

J.

>On Wed, Jun 4, 2008 at 1:48 AM, J. Stark <j.stark at imperial.ac.uk> wrote:
>>  On this topic, I would be interested to hear people's advice on using
>>  the system provided Python v an independent install. In 25 years of
>>  using Macs I have learned through several painful lessons that its
>>  wise to customize the system as little as possible: this minimizes
>>  both conflicts and reduces problems when doing system upgrades. I
>>  have therefore always used the default Python provided by OSX, so far
>>  with no obvious disadvantages for the types of scripts I use
>>  (primarily home written SciPy scientific code). However, I note that
>>  many people run either the pythomac.org distribution, or the
>>  ActiveState. What are the advantages to this?
>
>By installing a separate Python, you are actually customizing the
>system *less* than if you used the system Python and installed a bunch
>of extra packages. Parts of Apple's software uses the system Python.
>If you upgrade packages inside there (like numpy!) you might run into
>problems.
>
>--
>Robert Kern
>
>"I have come to believe that the whole world is an enigma, a harmless
>enigma that is made terrible by our own mad attempt to interpret it as
>though it had an underlying truth."
>  -- Umberto Eco


From robince at gmail.com  Wed Jun  4 05:40:56 2008
From: robince at gmail.com (Robin)
Date: Wed, 4 Jun 2008 10:40:56 +0100
Subject: [Numpy-discussion] Which Python to Use on OSX,
	Was: 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <p06240809c46c00857f66@10.0.0.3>
References: <p06240809c46c00857f66@10.0.0.3>
Message-ID: <b14b627d0806040240l26287472r793516638051a311@mail.gmail.com>

On Wed, Jun 4, 2008 at 9:25 AM, J. Stark <j.stark at imperial.ac.uk> wrote:
> Robert,
>
> I see your point, but why not just install a separate NumPy to run
> with the system Python? That is what I have always done in the past
> without problems.

I think the problem is the system python already comes with a (much
older) cut down version of numpy which you can find in:
/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/numpy
This makes all sorts of problems when installing a new version...
Obviously you can't have two different versions of the same package
with the same name in the same python installation (how do you choose
which one you mean with import numpy.)
I think there were problems with the path so when a new numpy is
installed in 2.5/Extras/lib/site-packages it is actually after the
existing one on the path and doesn't get picked up. Even if it does
work, the worry is that you're changing a supplied component and Apple
stuff might depend upon the version supplied (or other software people
distribute to use the 'system' python might expect it to be there).

I think theres much less chance of problems using the system python
for system things and leaving it well alone - and installing the
python.org for everyday use. The only problem with this is that the
system python works with dtrace while the normal one doesn't...

Cheers

Robin
>
> I guess I always feel a sense of uncertainty with having two separate
> Python installations as to which actually gets used in any particular
> situation. I appreciate that for experts who use Python daily, this
> isn't an issue, but for someone like myself who may have gaps of
> several months between projects that use Python, this is a real issue
> as I forget those kinds of subtleties.
>
> J.


From vincent.noel at gmail.com  Wed Jun  4 06:02:30 2008
From: vincent.noel at gmail.com (Vincent Noel)
Date: Wed, 4 Jun 2008 12:02:30 +0200
Subject: [Numpy-discussion] Which Python to Use on OSX,
	Was: 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <p06240809c46c00857f66@10.0.0.3>
References: <p06240809c46c00857f66@10.0.0.3>
Message-ID: <d9c4a7240806040302x5876c973q2e05012df84e7128@mail.gmail.com>

Another way to do things which might be useful, if you're not afraid
to modify the system python install, (more-or-less suggested at
http://wiki.python.org/moin/MacPython/Leopard), is to create a
symbolic link to make everything look as if you had installed
macpython, ie

sudo ln -s /System/Library/Frameworks/Python.framework/
/Library/Frameworks/Python.framework

Since, according to the MacPython page, the Leopard python is the same
as the MacPython (2.5.1),
all the packages you'll find on the web that suppose you have
MacPython installed should be happy (easy_installing eggs works fine
as well). HOWEVER you gotta add

export PATH=/Library/Frameworks/Python.framework/Versions/Current/bin:$PATH
export PYTHONPATH=/Library/Frameworks/Python.framework/Versions/Current/lib/python2.5/site-packages

in your ~/.bash_profile, otherwise the (older) system numpy will get
used. This is because the system python adds /System/.../2.5/Extras in
front of the /site-packages directory (weird, but hey).

Following this road, I was able to install NumPy 1.1, matplotlib 0.98
and ipython without any problem -- the best thing is that the system
wxPython is used, when it can be a PITA to setup correctly through
other ways. As was said by others, I guess there might be unforeseen
consequences, but everything seems to work fine for now.

Cheers
Vincent


On Wed, Jun 4, 2008 at 10:25 AM, J. Stark <j.stark at imperial.ac.uk> wrote:
> Robert,
>
> I see your point, but why not just install a separate NumPy to run
> with the system Python? That is what I have always done in the past
> without problems.
>
> I guess I always feel a sense of uncertainty with having two separate
> Python installations as to which actually gets used in any particular
> situation. I appreciate that for experts who use Python daily, this
> isn't an issue, but for someone like myself who may have gaps of
> several months between projects that use Python, this is a real issue
> as I forget those kinds of subtleties.
>
> J.
>
>>On Wed, Jun 4, 2008 at 1:48 AM, J. Stark <j.stark at imperial.ac.uk> wrote:
>>>  On this topic, I would be interested to hear people's advice on using
>>>  the system provided Python v an independent install. In 25 years of
>>>  using Macs I have learned through several painful lessons that its
>>>  wise to customize the system as little as possible: this minimizes
>>>  both conflicts and reduces problems when doing system upgrades. I
>>>  have therefore always used the default Python provided by OSX, so far
>>>  with no obvious disadvantages for the types of scripts I use
>>>  (primarily home written SciPy scientific code). However, I note that
>>>  many people run either the pythomac.org distribution, or the
>>>  ActiveState. What are the advantages to this?
>>
>>By installing a separate Python, you are actually customizing the
>>system *less* than if you used the system Python and installed a bunch
>>of extra packages. Parts of Apple's software uses the system Python.
>>If you upgrade packages inside there (like numpy!) you might run into
>>problems.
>>
>>--
>>Robert Kern
>>
>>"I have come to believe that the whole world is an enigma, a harmless
>>enigma that is made terrible by our own mad attempt to interpret it as
>>though it had an underlying truth."
>>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From lbolla at gmail.com  Wed Jun  4 06:10:19 2008
From: lbolla at gmail.com (lorenzo bolla)
Date: Wed, 4 Jun 2008 12:10:19 +0200
Subject: [Numpy-discussion] bvp on 64 bits machine
Message-ID: <80c99e790806040310p2ccc2518j9423832eec231754@mail.gmail.com>

Hello all.

I'm not sure that this is the correct mailing list to post to: please excuse
me if it's not.

I've been using bvp (http://www.elisanet.fi/ptvirtan/software/bvp/index.html)
by Pauli Virtanen happily on 32 bits machines.
When I used it on 64 bits machines I found a bug that I think I've solved
with the following patch:

=================

$ diff colnew.py.old colnew.py
347c347
<     ispace = _N.empty([nispace], _N.int_)
---
>     ispace = _N.empty([nispace], _N.int32)
402c402
<         ], _N.int_)
---
>         ], _N.int32)

=================

The problem is cause by the fact that _N.int_ is different for 32 and 64
bits machines. Forcing it to be an _N.int32 did the trick.
Pauli, would you like to commit it to your source distribution?

Regards,
Lorenzo.

-- 
"Whereof one cannot speak, thereof one must be silent." -- Ludwig
Wittgenstein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/c4fb5444/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Wed Jun  4 05:59:19 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 04 Jun 2008 18:59:19 +0900
Subject: [Numpy-discussion] Which Python to Use on OSX,
 Was: 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <b14b627d0806040240l26287472r793516638051a311@mail.gmail.com>
References: <p06240809c46c00857f66@10.0.0.3>
	<b14b627d0806040240l26287472r793516638051a311@mail.gmail.com>
Message-ID: <48466777.3040905@ar.media.kyoto-u.ac.jp>

Robin wrote:
> I think theres much less chance of problems using the system python
> for system things and leaving it well alone - and installing the
> python.org for everyday use. The only problem with this is that the
> system python works with dtrace while the normal one doesn't...
>   

The source for dtrace integration are not freely available ?

David


From robince at gmail.com  Wed Jun  4 06:14:42 2008
From: robince at gmail.com (Robin)
Date: Wed, 4 Jun 2008 11:14:42 +0100
Subject: [Numpy-discussion] Which Python to Use on OSX,
	Was: 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <48466777.3040905@ar.media.kyoto-u.ac.jp>
References: <p06240809c46c00857f66@10.0.0.3>
	<b14b627d0806040240l26287472r793516638051a311@mail.gmail.com>
	<48466777.3040905@ar.media.kyoto-u.ac.jp>
Message-ID: <b14b627d0806040314q48d02543ub19dab3fbe564382@mail.gmail.com>

On Wed, Jun 4, 2008 at 10:59 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Robin wrote:
>> I think theres much less chance of problems using the system python
>> for system things and leaving it well alone - and installing the
>> python.org for everyday use. The only problem with this is that the
>> system python works with dtrace while the normal one doesn't...
>>
>
> The source for dtrace integration are not freely available ?

Perhaps they are (I suppose they should be) but I can't find them - in
any case I don't think I would patch python distribution and build
from source just for this feature... I only mentioned it in passing.


From tgrav at mac.com  Wed Jun  4 08:38:35 2008
From: tgrav at mac.com (Tommy Grav)
Date: Wed, 4 Jun 2008 08:38:35 -0400
Subject: [Numpy-discussion] Which Python to Use on OSX,
	Was: 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <d9c4a7240806040302x5876c973q2e05012df84e7128@mail.gmail.com>
References: <p06240809c46c00857f66@10.0.0.3>
	<d9c4a7240806040302x5876c973q2e05012df84e7128@mail.gmail.com>
Message-ID: <EE7E4DDC-A0B7-4D14-80DD-C655320DFEA5@mac.com>

You have to very careful when you do this. For example
the system numpy is in ../python2.5/Extras/lib/ under the
framework, while I think the numpy binary installer installs
things in ../python2.5/lib/site-packages/. So if one is not
careful one ends up with two numpy packages with all
the problems that can cause.

I have installed Activepython on my machine (PPC w/ 10.5.3)
and it has worked more or less flawlessly. The system python
is still there and is untouched since I installed Leopard and
I do all my development against the activepython distribution.

Cheers
    Tommy


On Jun 4, 2008, at 6:02 AM, Vincent Noel wrote:

> Another way to do things which might be useful, if you're not afraid
> to modify the system python install, (more-or-less suggested at
> http://wiki.python.org/moin/MacPython/Leopard), is to create a
> symbolic link to make everything look as if you had installed
> macpython, ie
>
> sudo ln -s /System/Library/Frameworks/Python.framework/
> /Library/Frameworks/Python.framework
>
> Since, according to the MacPython page, the Leopard python is the same
> as the MacPython (2.5.1),
> all the packages you'll find on the web that suppose you have
> MacPython installed should be happy (easy_installing eggs works fine
> as well). HOWEVER you gotta add
>
> export PATH=/Library/Frameworks/Python.framework/Versions/Current/ 
> bin:$PATH
> export PYTHONPATH=/Library/Frameworks/Python.framework/Versions/ 
> Current/lib/python2.5/site-packages
>
> in your ~/.bash_profile, otherwise the (older) system numpy will get
> used. This is because the system python adds /System/.../2.5/Extras in
> front of the /site-packages directory (weird, but hey).
>
> Following this road, I was able to install NumPy 1.1, matplotlib 0.98
> and ipython without any problem -- the best thing is that the system
> wxPython is used, when it can be a PITA to setup correctly through
> other ways. As was said by others, I guess there might be unforeseen
> consequences, but everything seems to work fine for now.
>
> Cheers
> Vincent
>
>
> On Wed, Jun 4, 2008 at 10:25 AM, J. Stark <j.stark at imperial.ac.uk>  
> wrote:
>> Robert,
>>
>> I see your point, but why not just install a separate NumPy to run
>> with the system Python? That is what I have always done in the past
>> without problems.
>>
>> I guess I always feel a sense of uncertainty with having two separate
>> Python installations as to which actually gets used in any particular
>> situation. I appreciate that for experts who use Python daily, this
>> isn't an issue, but for someone like myself who may have gaps of
>> several months between projects that use Python, this is a real issue
>> as I forget those kinds of subtleties.
>>
>> J.
>>
>>> On Wed, Jun 4, 2008 at 1:48 AM, J. Stark <j.stark at imperial.ac.uk>  
>>> wrote:
>>>> On this topic, I would be interested to hear people's advice on  
>>>> using
>>>> the system provided Python v an independent install. In 25 years of
>>>> using Macs I have learned through several painful lessons that its
>>>> wise to customize the system as little as possible: this minimizes
>>>> both conflicts and reduces problems when doing system upgrades. I
>>>> have therefore always used the default Python provided by OSX, so  
>>>> far
>>>> with no obvious disadvantages for the types of scripts I use
>>>> (primarily home written SciPy scientific code). However, I note  
>>>> that
>>>> many people run either the pythomac.org distribution, or the
>>>> ActiveState. What are the advantages to this?
>>>
>>> By installing a separate Python, you are actually customizing the
>>> system *less* than if you used the system Python and installed a  
>>> bunch
>>> of extra packages. Parts of Apple's software uses the system Python.
>>> If you upgrade packages inside there (like numpy!) you might run  
>>> into
>>> problems.
>>>
>>> --
>>> Robert Kern
>>>
>>> "I have come to believe that the whole world is an enigma, a  
>>> harmless
>>> enigma that is made terrible by our own mad attempt to interpret  
>>> it as
>>> though it had an underlying truth."
>>> -- Umberto Eco
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From Chris.Barker at noaa.gov  Wed Jun  4 12:36:32 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 04 Jun 2008 09:36:32 -0700
Subject: [Numpy-discussion] Which Python to Use on OSX,
	Was: 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <EE7E4DDC-A0B7-4D14-80DD-C655320DFEA5@mac.com>
References: <p06240809c46c00857f66@10.0.0.3>
	<d9c4a7240806040302x5876c973q2e05012df84e7128@mail.gmail.com>
	<EE7E4DDC-A0B7-4D14-80DD-C655320DFEA5@mac.com>
Message-ID: <4846C490.7000604@noaa.gov>

Tommy Grav wrote:
> I have installed Activepython on my machine (PPC w/ 10.5.3)
> and it has worked more or less flawlessly.

And I've been using the python.org one for ages, also with NO issues. I 
tried to use Apple's Python for a while back with 10.2, but there were 
always problems, and Apple's has never patched or upgraded anything 
python within an OS version. This really is an issue: I just ran into a 
python bug that was fixed between 2.5.1 and 2.5.2 -- do you really not 
want to have the option of fixing that? You've also got an old wxPython, 
and old numpy, an old who knows what?

Also, it seems some folks have been having breakage of already compiled 
pyc files when upgrading to the latest 10.5

If you install your own Python, you have control over everything you 
need to, and you won't break anything Apple is using.

One other issue -- if you want to build re-distributable apps with 
py2app, or Universal binaries of packages, you'll need the python.,org 
python (Or some monkey-patching of the system python).

>> export PYTHONPATH=/Library/Frameworks/Python.framework/Versions/ 
>> Current/lib/python2.5/site-packages

This will probably not break anything Apple did, as their stuff won't be 
using your bash_profile, but it still scares me a bit.

 >> The best thing is that the system
>> wxPython is used, when it can be a PITA to setup correctly through
>> other ways.

huh? The installer provided at the wxPython pages has always worked 
flawlessly for me (for the python.org build) -- what could be easier?

By the way, this was discussed a lot on the pythonmac list -- though 
with no consensus reached :-( . For historical interest however, the 
Python supplied with 10.5 is the first one that core pythonmac folks 
have considered not too broken to use.

One more plug for my opinion -- the Python-on-OS-X world is far too 
fractured -- Apple's build, python.org's, Activestate's, fink, macports, 
build-from tarball... It's a mess, and a real pain for folks that want 
to just build/install a binary package. If the community could settle on 
one build to support, life would be so much easier. Python.org's 2.5 
build is really the only option for that (I think ActiveState's may be 
compatible), as it is the only one that supports:

10.3.9 and 10.4 and 10.5
PPC and Intel
py2app
building Universal binary packages.

That's why the binary distributed by numpy is for that build, and it's a 
good choice. (the working should be change,d tough, it's not the 
"system" python).

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From vincent.noel at gmail.com  Wed Jun  4 13:15:04 2008
From: vincent.noel at gmail.com (Vincent Noel)
Date: Wed, 4 Jun 2008 19:15:04 +0200
Subject: [Numpy-discussion] Which Python to Use on OSX,
	Was: 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <4846C490.7000604@noaa.gov>
References: <p06240809c46c00857f66@10.0.0.3>
	<d9c4a7240806040302x5876c973q2e05012df84e7128@mail.gmail.com>
	<EE7E4DDC-A0B7-4D14-80DD-C655320DFEA5@mac.com>
	<4846C490.7000604@noaa.gov>
Message-ID: <d9c4a7240806041015r1aa4cd76l6c29a8532de3d8b0@mail.gmail.com>

On Wed, Jun 4, 2008 at 6:36 PM, Christopher Barker
<Chris.Barker at noaa.gov> wrote:
>  >> The best thing is that the system
>>> wxPython is used, when it can be a PITA to setup correctly through
>>> other ways.
>
> huh? The installer provided at the wxPython pages has always worked
> flawlessly for me (for the python.org build) -- what could be easier?

Not installing anything is easier :-)
More seriously though, today I understand how things are supposed to work
and the distinction between the pythons in /System/Library/ and /Library,
but it took me a while to figure it out, and during the process I ended up
with lots of things installed in weird places (I still have some stuff in
/Library/Python that I'm not sure what is for) and not being recognized
(since I was not sure which Python to use). My remark came from these
bad memories (including a few times recompiling wxPython),
I apologize if things usually go smoothly.

I guess it's just another way to say it might be good, as you suggest,
to standardize on a single
specific distribution, I guess. It would limit the confusion.

Out of curiosity, what is the difference between python.org's python
and macpython's ?
I couldn't find a clear explanation. Is there any reason to use
macpython's package instead
of python.org's ?

Cheers
Vincent


From cburns at berkeley.edu  Wed Jun  4 13:32:32 2008
From: cburns at berkeley.edu (Christopher Burns)
Date: Wed, 4 Jun 2008 10:32:32 -0700
Subject: [Numpy-discussion] 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <p06240802c46be8fafa51@10.0.0.3>
References: <mailman.35258.1212550718.27196.numpy-discussion@scipy.org>
	<p06240802c46be8fafa51@10.0.0.3>
Message-ID: <764e38540806041032y3760af15wc9ae5045891ea91e@mail.gmail.com>

> many thanks. Could I suggest that this information be featured
> prominently in the Read Me in the Installer, and perhaps also at
> http://www.scipy.org/Download where this is given as the official
> binary distribution for MacOSX.

Good point, I added a note to the scipy download page.

>You might want to change the error
> message too, since I think that some people will interpret "System
> Python" to mean the default Python provided by the standard system
> install. Since this is 2.5.1 on Leopard, the error message could be
> confusing.

Agreed, the message is confusing.  I'll fix that on the next release.

>
> On this topic, I would be interested to hear people's advice on using
> the system provided Python v an independent install. In 25 years of
> using Macs I have learned through several painful lessons that its
> wise to customize the system as little as possible: this minimizes
> both conflicts and reduces problems when doing system upgrades. I
> have therefore always used the default Python provided by OSX, so far
> with no obvious disadvantages for the types of scripts I use
> (primarily home written SciPy scientific code). However, I note that
> many people run either the pythomac.org distribution, or the
> ActiveState. What are the advantages to this?

We should leave the System Python to Apple.  They use it, and have
numpy 1.0.1 installed with it.  Installing your own python with your
own packages (numpy 1.1, scipy, etc...) provides a clean division and
is less likely to create conflicts.

>From  http://developer.apple.com/tools/installerpolicy.html

" Off Limits
The folder /System is for Apple use only. It holds important operating
system files, and everyone using the machine relies on this folder.
Apple-provided system libraries and file systems live here. If you do
not know if your software belongs in here, chances are that it does
not. Mucking around in /System can lead to machine instability."

Thanks for your feedback,
Chris


From zachary.pincus at yale.edu  Wed Jun  4 13:37:04 2008
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Wed, 4 Jun 2008 10:37:04 -0700
Subject: [Numpy-discussion] numpy, py2exe, and SSE
Message-ID: <3B802A64-5EDB-4BFA-AE75-9B5FD1928AA9@yale.edu>

Hello all,

I've been toying around with bundling up a numpy-using python program  
for windows by using py2exe. All in all, it works great, except for  
one thing: the numpy superpack installer for windows has (correctly)  
selected SSE3 binary libraries to install on my machine. This causes  
(of course) crashes when the binary py2exe bundle, which includes the  
numpy libraries installed on my machine, is run on a non-SSE3 machine.

So, two questions:
  1) What's the best target for "safe" binaries on i386 machines? No  
SSE at all? SSE? SSE2?
  2) What's the best way to get the libraries desired installed for  
numpy? E.g. compile  if myself with some --no-sse option? (And if so,  
any clue as to the required options.) Or is there some way to get the  
numpy windows installer to install a specific kind of binary?

Thanks,
Zach Pincus


From Chris.Barker at noaa.gov  Wed Jun  4 14:19:39 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 04 Jun 2008 11:19:39 -0700
Subject: [Numpy-discussion] Which Python to Use on OSX,
	Was: 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <d9c4a7240806041015r1aa4cd76l6c29a8532de3d8b0@mail.gmail.com>
References: <p06240809c46c00857f66@10.0.0.3>
	<d9c4a7240806040302x5876c973q2e05012df84e7128@mail.gmail.com>
	<EE7E4DDC-A0B7-4D14-80DD-C655320DFEA5@mac.com>
	<4846C490.7000604@noaa.gov>
	<d9c4a7240806041015r1aa4cd76l6c29a8532de3d8b0@mail.gmail.com>
Message-ID: <4846DCBB.3090802@noaa.gov>

Vincent Noel wrote:
> Out of curiosity, what is the difference between python.org's python
> and macpython's ?

Nothing. They are the same. They didn't used to be, though, hence the 
confusion.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From barrywark at gmail.com  Wed Jun  4 14:23:54 2008
From: barrywark at gmail.com (Barry Wark)
Date: Wed, 4 Jun 2008 11:23:54 -0700
Subject: [Numpy-discussion] Which Python to Use on OSX,
	Was: 1.1.0 OSX Installer Fails Under 10.5.3?
In-Reply-To: <b14b627d0806040240l26287472r793516638051a311@mail.gmail.com>
References: <p06240809c46c00857f66@10.0.0.3>
	<b14b627d0806040240l26287472r793516638051a311@mail.gmail.com>
Message-ID: <cd7634ce0806041123h5610fd5fp6c0872d5d57cf1e7@mail.gmail.com>

On Wed, Jun 4, 2008 at 2:40 AM, Robin <robince at gmail.com> wrote:
> On Wed, Jun 4, 2008 at 9:25 AM, J. Stark <j.stark at imperial.ac.uk> wrote:
>> Robert,
>>
>> I see your point, but why not just install a separate NumPy to run
>> with the system Python? That is what I have always done in the past
>> without problems.
>
> I think the problem is the system python already comes with a (much
> older) cut down version of numpy which you can find in:
> /System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/numpy
> This makes all sorts of problems when installing a new version...
> Obviously you can't have two different versions of the same package
> with the same name in the same python installation (how do you choose
> which one you mean with import numpy.)
> I think there were problems with the path so when a new numpy is
> installed in 2.5/Extras/lib/site-packages it is actually after the
> existing one on the path and doesn't get picked up. Even if it does
> work, the worry is that you're changing a supplied component and Apple
> stuff might depend upon the version supplied (or other software people
> distribute to use the 'system' python might expect it to be there).

This is not entirely accurate on OS X 10.5. If you install anything
using the system python, it is put in
/Library/Python/2.5/site-packages. Apple's system-supplied packages
are kept in /System/Library/Frameworks/Python.framework/... and, as
you note, the site-packages directory in
/System/Library/Frameworks/Python.framework does come first on the
sys.path. In this way, it is very difficult to override the
Apple-provided packages in a way that breaks system tools that depend
on them. HOWEVER, if you install your packages using setuptools
(including an updated numpy), setuptools will place them first on the
sys.path at import (Apple's tools intentionally restrict their
sys.path to the System/Library/Frameworks/Python.framework
site-packages for this reason). So, if you use setuptools to install
numpy, matplotlib, even Twisted (as of version 8, it is
easy_install-able), etc., you can continue to use the system python
without fear of messing up apple's system tools and without having to
install a separate python instance in /Library/Frameworks.

Apple did a pretty good job on this one for Leopard. As others have
noted, sticking with this system python gains you dtrace support, and
continued improvements from Apple (many apple tools now use python
and/or PyObjC).

>
> I think theres much less chance of problems using the system python
> for system things and leaving it well alone - and installing the
> python.org for everyday use. The only problem with this is that the
> system python works with dtrace while the normal one doesn't...
>
> Cheers
>
> Robin
>>
>> I guess I always feel a sense of uncertainty with having two separate
>> Python installations as to which actually gets used in any particular
>> situation. I appreciate that for experts who use Python daily, this
>> isn't an issue, but for someone like myself who may have gaps of
>> several months between projects that use Python, this is a real issue
>> as I forget those kinds of subtleties.
>>
>> J.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From sameerslists at gmail.com  Wed Jun  4 15:11:00 2008
From: sameerslists at gmail.com (Sameer DCosta)
Date: Wed, 4 Jun 2008 14:11:00 -0500
Subject: [Numpy-discussion] Renaming record array fields (bug)
Message-ID: <8fb8cc060806041211i2454fab8g6d4ef6056ebfcc21@mail.gmail.com>

Hi,

There is a bug renaming record array fields if some field names are
the same. I reopened this ticket
http://scipy.org/scipy/numpy/ticket/674 and attached a tiny patch.
Maybe I should have opened a new ticket. Anyway, here is an example
that causes a segfault on the latest svn version.

import numpy as np
dt = np.dtype([('foo', float), ('bar', float)])
a = np.zeros(10, dt)
b = list(a.dtype.names);
b[0] = "notfoo"
#b[1] = "notbar" # uncomment this line for no segfault
a.dtype.names = b
print a, a.dtype # this will crash r5253


Sameer


From orest.kozyar at gmail.com  Wed Jun  4 15:12:40 2008
From: orest.kozyar at gmail.com (Orest Kozyar)
Date: Wed, 4 Jun 2008 15:12:40 -0400
Subject: [Numpy-discussion] PyArray_Resize with scipy.weave
Message-ID: <d635fdd00806041212s337ba48du9f27d660a7e42140@mail.gmail.com>

The following code fails:

from scipy import weave
from numpy import zeros

arr = zeros((10,2))
code = """
PyArray_Dims dims;
dims.len = 2;
dims.ptr = Narr;
dims.ptr[0] += 10;
PyArray_Resize(arr_array, &dims, 1);
"""
weave.inline(code, ['arr'], verbose=1)

The error message is:
In function 'PyObject* compiled_func(PyObject*, PyObject*)':
<filename>:678: error: too few arguments to function

678 is the line number for PyArray_Resize.  According to the NumPy
Handbook, PyArray_Resize requires three arguments, which I am
providing. Am I missing something obvious here?  There are times when
I need to be able to resize the array in C++ because I cannot predict
exactly how big the array needs to be before I pass it to weave.  Any
advice or pointers greatly appreciated!

Thanks,
Orest


From vinjvinj at gmail.com  Wed Jun  4 20:39:29 2008
From: vinjvinj at gmail.com (Vineet Jain (gmail))
Date: Wed, 4 Jun 2008 20:39:29 -0400
Subject: [Numpy-discussion] Is there a function to calculate ecnomic beta
	coefficient in numpy given two time series data.
Message-ID: <000c01c8c6a4$9f0fad00$dd2f0700$@com>

Timeseries1 = daily or weekly close of stock a 

Timeseries2 = daily or weekly close of market index (spx, qqqq, etc)

 

Beta of stock a is what I would like to compute as explained in this article
on Wikipedia:

 

http://en.wikipedia.org/wiki/Beta_coefficient

 

I'm trying to compute the beta of entire stock market (about 15,000
instruments) one stock at a time and would like to use the spiders and qqqq
to represent the overall market. 

 

Thanks,

 

Vineet

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/aacc2073/attachment.html>

From dyamins at gmail.com  Wed Jun  4 20:42:10 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Wed, 4 Jun 2008 20:42:10 -0400
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
Message-ID: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>

I'm using python 2.5.2 on OS X, with 8 GB of ram, and a 64-bit processor.
In
this, setting, I'm working with large arrays of binary data.  E.g, I want to
make calls like:
               Z = numpy.inner(a,b)
where and b are fairly large  -- e.g. 20000 rows by 100 columns.

However, when such a call is made, I get a memory error that I don't
understand.
Specifically:

>>> s = numpy.random.binomial(1,.5,(20000,100))   #creates 20000x100 bin.
array
>>> r = numpy.inner(s,s)
Python(1714) malloc: *** mmap(size=1600000000) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

Naively, the numpy.inner call should be fine on my system, since my computer
has
enough memory. (And, when it's run, I have checked to see that at least 5 GB
of
RAM is free.)  The error message thus suggests there's some problem to do
with
memory mapping going on here: that somehow, numpy.inner is calling the mmap
modeul, and that the address space is being exceeded.  And that all my extra
RAM
isn't being used here at all.

So, I have three questions about this:
    1) Why is mmap being called in the first place?  I've written to Travis
Oliphant, and he's explained that numpy.inner does NOT directly do any
memory
mapping and shouldn't call mmap.  Instead, it should just operate with
things in
memory -- in which case my 8 GB should allow the computation to go through
just
fine.  What's going on?

    2) How can I stop this from happening?  I want to be able to leverage
large
amounts of ram on my machine to scale up my computations and not be
dependent on
the limitations of the address space size.  If the mmap is somehow being
called
by the OS, is there some option I can set that will make it do things in
regular
memory instead?  (Sorry if this is a stupid question.)

    3) Even if I had to use memory mapping, why is the 1.6 GB requirement
failing?  I'm using a recent enough version of python, and I have a 64-bit
processor with sufficient amount of memory.  I should be able to allocate at
least 4 GB of address space, right?  But the system seems to be balking at
the
1.6 GB request. (Again, sorry if this is stupid.)

Any help would be greatly appreciated!  Thanks,
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/852030bc/attachment.html>

From kwgoodman at gmail.com  Wed Jun  4 21:04:20 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 4 Jun 2008 18:04:20 -0700
Subject: [Numpy-discussion] Is there a function to calculate ecnomic
	beta coefficient in numpy given two time series data.
In-Reply-To: <000c01c8c6a4$9f0fad00$dd2f0700$@com>
References: <000c01c8c6a4$9f0fad00$dd2f0700$@com>
Message-ID: <f4f93d420806041804n5c0302b3le70c093673140a79@mail.gmail.com>

On Wed, Jun 4, 2008 at 5:39 PM, Vineet Jain (gmail) <vinjvinj at gmail.com> wrote:
> Timeseries1 = daily or weekly close of stock a
>
> Timeseries2 = daily or weekly close of market index (spx, qqqq, etc)
>
>
>
> Beta of stock a is what I would like to compute as explained in this article
> on Wikipedia:
>
>
>
> http://en.wikipedia.org/wiki/Beta_coefficient
>
>
>
> I'm trying to compute the beta of entire stock market (about 15,000
> instruments) one stock at a time and would like to use the spiders and qqqq
> to represent the overall market.

Unless you run out of memory (or if you want to handle missing returns
which may occur on different dates in each series) there is no need to
do it one stock at a time:

>> import numpy.matlib as mp
>> mrkt = mp.randn(250,1)  # <----- 250 days of returns
>> stocks = mp.randn(250, 4)  # <---- 4 stocks
>> beta, resids, rank, s = mp.linalg.lstsq(mrkt, stocks)
>> beta
   matrix([[-0.01701467,  0.11242168,  0.00207398,  0.03920687]])

And you can use mp.log to convert the price ratios to log returns.

It might also be useful to shuffle (mp.random.shuffle) the market
returns and repeat the beta calculation many times to estimate the
noise level of your beta estimates.


From charlesr.harris at gmail.com  Wed Jun  4 21:06:12 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 4 Jun 2008 19:06:12 -0600
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
Message-ID: <e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>

On Wed, Jun 4, 2008 at 6:42 PM, Dan Yamins <dyamins at gmail.com> wrote:

> I'm using python 2.5.2 on OS X, with 8 GB of ram, and a 64-bit processor.
> In
> this, setting, I'm working with large arrays of binary data.  E.g, I want
> to
> make calls like:
>                Z = numpy.inner(a,b)
> where and b are fairly large  -- e.g. 20000 rows by 100 columns.
>
> However, when such a call is made, I get a memory error that I don't
> understand.
> Specifically:
>
> >>> s = numpy.random.binomial(1,.5,(20000,100))   #creates 20000x100 bin.
> array
> >>> r = numpy.inner(s,s)
> Python(1714) malloc: *** mmap(size=1600000000) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
>
> Naively, the numpy.inner call should be fine on my system, since my
> computer has
> enough memory. (And, when it's run, I have checked to see that at least 5
> GB of
> RAM is free.)  The error message thus suggests there's some problem to do
> with
> memory mapping going on here: that somehow, numpy.inner is calling the mmap
> modeul, and that the address space is being exceeded.  And that all my
> extra RAM
> isn't being used here at all.
>
> So, I have three questions about this:
>     1) Why is mmap being called in the first place?  I've written to Travis
> Oliphant, and he's explained that numpy.inner does NOT directly do any
> memory
> mapping and shouldn't call mmap.  Instead, it should just operate with
> things in
> memory -- in which case my 8 GB should allow the computation to go through
> just
> fine.  What's going on?
>
>     2) How can I stop this from happening?  I want to be able to leverage
> large
> amounts of ram on my machine to scale up my computations and not be
> dependent on
> the limitations of the address space size.  If the mmap is somehow being
> called
> by the OS, is there some option I can set that will make it do things in
> regular
> memory instead?  (Sorry if this is a stupid question.)
>
>     3) Even if I had to use memory mapping, why is the 1.6 GB requirement
> failing?  I'm using a recent enough version of python, and I have a 64-bit
> processor with sufficient amount of memory.  I should be able to allocate
> at
> least 4 GB of address space, right?  But the system seems to be balking at
> the
> 1.6 GB request. (Again, sorry if this is stupid.)
>

Are both python and your version of OS X fully 64 bits?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/93c0238c/attachment.html>

From kwgoodman at gmail.com  Wed Jun  4 21:11:17 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 4 Jun 2008 18:11:17 -0700
Subject: [Numpy-discussion] Is there a function to calculate ecnomic
	beta coefficient in numpy given two time series data.
In-Reply-To: <f4f93d420806041804n5c0302b3le70c093673140a79@mail.gmail.com>
References: <000c01c8c6a4$9f0fad00$dd2f0700$@com>
	<f4f93d420806041804n5c0302b3le70c093673140a79@mail.gmail.com>
Message-ID: <f4f93d420806041811s7e5dc76bp15845961e09b8c23@mail.gmail.com>

On Wed, Jun 4, 2008 at 6:04 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> It might also be useful to shuffle (mp.random.shuffle) the market
> returns and repeat the beta calculation many times to estimate the
> noise level of your beta estimates.

I guess that is more of a measure of how different from zero your
betas are. Maybe use a bootstrap to measure the noise in beta.


From peridot.faceted at gmail.com  Wed Jun  4 21:12:45 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 4 Jun 2008 21:12:45 -0400
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
Message-ID: <ce557a360806041812hda6ddy83aeb247d2ec257@mail.gmail.com>

2008/6/4 Dan Yamins <dyamins at gmail.com>:

> So, I have three questions about this:
>     1) Why is mmap being called in the first place?  I've written to Travis
> Oliphant, and he's explained that numpy.inner does NOT directly do any
> memory
> mapping and shouldn't call mmap.  Instead, it should just operate with
> things in
> memory -- in which case my 8 GB should allow the computation to go through
> just
> fine.  What's going on?
>
>     2) How can I stop this from happening?  I want to be able to leverage
> large
> amounts of ram on my machine to scale up my computations and not be
> dependent on
> the limitations of the address space size.  If the mmap is somehow being
> called
> by the OS, is there some option I can set that will make it do things in
> regular
> memory instead?  (Sorry if this is a stupid question.)

I don't know much about OSX, but I do know that many malloc()
implementations take advantage of a modern operating system's virtual
memory when allocating large blocks of memory. For small blocks,
malloc uses memory arenas, but if you ask for a large block malloc()
will request a whole bunch of pages from the operating system. This
way when the memory is freed, free() can easily return the chunk of
memory to the OS. On some systems, one way to get such a big hunk of
memory from the system is with an "anonymous mmap()". I think that's
what's going on here. So I don't think you want to stop malloc() from
using mmap().

You do, of course, want the memory allocation to succeed, and I'm
afraid I don't have any idea why it can't. Under Linux, I know that
you can run a 64-bit processor in 32-bit mode, which gives you the
usual 4 GB address space limit. I've no idea what OSX does.

Good luck,
Anne


From vinjvinj at gmail.com  Wed Jun  4 21:24:12 2008
From: vinjvinj at gmail.com (Vineet Jain (gmail))
Date: Wed, 4 Jun 2008 21:24:12 -0400
Subject: [Numpy-discussion] Is there a function to calculate
	ecnomic	beta coefficient in numpy given two time series data.
In-Reply-To: <f4f93d420806041804n5c0302b3le70c093673140a79@mail.gmail.com>
References: <000c01c8c6a4$9f0fad00$dd2f0700$@com>
	<f4f93d420806041804n5c0302b3le70c093673140a79@mail.gmail.com>
Message-ID: <001d01c8c6aa$de6c5240$9b44f6c0$@com>

Thanks Keith!

-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Keith Goodman
Sent: Wednesday, June 04, 2008 9:04 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Is there a function to calculate ecnomic
beta coefficient in numpy given two time series data.

On Wed, Jun 4, 2008 at 5:39 PM, Vineet Jain (gmail) <vinjvinj at gmail.com>
wrote:
> Timeseries1 = daily or weekly close of stock a
>
> Timeseries2 = daily or weekly close of market index (spx, qqqq, etc)
>
>
>
> Beta of stock a is what I would like to compute as explained in this
article
> on Wikipedia:
>
>
>
> http://en.wikipedia.org/wiki/Beta_coefficient
>
>
>
> I'm trying to compute the beta of entire stock market (about 15,000
> instruments) one stock at a time and would like to use the spiders and
qqqq
> to represent the overall market.

Unless you run out of memory (or if you want to handle missing returns
which may occur on different dates in each series) there is no need to
do it one stock at a time:

>> import numpy.matlib as mp
>> mrkt = mp.randn(250,1)  # <----- 250 days of returns
>> stocks = mp.randn(250, 4)  # <---- 4 stocks
>> beta, resids, rank, s = mp.linalg.lstsq(mrkt, stocks)
>> beta
   matrix([[-0.01701467,  0.11242168,  0.00207398,  0.03920687]])

And you can use mp.log to convert the price ratios to log returns.

It might also be useful to shuffle (mp.random.shuffle) the market
returns and repeat the beta calculation many times to estimate the
noise level of your beta estimates.
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From dyamins at gmail.com  Wed Jun  4 21:38:17 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Wed, 4 Jun 2008 21:38:17 -0400
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <ce557a360806041812hda6ddy83aeb247d2ec257@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<ce557a360806041812hda6ddy83aeb247d2ec257@mail.gmail.com>
Message-ID: <15e4667e0806041838x4ee8d54dgea72ab4cd176012a@mail.gmail.com>

I don't know much about OSX, but I do know that many malloc()
> implementations take advantage of a modern operating system's virtual
> memory when allocating large blocks of memory. For small blocks,
> malloc uses memory arenas, but if you ask for a large block malloc()
> will request a whole bunch of pages from the operating system. This
> way when the memory is freed, free() can easily return the chunk of
> memory to the OS. On some systems, one way to get such a big hunk of
> memory from the system is with an "anonymous mmap()". I think that's
> what's going on here. So I don't think you want to stop malloc() from
> using mmap().
>
> You do, of course, want the memory allocation to succeed, and I'm
> afraid I don't have any idea why it can't. Under Linux, I know that
> you can run a 64-bit processor in 32-bit mode, which gives you the
> usual 4 GB address space limit. I've no idea what OSX does.
>
> Good luck,
> Anne
>

Anne, thanks so much for your help.  I still a little confused.   If your
scenario about the the memory allocation is working is right, does that mean
that even if I put a lot of ram on the machine, e.g. > 16GB, I still can't
request it in blocks larger than the limit imposed by the processor
architecture (e.g. 4 GB for 32, 8 GB for 64-bit)?    What I really want is
to be able to have my ability to request memory limited just by the amount
of memory on the machine, and not have it depend on something about
paging/memory mapping limits.  Is this a stupid/naive thing?

(Sorry for my ignorance, and thanks again for the help!)

best,
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/dec57a52/attachment.html>

From dyamins at gmail.com  Wed Jun  4 21:41:19 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Wed, 4 Jun 2008 21:41:19 -0400
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>
Message-ID: <15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>

On Wed, Jun 4, 2008 at 9:06 PM, Charles R Harris <charlesr.harris at gmail.com>
wrote:

>
>
> On Wed, Jun 4, 2008 at 6:42 PM, Dan Yamins <dyamins at gmail.com> wrote:
>
>> I'm using python 2.5.2 on OS X, with 8 GB of ram, and a 64-bit processor.
>> In
>> this, setting, I'm working with large arrays of binary data.  E.g, I want
>> to
>> make calls like:
>>                Z = numpy.inner(a,b)
>> where and b are fairly large  -- e.g. 20000 rows by 100 columns.
>>
>> However, when such a call is made, I get a memory error that I don't
>> understand.
>> Specifically:
>>
>> >>> s = numpy.random.binomial(1,.5,(20000,100))   #creates 20000x100 bin.
>> array
>> >>> r = numpy.inner(s,s)
>> Python(1714) malloc: *** mmap(size=1600000000) failed (error code=12)
>> *** error: can't allocate region
>> *** set a breakpoint in malloc_error_break to debug
>>
>>
>>
> Are both python and your version of OS X fully 64 bits?
>


I'm not sure.   My version of OS X is the most recent version, the one that
ships with a new MacPro Dual Quad-core Xeon 3.2MHz chipset.  The processor
is definitely 64-bit, so I think the operating system probably is enable for
that, but am not sure. (How would I find out?)  As for the python version, I
thought that 2.5 and above were 64-enabled, but I'm not sure how I'd check
it.

Thanks!
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/955b40a4/attachment.html>

From peridot.faceted at gmail.com  Wed Jun  4 21:59:13 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 4 Jun 2008 21:59:13 -0400
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <15e4667e0806041838x4ee8d54dgea72ab4cd176012a@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<ce557a360806041812hda6ddy83aeb247d2ec257@mail.gmail.com>
	<15e4667e0806041838x4ee8d54dgea72ab4cd176012a@mail.gmail.com>
Message-ID: <ce557a360806041859p5aa9627dtabd312274ca773f8@mail.gmail.com>

2008/6/4 Dan Yamins <dyamins at gmail.com>:

> Anne, thanks so much for your help.  I still a little confused.   If your
> scenario about the the memory allocation is working is right, does that mean
> that even if I put a lot of ram on the machine, e.g. > 16GB, I still can't
> request it in blocks larger than the limit imposed by the processor
> architecture (e.g. 4 GB for 32, 8 GB for 64-bit)?    What I really want is
> to be able to have my ability to request memory limited just by the amount
> of memory on the machine, and not have it depend on something about
> paging/memory mapping limits.  Is this a stupid/naive thing?

No, that's a perfectly reasonable thing to want, and it *should* be
possible with your hardware. Maybe I'm barking up the wrong tree and
the problem is something else. But if you'll bear with me:

For the last ten years or so, normal computers have used 32 bits to
address memory. This means  that any given process can only get at
2**32 different bytes of memory, which is 4 GB. The magic of virtual
memory means you can possibly have a number of different processes
addressing different 4 GB chunks, some of which may temporarily be
stored on disk at any given time. If you want to get at a chunk of
memory bigger than 4 GB, you need all of these things:

* Your CPU must be 64-bit, or else it has no hope of being able to
access more than 4 GB (in fact more than 3 GB for complicated reasons)
of physical memory.
* Your operating system must be 64-bit (64-bit PC CPUs have a 32-bit
compatibility mode they often find themselves running in) so that it
can know about all the real RAM you have installed and so that it can
support 64-bit programs.
* Your program must be 64-bit, so that it can access huge chunks of RAM.

For compatibility reasons, many 64-bit machines have most of their
software compiled in 32-bit mode. This isn't necessarily a problem -
the programs probably even run faster - but they can't access more
than 4 GB each. So it's worth finding out whether your python
interpreter is compiled as a 64-bit program (and whether that's even
possible under your version of OSX). Unfortunately I  don't know how
to find that out in your case.

Anne


From charlesr.harris at gmail.com  Wed Jun  4 21:59:40 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 4 Jun 2008 19:59:40 -0600
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>
	<15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
Message-ID: <e06186140806041859v335104e6rd08d6c8d68d4b5f6@mail.gmail.com>

On Wed, Jun 4, 2008 at 7:41 PM, Dan Yamins <dyamins at gmail.com> wrote:

>
>
> On Wed, Jun 4, 2008 at 9:06 PM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>>
>>
>> On Wed, Jun 4, 2008 at 6:42 PM, Dan Yamins <dyamins at gmail.com> wrote:
>>
>>> I'm using python 2.5.2 on OS X, with 8 GB of ram, and a 64-bit
>>> processor.  In
>>> this, setting, I'm working with large arrays of binary data.  E.g, I want
>>> to
>>> make calls like:
>>>                Z = numpy.inner(a,b)
>>> where and b are fairly large  -- e.g. 20000 rows by 100 columns.
>>>
>>> However, when such a call is made, I get a memory error that I don't
>>> understand.
>>> Specifically:
>>>
>>> >>> s = numpy.random.binomial(1,.5,(20000,100))   #creates 20000x100 bin.
>>> array
>>> >>> r = numpy.inner(s,s)
>>> Python(1714) malloc: *** mmap(size=1600000000) failed (error code=12)
>>> *** error: can't allocate region
>>> *** set a breakpoint in malloc_error_break to debug
>>>
>>>
>>>
>> Are both python and your version of OS X fully 64 bits?
>>
>
>
> I'm not sure.   My version of OS X is the most recent version, the one that
> ships with a new MacPro Dual Quad-core Xeon 3.2MHz chipset.  The processor
> is definitely 64-bit, so I think the operating system probably is enable for
> that, but am not sure. (How would I find out?)  As for the python version, I
> thought that 2.5 and above were 64-enabled, but I'm not sure how I'd check
> it.
>

Hmm,

In [1]: s = numpy.random.binomial(1,.5,(20000,100))

In [2]: inner(s,s)
Out[2]:
array([[45, 22, 17, ..., 20, 26, 23],
       [22, 52, 26, ..., 23, 33, 24],
       [17, 26, 52, ..., 27, 27, 19],
       ...,
       [20, 23, 27, ..., 46, 26, 22],
       [26, 33, 27, ..., 26, 54, 25],
       [23, 24, 19, ..., 22, 25, 44]])

This on 32 bit fedora 8 with 2GiB of actual memory. It was slow and a couple
of hundred megs of something went into swap, but it did complete. So this
looks to me like an OS X problem. Are there any limitations on the user
memory sizes? There might be some system setting accounting for this.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/0c102cf2/attachment.html>

From cournapeau at cslab.kecl.ntt.co.jp  Wed Jun  4 22:07:03 2008
From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau)
Date: Thu, 05 Jun 2008 11:07:03 +0900
Subject: [Numpy-discussion] A memory problem: why does mmap come up
	in	numpy.inner?
In-Reply-To: <15e4667e0806041838x4ee8d54dgea72ab4cd176012a@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<ce557a360806041812hda6ddy83aeb247d2ec257@mail.gmail.com>
	<15e4667e0806041838x4ee8d54dgea72ab4cd176012a@mail.gmail.com>
Message-ID: <1212631623.13915.15.camel@bbc8>

On Wed, 2008-06-04 at 21:38 -0400, Dan Yamins wrote:

> 
> Anne, thanks so much for your help.  I still a little confused.   If
> your scenario about the the memory allocation is working is right,
> does that mean that even if I put a lot of ram on the machine, e.g. >
> 16GB, I still can't request it in blocks larger than the limit imposed
> by the processor architecture (e.g. 4 GB for 32, 8 GB for 64-bit)?

Definitely. For 32 bits, it is actually smaller than 4 Gb, because the
whole address range is used by the processus and the OS, and the OS
generally needs at least 1 Gb (I don't know the default for mac os X,
but both linux and windows default to 2 Gb: so a processus cannot use
more than 2 Gb of memory).

The whole point of having a 64 bits CPU in 64 bits mode is to have more
than 32 bits for the (virtual) address space. If it is not in 64 bits
mode, the pointer is 32 bits (void* is 4 bytes, as returned by malloc),
and you would have a hard time accessing anything above 32 bits. It is
possible to address more than 32 bits on 32 bits OS using some weird
extensions, with a "semi segmented" address (you address the memory with
an offset + an address, like in the old times), but I have never used
it.

IOW: with more than 4 Gb of "real" memory, the hardware can address it,
in 32 bits or 64 bits, but *you* cannot address  it in 32 bits mode
because your pointers are still 4 bytes. To address more than 4 Gb, you
need 8 bytes pointers. 

So the real question is how to do it on mac os X, and unfortunately, I
cannot help you. I know that mac os X supports executing both 32 bits
and 64 bits ("super fat" binaries with the ability to put up to 4
differents kind of binaries: ppc and x86, 32 bits and 64 bits in both
cases), but I don't know which subsystem you can use, and how to compile
it (you will certainly have to compile your application differently). I
remember having seen mentioned several times that for Leopard, you
cannot build a 64 bits binary using GUI, because no GUI subsystem is 64
bits compatible yet.

>     What I really want is to be able to have my ability to request
> memory limited just by the amount of memory on the machine, and not
> have it depend on something about paging/memory mapping limits.  Is
> this a stupid/naive thing? 

It is not stupid, I think everyone whished it was possible.
Unfortunately, in mac os X case at least, you cannot do it so simply
(yet).

David




From oliphant at enthought.com  Wed Jun  4 22:10:07 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 04 Jun 2008 21:10:07 -0500
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
 numpy.inner?
In-Reply-To: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
Message-ID: <48474AFF.5050400@enthought.com>

Dan Yamins wrote:
> I'm using python 2.5.2 on OS X, with 8 GB of ram, and a 64-bit 
> processor.  In
> this, setting, I'm working with large arrays of binary data.  E.g, I 
> want to
> make calls like:
>                Z = numpy.inner(a,b)
> where and b are fairly large  -- e.g. 20000 rows by 100 columns.

Hey Dan.   Now, that you mention you are using OS X, I'm fairly 
confident that the problem is that you are using a 32-bit version of 
Python (i.e. you are not running in full 64-bit mode and so the 4GB 
limit applies).

The most common Python on OS X is 32-bit python.  I think a few people 
in the SAGE project have successfully built Python in 64-bit mode on OSX 
(but I don't think they have released anything yet).  You would have to 
use a 64-bit version of Python to compile NumPy if you want to access 
large memory.

-Travis



From dyamins at gmail.com  Wed Jun  4 22:34:37 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Wed, 4 Jun 2008 22:34:37 -0400
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <1212631623.13915.15.camel@bbc8>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<ce557a360806041812hda6ddy83aeb247d2ec257@mail.gmail.com>
	<15e4667e0806041838x4ee8d54dgea72ab4cd176012a@mail.gmail.com>
	<1212631623.13915.15.camel@bbc8>
Message-ID: <15e4667e0806041934r2815d979v6a41e390ed2542b0@mail.gmail.com>

On Wed, Jun 4, 2008 at 10:07 PM, David Cournapeau <
cournapeau at cslab.kecl.ntt.co.jp> wrote:

> On Wed, 2008-06-04 at 21:38 -0400, Dan Yamins wrote:
>
> >
> > Anne, thanks so much for your help.  I still a little confused.   If
> > your scenario about the the memory allocation is working is right,
> > does that mean that even if I put a lot of ram on the machine, e.g. >
> > 16GB, I still can't request it in blocks larger than the limit imposed
> > by the processor architecture (e.g. 4 GB for 32, 8 GB for 64-bit)?
>
> Definitely. For 32 bits, it is actually smaller than 4 Gb, because the
> whole address range is used by the processus and the OS, and the OS
> generally needs at least 1 Gb (I don't know the default for mac os X,
> but both linux and windows default to 2 Gb: so a processus cannot use
> more than 2 Gb of memory).
>
> It is not stupid, I think everyone whished it was possible.
> Unfortunately, in mac os X case at least, you cannot do it so simply
> (yet).
>
> David
>


Anne and David, thank you both so much for your very lucid explanations.  (I
apologize for my ignorance -- with your help and some additional reading, I
think I understand how memory allocation works a bit better now.)

Best,
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/8fd4627d/attachment.html>

From charlesr.harris at gmail.com  Wed Jun  4 22:36:08 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 4 Jun 2008 20:36:08 -0600
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>
	<15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
Message-ID: <e06186140806041936q356d7d98j36a8d2bd2ab5c979@mail.gmail.com>

On Wed, Jun 4, 2008 at 7:41 PM, Dan Yamins <dyamins at gmail.com> wrote:

>
>
> On Wed, Jun 4, 2008 at 9:06 PM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>>
>>
>> On Wed, Jun 4, 2008 at 6:42 PM, Dan Yamins <dyamins at gmail.com> wrote:
>>
>>> I'm using python 2.5.2 on OS X, with 8 GB of ram, and a 64-bit
>>> processor.  In
>>> this, setting, I'm working with large arrays of binary data.  E.g, I want
>>> to
>>> make calls like:
>>>                Z = numpy.inner(a,b)
>>> where and b are fairly large  -- e.g. 20000 rows by 100 columns.
>>>
>>> However, when such a call is made, I get a memory error that I don't
>>> understand.
>>> Specifically:
>>>
>>> >>> s = numpy.random.binomial(1,.5,(20000,100))   #creates 20000x100 bin.
>>> array
>>> >>> r = numpy.inner(s,s)
>>> Python(1714) malloc: *** mmap(size=1600000000) failed (error code=12)
>>> *** error: can't allocate region
>>> *** set a breakpoint in malloc_error_break to debug
>>>
>>>
>>>
>> Are both python and your version of OS X fully 64 bits?
>>
>
>
> I'm not sure.   My version of OS X is the most recent version, the one that
> ships with a new MacPro Dual Quad-core Xeon 3.2MHz chipset.  The processor
> is definitely 64-bit, so I think the operating system probably is enable for
> that, but am not sure. (How would I find out?)  As for the python version, I
> thought that 2.5 and above were 64-enabled, but I'm not sure how I'd check
> it.
>

Try

In [3]: numpy.dtype(numpy.uintp).itemsize
Out[3]: 4

which is the size in bytes of the integer needed to hold a pointer. The
output above is for 32 bit python/numpy.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/2a68d5ee/attachment.html>

From dyamins at gmail.com  Wed Jun  4 22:48:35 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Wed, 4 Jun 2008 22:48:35 -0400
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <48474AFF.5050400@enthought.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<48474AFF.5050400@enthought.com>
Message-ID: <15e4667e0806041948o32dac0feg7b20569e314ecf7f@mail.gmail.com>

>
> Hey Dan.   Now, that you mention you are using OS X, I'm fairly
> confident that the problem is that you are using a 32-bit version of
> Python (i.e. you are not running in full 64-bit mode and so the 4GB
> limit applies).
>
> The most common Python on OS X is 32-bit python.  I think a few people
> in the SAGE project have successfully built Python in 64-bit mode on OSX
> (but I don't think they have released anything yet).  You would have to
> use a 64-bit version of Python to compile NumPy if you want to access
> large memory.
>
> -Travis
>
>
Travis, thanks for the message.   I think you're probably right -- I didn't
build python myself but instead downloaded the universal OSX binary from the
python download page -- and that surely wasn't built for 64-bit system.   So
I guess I'll have to figure out to do the 64-bit build.

Which leaves a question that I think Chuck brought up in a way:

> In [1]: s = numpy.random.binomial(1,.5,(20000,100))
> In [2]: inner(s,s)
> Out[2]:
> array([[45, 22, 17, ..., 20, 26, 23],
>      [22, 52, 26, ..., 23, 33, 24],
>       [17, 26, 52, ..., 27, 27, 19],
>       ...,
>       [20, 23, 27, ..., 46, 26, 22],
>       [26, 33, 27, ..., 26, 54, 25],
>       [23, 24, 19, ..., 22, 25, 44]])

>This on 32 bit fedora 8 with 2GiB of actual memory. It was slow and a
couple of hundred megs of something went into swap, but it >did complete. So
this looks to me like an OS X problem. Are there any limitations on the user
memory sizes? There might be some >system setting accounting for this.

Chuck, is this another way of asking:  why is my OS X system not paging
memory the way you'd expect a system to respond to the malloc command?    Is
python somehow overloaded the malloc command so that when the OS says a swap
would have to occur, somehow instead of just the swap, that an error message
involving mmap is somehow triggered?   (Sorry if this makes no sense.)   I
should add that the tried the same code on a 32-bit windows machine and got
the same error as on OS X.   Maybe the Linux python  builds manage this
stuff better.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/c46b5901/attachment.html>

From dyamins at gmail.com  Wed Jun  4 22:50:25 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Wed, 4 Jun 2008 22:50:25 -0400
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <e06186140806041936q356d7d98j36a8d2bd2ab5c979@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>
	<15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
	<e06186140806041936q356d7d98j36a8d2bd2ab5c979@mail.gmail.com>
Message-ID: <15e4667e0806041950j2ae0b0e0h2678725c7d476bbc@mail.gmail.com>

> Try
>
> In [3]: numpy.dtype(numpy.uintp).itemsize
> Out[3]: 4
>
> which is the size in bytes of the integer needed to hold a pointer. The
> output above is for 32 bit python/numpy.
>
> Chuck
>

Check, the answer is 4, as you got for the 32-bit.   What would the answer
be on a 64-bit architecture?  Why is this diagnostic?

Thanks!
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/60e073a4/attachment.html>

From peridot.faceted at gmail.com  Wed Jun  4 23:07:15 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 4 Jun 2008 23:07:15 -0400
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <15e4667e0806041950j2ae0b0e0h2678725c7d476bbc@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>
	<15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
	<e06186140806041936q356d7d98j36a8d2bd2ab5c979@mail.gmail.com>
	<15e4667e0806041950j2ae0b0e0h2678725c7d476bbc@mail.gmail.com>
Message-ID: <ce557a360806042007x5cd53e19i40393e4ac5ea84d6@mail.gmail.com>

2008/6/4 Dan Yamins <dyamins at gmail.com>:
>
>
>
>>
>> Try
>>
>> In [3]: numpy.dtype(numpy.uintp).itemsize
>> Out[3]: 4
>>
>> which is the size in bytes of the integer needed to hold a pointer. The
>> output above is for 32 bit python/numpy.
>>
>> Chuck
>
> Check, the answer is 4, as you got for the 32-bit.   What would the answer
> be on a 64-bit architecture?  Why is this diagnostic?

In a 64-bit setting, a pointer needs to be 64 bits long, that is,
eight bytes, not four.

What Charles pointed out was that while the inner product is very big,
it seems to fit into memory on his 32-bit Linux machine; is it
possible that OSX is preventing your python process from using even
the meager 2-3 GB that a 32-bit process ought to get? In particular,
try running Charles' script in a fresh python interpreter and see if
it works; it may be that other arrays you had allocated are taking up
some of the space that this one could.

You will probably still want a 64-bit python, though, in order to have
a little elbow room.

Anne


From wnbell at gmail.com  Wed Jun  4 23:08:07 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Wed, 4 Jun 2008 22:08:07 -0500
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <15e4667e0806041950j2ae0b0e0h2678725c7d476bbc@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>
	<15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
	<e06186140806041936q356d7d98j36a8d2bd2ab5c979@mail.gmail.com>
	<15e4667e0806041950j2ae0b0e0h2678725c7d476bbc@mail.gmail.com>
Message-ID: <d05265cb0806042008ue754eafkc379a63a5eb0b702@mail.gmail.com>

On Wed, Jun 4, 2008 at 9:50 PM, Dan Yamins <dyamins at gmail.com> wrote:
>>
>> In [3]: numpy.dtype(numpy.uintp).itemsize
>> Out[3]: 4
>>
>> which is the size in bytes of the integer needed to hold a pointer. The
>> output above is for 32 bit python/numpy.
>>
>
> Check, the answer is 4, as you got for the 32-bit.   What would the answer
> be on a 64-bit architecture?  Why is this diagnostic?

It would be 8 on a 64-bit architecture (with a 64-bit binary):  8
bytes = 64 bits, 4 bytes = 32 bits.

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From charlesr.harris at gmail.com  Wed Jun  4 23:18:25 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 4 Jun 2008 21:18:25 -0600
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <15e4667e0806041950j2ae0b0e0h2678725c7d476bbc@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>
	<15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
	<e06186140806041936q356d7d98j36a8d2bd2ab5c979@mail.gmail.com>
	<15e4667e0806041950j2ae0b0e0h2678725c7d476bbc@mail.gmail.com>
Message-ID: <e06186140806042018uec05ebfp3d07c7d43592053@mail.gmail.com>

Hi Dan,

On Wed, Jun 4, 2008 at 8:50 PM, Dan Yamins <dyamins at gmail.com> wrote:

>
>
>
>
>> Try
>>
>> In [3]: numpy.dtype(numpy.uintp).itemsize
>> Out[3]: 4
>>
>> which is the size in bytes of the integer needed to hold a pointer. The
>> output above is for 32 bit python/numpy.
>>
>> Chuck
>>
>
> Check, the answer is 4, as you got for the 32-bit.   What would the answer
> be on a 64-bit architecture?  Why is this diagnostic?
>

On 64 bit ubuntu

In [1]: import numpy

In [2]: numpy.dtype(numpy.uintp).itemsize
Out[2]: 8


This is diagnostic because a pointer of 4 bytes, 32 bits, can only address 4
GiB whereas an 8 byte pointer has 64 bits to address memory. So it looks
like your numpy was compiled by a 32 python executable.

I've been been googling this a bit and it looks like the standard executable
python on the mac is 32 bits, even though 64 bit libraries are available if
you can get a 64 bit version up and running. A lot of libraries are
available in both 32 and 64 bit versions.

Chuck

>
> Thanks!
> Dan
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/1ed84b1f/attachment.html>

From charlesr.harris at gmail.com  Wed Jun  4 23:26:44 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 4 Jun 2008 21:26:44 -0600
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <ce557a360806042007x5cd53e19i40393e4ac5ea84d6@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>
	<15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
	<e06186140806041936q356d7d98j36a8d2bd2ab5c979@mail.gmail.com>
	<15e4667e0806041950j2ae0b0e0h2678725c7d476bbc@mail.gmail.com>
	<ce557a360806042007x5cd53e19i40393e4ac5ea84d6@mail.gmail.com>
Message-ID: <e06186140806042026h1d47ae7bke95de153f4dbbdfc@mail.gmail.com>

On Wed, Jun 4, 2008 at 9:07 PM, Anne Archibald <peridot.faceted at gmail.com>
wrote:

> 2008/6/4 Dan Yamins <dyamins at gmail.com>:
> >
> >
> >
> >>
> >> Try
> >>
> >> In [3]: numpy.dtype(numpy.uintp).itemsize
> >> Out[3]: 4
> >>
> >> which is the size in bytes of the integer needed to hold a pointer. The
> >> output above is for 32 bit python/numpy.
> >>
> >> Chuck
> >
> > Check, the answer is 4, as you got for the 32-bit.   What would the
> answer
> > be on a 64-bit architecture?  Why is this diagnostic?
>
> In a 64-bit setting, a pointer needs to be 64 bits long, that is,
> eight bytes, not four.
>
> What Charles pointed out was that while the inner product is very big,
> it seems to fit into memory on his 32-bit Linux machine; is it
> possible that OSX is preventing your python process from using even
> the meager 2-3 GB that a 32-bit process ought to get? In particular,
> try running Charles' script in a fresh python interpreter and see if
> it works; it may be that other arrays you had allocated are taking up
> some of the space that this one could.
>
> You will probably still want a 64-bit python, though, in order to have
> a little elbow room.
>

I think the difference is that Linux takes up 1 GiB, leaving 3 GiB to the
process, while I suspect OS X is taking up 2 GiB. I don't have a Mac, so I
don't really know. When I ran the the problem it took about 1.8 GiB, making
it a close run thing if the Mac only gives 32 bit proccesses 2 GiB.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/3b24b3bd/attachment.html>

From dyamins at gmail.com  Wed Jun  4 23:29:15 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Wed, 4 Jun 2008 23:29:15 -0400
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
	numpy.inner?
In-Reply-To: <ce557a360806042007x5cd53e19i40393e4ac5ea84d6@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>
	<15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
	<e06186140806041936q356d7d98j36a8d2bd2ab5c979@mail.gmail.com>
	<15e4667e0806041950j2ae0b0e0h2678725c7d476bbc@mail.gmail.com>
	<ce557a360806042007x5cd53e19i40393e4ac5ea84d6@mail.gmail.com>
Message-ID: <15e4667e0806042029v64218c1ct8a69b0c9ee0f56bf@mail.gmail.com>

What Charles pointed out was that while the inner product is very big,
> it seems to fit into memory on his 32-bit Linux machine; is it
> possible that OSX is preventing your python process from using even
> the meager 2-3 GB that a 32-bit process ought to get?



Yes -- I think this is what is happening, because it's choking on calling up
1.6 GiB



> In particular,
> try running Charles' script in a fresh python interpreter and see if
> it works; it may be that other arrays you had allocated are taking up
> some of the space that this one could.



I did try this a number of times.   The result was by starting python
freshly or deleting other arrays from memory, I am able to increase the size
of the largest array I can compute the inner product on.   However, even
with nothing else in memory other than numpy and the original matrix whose
inner product I'm taking
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/45355039/attachment.html>

From mabshoff at googlemail.com  Wed Jun  4 23:59:09 2008
From: mabshoff at googlemail.com (Michael Abshoff)
Date: Thu, 05 Jun 2008 05:59:09 +0200
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
 numpy.inner?
In-Reply-To: <15e4667e0806041950j2ae0b0e0h2678725c7d476bbc@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>	<15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>	<e06186140806041936q356d7d98j36a8d2bd2ab5c979@mail.gmail.com>
	<15e4667e0806041950j2ae0b0e0h2678725c7d476bbc@mail.gmail.com>
Message-ID: <4847648D.9010503@googlemail.com>

Dan Yamins wrote:
>
>
Hello folks,

I did port Sage and hence Python with numpy and scipy to 64 bit OSX and 
below are some sample build instructions for just building python and 
numpy in 64 bit mode.
>
>
>     Try
>
>     In [3]: numpy.dtype(numpy.uintp).itemsize
>     Out[3]: 4
>
>     which is the size in bytes of the integer needed to hold a
>     pointer. The output above is for 32 bit python/numpy.
>
>     Chuck
>
>
> Check, the answer is 4, as you got for the 32-bit.   What would the 
> answer be on a 64-bit architecture?  Why is this diagnostic?
>
> Thanks!
> Dan
>
First try:

export OPT="-g -fwrapv -O3 -m64 -Wall -Wstrict-prototypes"
./configure --disable-toolbox-glue 
--prefix=/Users/mabshoff/64bitnumpy/python-2.5.2-bin  

<SNIP>
checking for int... yes
checking size of int... 4
checking for long... yes
checking size of long... 4
<SNIP>

Oops, make fail because of the above. Let's try again:

 ./configure --disable-toolbox-glue 
--prefix=/Users/mabshoff/64bitnumpy/python-2.5.2-bin --with-gcc="gcc -m64"

<SNIP>
checking for int... yes
checking size of int... 4
checking for long... yes
checking size of long... 8
<SNIP>

make && make install

then:

bsd:python-2.5.2-bin mabshoff$ file bin/python
bin/python: Mach-O 64-bit executable x86_64

Let's make the 64 bit python default:

bsd:64bitnumpy mabshoff$ export 
PATH=/Users/mabshoff/64bitnumpy/python-2.5.2-bin/bin/:$PATH
bsd:64bitnumpy mabshoff$ which python
/Users/mabshoff/64bitnumpy/python-2.5.2-bin/bin//python
bsd:64bitnumpy mabshoff$ file `which python`
/Users/mabshoff/64bitnumpy/python-2.5.2-bin/bin//python: Mach-O 64-bit 
executable x86_64

Let's build numpy 1.1.0:

bsd:64bitnumpy mabshoff$ tar xf numpy-1.1.0.tar.gz
bsd:64bitnumpy mabshoff$ cd numpy-1.1.0
bsd:numpy-1.1.0 mabshoff$ python setup.py install
<SNIP>

bsd:python-2.5.2-bin mabshoff$ python
Python 2.5.2 (r252:60911, Jun  4 2008, 20:47:16)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> import numpy
 >>> numpy.dtype(numpy.uintp).itemsize
8
 >>> ^D
bsd:python-2.5.2-bin mabshoff$  

Voila ;)

Wi th numpy 1.0.4svn from 20080104 the numpy setup did not work since a 
conftest failed. I did report that to Stefan V. in IRC via #sage-devel 
and I also thought that this was still a problem with numpy 1.1.0, but 
fortuantely that was fixed.

Now on to the test suite:

 >>> numpy.test()
Numpy is installed in 
/Users/mabshoff/64bitnumpy/python-2.5.2-bin/lib/python2.5/site-packages/numpy
Numpy version 1.1.0
Python version 2.5.2 (r252:60911, Jun  4 2008, 20:47:16) [GCC 4.0.1 
(Apple Inc. build 5465)]
  Found 18/18 tests for numpy.core.tests.test_defmatrix
  Found 3/3 tests for numpy.core.tests.test_errstate
  Found 3/3 tests for numpy.core.tests.test_memmap
  Found 286/286 tests for numpy.core.tests.test_multiarray
  Found 70/70 tests for numpy.core.tests.test_numeric
  Found 36/36 tests for numpy.core.tests.test_numerictypes
  Found 12/12 tests for numpy.core.tests.test_records
  Found 143/143 tests for numpy.core.tests.test_regression
  Found 7/7 tests for numpy.core.tests.test_scalarmath
  Found 2/2 tests for numpy.core.tests.test_ufunc
  Found 16/16 tests for numpy.core.tests.test_umath
  Found 63/63 tests for numpy.core.tests.test_unicode
  Found 4/4 tests for numpy.distutils.tests.test_fcompiler_gnu
  Found 5/5 tests for numpy.distutils.tests.test_misc_util
  Found 2/2 tests for numpy.fft.tests.test_fftpack
  Found 3/3 tests for numpy.fft.tests.test_helper
  Found 24/24 tests for numpy.lib.tests.test__datasource
  Found 10/10 tests for numpy.lib.tests.test_arraysetops
  Found 1/1 tests for numpy.lib.tests.test_financial
  Found 53/53 tests for numpy.lib.tests.test_function_base
  Found 5/5 tests for numpy.lib.tests.test_getlimits
  Found 6/6 tests for numpy.lib.tests.test_index_tricks
  Found 15/15 tests for numpy.lib.tests.test_io
  Found 1/1 tests for numpy.lib.tests.test_machar
  Found 4/4 tests for numpy.lib.tests.test_polynomial
  Found 1/1 tests for numpy.lib.tests.test_regression
  Found 49/49 tests for numpy.lib.tests.test_shape_base
  Found 15/15 tests for numpy.lib.tests.test_twodim_base
  Found 43/43 tests for numpy.lib.tests.test_type_check
  Found 1/1 tests for numpy.lib.tests.test_ufunclike
  Found 89/89 tests for numpy.linalg.tests.test_linalg
  Found 3/3 tests for numpy.linalg.tests.test_regression
  Found 94/94 tests for numpy.ma.tests.test_core
  Found 15/15 tests for numpy.ma.tests.test_extras
  Found 17/17 tests for numpy.ma.tests.test_mrecords
  Found 36/36 tests for numpy.ma.tests.test_old_ma
  Found 4/4 tests for numpy.ma.tests.test_subclassing
  Found 7/7 tests for numpy.tests.test_random
  Found 16/16 tests for numpy.testing.tests.test_utils
  Found 5/5 tests for numpy.tests.test_ctypeslib
......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ctypes 
is not available on this python: skipping the test (import error was: 
ctypes is not available.)
.....
----------------------------------------------------------------------
Ran 1275 tests in 1.420s

OK
<unittest._TextTestResult run=1275 errors=0 failures=0>

I did also fixe the ctypes build in 64 bit OSX, but that is right now a 
truly horrible hack. Sage needs it for twisted, so it isn't an option. I 
will clean up my fix and post it to the python list soon [I have been 
busy with $WORK, so no clean version yet], but if anybody cares here I 
can post a simple step by step workaround. The issue here is that the 
libffi used by ctypes is way out of date and as is does not have 64 bit 
OSX support.

Hope this helps.

Congratulations on numpy 1.1.0 - I mostly lurk here, so I didn't have a 
chance yet.

Cheers,

Michael

> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   




From wright at esrf.fr  Thu Jun  5 01:35:00 2008
From: wright at esrf.fr (Jonathan Wright)
Date: Thu, 05 Jun 2008 07:35:00 +0200
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
 numpy.inner?
In-Reply-To: <15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>
	<15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
Message-ID: <48477B04.6070007@esrf.fr>

Dan Yamins wrote:
> On Wed, Jun 4, 2008 at 9:06 PM, Charles R Harris 
> <charlesr.harris at gmail.com <mailto:charlesr.harris at gmail.com>> wrote:
>
>
>         Are both python and your version of OS X fully 64 bits?
>
>
> I'm not sure.  
>From  python:

python2.5 -c 'import platform;print platform.architecture()'
('32bit', 'ELF')

versus :

('64bit', 'ELF')

You can also try the unix file command (eg: from a terminal):

$ file `which python2.5`
/sware/exp/fable/standalone/redhate4-a64/bin/python: ELF 64-bit LSB 
executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, 
dynamically linked (uses shared libs), not stripped

...etc. We needed this for generating the .so library file name for 
ctypes, and got the answer from comp.lang.python. I hope it also works 
for OS X.

Best,

Jon






From mabshoff at googlemail.com  Thu Jun  5 02:24:26 2008
From: mabshoff at googlemail.com (Michael Abshoff)
Date: Thu, 05 Jun 2008 08:24:26 +0200
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
 numpy.inner?
In-Reply-To: <48477B04.6070007@esrf.fr>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>	<15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
	<48477B04.6070007@esrf.fr>
Message-ID: <4847869A.3020105@googlemail.com>

Jonathan Wright wrote:
> Dan Yamins wrote:
>   
>> On Wed, Jun 4, 2008 at 9:06 PM, Charles R Harris 
>> <charlesr.harris at gmail.com <mailto:charlesr.harris at gmail.com>> wrote:
>>
>>
>>         Are both python and your version of OS X fully 64 bits?
>>
>>
>> I'm not sure.  
>>     
> >From  python:
>
> python2.5 -c 'import platform;print platform.architecture()'
> ('32bit', 'ELF')
>
> versus :
>
> ('64bit', 'ELF')
>
> You can also try the unix file command (eg: from a terminal):
>
>   
Hi Jon,

as described in the other email in this thread about an hour ago I did 
build python 2.5.2 on OSX in 64 bit mode. As is the ctypes extension 
does not build due to libffi being too old. I manually got it to build 
[hackish, long story, details will go to the python dev list in a couple 
days] and now it works, i.e. twisted works correctly and the Sage 
notebook which depends on twisted passes doctests.

> $ file `which python2.5`
> /sware/exp/fable/standalone/redhate4-a64/bin/python: ELF 64-bit LSB 
> executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, 
> dynamically linked (uses shared libs), not stripped
>
> ...etc. We needed this for generating the .so library file name for 
> ctypes, and got the answer from comp.lang.python. I hope it also works 
> for OS X.
>
>   

Can you elaborate on this a little? I have a 64 bit python 2.5.2 and 
ctypes imports fine:

bsd:64bitnumpy mabshoff$ file `which python`
/Users/mabshoff/64bitnumpy/python-2.5.2-bin/bin//python: Mach-O 64-bit 
executable x86_64
bsd:64bitnumpy mabshoff$ python
Python 2.5.2 (r252:60911, Jun  4 2008, 21:59:02)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> import _ctypes
 >>> ^D

But  since I manually build it and copied it over python might not 
"know" it exists [excuse much lack of python-internals foo here]. Since 
I tested numpy-1.1 before I copied over the ctypes extension I deleted 
numpy-1.1 from site-packages and rebuild from fresh sources, but despite 
the ctypes extension existing and working it still fails the numpy 
ctypes test:

<SNIP other tests>
  Found 5/5 tests for numpy.tests.test_ctypeslib
<SNIP many dots>ctypes is not available on this python: skipping the 
test (import error was: ctypes is not available.)
.....
----------------------------------------------------------------------
Ran 1275 tests in 1.235s

I am not what I would call familiar with numpy internals, so is there a 
magic thing I can do to make numpy aware that ctypes exists? I am 
willing to find out myself, but do not have the time today to go off and 
spend a day or two on this, especially if somebody on the list can just 
point me to the right spot ;)

Any input is appreciated.

> Best,
>
> Jon
>
>
>   
Cheers,

Michael

>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   



From cournapeau at cslab.kecl.ntt.co.jp  Thu Jun  5 02:45:59 2008
From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau)
Date: Thu, 05 Jun 2008 15:45:59 +0900
Subject: [Numpy-discussion] A memory problem: why does mmap come up
	in	numpy.inner?
In-Reply-To: <4847869A.3020105@googlemail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>
	<15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
	<48477B04.6070007@esrf.fr>  <4847869A.3020105@googlemail.com>
Message-ID: <1212648359.13915.56.camel@bbc8>

On Thu, 2008-06-05 at 08:24 +0200, Michael Abshoff wrote:

> I am not what I would call familiar with numpy internals, so is there a 
> magic thing I can do to make numpy aware that ctypes exists? 

I don't think any magic is needed. As long as importing ctypes works
from the python you used to build numpy, it should work.

David



From wright at esrf.fr  Thu Jun  5 08:06:42 2008
From: wright at esrf.fr (Jon Wright)
Date: Thu, 05 Jun 2008 14:06:42 +0200
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
 numpy.inner?
In-Reply-To: <4847869A.3020105@googlemail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>	<15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>	<48477B04.6070007@esrf.fr>
	<4847869A.3020105@googlemail.com>
Message-ID: <4847D6D2.4050106@esrf.fr>

Michael Abshoff wrote:
> Jonathan Wright wrote:
>>
>> ...etc. We needed this for generating the .so library file name for 
>> ctypes
>
> Can you elaborate on this a little? 

The "we" refered to another project (not numpy) where we needed to 
distinguish 32 bit from 64 bit platforms. We have code for picking which 
shared library file to use similar to that below.  Apologies if I've 
confused the issue.

-Jon
----

if sys.platform == "linux2":
     if platform.architecture()[0] == '32bit':
         libfilename = 'libKLT32.so'
     if platform.architecture()[0] == '64bit':
         libfilename = 'libKLT64.so'
if sys.platform == "win32":
     libfilename = 'klt.dll'

_libraries['klt_bin'] = CDLL(
         os.path.join( os.path.split(__file__)[0],
	libfilename )
             )

# If you still don't have it you probably have a mac??





From doutriaux1 at llnl.gov  Thu Jun  5 12:25:54 2008
From: doutriaux1 at llnl.gov (Charles Doutriaux)
Date: Thu, 05 Jun 2008 09:25:54 -0700
Subject: [Numpy-discussion] One more wrinkle in going to 5.0
In-Reply-To: <48480F16.6020504@iri.columbia.edu>
References: <1E3A72A9-08C0-4906-AE1E-6842FDDEC5E7@atmos.colostate.edu>	<47EBFB1E.5000703@llnl.gov>
	<4847F6D0.6020309@iri.columbia.edu> <4848052E.9030509@llnl.gov>
	<48480F16.6020504@iri.columbia.edu>
Message-ID: <48481392.4030405@llnl.gov>

Arthur
I'm forwarding your question to the numpy list, I'm hoping somebody 
there will be able to help you with that.

C.

Arthur M. Greene wrote:
> Hi All,
>
> This does not involve the CDAT-5 code, but rather files pickled under 
> earlier versions of CDAT. These files store the variable type along 
> with the data, but some types from 4.x are absent in 5.0, making the 
> pickled files unreadable. Example:
>
> $ cdat2
> Executing /home/amg/usr/local/cdat/5.0.0.beta1/bin/python
> Python 2.5.2 (r252:60911, May 30 2008, 11:00:23)
> [GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import cPickle as pickle
> >>> f=open('pickled/prTS2p1_Ind.p')
> >>> pr=pickle.load(f)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ImportError: No module named cdms.tvariable
> >>>
>
> $ grep cdms pickled/prTS2p1_Ind.p
> (ccdms.tvariable
> (icdms.grid
> (icdms.axis
> (icdms.axis
> ((icdms.axis
>
> If I replace "cdms" with "cdms2" (using sed) and try again, a new 
> error comes up:
>
> >>> f=open('prTS2p1_Ind1.p')
> >>> pr=pickle.load(f)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ImportError: No module named Numeric
>
> $ grep -B 3 -A 3 Numeric prTS2p1_Ind1.p
> (dp22
> S'gridtype'
> p23
> cNumeric
> array_constructor
> p24
> ((I1
>
> Numpy does not have an array_constructor attribute, so can't simply 
> replace "Numeric" with "numpy" in this case. If I replace instead with 
> "numpy.oldnumeric," trying to unpickle produces what appears to be a 
> screen dump (in the python console) of the pickled file as text, but 
> there is no assignment. So I'm presently at a loss as to how to read 
> such files. I have hundreds of files that were pickled under 4.x, so 
> this could be a big headache...
>
> Thx for any ideas,
>
> Arthur
>
>
> *^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*
> Arthur M. Greene, Ph.D.
> The International Research Institute for Climate and Society
> The Earth Institute, Columbia University, Lamont Campus
> amg -at- iri -dot- columbia -dot- edu
> *^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*
>
>
>
> -- 



From oliphant at enthought.com  Thu Jun  5 13:22:55 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 05 Jun 2008 12:22:55 -0500
Subject: [Numpy-discussion] One more wrinkle in going to 5.0
In-Reply-To: <48481392.4030405@llnl.gov>
References: <1E3A72A9-08C0-4906-AE1E-6842FDDEC5E7@atmos.colostate.edu>	<47EBFB1E.5000703@llnl.gov>	<4847F6D0.6020309@iri.columbia.edu>
	<4848052E.9030509@llnl.gov>	<48480F16.6020504@iri.columbia.edu>
	<48481392.4030405@llnl.gov>
Message-ID: <484820EF.9090605@enthought.com>

Charles Doutriaux wrote:
> Arthur
> I'm forwarding your question to the numpy list, I'm hoping somebody 
> there will be able to help you with that.
>   

Try using numpy.oldnumeric.load(f).

Or, just replace in the pickle stream:

Numeric --> numpy.oldnumeric

It should work fine.  If you have problems let us know.

-Travis




From christopherlmarshall at yahoo.com  Thu Jun  5 19:54:17 2008
From: christopherlmarshall at yahoo.com (Christopher Marshall)
Date: Thu, 5 Jun 2008 16:54:17 -0700 (PDT)
Subject: [Numpy-discussion] calculating the mean and variance of a large
	float vector
Message-ID: <524025.20392.qm@web31811.mail.mud.yahoo.com>

I will be calculating the mean and variance of a vector with millions of elements.

I was wondering how well numpy's mean and variance functions handle the numerical stability of such a calculation.


From kwgoodman at gmail.com  Thu Jun  5 21:06:28 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 5 Jun 2008 18:06:28 -0700
Subject: [Numpy-discussion] calculating the mean and variance of a large
	float vector
In-Reply-To: <524025.20392.qm@web31811.mail.mud.yahoo.com>
References: <524025.20392.qm@web31811.mail.mud.yahoo.com>
Message-ID: <f4f93d420806051806u46c502c6n375c77aa0f30c3dd@mail.gmail.com>

On Thu, Jun 5, 2008 at 4:54 PM, Christopher Marshall
<christopherlmarshall at yahoo.com> wrote:
> I will be calculating the mean and variance of a vector with millions of elements.
>
> I was wondering how well numpy's mean and variance functions handle the numerical stability of such a calculation.

How's this for stability?

>> import numpy as np
>> x = 1e10 * np.ones(100000000)
>> x.mean()
   10000000000.0

Are you worried that the mean might overflow on the intermediate sum?
Yeah, that could be a problem if you are counting stars or bugs in my
code:

>> x = 1e305 * np.ones(10000000)
>> y = x.mean()
Warning: overflow encountered in reduce
>> y
   inf

But at least you get a warning.


From alan.mcintyre at gmail.com  Thu Jun  5 21:55:19 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Thu, 5 Jun 2008 21:55:19 -0400
Subject: [Numpy-discussion] calculating the mean and variance of a large
	float vector
In-Reply-To: <f4f93d420806051806u46c502c6n375c77aa0f30c3dd@mail.gmail.com>
References: <524025.20392.qm@web31811.mail.mud.yahoo.com>
	<f4f93d420806051806u46c502c6n375c77aa0f30c3dd@mail.gmail.com>
Message-ID: <1d36917a0806051855n42bf17fbgf196b99b6f574701@mail.gmail.com>

On Thu, Jun 5, 2008 at 9:06 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> On Thu, Jun 5, 2008 at 4:54 PM, Christopher Marshall
> Are you worried that the mean might overflow on the intermediate sum?

I suspect (but please correct me if I'm wrong, Christopher) he's
asking whether there's cases where small variations in the contents of
the vector can produce relatively large changes in the value given as
the mean or variance.  This is a wild guess, but if the intermediate
sums are large enough, you could have a situation where (for example)
the last half-million values aren't counted in the intermediate sum
because they're too small relative to the intermediate sum.  (I hope
my numerics prof from last year doesn't read this list...I should
really have no trouble figuring out the condition number for mean/var
:).

What kinds of values are in your vectors, Christopher?  If nobody has
a sure answer for stability of mean/var, I'll see if I can figure it
out.

Cheers,
Alan


From kwgoodman at gmail.com  Thu Jun  5 22:16:30 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 5 Jun 2008 19:16:30 -0700
Subject: [Numpy-discussion] calculating the mean and variance of a large
	float vector
In-Reply-To: <1d36917a0806051855n42bf17fbgf196b99b6f574701@mail.gmail.com>
References: <524025.20392.qm@web31811.mail.mud.yahoo.com>
	<f4f93d420806051806u46c502c6n375c77aa0f30c3dd@mail.gmail.com>
	<1d36917a0806051855n42bf17fbgf196b99b6f574701@mail.gmail.com>
Message-ID: <f4f93d420806051916g40da988ay27122454e861e8f6@mail.gmail.com>

On Thu, Jun 5, 2008 at 6:55 PM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> On Thu, Jun 5, 2008 at 9:06 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>> On Thu, Jun 5, 2008 at 4:54 PM, Christopher Marshall
>> Are you worried that the mean might overflow on the intermediate sum?
>
> I suspect (but please correct me if I'm wrong, Christopher) he's
> asking whether there's cases where small variations in the contents of
> the vector can produce relatively large changes in the value given as
> the mean or variance.  This is a wild guess, but if the intermediate
> sums are large enough, you could have a situation where (for example)
> the last half-million values aren't counted in the intermediate sum
> because they're too small relative to the intermediate sum.  (I hope
> my numerics prof from last year doesn't read this list...I should
> really have no trouble figuring out the condition number for mean/var
> :).

How can that lead to instability? If the last half-million values are
small then they won't have a big impact on the mean even if they are
ignored. The variance is a mean too (of the squares), so it should be
stable too. Or am I, once again, missing the point?


From charlesr.harris at gmail.com  Thu Jun  5 22:20:13 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 5 Jun 2008 20:20:13 -0600
Subject: [Numpy-discussion] calculating the mean and variance of a large
	float vector
In-Reply-To: <1d36917a0806051855n42bf17fbgf196b99b6f574701@mail.gmail.com>
References: <524025.20392.qm@web31811.mail.mud.yahoo.com>
	<f4f93d420806051806u46c502c6n375c77aa0f30c3dd@mail.gmail.com>
	<1d36917a0806051855n42bf17fbgf196b99b6f574701@mail.gmail.com>
Message-ID: <e06186140806051920l5374121doabd8cf535cafc502@mail.gmail.com>

On Thu, Jun 5, 2008 at 7:55 PM, Alan McIntyre <alan.mcintyre at gmail.com>
wrote:

> On Thu, Jun 5, 2008 at 9:06 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> > On Thu, Jun 5, 2008 at 4:54 PM, Christopher Marshall
> > Are you worried that the mean might overflow on the intermediate sum?
>
> I suspect (but please correct me if I'm wrong, Christopher) he's
> asking whether there's cases where small variations in the contents of
> the vector can produce relatively large changes in the value given as
> the mean or variance.  This is a wild guess, but if the intermediate
> sums are large enough, you could have a situation where (for example)
> the last half-million values aren't counted in the intermediate sum
> because they're too small relative to the intermediate sum.  (I hope
> my numerics prof from last year doesn't read this list...I should
> really have no trouble figuring out the condition number for mean/var
> :).
>
> What kinds of values are in your vectors, Christopher?  If nobody has
> a sure answer for stability of mean/var, I'll see if I can figure it
> out.
>

If it is a real concern, add adjacent pairs, then add adjacent results, so
on and so forth. This is basically computing the DC value using an fft. For
that extra fillip, add the two smallest, put the sum back in the list, add
the two new smallest, so on and so forth. But several million values isn't
that many when done in double precision. The real question is: how much
accuracy is needed? Then design the algorithm to fit the need.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080605/f2f22b04/attachment.html>

From alan.mcintyre at gmail.com  Fri Jun  6 00:30:09 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Fri, 6 Jun 2008 00:30:09 -0400
Subject: [Numpy-discussion] calculating the mean and variance of a large
	float vector
In-Reply-To: <f4f93d420806051916g40da988ay27122454e861e8f6@mail.gmail.com>
References: <524025.20392.qm@web31811.mail.mud.yahoo.com>
	<f4f93d420806051806u46c502c6n375c77aa0f30c3dd@mail.gmail.com>
	<1d36917a0806051855n42bf17fbgf196b99b6f574701@mail.gmail.com>
	<f4f93d420806051916g40da988ay27122454e861e8f6@mail.gmail.com>
Message-ID: <1d36917a0806052130r264e41akb543dc2941018d@mail.gmail.com>

On Thu, Jun 5, 2008 at 10:16 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> How can that lead to instability? If the last half-million values are
> small then they won't have a big impact on the mean even if they are
> ignored. The variance is a mean too (of the squares), so it should be
> stable too. Or am I, once again, missing the point?

No, I just didn't think about it long enough, and I shouldn't have
tried to make an example off the cuff. ;)   After thinking about it
again, I think some loss of accuracy is probably the worst that can
happen.


From Chris.Barker at noaa.gov  Fri Jun  6 00:47:23 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 05 Jun 2008 21:47:23 -0700
Subject: [Numpy-discussion] A memory problem: why does mmap come up in
 numpy.inner?
In-Reply-To: <15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
References: <15e4667e0806041742n7a93c5a2r2a6140c5163cba7d@mail.gmail.com>
	<e06186140806041806y3bcf70edj8b9ea2ef43754d3f@mail.gmail.com>
	<15e4667e0806041841u39ad05bbt2dce5a8014bf6a5e@mail.gmail.com>
Message-ID: <4848C15B.6080908@noaa.gov>

There is a good discussion of the future of 64 bit python on the 
pythonmac list.

Also, Apple seems to be indicating that 10.6 will only support Intel64 
-- so some day this will all be default!


by the way, sys.maxint is another easy way to check, without needing numpy.

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer

NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception


From millman at berkeley.edu  Fri Jun  6 05:24:09 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Fri, 6 Jun 2008 02:24:09 -0700
Subject: [Numpy-discussion] ANN: SciPy 2008 Conference
Message-ID: <c7009a550806060224q1523d63fx547abed843b8434a@mail.gmail.com>

Greetings,

The SciPy 2008 Conference website is now open:  http://conference.scipy.org

This year's conference will be at Caltech from August 19-24:

Tutorials:        August 19-20 (Tuesday and Wednesday)
Conference:  August 21-22 (Thursday and Friday)
Sprints:           August 23-24 (Saturday and Sunday)

Exciting things are happening in the Python community, and the SciPy
2008 Conference is an excellent opportunity to exchange ideas, learn
techniques, contribute code and affect the direction of scientific
computing (or just to learn what all the fuss is about).  We'll be
announcing the Keynote Speaker and providing a detailed schedule in
the coming weeks.

This year we are asking presenters to submit short papers to be included
in the conference proceedings:  http://conference.scipy.org/call_for_papers

We will be sending out more details over the next few days and will publish
this announcement much more widely, but wanted to make the announcement
to a smaller group to start.

Cheers,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From dyamins at gmail.com  Fri Jun  6 09:52:32 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Fri, 6 Jun 2008 09:52:32 -0400
Subject: [Numpy-discussion] Building 64-bit numpy on OSX
Message-ID: <15e4667e0806060652n7bcaa2aes2c21e15f9a1273ae@mail.gmail.com>

>./configure --disable-toolbox-glue
--prefix=/Users/mabshoff/64bitnumpy/python-2.5.2-bin --with-gcc="gcc -m64"


> Let's build numpy 1.1.0:
>
> bsd:64bitnumpy mabshoff$ tar xf numpy-1.1.0.tar.gz
> bsd:64bitnumpy mabshoff$ cd numpy-1.1.0
> bsd:numpy-1.1.0 mabshoff$ python setup.py install
> <SNIP>
>
>

Michael thanks for your help on building 64-bit python/numpy.     I followed
your instructions, and was able to build the python portion.  (I did the
test "python -c 'import platform;print platform.architecture()' "  suggested
by Jon and I got the 64-bit result, so I think it worked properly.)
Thanks very much for this!   However, I was unable to get the numpy build to
work.   When running the setup.py install, I get all sorts of permission
errors, so I'm forced to run it as su.  (Is this a bad idea?)     Anyhow,
when I do run "sudo python setup.py install" in the numpy-1.1.0 directory I
downloaded from SciPy website, the build apparently works.  But when I go
into python, I'm not able to import numpy.  Specifically, I get:

   daniel-yaminss-mac-pro:Desktop danielyamins$ python
   Python 2.5.2 (r252:60911, Jun  5 2008, 19:13:50)
   [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import numpy
  Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/usr/local/lib/python2.5/site-packages/numpy/__init__.py", line
93, in <module>
       import add_newdocs
     File "/usr/local/lib/python2.5/site-packages/numpy/add_newdocs.py",
line 9, in <module>
       from lib import add_newdoc
     File "/usr/local/lib/python2.5/site-packages/numpy/lib/__init__.py",
line 4, in <module>
       from type_check import *
     File "/usr/local/lib/python2.5/site-packages/numpy/lib/type_check.py",
line 8, in <module>
       import numpy.core.numeric as _nx
     File "/usr/local/lib/python2.5/site-packages/numpy/core/__init__.py",
line 5, in <module>
       import multiarray
  ImportError:
dlopen(/usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so, 2):
no suitable image found.  Did find:
      /usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so: no
matching architecture in universal wrapper


Does anyone know what this error means, and/or how to fix it?   Sorry if
it's obvious and I should know it alread.y


Thanks!
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080606/e060a61e/attachment.html>

From bsouthey at gmail.com  Fri Jun  6 09:56:50 2008
From: bsouthey at gmail.com (Bruce Southey)
Date: Fri, 06 Jun 2008 08:56:50 -0500
Subject: [Numpy-discussion] calculating the mean and variance of a large
 float vector
In-Reply-To: <1d36917a0806052130r264e41akb543dc2941018d@mail.gmail.com>
References: <524025.20392.qm@web31811.mail.mud.yahoo.com>	<f4f93d420806051806u46c502c6n375c77aa0f30c3dd@mail.gmail.com>	<1d36917a0806051855n42bf17fbgf196b99b6f574701@mail.gmail.com>	<f4f93d420806051916g40da988ay27122454e861e8f6@mail.gmail.com>
	<1d36917a0806052130r264e41akb543dc2941018d@mail.gmail.com>
Message-ID: <48494222.6060808@gmail.com>

Alan McIntyre wrote:
> On Thu, Jun 5, 2008 at 10:16 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>   
>> How can that lead to instability? If the last half-million values are
>> small then they won't have a big impact on the mean even if they are
>> ignored. The variance is a mean too (of the squares), so it should be
>> stable too. Or am I, once again, missing the point?
>>     
>
> No, I just didn't think about it long enough, and I shouldn't have
> tried to make an example off the cuff. ;)   After thinking about it
> again, I think some loss of accuracy is probably the worst that can
> happen.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   
Any problems are going to mainly due to the distribution of numbers 
especially if there are very small numbers and very large numbers. This 
is mitigated by numerical precision and algorithm - my guess is that it 
will take a rather extreme case to cause you any problems.

Python and NumPy are already using high numerical precision (may depend 
on architecture)  and NumPy defines 32-bit, 64-bit and 128-bit precision 
if you want to go higher (or lower). This means that calculations are 
rather insensitive to numbers used so typically there is no reason for 
any concern (ignoring the old Pentium FDIV bug, 
http://en.wikipedia.org/wiki/Pentium_FDIV_bug ).

The second issue is the algorithm where you need to balance performance 
with precision. For simple calculations:
http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance

Bruce







From bsouthey at gmail.com  Fri Jun  6 10:16:32 2008
From: bsouthey at gmail.com (Bruce Southey)
Date: Fri, 06 Jun 2008 09:16:32 -0500
Subject: [Numpy-discussion] calculating the mean and variance of a large
 float vector
In-Reply-To: <48494222.6060808@gmail.com>
References: <524025.20392.qm@web31811.mail.mud.yahoo.com>	<f4f93d420806051806u46c502c6n375c77aa0f30c3dd@mail.gmail.com>	<1d36917a0806051855n42bf17fbgf196b99b6f574701@mail.gmail.com>	<f4f93d420806051916g40da988ay27122454e861e8f6@mail.gmail.com>
	<1d36917a0806052130r264e41akb543dc2941018d@mail.gmail.com>
	<48494222.6060808@gmail.com>
Message-ID: <484946C0.5050600@gmail.com>

Bruce Southey wrote:
> Alan McIntyre wrote:
>> On Thu, Jun 5, 2008 at 10:16 PM, Keith Goodman <kwgoodman at gmail.com> 
>> wrote:
>>  
>>> How can that lead to instability? If the last half-million values are
>>> small then they won't have a big impact on the mean even if they are
>>> ignored. The variance is a mean too (of the squares), so it should be
>>> stable too. Or am I, once again, missing the point?
>>>     
>>
>> No, I just didn't think about it long enough, and I shouldn't have
>> tried to make an example off the cuff. ;)   After thinking about it
>> again, I think some loss of accuracy is probably the worst that can
>> happen.
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>>   
> Any problems are going to mainly due to the distribution of numbers 
> especially if there are very small numbers and very large numbers. 
> This is mitigated by numerical precision and algorithm - my guess is 
> that it will take a rather extreme case to cause you any problems.
>
> Python and NumPy are already using high numerical precision (may 
> depend on architecture)  and NumPy defines 32-bit, 64-bit and 128-bit 
> precision if you want to go higher (or lower). This means that 
> calculations are rather insensitive to numbers used so typically there 
> is no reason for any concern (ignoring the old Pentium FDIV bug, 
> http://en.wikipedia.org/wiki/Pentium_FDIV_bug ).
>
> The second issue is the algorithm where you need to balance 
> performance with precision. For simple calculations:
> http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
>
> Bruce
>
I forgot to add:
import numpy
x = 1e305 * numpy.ones(10000000, np.float128)
type(x[0]) #  gives <type 'numpy.float128'>
x.mean()  # gives 1.000000000000036542e+305

Bruce


From mabshoff at googlemail.com  Fri Jun  6 14:08:34 2008
From: mabshoff at googlemail.com (Michael Abshoff)
Date: Fri, 06 Jun 2008 20:08:34 +0200
Subject: [Numpy-discussion] Building 64-bit numpy on OSX
In-Reply-To: <15e4667e0806060652n7bcaa2aes2c21e15f9a1273ae@mail.gmail.com>
References: <15e4667e0806060652n7bcaa2aes2c21e15f9a1273ae@mail.gmail.com>
Message-ID: <48497D22.10705@googlemail.com>

Dan Yamins wrote:

Hi Dan,
> >./configure --disable-toolbox-glue 
> --prefix=/Users/mabshoff/64bitnumpy/python-2.5.2-bin --with-gcc="gcc 
> -m64"
>
>
>     Let's build numpy 1.1.0:
>
>     bsd:64bitnumpy mabshoff$ tar xf numpy-1.1.0.tar.gz
>     bsd:64bitnumpy mabshoff$ cd numpy-1.1.0
>     bsd:numpy-1.1.0 mabshoff$ python setup.py install
>     <SNIP>
>
>
>  
> Michael thanks for your help on building 64-bit python/numpy.     I 
> followed your instructions, and was able to build the python portion.  
> (I did the test "python -c 'import platform;print 
> platform.architecture()' "  suggested by Jon and I got the 64-bit 
> result, so I think it worked properly.)    Thanks very much for 
> this!   However, I was unable to get the numpy build to work.   When 
> running the setup.py install, I get all sorts of permission errors, so 
> I'm forced to run it as su.  (Is this a bad idea?)     Anyhow, when I 
> do run "sudo python setup.py install" in the numpy-1.1.0 directory I 
> downloaded from SciPy website, the build apparently works.
If you were able to install python without sudo you should not need to 
use sudo to build numpy. I certainly did not need to do so. Can you post 
the exact sequence of commands you ran?

>   But when I go into python, I'm not able to import numpy.  
> Specifically, I get:
>
>    daniel-yaminss-mac-pro:Desktop danielyamins$ python
>    Python 2.5.2 (r252:60911, Jun  5 2008, 19:13:50)
>    [GCC 4.0.1 (Apple Inc. build 5465)] on darwin  
>    Type "help", "copyright", "credits" or "license" for more information.
>    >>> import numpy
>   Traceback (most recent call last):
>      File "<stdin>", line 1, in <module>
>      File "/usr/local/lib/python2.5/site-packages/numpy/__init__.py", 
> line 93, in <module>
>        import add_newdocs
>      File 
> "/usr/local/lib/python2.5/site-packages/numpy/add_newdocs.py", line 9, 
> in <module>
>        from lib import add_newdoc
>      File 
> "/usr/local/lib/python2.5/site-packages/numpy/lib/__init__.py", line 
> 4, in <module>
>        from type_check import *
>      File 
> "/usr/local/lib/python2.5/site-packages/numpy/lib/type_check.py", line 
> 8, in <module>
>        import numpy.core.numeric as _nx
>      File 
> "/usr/local/lib/python2.5/site-packages/numpy/core/__init__.py", line 
> 5, in <module>
>        import multiarray
>   ImportError: 
> dlopen(/usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so, 
> 2): no suitable image found.  Did find:
>       /usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so: 
> no matching architecture in universal wrapper
>
>
> Does anyone know what this error means, and/or how to fix it?   Sorry 
> if it's obvious and I should know it alread.y
It is not obvious to me, but I do't see what is wrong either. Apple's 
python is not in /usr/local, so it does not sound like when you build 
numpy you ended up using the "wrong" python.

>
>
> Thanks!
> Dan
>
>
Cheers,

Michael
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From kwgoodman at gmail.com  Fri Jun  6 14:24:10 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Fri, 6 Jun 2008 11:24:10 -0700
Subject: [Numpy-discussion] Inplace shift
Message-ID: <f4f93d420806061124s494eda58m405cbab2befc51d9@mail.gmail.com>

I'd like to shift the columns of a 2d array one column to the right.
Is there a way to do that without making a copy?

This doesn't work:

>> import numpy as np
>> x = np.random.rand(2,3)
>> x[:,1:] = x[:,:-1]
>> x

array([[ 0.44789223,  0.44789223,  0.44789223],
       [ 0.80600897,  0.80600897,  0.80600897]])

What I'm trying to do (but without the copy):

x2 = x.copy()
x[:,1:] = x2[:,:-1]


From zachary.pincus at yale.edu  Fri Jun  6 15:08:26 2008
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Fri, 6 Jun 2008 15:08:26 -0400
Subject: [Numpy-discussion] Inplace shift
In-Reply-To: <f4f93d420806061124s494eda58m405cbab2befc51d9@mail.gmail.com>
References: <f4f93d420806061124s494eda58m405cbab2befc51d9@mail.gmail.com>
Message-ID: <56ABCC49-6AFD-4CE6-B7EA-FF130BBE98C8@yale.edu>

> I'd like to shift the columns of a 2d array one column to the right.
> Is there a way to do that without making a copy?


I think what you want is numpy.roll?

Definition:	numpy.roll(a, shift, axis=None)
Docstring:
     Roll the elements in the array by 'shift' positions along
     the given axis.

     >>> from numpy import roll
     >>> arange(10)
     array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
     >>> roll(arange(10), 2)
     array([8, 9, 0, 1, 2, 3, 4, 5, 6, 7])

Zach


From robert.kern at gmail.com  Fri Jun  6 15:16:43 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 6 Jun 2008 14:16:43 -0500
Subject: [Numpy-discussion] Inplace shift
In-Reply-To: <56ABCC49-6AFD-4CE6-B7EA-FF130BBE98C8@yale.edu>
References: <f4f93d420806061124s494eda58m405cbab2befc51d9@mail.gmail.com>
	<56ABCC49-6AFD-4CE6-B7EA-FF130BBE98C8@yale.edu>
Message-ID: <3d375d730806061216k5943e934s723a87283e463740@mail.gmail.com>

On Fri, Jun 6, 2008 at 2:08 PM, Zachary Pincus <zachary.pincus at yale.edu> wrote:
>> I'd like to shift the columns of a 2d array one column to the right.
>> Is there a way to do that without making a copy?
>
> I think what you want is numpy.roll?

It doesn't work inplace. It creates a new array, thereby effectively
copying the data.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From pgmdevlist at gmail.com  Fri Jun  6 15:54:57 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Fri, 6 Jun 2008 15:54:57 -0400
Subject: [Numpy-discussion] Methods accepting out as arguments
Message-ID: <200806061554.57726.pgmdevlist@gmail.com>

Folks,
Where could I find a list of the ndarray methods accepting out as an 
argument ?
Thx a lot in advance.


From charlesr.harris at gmail.com  Fri Jun  6 16:27:19 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 6 Jun 2008 14:27:19 -0600
Subject: [Numpy-discussion] Methods accepting out as arguments
In-Reply-To: <200806061554.57726.pgmdevlist@gmail.com>
References: <200806061554.57726.pgmdevlist@gmail.com>
Message-ID: <e06186140806061327m2235f221p2258f7f799212363@mail.gmail.com>

On Fri, Jun 6, 2008 at 1:54 PM, Pierre GM <pgmdevlist at gmail.com> wrote:

> Folks,
> Where could I find a list of the ndarray methods accepting out as an
> argument ?


Alan McIntyre was working on that. I don't know what the current status of
the list is.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080606/2c14c94a/attachment.html>

From alan.mcintyre at gmail.com  Fri Jun  6 16:41:21 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Fri, 6 Jun 2008 16:41:21 -0400
Subject: [Numpy-discussion] Methods accepting out as arguments
In-Reply-To: <200806061554.57726.pgmdevlist@gmail.com>
References: <200806061554.57726.pgmdevlist@gmail.com>
Message-ID: <1d36917a0806061341v6d2b62edue7a65bd8d808d6fd@mail.gmail.com>

Here's an automatically generated list from the current numpy trunk.

I should really post the script I used to make this somewhere.
Anybody have any suggestions on a good place to put it?

On Fri, Jun 6, 2008 at 3:54 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
> Folks,
> Where could I find a list of the ndarray methods accepting out as an
> argument ?
> Thx a lot in advance.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: numpy-out-func-like.txt.gz
Type: application/x-gzip
Size: 4562 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080606/d70d30f6/attachment.bin>

From kwgoodman at gmail.com  Fri Jun  6 16:50:32 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Fri, 6 Jun 2008 13:50:32 -0700
Subject: [Numpy-discussion] Methods accepting out as arguments
In-Reply-To: <1d36917a0806061341v6d2b62edue7a65bd8d808d6fd@mail.gmail.com>
References: <200806061554.57726.pgmdevlist@gmail.com>
	<1d36917a0806061341v6d2b62edue7a65bd8d808d6fd@mail.gmail.com>
Message-ID: <f4f93d420806061350g21d1bb7fh3e84813c3f6a880d@mail.gmail.com>

On Fri, Jun 6, 2008 at 1:41 PM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> Here's an automatically generated list from the current numpy trunk.
>
> I should really post the script I used to make this somewhere.
> Anybody have any suggestions on a good place to put it?

I don't know where to put it (the wiki?). But I'd like to see how you did it


From alan.mcintyre at gmail.com  Fri Jun  6 17:11:03 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Fri, 6 Jun 2008 17:11:03 -0400
Subject: [Numpy-discussion] Methods accepting out as arguments
In-Reply-To: <f4f93d420806061350g21d1bb7fh3e84813c3f6a880d@mail.gmail.com>
References: <200806061554.57726.pgmdevlist@gmail.com>
	<1d36917a0806061341v6d2b62edue7a65bd8d808d6fd@mail.gmail.com>
	<f4f93d420806061350g21d1bb7fh3e84813c3f6a880d@mail.gmail.com>
Message-ID: <1d36917a0806061411t17e07ecfi4ecb9d6e71ae2d1f@mail.gmail.com>

On Fri, Jun 6, 2008 at 4:50 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> On Fri, Jun 6, 2008 at 1:41 PM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
>> Here's an automatically generated list from the current numpy trunk.
>>
>> I should really post the script I used to make this somewhere.
>> Anybody have any suggestions on a good place to put it?
>
> I don't know where to put it (the wiki?). But I'd like to see how you did it

It's not much to look at; I put it here for now:
http://alan-mcintyre.blogspot.com/2008/06/just-so-its-somewhere.html


From dyamins at gmail.com  Fri Jun  6 17:50:02 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Fri, 6 Jun 2008 17:50:02 -0400
Subject: [Numpy-discussion] Building 64-bit numpy on OSX
In-Reply-To: <48497D22.10705@googlemail.com>
References: <15e4667e0806060652n7bcaa2aes2c21e15f9a1273ae@mail.gmail.com>
	<48497D22.10705@googlemail.com>
Message-ID: <15e4667e0806061450g6ca4154ep96c742c0b6ccea8c@mail.gmail.com>

> > I'm forced to run it as su.  (Is this a bad idea?)     Anyhow, when I
> > do run "sudo python setup.py install" in the numpy-1.1.0 directory I
> > downloaded from SciPy website, the build apparently works.
> If you were able to install python without sudo you should not need to
> use sudo to build numpy. I certainly did not need to do so. Can you post
> the exact sequence of commands you ran?
>

Here they are:

   daniel-yaminss-mac-pro:numpy-1.1.0 danielyamins$ python setup.py install
   Running from numpy source directory.

   ....[a whole bunch of output, things apparently working for a little
while. then:]

   copying build/lib.macosx-10.3-i386-2.5/numpy/__config__.py ->
/usr/local/lib/python2.5/site-packages/numpy
   error: could not delete
'/usr/local/lib/python2.5/site-packages/numpy/__config__.py': Permission
denied

This is what happens when I try to run the install without "sudo".  When I
do "sudo python setup.py install" instead, the process finishes but then, as
I said before, when I open a python interpreter and try to import numpy it
fails in the way I posted previously.


>It is not obvious to me, but I do't see what is wrong either. Apple's
>python is not in /usr/local, so it does not sound like when you build
>numpy you ended up using the "wrong" python.

So, I think it's using the "right" python since I modified my .profile to
make the 64-bit version stored in /usr/local the default.    (and when I
open python the "build" information gives the time of the new one I built.)


thanks,
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080606/7b6c7c16/attachment.html>

From thrabe at burnham.org  Fri Jun  6 18:10:25 2008
From: thrabe at burnham.org (Thomas Hrabe)
Date: Fri, 6 Jun 2008 15:10:25 -0700
Subject: [Numpy-discussion] embedded arrays
Message-ID: <A7E1AE993C63414B8B2F762E13BF4D898E4E9F@MAIL.burnham.org>

Hi all,

while writing a extension module in C++ for python & numpy, I find a strange error.

I can send and retrieve numpy arrays to and from my module.
But python stops if I do the following:

a = numpy.array([[1.1,2,3],[4,5,6]])
PM.put(a,'a')   //send a to the module
b = PM.get('a') //get a identical copy from the module
print b
array([[ 1.1,  2. ,  3. ],
       [ 4. ,  5. ,  6. ]])

b.__class__
Out[36]: <type 'numpy.ndarray'>


perfect, until
a == b

the interpreter does not continue from here... 
I can add values to to b, everything, but a == b simply crashes ...?

Does anybody have a clue for this problem?

Have a nice weekend,
Thomas



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080606/1b63de1d/attachment.html>

From robert.kern at gmail.com  Fri Jun  6 18:27:22 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 6 Jun 2008 17:27:22 -0500
Subject: [Numpy-discussion] embedded arrays
In-Reply-To: <A7E1AE993C63414B8B2F762E13BF4D898E4E9F@MAIL.burnham.org>
References: <A7E1AE993C63414B8B2F762E13BF4D898E4E9F@MAIL.burnham.org>
Message-ID: <3d375d730806061527w42fccb9dt8e5cc857934d4196@mail.gmail.com>

On Fri, Jun 6, 2008 at 17:10, Thomas Hrabe <thrabe at burnham.org> wrote:
> Hi all,
>
> while writing a extension module in C++ for python & numpy, I find a strange
> error.
>
> I can send and retrieve numpy arrays to and from my module.
> But python stops if I do the following:
>
> a = numpy.array([[1.1,2,3],[4,5,6]])
> PM.put(a,'a')   //send a to the module
> b = PM.get('a') //get a identical copy from the module
> print b
> array([[ 1.1,  2. ,  3. ],
>        [ 4. ,  5. ,  6. ]])
>
> b.__class__
> Out[36]: <type 'numpy.ndarray'>
>
>
> perfect, until
> a == b
>
> the interpreter does not continue from here...
> I can add values to to b, everything, but a == b simply crashes ...?
>
> Does anybody have a clue for this problem?

Not really. It probably depends on some details with your interfacing.
Since we don't have access to your code, we don't have much to go on.
You might have buggy reference counting or perhaps you gave the numpy
ndarray ownership of the array's memory when it actually shouldn't.
Memory issues can be a bit finicky where everything will work for a
while, then crash.

Try running your program under a C debugger like gdb so we can get a
backtrace. That might give us some more ideas about exactly where
problems are occurring.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From thrabe at burnham.org  Fri Jun  6 18:25:44 2008
From: thrabe at burnham.org (Thomas Hrabe)
Date: Fri, 6 Jun 2008 15:25:44 -0700
Subject: [Numpy-discussion] embedded arrays
References: <A7E1AE993C63414B8B2F762E13BF4D898E4E9F@MAIL.burnham.org>
Message-ID: <A7E1AE993C63414B8B2F762E13BF4D898E4EA1@MAIL.burnham.org>

Furthermore, I sometimes get a 
Segmentation fault 
Illegal instruction 

and sometimes it works

It might be a memory leak, due to the segfault and the arbitrary behavior.?

Best

-----Urspr?ngliche Nachricht-----
Von: numpy-discussion-bounces at scipy.org im Auftrag von Thomas Hrabe
Gesendet: Fr 06.06.2008 15:10
An: numpy-discussion at scipy.org
Betreff: [Numpy-discussion] embedded arrays
 
Hi all,

while writing a extension module in C++ for python & numpy, I find a strange error.

I can send and retrieve numpy arrays to and from my module.
But python stops if I do the following:

a = numpy.array([[1.1,2,3],[4,5,6]])
PM.put(a,'a')   //send a to the module
b = PM.get('a') //get a identical copy from the module
print b
array([[ 1.1,  2. ,  3. ],
       [ 4. ,  5. ,  6. ]])

b.__class__
Out[36]: <type 'numpy.ndarray'>


perfect, until
a == b

the interpreter does not continue from here... 
I can add values to to b, everything, but a == b simply crashes ...?

Does anybody have a clue for this problem?

Have a nice weekend,
Thomas




-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 3107 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080606/0227a8a3/attachment.bin>

From pgmdevlist at gmail.com  Fri Jun  6 18:35:41 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Fri, 6 Jun 2008 18:35:41 -0400
Subject: [Numpy-discussion] Methods accepting out as arguments
In-Reply-To: <1d36917a0806061411t17e07ecfi4ecb9d6e71ae2d1f@mail.gmail.com>
References: <200806061554.57726.pgmdevlist@gmail.com>
	<f4f93d420806061350g21d1bb7fh3e84813c3f6a880d@mail.gmail.com>
	<1d36917a0806061411t17e07ecfi4ecb9d6e71ae2d1f@mail.gmail.com>
Message-ID: <200806061835.41781.pgmdevlist@gmail.com>

On Friday 06 June 2008 17:11:03 Alan McIntyre wrote:
> >> Here's an automatically generated list from the current numpy trunk.

OK, thanks an awful lot ! 
Now I just have to process your file to get the methods, but that's a nice 
little exercise.
Thx again !



From charlesr.harris at gmail.com  Fri Jun  6 19:28:47 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 6 Jun 2008 17:28:47 -0600
Subject: [Numpy-discussion] Building 64-bit numpy on OSX
In-Reply-To: <15e4667e0806061450g6ca4154ep96c742c0b6ccea8c@mail.gmail.com>
References: <15e4667e0806060652n7bcaa2aes2c21e15f9a1273ae@mail.gmail.com>
	<48497D22.10705@googlemail.com>
	<15e4667e0806061450g6ca4154ep96c742c0b6ccea8c@mail.gmail.com>
Message-ID: <e06186140806061628k4ab29e5fjccacab12a2866787@mail.gmail.com>

On Fri, Jun 6, 2008 at 3:50 PM, Dan Yamins <dyamins at gmail.com> wrote:

>
>
>
>
>> > I'm forced to run it as su.  (Is this a bad idea?)     Anyhow, when I
>> > do run "sudo python setup.py install" in the numpy-1.1.0 directory I
>> > downloaded from SciPy website, the build apparently works.
>> If you were able to install python without sudo you should not need to
>> use sudo to build numpy. I certainly did not need to do so. Can you post
>> the exact sequence of commands you ran?
>>
>
> Here they are:
>
>    daniel-yaminss-mac-pro:numpy-1.1.0 danielyamins$ python setup.py install
>
>    Running from numpy source directory.
>
>    ....[a whole bunch of output, things apparently working for a little
> while. then:]
>
>    copying build/lib.macosx-10.3-i386-2.5/numpy/__config__.py ->
> /usr/local/lib/python2.5/site-packages/numpy
>    error: could not delete
> '/usr/local/lib/python2.5/site-packages/numpy/__config__.py': Permission
> denied
>
> This is what happens when I try to run the install without "sudo".  When I
> do "sudo python setup.py install" instead, the process finishes but then, as
> I said before, when I open a python interpreter and try to import numpy it
> fails in the way I posted previously.
>
>
> >It is not obvious to me, but I do't see what is wrong either. Apple's
> >python is not in /usr/local, so it does not sound like when you build
> >numpy you ended up using the "wrong" python.
>
> So, I think it's using the "right" python since I modified my .profile to
> make the 64-bit version stored in /usr/local the default.    (and when I
> open python the "build" information gives the time of the new one I built.)
>
>

I don't think the permissions problem really matters, it just means you
don't write permissions in /usr/local/lib/python2.5/site-packages, which is
normal.  What I think matters is "no matching architecture in universal
wrapper". Hmmm. I wonder if you and Michael have the same versions of OS X?
And why is dlopen looking for a universal library? One would hope that
distutils would have taken care of that.

Out of curiosity, where are the 32/64 bit libraries normally put? Do you
have a /usr/local/lib32 or a /usr/local/lib64? What does

file /usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so

do?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080606/cf08b26b/attachment.html>

From charlesr.harris at gmail.com  Fri Jun  6 19:34:39 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 6 Jun 2008 17:34:39 -0600
Subject: [Numpy-discussion] Building 64-bit numpy on OSX
In-Reply-To: <e06186140806061628k4ab29e5fjccacab12a2866787@mail.gmail.com>
References: <15e4667e0806060652n7bcaa2aes2c21e15f9a1273ae@mail.gmail.com>
	<48497D22.10705@googlemail.com>
	<15e4667e0806061450g6ca4154ep96c742c0b6ccea8c@mail.gmail.com>
	<e06186140806061628k4ab29e5fjccacab12a2866787@mail.gmail.com>
Message-ID: <e06186140806061634t1e08bfb8tf968a76e50c37750@mail.gmail.com>

On Fri, Jun 6, 2008 at 5:28 PM, Charles R Harris <charlesr.harris at gmail.com>
wrote:

>
>
> On Fri, Jun 6, 2008 at 3:50 PM, Dan Yamins <dyamins at gmail.com> wrote:
>
>>
>>
>>
>>
>>> > I'm forced to run it as su.  (Is this a bad idea?)     Anyhow, when I
>>> > do run "sudo python setup.py install" in the numpy-1.1.0 directory I
>>> > downloaded from SciPy website, the build apparently works.
>>> If you were able to install python without sudo you should not need to
>>> use sudo to build numpy. I certainly did not need to do so. Can you post
>>> the exact sequence of commands you ran?
>>>
>>
>> Here they are:
>>
>>    daniel-yaminss-mac-pro:numpy-1.1.0 danielyamins$ python setup.py
>> install
>>    Running from numpy source directory.
>>
>>    ....[a whole bunch of output, things apparently working for a little
>> while. then:]
>>
>>    copying build/lib.macosx-10.3-i386-2.5/numpy/__config__.py ->
>> /usr/local/lib/python2.5/site-packages/numpy
>>    error: could not delete
>> '/usr/local/lib/python2.5/site-packages/numpy/__config__.py': Permission
>> denied
>>
>> This is what happens when I try to run the install without "sudo".  When I
>> do "sudo python setup.py install" instead, the process finishes but then, as
>> I said before, when I open a python interpreter and try to import numpy it
>> fails in the way I posted previously.
>>
>>
>> >It is not obvious to me, but I do't see what is wrong either. Apple's
>> >python is not in /usr/local, so it does not sound like when you build
>> >numpy you ended up using the "wrong" python.
>>
>> So, I think it's using the "right" python since I modified my .profile to
>> make the 64-bit version stored in /usr/local the default.    (and when I
>> open python the "build" information gives the time of the new one I built.)
>>
>>
>
> I don't think the permissions problem really matters, it just means you
> don't write permissions in /usr/local/lib/python2.5/site-packages, which is
> normal.  What I think matters is "no matching architecture in universal
> wrapper". Hmmm. I wonder if you and Michael have the same versions of OS X?
> And why is dlopen looking for a universal library? One would hope that
> distutils would have taken care of that.
>
> Out of curiosity, where are the 32/64 bit libraries normally put? Do you
> have a /usr/local/lib32 or a /usr/local/lib64? What does
>
> file /usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so
>
> do?
>

If you google for "no matching architecture in universal wrapper" you will
turn up a bunch of references to leopard (OS X 10.5?). So this problem has
been seen before, but I didn't follow any of the threads far enough to know
what the solution might be.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080606/50e3ae68/attachment.html>

From mabshoff at googlemail.com  Fri Jun  6 19:33:45 2008
From: mabshoff at googlemail.com (Michael Abshoff)
Date: Sat, 07 Jun 2008 01:33:45 +0200
Subject: [Numpy-discussion] Building 64-bit numpy on OSX
In-Reply-To: <e06186140806061628k4ab29e5fjccacab12a2866787@mail.gmail.com>
References: <15e4667e0806060652n7bcaa2aes2c21e15f9a1273ae@mail.gmail.com>	<48497D22.10705@googlemail.com>	<15e4667e0806061450g6ca4154ep96c742c0b6ccea8c@mail.gmail.com>
	<e06186140806061628k4ab29e5fjccacab12a2866787@mail.gmail.com>
Message-ID: <4849C959.30700@googlemail.com>

Charles R Harris wrote:
>
>
> On Fri, Jun 6, 2008 at 3:50 PM, Dan Yamins <dyamins at gmail.com 
> <mailto:dyamins at gmail.com>> wrote:
>
>
>
>
>
>         > I'm forced to run it as su.  (Is this a bad idea?)    
>         Anyhow, when I
>         > do run "sudo python setup.py install" in the numpy-1.1.0
>         directory I
>         > downloaded from SciPy website, the build apparently works.
>         If you were able to install python without sudo you should not
>         need to
>         use sudo to build numpy. I certainly did not need to do so.
>         Can you post
>         the exact sequence of commands you ran?
>
>
>     Here they are:
>
>        daniel-yaminss-mac-pro:numpy-1.1.0 danielyamins$ python
>     setup.py install
>        Running from numpy source directory.
>
>        ....[a whole bunch of output, things apparently working for a
>     little while. then:]
>
>        copying build/lib.macosx-10.3-i386-2.5/numpy/__config__.py ->
>     /usr/local/lib/python2.5/site-packages/numpy
>        error: could not delete
>     '/usr/local/lib/python2.5/site-packages/numpy/__config__.py':
>     Permission denied
>
>     This is what happens when I try to run the install without
>     "sudo".  When I do "sudo python setup.py install" instead, the
>     process finishes but then, as I said before, when I open a python
>     interpreter and try to import numpy it fails in the way I posted
>     previously.
>
>
>
>     >It is not obvious to me, but I do't see what is wrong either. Apple's
>     >python is not in /usr/local, so it does not sound like when you build
>     >numpy you ended up using the "wrong" python.
>
>     So, I think it's using the "right" python since I modified my
>     .profile to make the 64-bit version stored in /usr/local the
>     default.    (and when I open python the "build" information gives
>     the time of the new one I built.) 
>
>
> I don't think the permissions problem really matters, it just means 
> you don't write permissions in /usr/local/lib/python2.5/site-packages, 
> which is normal.  What I think matters is "no matching architecture in 
> universal wrapper". Hmmm. I wonder if you and Michael have the same 
> versions of OS X?

For the record: I am running OSX 10.5.2 on an Intel Mac.
> And why is dlopen looking for a universal library? One would hope that 
> distutils would have taken care of that.
>
Maybe Dan did overwrite some older 32 bit Python with the new 64 bit build?

> Out of curiosity, where are the 32/64 bit libraries normally put? Do 
> you have a /usr/local/lib32 or a /usr/local/lib64? What does
>
> file /usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so
>
> do?
>
> Chuck
>
I will be traveling, so I will drop off the net for the next day or two, 
but I expect to catch up with email then.

Cheers,

Michael

>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From peridot.faceted at gmail.com  Sat Jun  7 01:46:58 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 7 Jun 2008 01:46:58 -0400
Subject: [Numpy-discussion] Inplace shift
In-Reply-To: <f4f93d420806061124s494eda58m405cbab2befc51d9@mail.gmail.com>
References: <f4f93d420806061124s494eda58m405cbab2befc51d9@mail.gmail.com>
Message-ID: <ce557a360806062246q6ad98541i882ed4fe2b2f0269@mail.gmail.com>

2008/6/6 Keith Goodman <kwgoodman at gmail.com>:
> I'd like to shift the columns of a 2d array one column to the right.
> Is there a way to do that without making a copy?
>
> This doesn't work:
>
>>> import numpy as np
>>> x = np.random.rand(2,3)
>>> x[:,1:] = x[:,:-1]
>>> x
>
> array([[ 0.44789223,  0.44789223,  0.44789223],
>       [ 0.80600897,  0.80600897,  0.80600897]])
>
> What I'm trying to do (but without the copy):
>
> x2 = x.copy()
> x[:,1:] = x2[:,:-1]

Aha, thank you. You have provided an example where numpy's behaviour
is undefined and subtle:

What you saw:

In [34]: x = np.random.rand(2,3)

In [35]: x[:,1:] = x[:,:-1]

In [36]: x
Out[36]:
array([[ 0.21559212,  0.21559212,  0.21559212],
       [ 0.02812742,  0.02812742,  0.02812742]])

In [37]: x = np.random.rand(3,2).T

If we transpose the array in memory it works:

In [38]: x[:,1:] = x[:,:-1]

In [39]: x
Out[39]:
array([[ 0.76877954,  0.76877954,  0.2793666 ],
       [ 0.09269335,  0.09269335,  0.52361756]])

As a workaround you can use backwards slices:

In [40]: x = np.random.rand(2,3)

In [41]: x[:,:0:-1] = x[:,-2::-1]

In [42]: x
Out[42]:
array([[ 0.20183084,  0.20183084,  0.08156887],
       [ 0.30611585,  0.30611585,  0.79001577]])


I can't explain exactly why this behaves this way, but the reason it
depends on memory layout is that when you write A[<some slice>] =
<some array>, numpy tries to reorder the loop to be as efficient as
possible. This depends on shape, size, and memory layout.
Unfortunately when the slices overlap, the different loop orderings
give different results.

I don't know what can be done about this, since detecting overlapping
slices is basically infeasible, and loop reordering could be an
extremely valuable optimization (and could therefore change to become
even more unpredictable in future, what with SIMD and cache issues).
The least painful option would be for numpy to offer the assurance
that it walks arrays in "virtual C order", that is, it walks the
indices from right to left. Then users who want efficient looping can
(try to) force their arrays to be laid out nicely in memory.
(Optimizing loops could be done when safe, that is, when the arrays
are known not to overlap.)

Less painful for numpy developers but more painful for users is to
warn them about the status quo: operations on overlapping slices can
happen in arbitrary order, depending on the memory layout of the input
arrays. This almost seems designed to make users tear their hair out
as their test cases work but real code fails. (I wonder how much of
numpy itself breaks on arrays with unusual memory layouts?)

I volunteer to write a quick test that will usually assert that arrays
cannot overlap (in fact I think I wrote one once). It won't detect all
cases where they don't overlap, but it should work on the majority.

Anne


From nwagner at iam.uni-stuttgart.de  Sat Jun  7 03:40:19 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sat, 07 Jun 2008 09:40:19 +0200
Subject: [Numpy-discussion] FAIL: Tests mask_rowcols
Message-ID: <web-102961265@uni-stuttgart.de>

Hi all,

I found one failure in numpy.test() with numpy 
1.2.0.dev5257

======================================================================
FAIL: Tests mask_rowcols.
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/numpy/ma/tests/test_extras.py", 
line 211, in check_mask_rowcols
     assert(mask_rowcols(x).all())
AssertionError

----------------------------------------------------------------------
Ran 1298 tests in 1.845s

FAILED (failures=1)
<unittest._TextTestResult run=1298 errors=0 failures=1>
>>> numpy.__version__
'1.2.0.dev5257'

  
Nils


From kwgoodman at gmail.com  Sat Jun  7 10:28:39 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Sat, 7 Jun 2008 07:28:39 -0700
Subject: [Numpy-discussion] Inplace shift
In-Reply-To: <ce557a360806062246q6ad98541i882ed4fe2b2f0269@mail.gmail.com>
References: <f4f93d420806061124s494eda58m405cbab2befc51d9@mail.gmail.com>
	<ce557a360806062246q6ad98541i882ed4fe2b2f0269@mail.gmail.com>
Message-ID: <f4f93d420806070728m5b8d701dj6be7e96d7e7cc1e8@mail.gmail.com>

On Fri, Jun 6, 2008 at 10:46 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> 2008/6/6 Keith Goodman <kwgoodman at gmail.com>:
>> I'd like to shift the columns of a 2d array one column to the right.
>> Is there a way to do that without making a copy?
>>
>> This doesn't work:
>>
>>>> import numpy as np
>>>> x = np.random.rand(2,3)
>>>> x[:,1:] = x[:,:-1]
>>>> x
>>
>> array([[ 0.44789223,  0.44789223,  0.44789223],
>>       [ 0.80600897,  0.80600897,  0.80600897]])
>
> As a workaround you can use backwards slices:
>worki
> In [40]: x = np.random.rand(2,3)
>
> In [41]: x[:,:0:-1] = x[:,-2::-1]
>
> In [42]: x
> Out[42]:
> array([[ 0.20183084,  0.20183084,  0.08156887],
>       [ 0.30611585,  0.30611585,  0.79001577]])

Neat. It makes sense to go backwards. Thank you.

> Less painful for numpy developers but more painful for users is to
> warn them about the status quo: operations on overlapping slices can
> happen in arbitrary order.

Now I'm confused. Could some corner case of memory layout cause numpy
to work from right to left, breaking the workaround? Or can I depend
on the workaround working with numpy 1.0.4?


From dyamins at gmail.com  Sat Jun  7 11:30:16 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Sat, 7 Jun 2008 11:30:16 -0400
Subject: [Numpy-discussion] Building 64-bit numpy on OSX
In-Reply-To: <4849C959.30700@googlemail.com>
References: <15e4667e0806060652n7bcaa2aes2c21e15f9a1273ae@mail.gmail.com>
	<48497D22.10705@googlemail.com>
	<15e4667e0806061450g6ca4154ep96c742c0b6ccea8c@mail.gmail.com>
	<e06186140806061628k4ab29e5fjccacab12a2866787@mail.gmail.com>
	<4849C959.30700@googlemail.com>
Message-ID: <15e4667e0806070830p11bfd8e1s575e27eb796204bb@mail.gmail.com>

.  What I think matters is "no matching architecture in

> > universal wrapper". Hmmm. I wonder if you and Michael have the same
> > versions of OS X?
>
>  > And why is dlopen looking for a universal library? One would hope that
> > distutils would have taken care of that.
>


I think you're right about the "no matching architecture" thing mattering
more than the permissions issue.  However, somewhere I read that using a
"sudo ... install" command could sometimes interfere with distutils.   Is
that plausible?


In any case, my overall situation is still that I cannot properly install
numpy on 64-bit python.   The immediate issue is that after running the
install command;
      sudo python setup.py install
in the numpy install directory, and then firing up a python 64-bit
interpreter, I cannot import numpy.   what happens is that it fails to be
able to import the multiarray.so module:

    >>>  import numpy
    .....

    ImportError:
dlopen(/usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so, 2):
no suitable image found.  Did find:
    /usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so: no
matching architecture in universal wrapper

I do not know what multiarray.so does.  However, I have tried the following
things:

      a) just importing multiarray.so from the 64-bit python interpreter
prompt.  this fails in the same way as the numpy import:

>>> import multiarray
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(./multiarray.so, 2): no suitable image found.  Did find:
    ./multiarray.so: no matching architecture in universal wrapper

     b) switching back to the 32-bit python install, and importing the numpy
version originall installed there.  This works fine.
     c) again back on the 32-bit python, importing multiarray.so -- the
version of it both in the 64-bit python file tree and the 32-bit python file
tree (the two versions of multiarray.so appear to be identical).  This also
works fine.

> Maybe Dan did overwrite some older 32 bit Python with the new 64 bit
build?

>
Since I can switch back on forth between the two builds, and the file
structures are all separate, I think this is probably not the problem,

> Out of curiosity, where are the 32/64 bit libraries normally put?

The 32-bit versin was built as a framework so its in:
/Library/Frameworks/Python.framework/Versions/Current/bin:${PATH}
The 64-bit version was not, and its in /usr/local/.


> What does
>
> file /usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so
>
> do?
>


This is telling.  I get:

   multiarray.so: Mach-O universal binary with 2 architectures
   multiarray.so (for architecture i386):    Mach-O bundle i386
   multiarray.so (for architecture ppc):    Mach-O bundle ppc

It seems to my like it's possible that somehow, I never ended up building
numpy as 64-bit in the first place.   But I'm basically stumped.


Thanks,
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080607/7f32ef91/attachment.html>

From charlesr.harris at gmail.com  Sat Jun  7 12:09:36 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 7 Jun 2008 10:09:36 -0600
Subject: [Numpy-discussion] Building 64-bit numpy on OSX
In-Reply-To: <15e4667e0806070830p11bfd8e1s575e27eb796204bb@mail.gmail.com>
References: <15e4667e0806060652n7bcaa2aes2c21e15f9a1273ae@mail.gmail.com>
	<48497D22.10705@googlemail.com>
	<15e4667e0806061450g6ca4154ep96c742c0b6ccea8c@mail.gmail.com>
	<e06186140806061628k4ab29e5fjccacab12a2866787@mail.gmail.com>
	<4849C959.30700@googlemail.com>
	<15e4667e0806070830p11bfd8e1s575e27eb796204bb@mail.gmail.com>
Message-ID: <e06186140806070909q7a9771f1pb831c2e6b8ecc80d@mail.gmail.com>

On Sat, Jun 7, 2008 at 9:30 AM, Dan Yamins <dyamins at gmail.com> wrote:

>
> .  What I think matters is "no matching architecture in
>
>>  > universal wrapper". Hmmm. I wonder if you and Michael have the same
>> > versions of OS X?
>>
>>  > And why is dlopen looking for a universal library? One would hope that
>> > distutils would have taken care of that.
>>
>
>
> I think you're right about the "no matching architecture" thing mattering
> more than the permissions issue.  However, somewhere I read that using a
> "sudo ... install" command could sometimes interfere with distutils.   Is
> that plausible?
>
> No, I sudo to install all the time. Usually I do python setup.py build
first so I can easily remove the build directory.

<snip>


>
>
> > What does
> >
> > file /usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so
> >
> > do?
> >
>
>
> This is telling.  I get:
>
>    multiarray.so: Mach-O universal binary with 2 architectures
>    multiarray.so (for architecture i386):    Mach-O bundle i386
>    multiarray.so (for architecture ppc):    Mach-O bundle ppc
>
> It seems to my like it's possible that somehow, I never ended up building
> numpy as 64-bit in the first place.   But I'm basically stumped.
>
>
A good thing to try is deleting both the build directory and the numpy
folder in site-packages just to make sure you are starting with a clean
slate. Then try the install again. You should also run which python just to
make sure you are using the right python, not that that seems to be the case
here. Also, put the build output into a log file like so: $ python setup.py
build &> log.txt. Then zip and attach the log to your post so we can get a
better idea what is going on.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080607/085dc2b3/attachment.html>

From alan.mcintyre at gmail.com  Sat Jun  7 13:02:58 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Sat, 7 Jun 2008 13:02:58 -0400
Subject: [Numpy-discussion] SCons install fails with rev 5258
Message-ID: <1d36917a0806071002l234fb6f9if028f1391bdd43f4@mail.gmail.com>

I just updated my NumPy tree from svn trunk, and the SCons install now
fails with the following message:

ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python intepreter from there.:
  File "/home/alan/numpy/numpy/linalg/SConstruct", line 18:
    mlib = scons_get_mathlib(env)
  File "/usr/local/lib/python2.5/site-packages/numscons-0.7.1-py2.5.egg/numscons/__init__.py",
line 37:
    from numpy.distutils.misc_util import get_mathlibs
  File "/home/alan/numpy/numpy/__init__.py", line 84:
    raise ImportError(msg)
status is 512
error: Error while executing scons command /usr/local/bin/python2.5
"/usr/local/lib/python2.5/site-packages/numscons-0.7.1-py2.5.egg/numscons/scons-local/scons.py"
-f numpy/linalg/SConstruct -I. scons_tool_path=""
src_dir="numpy/linalg" pkg_name="numpy.linalg"
distutils_libdir="build/lib.linux-i686-2.5" cc_opt=gcc
cc_opt_path="/usr/bin" f77_opt=gfortran f77_opt_path="/usr/bin"
cxx_opt=g++ cxx_opt_path="/usr/bin"
include_bootstrap=numpy/core/include:build/scons/numpy/core silent=0
bootstrapping=1 (see above)


From lists at informa.tiker.net  Sat Jun  7 14:11:34 2008
From: lists at informa.tiker.net (Andreas =?utf-8?q?Kl=C3=B6ckner?=)
Date: Sat, 7 Jun 2008 14:11:34 -0400
Subject: [Numpy-discussion] embedded arrays
In-Reply-To: <A7E1AE993C63414B8B2F762E13BF4D898E4EA1@MAIL.burnham.org>
References: <A7E1AE993C63414B8B2F762E13BF4D898E4E9F@MAIL.burnham.org>
	<A7E1AE993C63414B8B2F762E13BF4D898E4EA1@MAIL.burnham.org>
Message-ID: <200806071411.35390.lists@informa.tiker.net>

On Freitag 06 Juni 2008, Thomas Hrabe wrote:
> Furthermore, I sometimes get a
> Segmentation fault
> Illegal instruction
>
> and sometimes it works
>
> It might be a memory leak, due to the segfault and the arbitrary behavior.?

Shameless plug: PyUblas [1] will take care of the nasty bits of wrapping C++ 
code for numpy. Including getting the refcounting right. :)

Andreas

[1] http://tiker.net/doc/pyublas/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080607/12e432fa/attachment.sig>

From ondrej at certik.cz  Sat Jun  7 15:37:02 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Sat, 7 Jun 2008 21:37:02 +0200
Subject: [Numpy-discussion] array vs matrix
Message-ID: <85b5c3130806071237g16582e6cr2a122ce8c38886@mail.gmail.com>

Hi,

what is the current plan with array and matrix with regard of calculating

sin(A)

? I.e. elementwise vs sin(A) = Q*sin(D)*Q^T? Is the current approach
(elementwise for array and Q*sin(D)*Q^T for matrix) the way to go?

We are solving the same problem in SymPy:

http://groups.google.com/group/sympy/browse_thread/thread/6cfe66b012b24948

so I'd like to have the exact same behavior as numpy has, so that we
are compatible. But I was wondering if the current status is ok, or if
many numpy developers (or users?) would like this to be changed.

I think a clean solution is to treat matrices as mathematical objects,
i.e. A*B and sin(A) being matrix multiplication and matrix sin
respectively, while leaving the array for elementwise operations.

But above all, I'd like to be compatible with numpy, otherwise it will
be a mess.

Thanks for any feedback,
Ondrej


From robert.kern at gmail.com  Sat Jun  7 15:44:13 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 7 Jun 2008 14:44:13 -0500
Subject: [Numpy-discussion] array vs matrix
In-Reply-To: <85b5c3130806071237g16582e6cr2a122ce8c38886@mail.gmail.com>
References: <85b5c3130806071237g16582e6cr2a122ce8c38886@mail.gmail.com>
Message-ID: <3d375d730806071244o7d19bbc4yf7e8f2b1477e7e6b@mail.gmail.com>

On Sat, Jun 7, 2008 at 14:37, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> what is the current plan with array and matrix with regard of calculating
>
> sin(A)
>
> ? I.e. elementwise vs sin(A) = Q*sin(D)*Q^T? Is the current approach
> (elementwise for array and Q*sin(D)*Q^T for matrix) the way to go?

I don't believe we intend to make numpy.matrix any more featureful. I
don't think it's a good idea for you to base sympy.Matrix's
capabilities in lock-step with numpy.matrix, though. There are very
different constraints at work. Please, do what you think is best for
sympy.Matrix.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From pgmdevlist at gmail.com  Sat Jun  7 17:46:56 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sat, 7 Jun 2008 17:46:56 -0400
Subject: [Numpy-discussion] FAIL: Tests mask_rowcols
In-Reply-To: <web-102961265@uni-stuttgart.de>
References: <web-102961265@uni-stuttgart.de>
Message-ID: <200806071746.57181.pgmdevlist@gmail.com>

Folks,
Can anybody confirm that ? The tests look like they pass OK on my machine...

On Saturday 07 June 2008 03:40:19 Nils Wagner wrote:
> Hi all,
>
> I found one failure in numpy.test() with numpy
> 1.2.0.dev5257
>
> ======================================================================
> FAIL: Tests mask_rowcols.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File
> "/usr/local/lib64/python2.5/site-packages/numpy/ma/tests/test_extras.py",
> line 211, in check_mask_rowcols
>      assert(mask_rowcols(x).all())
> AssertionError


From charlesr.harris at gmail.com  Sat Jun  7 18:15:45 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 7 Jun 2008 16:15:45 -0600
Subject: [Numpy-discussion] FAIL: Tests mask_rowcols
In-Reply-To: <200806071746.57181.pgmdevlist@gmail.com>
References: <web-102961265@uni-stuttgart.de>
	<200806071746.57181.pgmdevlist@gmail.com>
Message-ID: <e06186140806071515j3a4d55d0ja3cd596cecb2fc07@mail.gmail.com>

On Sat, Jun 7, 2008 at 3:46 PM, Pierre GM <pgmdevlist at gmail.com> wrote:

> Folks,
> Can anybody confirm that ? The tests look like they pass OK on my
> machine...


Yep,

======================================================================
FAIL: Tests mask_rowcols.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/numpy/ma/tests/test_extras.py",
line 211, in check_mask_rowcols
    assert(mask_rowcols(x).all())
AssertionError

----------------------------------------------------------------------
Ran 1371 tests in 1.750s

FAILED (failures=1)
Out[1]: <unittest._TextTestResult run=1371 errors=0 failures=1>

This is on 32bit f8.

Compilation also gives a new warning:

gcc: numpy/core/src/multiarraymodule.c
numpy/core/src/multiarraymodule.c: In function '_zerofill':
numpy/core/src/arrayobject.c:6661: warning: control reaches end of non-void
function

But I don't think that is  yours ;)

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080607/070fd455/attachment.html>

From zachary.pincus at yale.edu  Sat Jun  7 18:55:05 2008
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Sat, 7 Jun 2008 18:55:05 -0400
Subject: [Numpy-discussion] Getting specific Win32 binaries? (was: numpy,
	py2exe, and SSE)
In-Reply-To: <3B802A64-5EDB-4BFA-AE75-9B5FD1928AA9@yale.edu>
References: <3B802A64-5EDB-4BFA-AE75-9B5FD1928AA9@yale.edu>
Message-ID: <DC773799-D113-4A14-B3AC-42707F6C2A2C@yale.edu>

Hello all,

I'd like to be able to grab specific Win32 binaries from the  
distribution -- e.g. the SSE, SSE2. or SSE3-specific files (so that I  
can make sure to distribute the lowest-common-denominator ones via  
py2exe). Is this possible with the current binary releases? Or ought I  
to just build my own as required?

Thanks,
Zach



On Jun 4, 2008, at 1:37 PM, Zachary Pincus wrote:

> Hello all,
>
> I've been toying around with bundling up a numpy-using python program
> for windows by using py2exe. All in all, it works great, except for
> one thing: the numpy superpack installer for windows has (correctly)
> selected SSE3 binary libraries to install on my machine. This causes
> (of course) crashes when the binary py2exe bundle, which includes the
> numpy libraries installed on my machine, is run on a non-SSE3 machine.
>
> So, two questions:
>  1) What's the best target for "safe" binaries on i386 machines? No
> SSE at all? SSE? SSE2?
>  2) What's the best way to get the libraries desired installed for
> numpy? E.g. compile  if myself with some --no-sse option? (And if so,
> any clue as to the required options.) Or is there some way to get the
> numpy windows installer to install a specific kind of binary?
>
> Thanks,
> Zach Pincus
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From peridot.faceted at gmail.com  Sat Jun  7 21:48:17 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 7 Jun 2008 21:48:17 -0400
Subject: [Numpy-discussion] Inplace shift
In-Reply-To: <f4f93d420806070728m5b8d701dj6be7e96d7e7cc1e8@mail.gmail.com>
References: <f4f93d420806061124s494eda58m405cbab2befc51d9@mail.gmail.com>
	<ce557a360806062246q6ad98541i882ed4fe2b2f0269@mail.gmail.com>
	<f4f93d420806070728m5b8d701dj6be7e96d7e7cc1e8@mail.gmail.com>
Message-ID: <ce557a360806071848m59f56989yacc37f3549d1bd0c@mail.gmail.com>

2008/6/7 Keith Goodman <kwgoodman at gmail.com>:
> On Fri, Jun 6, 2008 at 10:46 PM, Anne Archibald
> <peridot.faceted at gmail.com> wrote:
>> 2008/6/6 Keith Goodman <kwgoodman at gmail.com>:
>>> I'd like to shift the columns of a 2d array one column to the right.
>>> Is there a way to do that without making a copy?
>>>
>>> This doesn't work:
>>>
>>>>> import numpy as np
>>>>> x = np.random.rand(2,3)
>>>>> x[:,1:] = x[:,:-1]
>>>>> x
>>>
>>> array([[ 0.44789223,  0.44789223,  0.44789223],
>>>       [ 0.80600897,  0.80600897,  0.80600897]])
>>
>> As a workaround you can use backwards slices:
>>worki
>> In [40]: x = np.random.rand(2,3)
>>
>> In [41]: x[:,:0:-1] = x[:,-2::-1]
>>
>> In [42]: x
>> Out[42]:
>> array([[ 0.20183084,  0.20183084,  0.08156887],
>>       [ 0.30611585,  0.30611585,  0.79001577]])
>
> Neat. It makes sense to go backwards. Thank you.
>
>> Less painful for numpy developers but more painful for users is to
>> warn them about the status quo: operations on overlapping slices can
>> happen in arbitrary order.
>
> Now I'm confused. Could some corner case of memory layout cause numpy
> to work from right to left, breaking the workaround? Or can I depend
> on the workaround working with numpy 1.0.4?

I'm afraid so. And it's not such a corner case as that: if the array
is laid out in "C contiguous" order, you have to go backwards, while
if the array is laid out in "FORTRAN contiguous" order you have to go
forwards.

Anne


From peridot.faceted at gmail.com  Sat Jun  7 21:54:44 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 7 Jun 2008 21:54:44 -0400
Subject: [Numpy-discussion] array vs matrix
In-Reply-To: <3d375d730806071244o7d19bbc4yf7e8f2b1477e7e6b@mail.gmail.com>
References: <85b5c3130806071237g16582e6cr2a122ce8c38886@mail.gmail.com>
	<3d375d730806071244o7d19bbc4yf7e8f2b1477e7e6b@mail.gmail.com>
Message-ID: <ce557a360806071854m8767bfbkd27afe2128ef52f9@mail.gmail.com>

2008/6/7 Robert Kern <robert.kern at gmail.com>:
> On Sat, Jun 7, 2008 at 14:37, Ondrej Certik <ondrej at certik.cz> wrote:
>> Hi,
>>
>> what is the current plan with array and matrix with regard of calculating
>>
>> sin(A)
>>
>> ? I.e. elementwise vs sin(A) = Q*sin(D)*Q^T? Is the current approach
>> (elementwise for array and Q*sin(D)*Q^T for matrix) the way to go?
>
> I don't believe we intend to make numpy.matrix any more featureful. I
> don't think it's a good idea for you to base sympy.Matrix's
> capabilities in lock-step with numpy.matrix, though. There are very
> different constraints at work. Please, do what you think is best for
> sympy.Matrix.

Let me elaborate somewhat:

We recently ran across some painful quirks in numpy's handling of
matrices, and they spawned massive discussion. As it stands now, there
is significant interest in reimplementing the matrix object from
scratch, with different behaviour. So emulating its current behaviour
is not a win.

For consistency, it makes a certain amount of sense to have sin(A)
compute a matrix sine, since A**n computes a matrix power. But looking
at the matrix exponential, I see that we have several implementations,
none of which is satisfactory for all matrices. I would expect the
matrix sine to be similar - particularly when faced with complex
matrices - so perhaps needing an explicit matrix sine is a good thing?

Also worth noting is that this problem can be evaded with namespaces;
matrix sin could be scipy.matrixmath.sin, abbreviated perhaps to
mm.sin, as opposed to np.sin.

Anne


From robert.kern at gmail.com  Sat Jun  7 21:57:50 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 7 Jun 2008 20:57:50 -0500
Subject: [Numpy-discussion] array vs matrix
In-Reply-To: <ce557a360806071854m8767bfbkd27afe2128ef52f9@mail.gmail.com>
References: <85b5c3130806071237g16582e6cr2a122ce8c38886@mail.gmail.com>
	<3d375d730806071244o7d19bbc4yf7e8f2b1477e7e6b@mail.gmail.com>
	<ce557a360806071854m8767bfbkd27afe2128ef52f9@mail.gmail.com>
Message-ID: <3d375d730806071857g71feea4cp90d1a91bb3bb05a8@mail.gmail.com>

On Sat, Jun 7, 2008 at 20:54, Anne Archibald <peridot.faceted at gmail.com> wrote:

> For consistency, it makes a certain amount of sense to have sin(A)
> compute a matrix sine, since A**n computes a matrix power. But looking
> at the matrix exponential, I see that we have several implementations,
> none of which is satisfactory for all matrices. I would expect the
> matrix sine to be similar - particularly when faced with complex
> matrices - so perhaps needing an explicit matrix sine is a good thing?

I think so. We already have scipy.linalg.sinm(), cosm(), expm(), expm2(), etc.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Sun Jun  8 02:11:14 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 08 Jun 2008 15:11:14 +0900
Subject: [Numpy-discussion] SCons install fails with rev 5258
In-Reply-To: <1d36917a0806071002l234fb6f9if028f1391bdd43f4@mail.gmail.com>
References: <1d36917a0806071002l234fb6f9if028f1391bdd43f4@mail.gmail.com>
Message-ID: <484B7802.6060104@ar.media.kyoto-u.ac.jp>

Alan McIntyre wrote:
> I just updated my NumPy tree from svn trunk, and the SCons install now
> fails with the following message:
>   

Updating to 0.7.2 should fix the problem (this is due to a change on how 
I handle bootstrapping numpy build: when numpy changed its bootstrapping 
implementation, I quickly adapt scons build in a way which was pretty 
stupid, and I fixed this in numpy trunk and numscons recently).

cheers,

David


From alan.mcintyre at gmail.com  Sun Jun  8 05:50:58 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Sun, 8 Jun 2008 05:50:58 -0400
Subject: [Numpy-discussion] SCons install fails with rev 5258
In-Reply-To: <484B7802.6060104@ar.media.kyoto-u.ac.jp>
References: <1d36917a0806071002l234fb6f9if028f1391bdd43f4@mail.gmail.com>
	<484B7802.6060104@ar.media.kyoto-u.ac.jp>
Message-ID: <1d36917a0806080250k41cbafd3wcffe27907ef879e9@mail.gmail.com>

That fixes it, thanks! Apparently I forgot the -U when I tried to
update numscons via easy_install yesterday. ;)

On Sun, Jun 8, 2008 at 2:11 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Alan McIntyre wrote:
>> I just updated my NumPy tree from svn trunk, and the SCons install now
>> fails with the following message:
>>
>
> Updating to 0.7.2 should fix the problem (this is due to a change on how
> I handle bootstrapping numpy build: when numpy changed its bootstrapping
> implementation, I quickly adapt scons build in a way which was pretty
> stupid, and I fixed this in numpy trunk and numscons recently).
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From alan.mcintyre at gmail.com  Sun Jun  8 07:15:40 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Sun, 8 Jun 2008 07:15:40 -0400
Subject: [Numpy-discussion] Switching to nose test framework (was: NumpyTest
	problem)
Message-ID: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>

Hi all,

Later this week (after finals are done ;) I'll be checking in some
changes to switch NumPy tests over to nose, and I just wanted to give
the list a heads up and see if there's any concerns, etc.

On Mon, Jun 2, 2008 at 11:04 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
> * is there a way to select only some tests ? It used to be a time not that
> long ago (less than 6 weeks) where I was able to run one specific test of a
> suite (for example, numpy/ma/tests/test_core.py), and that's no longer the
> case.

As Christopher pointed out, you should be able to use the nosetests
script to run tests that way if you rename your methods.  Once I check
in my changes, all the test modules will have the test and utility
function/method names updated so that they'll work with the nose
scheme.

Right now there's "if __name__ == '__main__'" boilerplate at the end
of every test module; nose can find and run tests without that being
there.  I wanted to get the list's take on removing that--it just
means that if you want to run the tests in one module, you have to do
"nosetests numpy/blah/test_foo.py" instead of "python
numpy/blah/test_foo.py".  Is there a reason not to remove that
boilerplate since we've already decided to use nose?

> * Running the whole test suite with import numpy; numpy.test() works fine, but
> I don't really need to test the rest of numpy when I'm messing with
> numpy.ma...

I will be including a "test" method in as many subpackages as
possible, so that you can do numpy.ma.test() and such.

If anybody has any concerns or requests for the test suite, please let
me know and I'll see what I can do.

Cheers,
Alan


From pav at iki.fi  Sun Jun  8 08:19:56 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 08 Jun 2008 15:19:56 +0300
Subject: [Numpy-discussion] (SPAM)  umath ufunc docstrings
In-Reply-To: <1212263377.8410.18.camel@localhost.localdomain>
References: <1212263377.8410.18.camel@localhost.localdomain>
Message-ID: <1212927597.8203.2.camel@localhost.localdomain>

la, 2008-05-31 kello 22:49 +0300, Pauli Virtanen kirjoitti:
> Hi,
> 
> I'd like to adjust the way numpy.core.umath ufunc docstrings are defined
> to make them more easy to handle in the ongoing documentation marathon:
> 
> - Remove the signature magic in ufunc_get_doc
> - Define ufunc docstrings in a separate module
>   instead of in generate_umath.py, in the same format as in 
>   add_newdocs.py
> 
> Suggested patch is attached; it passes numpy tests. Any thoughts on
> whether it should go in?
[clip]

I moved the docstrings (r5262), but left the automatic ufunc signatures
in place.

	Pauli

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digitaalisesti allekirjoitettu viestin osa
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080608/6c55ac95/attachment.sig>

From vinjvinj at gmail.com  Sun Jun  8 10:02:36 2008
From: vinjvinj at gmail.com (Vineet Jain (gmail))
Date: Sun, 8 Jun 2008 10:02:36 -0400
Subject: [Numpy-discussion] Is there a function to calculate
	ecnomic	beta coefficient in numpy given two time series data.
References: <000c01c8c6a4$9f0fad00$dd2f0700$@com>
	<f4f93d420806041804n5c0302b3le70c093673140a79@mail.gmail.com> 
Message-ID: <000f01c8c970$50245e60$f06d1b20$@com>

Currently my code handles market returns and stocks as 1d arrays. While the
function below expects a matrix. Is there an equivalent of the function
below which works with numpy arrays?

I'd like to do:

beta, resids, rank, s = mp.linalg.lstsq(mrkt_1d_array, stock_1d_array)

If not, how do:

1. create an empty matrix object and then keep adding arrays objects to it
in a loop?

Thanks,

Vineet

>> import numpy.matlib as mp
>> mrkt = mp.randn(250,1)  # <----- 250 days of returns
>> stocks = mp.randn(250, 4)  # <---- 4 stocks
>> beta, resids, rank, s = mp.linalg.lstsq(mrkt, stocks)
>> matrix([[-0.01701467,  0.11242168,  0.00207398,  0.03920687]])





From kwgoodman at gmail.com  Sun Jun  8 10:57:27 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Sun, 8 Jun 2008 07:57:27 -0700
Subject: [Numpy-discussion] Is there a function to calculate ecnomic
	beta coefficient in numpy given two time series data.
In-Reply-To: <000f01c8c970$50245e60$f06d1b20$@com>
References: <000c01c8c6a4$9f0fad00$dd2f0700$@com>
	<f4f93d420806041804n5c0302b3le70c093673140a79@mail.gmail.com>
	<000f01c8c970$50245e60$f06d1b20$@com>
Message-ID: <f4f93d420806080757m6c101462q934c834a736ddb22@mail.gmail.com>

On Sun, Jun 8, 2008 at 7:02 AM, Vineet Jain (gmail) <vinjvinj at gmail.com> wrote:
> Currently my code handles market returns and stocks as 1d arrays. While the
> function below expects a matrix. Is there an equivalent of the function
> below which works with numpy arrays?
>
> I'd like to do:
>
> beta, resids, rank, s = mp.linalg.lstsq(mrkt_1d_array, stock_1d_array)

lstsq works with arrays if the dimensions are right. Here's one way to do it:

>> import numpy as np
>> mrkt = np.random.randn(250,1)  # <----- 250 days of returns
>> stocks = np.random.randn(250, 4)  # <---- 4 stocks
>> beta, resids, rank, s = np.linalg.lstsq(np.atleast_2d(mrkt), stocks)
>> beta
   array([[-0.02000282,  0.11898366,  0.01010665, -0.03257696]])

Let's make sure the results are the same as the matrix case:

>> mrkt = np.asmatrix(mrkt)
>> mrkt.shape
   (250, 1)
>> stocks = np.asmatrix(stocks)
>> stocks.shape
   (250, 4)
>> beta, resids, rank, s = np.linalg.lstsq(mrkt, stocks)
>> beta
   matrix([[-0.02000282,  0.11898366,  0.01010665, -0.03257696]])  #
<--- Yep, they are the same

There is also np.clip if you want to clip returns. And np.linalg.eigh
if you want to estimate the betas from an estimate of the covariance
matrix, where you throw out the eigenvectors with small eigenvalues.


From pgmdevlist at gmail.com  Sun Jun  8 14:31:57 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 8 Jun 2008 14:31:57 -0400
Subject: [Numpy-discussion] Ticket #812 - numpy.sum, numpy.std,
	numpy.var fail on masked arrays
Message-ID: <200806081431.57529.pgmdevlist@gmail.com>

Folks,
I recently revamped some of the functions of numpy.ma to handle explicit 
output parameters (eg, sum, std, var...). As a consequence, ticket #812 
should be solved. Could anybody give a recent SVN (>=5259) a try and let me 
know if I can close the ticket ?
Thanks a lot in advance.
P.


From pgmdevlist at gmail.com  Sun Jun  8 14:35:09 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 8 Jun 2008 14:35:09 -0400
Subject: [Numpy-discussion] Keywords in wrapped functions
Message-ID: <200806081435.09817.pgmdevlist@gmail.com>

All,

is there a reason why in some functions (min, max...) optional parameters are 
parsed by position instead of by keyword ?
OK, let me give you an example:
#-----------------------------------------------
def amin(a, axis=None, out=None):
    try:
        amin = a.min
    except AttributeError:
        return _wrapit(a, 'min', axis, out)
    return amin(axis, out)
#-----------------------------------------------

* Why using amin(axis,out) instead of amin(axis=axis,out=out) ?

* Would it be possible to add some optional keywords to amin/amax, so that 
a "fill_value" parameter would be recognized by numpy.min when dealing with 
masked arrays ? A modified amin would then be like:

def amin(a, axis=None, out=None, **kwargs):
    try:
        amin = a.min
    except AttributeError:
        return _wrapit(a, 'min', axis=axis, out=out, **kwargs)
    return amin(axis=axis, out=out, **kwargs)

Am I missing something ?


From wnbell at gmail.com  Sun Jun  8 18:49:59 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Sun, 8 Jun 2008 17:49:59 -0500
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
Message-ID: <d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>

On Sun, Jun 8, 2008 at 6:15 AM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> Right now there's "if __name__ == '__main__'" boilerplate at the end
> of every test module; nose can find and run tests without that being
> there.  I wanted to get the list's take on removing that--it just
> means that if you want to run the tests in one module, you have to do
> "nosetests numpy/blah/test_foo.py" instead of "python
> numpy/blah/test_foo.py".  Is there a reason not to remove that
> boilerplate since we've already decided to use nose?
>

When making frequent changes to test_foo.py, it's often nice to run
test_foo.py directly, rather than installing the whole package and
then testing via nose.

I would leave the decision up to the maintainers of the individual
submodules.  Personally, I find it useful to have.

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From alan.mcintyre at gmail.com  Sun Jun  8 19:18:07 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Sun, 8 Jun 2008 19:18:07 -0400
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>
Message-ID: <1d36917a0806081618g5875fa3bt1bfe5f379e9a797a@mail.gmail.com>

On Sun, Jun 8, 2008 at 6:49 PM, Nathan Bell <wnbell at gmail.com> wrote:
> On Sun, Jun 8, 2008 at 6:15 AM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
>> Right now there's "if __name__ == '__main__'" boilerplate at the end
>> of every test module; nose can find and run tests without that being
>> there.  I wanted to get the list's take on removing that--it just
>> means that if you want to run the tests in one module, you have to do
>> "nosetests numpy/blah/test_foo.py" instead of "python
>> numpy/blah/test_foo.py".  Is there a reason not to remove that
>> boilerplate since we've already decided to use nose?
>>
>
> When making frequent changes to test_foo.py, it's often nice to run
> test_foo.py directly, rather than installing the whole package and
> then testing via nose.
>
> I would leave the decision up to the maintainers of the individual
> submodules.  Personally, I find it useful to have.

You can run the test scripts from the source tree without installing
them by using nosetests, like this:

nosetests nump/numpy/ilb/tests/test_getlimits.py
or
nosetests nump/numpy/ilb/tests


From pgmdevlist at gmail.com  Sun Jun  8 19:17:38 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 8 Jun 2008 19:17:38 -0400
Subject: [Numpy-discussion] FAIL: Tests mask_rowcols
In-Reply-To: <e06186140806071515j3a4d55d0ja3cd596cecb2fc07@mail.gmail.com>
References: <web-102961265@uni-stuttgart.de>
	<200806071746.57181.pgmdevlist@gmail.com>
	<e06186140806071515j3a4d55d0ja3cd596cecb2fc07@mail.gmail.com>
Message-ID: <200806081917.39052.pgmdevlist@gmail.com>

On Saturday 07 June 2008 18:15:45 Charles R Harris wrote:
> On Sat, Jun 7, 2008 at 3:46 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
> > Folks,
> > Can anybody confirm that ? The tests look like they pass OK on my
> > machine...
>
> Yep,
>
> ======================================================================
> FAIL: Tests mask_rowcols.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/numpy/ma/tests/test_extras.py",
> line 211, in check_mask_rowcols
>     assert(mask_rowcols(x).all())
> AssertionError

OK, my bad, I guess I still have some nose problems.
Should be fixed in SVN5264.





From gael.varoquaux at normalesup.org  Mon Jun  9 02:25:17 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 9 Jun 2008 08:25:17 +0200
Subject: [Numpy-discussion] Installation info
In-Reply-To: <5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>
	<6be8b94a0805300431p14eb1ae0pc5fd111db9b0503@mail.gmail.com>
	<789d27b10805300435t361d12a8j59f69aadefd3f9e6@mail.gmail.com>
	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>
Message-ID: <20080609062517.GC24990@phare.normalesup.org>

On Sat, May 31, 2008 at 12:01:10PM +0900, David Cournapeau wrote:
> On Fri, May 30, 2008 at 8:35 PM, Hanni Ali <hanni.ali at gmail.com> wrote:
> > I would also like to see a 64bit build for windows as well if possible.



> Unfortunately, this is difficult: windows 64 is not commonly available
> (I don't have any access to it personally, for example), and mingw is
> not available yet for windows 64 either.

I was just wondering, could we ask Microsoft for some help here. A build
bot, or a Windows 64 license... They are helping porting the SAGE project
to windows, so they do have interest in getting open source scientific
software working on windows.

My 2 cents,

Ga?l


From david at ar.media.kyoto-u.ac.jp  Mon Jun  9 02:25:03 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 09 Jun 2008 15:25:03 +0900
Subject: [Numpy-discussion] Installation info
In-Reply-To: <20080609062517.GC24990@phare.normalesup.org>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>	<6be8b94a0805300431p14eb1ae0pc5fd111db9b0503@mail.gmail.com>	<789d27b10805300435t361d12a8j59f69aadefd3f9e6@mail.gmail.com>	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>
	<20080609062517.GC24990@phare.normalesup.org>
Message-ID: <484CCCBF.5010105@ar.media.kyoto-u.ac.jp>

Gael Varoquaux wrote:
>
> I was just wondering, could we ask Microsoft for some help here. A build
> bot, or a Windows 64 license... They are helping porting the SAGE project
> to windows, so they do have interest in getting open source scientific
> software working on windows.
>   

Windows 64 is actually the easiest problem: you can find free trial 
license for Windows server 2008 on MS website.

The main problem is the compiler and BLAS/LAPACK: mingw cannot target 64 
bits yet (there is a mingw-w64 project, but after some digging, I found 
out that it was legally dubious, hence not integrated yet in the 'real' 
mingw project). Now, you could think about using MS compiler, but I 
don't like the idea very much, and there is the problem of blas/lapack. 
cygwin, already slow, is dog slow on windows 64 (because it runs in 32 
bits, I guess ?), and is needed to build atlas (blas and lapack too, but 
it is at least theoretically possible to build them without makefiles; 
atlas build system is so complicated that it makes autotools feel 
extremely enjoyable).

cheers,

David


From hanni.ali at gmail.com  Mon Jun  9 03:23:07 2008
From: hanni.ali at gmail.com (Hanni Ali)
Date: Mon, 9 Jun 2008 08:23:07 +0100
Subject: [Numpy-discussion] Installation info
In-Reply-To: <484CCCBF.5010105@ar.media.kyoto-u.ac.jp>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>
	<6be8b94a0805300431p14eb1ae0pc5fd111db9b0503@mail.gmail.com>
	<789d27b10805300435t361d12a8j59f69aadefd3f9e6@mail.gmail.com>
	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>
	<20080609062517.GC24990@phare.normalesup.org>
	<484CCCBF.5010105@ar.media.kyoto-u.ac.jp>
Message-ID: <789d27b10806090023i727586betba0603f06b80be9@mail.gmail.com>

I have used the trail of Visual Studio 2008 (on full Server 2003) it
copmiles and runs well if you comment out the lines meantioned earlier on in
this thread.

I am planning to spend some time with the intel fortran compiler to try to
compile blas/lapack in the next few weeks. It is part of my current project,
so I will report back, Intel are usually very friendly to open source and if
that means you offer intel optimised binaries they should hopefully be
willing.

Hanni

(P.S. I sent an output from the compile in an earlier thread, but I think it
was filtered I will re-post it and try to compress it a bit)



2008/6/9 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:

> Gael Varoquaux wrote:
> >
> > I was just wondering, could we ask Microsoft for some help here. A build
> > bot, or a Windows 64 license... They are helping porting the SAGE project
> > to windows, so they do have interest in getting open source scientific
> > software working on windows.
> >
>
> Windows 64 is actually the easiest problem: you can find free trial
> license for Windows server 2008 on MS website.
>
> The main problem is the compiler and BLAS/LAPACK: mingw cannot target 64
> bits yet (there is a mingw-w64 project, but after some digging, I found
> out that it was legally dubious, hence not integrated yet in the 'real'
> mingw project). Now, you could think about using MS compiler, but I
> don't like the idea very much, and there is the problem of blas/lapack.
> cygwin, already slow, is dog slow on windows 64 (because it runs in 32
> bits, I guess ?), and is needed to build atlas (blas and lapack too, but
> it is at least theoretically possible to build them without makefiles;
> atlas build system is so complicated that it makes autotools feel
> extremely enjoyable).
>
> cheers,
>
> David
>  _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080609/0dc608f6/attachment.html>

From hanni.ali at gmail.com  Mon Jun  9 03:24:23 2008
From: hanni.ali at gmail.com (Hanni Ali)
Date: Mon, 9 Jun 2008 08:24:23 +0100
Subject: [Numpy-discussion] Installation info
In-Reply-To: <3d375d730806031148s39f59d4dl9b7bb4b6ce2f611a@mail.gmail.com>
References: <6be8b94a0805300429y46ab7da4ga5adfc6605c376e8@mail.gmail.com>
	<789d27b10805300435t361d12a8j59f69aadefd3f9e6@mail.gmail.com>
	<5b8d13220805302001p65e05d4ep4162aca1947c4f80@mail.gmail.com>
	<789d27b10806020141m43d8d699u8d9dbe8d593d8e3e@mail.gmail.com>
	<4843B4AF.4070605@ar.media.kyoto-u.ac.jp>
	<789d27b10806020230l1d88f270l52a62ec90954aeb7@mail.gmail.com>
	<4843BDD2.8060002@ar.media.kyoto-u.ac.jp>
	<789d27b10806020526h47d08418v4f0e0752bbcabb1a@mail.gmail.com>
	<789d27b10806030121s3a3df8a0q3a7bbdd173515ba7@mail.gmail.com>
	<3d375d730806031148s39f59d4dl9b7bb4b6ce2f611a@mail.gmail.com>
Message-ID: <789d27b10806090024g398366c4r7a2fe1d6afb01200@mail.gmail.com>

Hi Robert,

Attached is the complete output, below is what I believe is the relevant
area of interest:

compiling C sources
creating build\temp.win-amd64-2.6\Release\build
creating build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6
creating build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6\numpy
creating build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6\numpy\core
creating
build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6\numpy\core\src
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c
/nologo /Ox /MD /W3 /GS- /DNDEBUG -Ibuild\src.win-amd64-2.6\numpy\core\src
-Inumpy\core\include -Ibuild\src.win-amd64-2.6\numpy\core -Inumpy\core\src
-Inumpy\core\include -IC:\Python26\include -IC:\Python26\PC
-IC:\Python26\include -IC:\Python26\PC
/Tcbuild\src.win-amd64-2.6\numpy\core\src\umathmodule.c
/Fobuild\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6\numpy\core\src\umathmodule.obj
umathmodule.c

numpy\core\src\umathmodule.c.src(64) : error C2059: syntax error : 'type'
numpy\core\src\umathmodule.c.src(70) : error C2059: syntax error : 'type'

numpy\core\src\ufuncobject.c(1645) : warning C4244: '=' : conversion from
'npy_intp' to 'int', possible loss of data
numpy\core\src\ufuncobject.c(2346) : warning C4244: '=' : conversion from
'npy_intp' to 'int', possible loss of data

What I find odd is that the instruction:

> #ifndef HAVE_FREXPF

should not produce the error (at least that's what I would have thought)
i.e. if it isn't recognised the compiler should simply be using the
alternate definition of the frexpf function defined, although I suspect it
is a little less efficient than the float one from the libs.

Hanni

2008/6/3 Robert Kern <robert.kern at gmail.com>:

> On Tue, Jun 3, 2008 at 3:21 AM, Hanni Ali <hanni.ali at gmail.com> wrote:
> > Hi David,
> >
> > I compiled numpy with MSVC 9.0 (vs 2008), I am just using the inbuilt LA
> > libs to minimise complexity.
> >
> > Although I have hacked it such that I can compile and all but one of the
> > regression tests passed:
> >
> > ======================================================================
> > ERROR: Tests reading from a text file.
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> >   File "C:\Python26\lib\site-packages\numpy\ma\tests\test_mrecords.py",
> line
> > 363
> > , in test_fromtextfile
> >     fname = 'tmp%s' % datetime.now().strftime("%y%m%d%H%M%S%s")
> > ValueError: Invalid format string
> > ----------------------------------------------------------------------
> > Ran 1267 tests in 1.141s
> > FAILED (errors=1)
> > <unittest._TextTestResult run=1267 errors=1 failures=0>
> >
> > This appears to be a problem with the strftime function in
> test_mrecords.py
> >
> > The error seems to be created by the millisecond formatting argument %s,
> > removing this caused the test to pass.
>
> Well, this should be using tempfile anyways.
>
> > So I think it's all ok really, however in order to get numpy to compile I
> > have commented out a small part which was causing compilation to fail:
> >
> > numpy\core\src\umathmodule.c.src(64) : error C2059: syntax error : 'type'
> > numpy\core\src\umathmodule.c.src(70) : error C2059: syntax error : 'type'
> >
> > This relates to this section of code:
> >
> > #ifndef HAVE_FREXPF
> > static float frexpf(float x, int * i)
> > {
> >     return (float)frexp((double)(x), i);
> > }
> > #endif
> > #ifndef HAVE_LDEXPF
> > static float ldexpf(float x, int i)
> > {
> >     return (float)ldexp((double)(x), i);
> > }
> > #endif
> >
> > The compiler directives HAVE_FREXPF and HAVE_LDEXPF do not appear to be
> > recognised by msvc 9 would you agree with that assessment?
> >
> > And a redefinition of a function present in the stdc library is
> occurring.
> >
> > What do you think? By just commenting out this piece of code numpy
> compiles
> > and appears to function.
>
> The presence of these functions should have been detected by the
> configuration process of numpy. HAVE_FREXPF and HAVE_LDEXPF would have
> been #define'd if we had detected them correctly. It is possible that
> our configuration process for this does not work correctly with VS
> 2008. From a clean checkout, can you please do the build again and
> copy-and-paste everything printed to the terminal?
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
>  Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080609/a2dccecc/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: output.tar.gz
Type: application/x-gzip
Size: 4221 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080609/a2dccecc/attachment.bin>

From stefan at sun.ac.za  Mon Jun  9 03:41:01 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 9 Jun 2008 09:41:01 +0200
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
Message-ID: <9457e7c80806090041r56c2bb9bjf0d4292bd3ecc1fd@mail.gmail.com>

2008/6/8 Alan McIntyre <alan.mcintyre at gmail.com>:
> Right now there's "if __name__ == '__main__'" boilerplate at the end
> of every test module; nose can find and run tests without that being
> there.  I wanted to get the list's take on removing that--it just
> means that if you want to run the tests in one module, you have to do
> "nosetests numpy/blah/test_foo.py" instead of "python
> numpy/blah/test_foo.py".  Is there a reason not to remove that
> boilerplate since we've already decided to use nose?

The less boilerplate code required the better.  So I'm +1.

>> * Running the whole test suite with import numpy; numpy.test() works fine, but
>> I don't really need to test the rest of numpy when I'm messing with
>> numpy.ma...
>
> I will be including a "test" method in as many subpackages as
> possible, so that you can do numpy.ma.test() and such.

This sounds great, thanks for your efforts!

I suggest we also remove ParametricTestCase now.

Cheers
St?fan


From alan.mcintyre at gmail.com  Mon Jun  9 10:37:08 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 9 Jun 2008 10:37:08 -0400
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <9457e7c80806090041r56c2bb9bjf0d4292bd3ecc1fd@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<9457e7c80806090041r56c2bb9bjf0d4292bd3ecc1fd@mail.gmail.com>
Message-ID: <1d36917a0806090737r5fea3bces6a3e5e223e28999e@mail.gmail.com>

On Mon, Jun 9, 2008 at 3:41 AM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> I suggest we also remove ParametricTestCase now.

I don't mind converting the existing uses (looks like it's only used 5
times) to something else, it was causing trouble for me with nose
anyway--whenever the test module imported it, nose wanted to run it
since it inherited from TestCase.  So if nobody really, really needs
it I'll make ParametricTestCase go away.


From strawman at astraw.com  Mon Jun  9 10:43:38 2008
From: strawman at astraw.com (Andrew Straw)
Date: Mon, 09 Jun 2008 07:43:38 -0700
Subject: [Numpy-discussion] Switching to nose test framework
 (was:	NumpyTest problem)
In-Reply-To: <1d36917a0806090737r5fea3bces6a3e5e223e28999e@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>	<9457e7c80806090041r56c2bb9bjf0d4292bd3ecc1fd@mail.gmail.com>
	<1d36917a0806090737r5fea3bces6a3e5e223e28999e@mail.gmail.com>
Message-ID: <484D419A.4040803@astraw.com>

Alan McIntyre wrote:
> On Mon, Jun 9, 2008 at 3:41 AM, St?fan van der Walt <stefan at sun.ac.za> wrote:
>   
>> I suggest we also remove ParametricTestCase now.
>>     
>
> I don't mind converting the existing uses (looks like it's only used 5
> times) to something else, it was causing trouble for me with nose
> anyway--whenever the test module imported it, nose wanted to run it
> since it inherited from TestCase.  So if nobody really, really needs
> it I'll make ParametricTestCase go away.
I'm using it in some of my code, but I'll happily switch to nose. It
will make my life easier, however, if I can see how you've converted it.
If you do this, can you indicate what svn revision made the switch?
Thanks, Andrew


From matthew.brett at gmail.com  Mon Jun  9 10:47:51 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 9 Jun 2008 15:47:51 +0100
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <1d36917a0806090737r5fea3bces6a3e5e223e28999e@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<9457e7c80806090041r56c2bb9bjf0d4292bd3ecc1fd@mail.gmail.com>
	<1d36917a0806090737r5fea3bces6a3e5e223e28999e@mail.gmail.com>
Message-ID: <1e2af89e0806090747od75bf0ay9ecf5c5fbf26d0c5@mail.gmail.com>

Hi,

I am sure you know this already, but you can just replace the tests
using ParametricTestCase with a nose test generator.  See:

http://www.scipy.org/scipy/scipy/wiki/TestingGuidelines

under Parametric tests.

Best,

Matthew

On Mon, Jun 9, 2008 at 3:37 PM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> On Mon, Jun 9, 2008 at 3:41 AM, St?fan van der Walt <stefan at sun.ac.za> wrote:
>> I suggest we also remove ParametricTestCase now.
>
> I don't mind converting the existing uses (looks like it's only used 5
> times) to something else, it was causing trouble for me with nose
> anyway--whenever the test module imported it, nose wanted to run it
> since it inherited from TestCase.  So if nobody really, really needs
> it I'll make ParametricTestCase go away.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From alan.mcintyre at gmail.com  Mon Jun  9 11:04:27 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 9 Jun 2008 11:04:27 -0400
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <484D419A.4040803@astraw.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<9457e7c80806090041r56c2bb9bjf0d4292bd3ecc1fd@mail.gmail.com>
	<1d36917a0806090737r5fea3bces6a3e5e223e28999e@mail.gmail.com>
	<484D419A.4040803@astraw.com>
Message-ID: <1d36917a0806090804q437e1fdcl6a2ea329c8c79ae5@mail.gmail.com>

On Mon, Jun 9, 2008 at 10:43 AM, Andrew Straw <strawman at astraw.com> wrote:
> I'm using it in some of my code, but I'll happily switch to nose. It
> will make my life easier, however, if I can see how you've converted it.
> If you do this, can you indicate what svn revision made the switch?

Certainly, once it's in svn I'll post a message to the list.


From alan.mcintyre at gmail.com  Mon Jun  9 11:07:55 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 9 Jun 2008 11:07:55 -0400
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <1e2af89e0806090747od75bf0ay9ecf5c5fbf26d0c5@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<9457e7c80806090041r56c2bb9bjf0d4292bd3ecc1fd@mail.gmail.com>
	<1d36917a0806090737r5fea3bces6a3e5e223e28999e@mail.gmail.com>
	<1e2af89e0806090747od75bf0ay9ecf5c5fbf26d0c5@mail.gmail.com>
Message-ID: <1d36917a0806090807i8e98f49q1417cd4e86f71b1d@mail.gmail.com>

On Mon, Jun 9, 2008 at 10:47 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> I am sure you know this already, but you can just replace the tests
> using ParametricTestCase with a nose test generator.

Thanks for pointing that out; it's a rather cool feature. :)


From tgrav at mac.com  Mon Jun  9 11:11:36 2008
From: tgrav at mac.com (Tommy Grav)
Date: Mon, 9 Jun 2008 11:11:36 -0400
Subject: [Numpy-discussion] Bug in numpy.histogram?
Message-ID: <95E6A26E-8E38-4C11-A580-9414A890DFEC@mac.com>

With the most recent change in numpy 1.1 it seems that numpy.histogram
was broken when wanting a normalized histogram. I thought the idea
was to leave the functionality of histogram as it was in 1.1 and then  
break
the api in 1.2?

import numpy
a = [0,1,2,3,4,5,6,7,8]
numpy.histogram(a)
/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
packages/numpy/lib/function_base.py:166: FutureWarning:
         The semantics of histogram will be modified in
         release 1.2 to improve outlier handling. The new behavior can  
be
         obtained using new=True. Note that the new version accepts/ 
returns
         the bin edges instead of the left bin edges.
         Please read the docstring for more information.
   Please read the docstring for more information.""", FutureWarning)
(array([1, 1, 1, 1, 0, 1, 1, 1, 1, 1]), array([ 0. ,  0.8,  1.6,   
2.4,  3.2,  4. ,  4.8,  5.6,  6.4,  7.2]))
data, bins = numpy.histogram(a)
len(data)
10
len(bins)
10
b = [1,3,5,6,9]
data, bins = numpy.histogram(a,b)
/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
packages/numpy/lib/function_base.py:193: FutureWarning:
             The semantic for bins will change in version 1.2.
             The bins will become the bin edges, instead of the left  
bin edges.

   """, FutureWarning)
data
array([2, 2, 1, 3, 0])
bins
array([1, 3, 5, 6, 9])
data, bins = numpy.histogram(a,b,normed=True)
Traceback (most recent call last):
   File "<console>", line 0, in <module>
   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ 
python2.5/site-packages/numpy/lib/function_base.py", line 189, in  
histogram
     raise ValueError, 'Use new=True to pass bin edges explicitly.'
ValueError: Use new=True to pass bin edges explicitly.




From pav at iki.fi  Mon Jun  9 11:17:16 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 09 Jun 2008 18:17:16 +0300
Subject: [Numpy-discussion] Bug in numpy.histogram?
In-Reply-To: <95E6A26E-8E38-4C11-A580-9414A890DFEC@mac.com>
References: <95E6A26E-8E38-4C11-A580-9414A890DFEC@mac.com>
Message-ID: <1213024636.6490.71.camel@localhost>

ma, 2008-06-09 kello 11:11 -0400, Tommy Grav kirjoitti:
> With the most recent change in numpy 1.1 it seems that numpy.histogram
> was broken when wanting a normalized histogram. I thought the idea was
> to leave the functionality of histogram as it was in 1.1 and then
> break the api in 1.2?
[clip]
> data, bins = numpy.histogram(a,b,normed=True)
> Traceback (most recent call last):
>    File "<console>", line 0, in <module>
>    File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ 
> python2.5/site-packages/numpy/lib/function_base.py", line 189, in  
> histogram
>      raise ValueError, 'Use new=True to pass bin edges explicitly.'
> ValueError: Use new=True to pass bin edges explicitly.

I think the point in this specific change was that numpy.histogram
previously returned invalid results when normed=True and explicit bins
were given; the previous code always normalized the results assuming the
bins were of equal size.

Moreover, I think it was not obvious what "normalized" results should
mean when one of the bins is of infinite size.

	Pauli




From tgrav at mac.com  Mon Jun  9 11:38:07 2008
From: tgrav at mac.com (Tommy Grav)
Date: Mon, 9 Jun 2008 11:38:07 -0400
Subject: [Numpy-discussion] Bug in numpy.histogram?
In-Reply-To: <1213024636.6490.71.camel@localhost>
References: <95E6A26E-8E38-4C11-A580-9414A890DFEC@mac.com>
	<1213024636.6490.71.camel@localhost>
Message-ID: <531B24CB-E945-4212-8955-AC8B30A060A3@mac.com>

I understand this and agree, but it still means that the API for  
histogram is
broken since normed can only be used with the new=True parameter. I  
though
the whole point of the future warning was to avoid this. It is not a  
big deal,
just means that one is forced to use the new API somewhat quicker :)

Cheers
   Tommy



On Jun 9, 2008, at 11:17 AM, Pauli Virtanen wrote:

> ma, 2008-06-09 kello 11:11 -0400, Tommy Grav kirjoitti:
>> With the most recent change in numpy 1.1 it seems that  
>> numpy.histogram
>> was broken when wanting a normalized histogram. I thought the idea  
>> was
>> to leave the functionality of histogram as it was in 1.1 and then
>> break the api in 1.2?
> [clip]
>> data, bins = numpy.histogram(a,b,normed=True)
>> Traceback (most recent call last):
>>   File "<console>", line 0, in <module>
>>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
>> python2.5/site-packages/numpy/lib/function_base.py", line 189, in
>> histogram
>>     raise ValueError, 'Use new=True to pass bin edges explicitly.'
>> ValueError: Use new=True to pass bin edges explicitly.
>
> I think the point in this specific change was that numpy.histogram
> previously returned invalid results when normed=True and explicit bins
> were given; the previous code always normalized the results assuming  
> the
> bins were of equal size.
>
> Moreover, I think it was not obvious what "normalized" results should
> mean when one of the bins is of infinite size.
>
> 	Pauli
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From charlesr.harris at gmail.com  Mon Jun  9 12:16:13 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 9 Jun 2008 10:16:13 -0600
Subject: [Numpy-discussion] Buildbot dying horrible, slow death.
Message-ID: <e06186140806090916yf6c559bqb4d8efe7a05bf6d4@mail.gmail.com>

Dear Friends,

Our wonderful buildbot is in declining health. The Mac can't update from
svn, Andrew's machines are offline, the Sparcs have lost their spark, and
bsd_64 is suffering from tolist() syndrome:

======================================================================
FAIL: Ticket #793, changeset r5215
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/numpy-buildbot/b12/numpy-install/lib/python2.4/site-packages/numpy/core/tests/test_regression.py",
line 1134, in check_recarray_tolist
    assert( a[1].tolist() == b[1])
AssertionError

----------------------------------------------------------------------
Ran 1316 tests in 5.440s

Only Windows_xp_64 still functions, but for how long? We need your help!

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080609/c645b2ea/attachment.html>

From strawman at astraw.com  Mon Jun  9 14:18:09 2008
From: strawman at astraw.com (Andrew Straw)
Date: Mon, 09 Jun 2008 11:18:09 -0700
Subject: [Numpy-discussion] Buildbot dying horrible, slow death.
In-Reply-To: <e06186140806090916yf6c559bqb4d8efe7a05bf6d4@mail.gmail.com>
References: <e06186140806090916yf6c559bqb4d8efe7a05bf6d4@mail.gmail.com>
Message-ID: <484D73E1.4010709@astraw.com>

Charles R Harris wrote:
> Dear Friends,
>
> Our wonderful buildbot is in declining health. The Mac can't update
> from svn, Andrew's machines are offline, the Sparcs have lost their
> spark, and bsd_64 is suffering from tolist() syndrome:
AFAIK, I don't have any machines on the buildbot... Is there another
Andrew with offline machines?


From alan.mcintyre at gmail.com  Mon Jun  9 14:34:40 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 9 Jun 2008 14:34:40 -0400
Subject: [Numpy-discussion] Buildbot dying horrible, slow death.
In-Reply-To: <e06186140806090916yf6c559bqb4d8efe7a05bf6d4@mail.gmail.com>
References: <e06186140806090916yf6c559bqb4d8efe7a05bf6d4@mail.gmail.com>
Message-ID: <1d36917a0806091134y5318aae9t8fd772e9b7b82eda@mail.gmail.com>

On Mon, Jun 9, 2008 at 12:16 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> Only Windows_xp_64 still functions, but for how long? We need your help!

What version of the MS compilers is needed for a Windows build?  I've
got a WinXP machine that could be used as a buildbot as long as I
don't have to buy any software. I could try to set up my Intel Mac too
(although the last time I tried to build NumPy on it I gave up and
just chose to do all my NumPy work on a Linux machine).

Alan


From bsouthey at gmail.com  Mon Jun  9 15:02:55 2008
From: bsouthey at gmail.com (Bruce Southey)
Date: Mon, 09 Jun 2008 14:02:55 -0500
Subject: [Numpy-discussion] Buildbot dying horrible, slow death.
In-Reply-To: <1d36917a0806091134y5318aae9t8fd772e9b7b82eda@mail.gmail.com>
References: <e06186140806090916yf6c559bqb4d8efe7a05bf6d4@mail.gmail.com>
	<1d36917a0806091134y5318aae9t8fd772e9b7b82eda@mail.gmail.com>
Message-ID: <484D7E5F.40608@gmail.com>

Alan McIntyre wrote:
> On Mon, Jun 9, 2008 at 12:16 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>   
>> Only Windows_xp_64 still functions, but for how long? We need your help!
>>     
>
> What version of the MS compilers is needed for a Windows build?  I've
> got a WinXP machine that could be used as a buildbot as long as I
> don't have to buy any software. I could try to set up my Intel Mac too
> (although the last time I tried to build NumPy on it I gave up and
> just chose to do all my NumPy work on a Linux machine).
>
> Alan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   
Hi,
I could spare a Linux systems depending on what is required.

What do you actually need and what are the requirements (especially 
firewall configurations)?

Bruce



From christopher.e.kees at usace.army.mil  Mon Jun  9 15:41:14 2008
From: christopher.e.kees at usace.army.mil (Chris Kees)
Date: Mon, 9 Jun 2008 14:41:14 -0500
Subject: [Numpy-discussion] problems building in cygwin under vmware
Message-ID: <F5BF4B3D-66F8-4D4E-BA47-0E394AE7C16F@usace.army.mil>

Hi,

I'm getting an assembler error "Error: suffix or operands invalid for  
`fnstsw'" while trying to build numpy on cygwin (running under windows  
XP running on vmware on a mac pro). I've tried the last two releases  
of numpy and the svn version.  Has anybody ever seen this before?

-Chris

tail end of 'python setup.py build' output:

compile options: '-g -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core/src  
-Inumpy/core/include -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core - 
Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -I/usr/ 
include/python2.5 -c'
gcc: build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/umathmodule.c
In file included from numpy/core/src/umathmodule.c.src:2183:
numpy/core/src/ufuncobject.c: In function `_extract_pyvals':
numpy/core/src/ufuncobject.c:1164: warning: int format, long int arg  
(arg 4)
numpy/core/src/ufuncobject.c:1164: warning: int format, long int arg  
(arg 5)
/cygdrive/c/DOCUME~1/ADMINI~1/LOCALS~1/Temp/cclP4Hfs.s: Assembler  
messages:
/cygdrive/c/DOCUME~1/ADMINI~1/LOCALS~1/Temp/cclP4Hfs.s:72160: Error:  
suffix or operands invalid for `fnstsw'
/cygdrive/c/DOCUME~1/ADMINI~1/LOCALS~1/Temp/cclP4Hfs.s:72415: Error:  
suffix or operands invalid for `fnstsw'
In file included from numpy/core/src/umathmodule.c.src:2183:
numpy/core/src/ufuncobject.c: In function `_extract_pyvals':
numpy/core/src/ufuncobject.c:1164: warning: int format, long int arg  
(arg 4)
numpy/core/src/ufuncobject.c:1164: warning: int format, long int arg  
(arg 5)
/cygdrive/c/DOCUME~1/ADMINI~1/LOCALS~1/Temp/cclP4Hfs.s: Assembler  
messages:
/cygdrive/c/DOCUME~1/ADMINI~1/LOCALS~1/Temp/cclP4Hfs.s:72160: Error:  
suffix or operands invalid for `fnstsw'
/cygdrive/c/DOCUME~1/ADMINI~1/LOCALS~1/Temp/cclP4Hfs.s:72415: Error:  
suffix or operands invalid for `fnstsw'
error: Command "gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall - 
Wstrict-prototypes -g -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core/ 
src -Inumpy/core/include -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core  
-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -I/usr/ 
include/python2.5 -c build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/ 
umathmodule.c -o build/temp.cygwin-1.5.25-i686-2.5/build/ 
src.cygwin-1.5.25-i686-2.5/numpy/core/src/umathmodule.o" failed with  
exit status 1


compiler information:

$ python -V
Python 2.5.1
$ gcc -v
Reading specs from /usr/lib/gcc/i686-pc-cygwin/3.4.4/specs
Configured with: /usr/build/package/orig/test.respin/gcc-3.4.4-3/ 
configure --verbose --prefix=/usr --exec-prefix=/usr --sysconfdir=/etc  
--libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man -- 
infodir=/usr/share/info --enable-languages=c,ada,c+ 
+,d,f77,pascal,java,objc --enable-nls --without-included-gettext -- 
enable-version-specific-runtime-libs --without-x --enable-libgcj -- 
disable-java-awt --with-system-zlib --enable-interpreter --disable- 
libgcj-debug --enable-threads=posix --enable-java-gc=boehm --disable- 
win32-registry --enable-sjlj-exceptions --enable-hash-synchronization  
--enable-libstdcxx-debug
Thread model: posix
gcc version 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)
$ g77 -v
Reading specs from /usr/lib/gcc/i686-pc-cygwin/3.4.4/specs
Configured with: /usr/build/package/orig/test.respin/gcc-3.4.4-3/ 
configure --verbose --prefix=/usr --exec-prefix=/usr --sysconfdir=/etc  
--libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man -- 
infodir=/usr/share/info --enable-languages=c,ada,c+ 
+,d,f77,pascal,java,objc --enable-nls --without-included-gettext -- 
enable-version-specific-runtime-libs --without-x --enable-libgcj -- 
disable-java-awt --with-system-zlib --enable-interpreter --disable- 
libgcj-debug --enable-threads=posix --enable-java-gc=boehm --disable- 
win32-registry --enable-sjlj-exceptions --enable-hash-synchronization  
--enable-libstdcxx-debug
Thread model: posix
gcc version 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)



From charlesr.harris at gmail.com  Mon Jun  9 16:23:52 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 9 Jun 2008 14:23:52 -0600
Subject: [Numpy-discussion] Buildbot dying horrible, slow death.
In-Reply-To: <484D73E1.4010709@astraw.com>
References: <e06186140806090916yf6c559bqb4d8efe7a05bf6d4@mail.gmail.com>
	<484D73E1.4010709@astraw.com>
Message-ID: <e06186140806091323h47c5fed3y41d64c516db5131a@mail.gmail.com>

On Mon, Jun 9, 2008 at 12:18 PM, Andrew Straw <strawman at astraw.com> wrote:

> Charles R Harris wrote:
> > Dear Friends,
> >
> > Our wonderful buildbot is in declining health. The Mac can't update
> > from svn, Andrew's machines are offline, the Sparcs have lost their
> > spark, and bsd_64 is suffering from tolist() syndrome:
> AFAIK, I don't have any machines on the buildbot... Is there another
> Andrew with offline machines?


Sorry, that would be Albert Strasheim.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080609/26d2a7fe/attachment.html>

From charlesr.harris at gmail.com  Mon Jun  9 16:25:25 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 9 Jun 2008 14:25:25 -0600
Subject: [Numpy-discussion] Buildbot dying horrible, slow death.
In-Reply-To: <484D7E5F.40608@gmail.com>
References: <e06186140806090916yf6c559bqb4d8efe7a05bf6d4@mail.gmail.com>
	<1d36917a0806091134y5318aae9t8fd772e9b7b82eda@mail.gmail.com>
	<484D7E5F.40608@gmail.com>
Message-ID: <e06186140806091325h528ad77dvbe26da01d18ac424@mail.gmail.com>

On Mon, Jun 9, 2008 at 1:02 PM, Bruce Southey <bsouthey at gmail.com> wrote:

> Alan McIntyre wrote:
> > On Mon, Jun 9, 2008 at 12:16 PM, Charles R Harris
> > <charlesr.harris at gmail.com> wrote:
> >
> >> Only Windows_xp_64 still functions, but for how long? We need your help!
> >>
> >
> > What version of the MS compilers is needed for a Windows build?  I've
> > got a WinXP machine that could be used as a buildbot as long as I
> > don't have to buy any software. I could try to set up my Intel Mac too
> > (although the last time I tried to build NumPy on it I gave up and
> > just chose to do all my NumPy work on a Linux machine).
> >
> > Alan
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> Hi,
> I could spare a Linux systems depending on what is required.
>
> What do you actually need and what are the requirements (especially
> firewall configurations)?
>

I think Stefan is the guy who  set up the buildbot and knows that
information.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080609/59627c6b/attachment.html>

From stefan at sun.ac.za  Mon Jun  9 16:37:03 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 9 Jun 2008 22:37:03 +0200
Subject: [Numpy-discussion] Buildbot dying horrible, slow death.
In-Reply-To: <e06186140806091323h47c5fed3y41d64c516db5131a@mail.gmail.com>
References: <e06186140806090916yf6c559bqb4d8efe7a05bf6d4@mail.gmail.com>
	<484D73E1.4010709@astraw.com>
	<e06186140806091323h47c5fed3y41d64c516db5131a@mail.gmail.com>
Message-ID: <9457e7c80806091337m10f64ec6rf6e65262a1cf0a60@mail.gmail.com>

2008/6/9 Charles R Harris <charlesr.harris at gmail.com>:
>
> On Mon, Jun 9, 2008 at 12:18 PM, Andrew Straw <strawman at astraw.com> wrote:
>>
>> Charles R Harris wrote:
>> > Dear Friends,
>> >
>> > Our wonderful buildbot is in declining health. The Mac can't update
>> > from svn, Andrew's machines are offline, the Sparcs have lost their
>> > spark, and bsd_64 is suffering from tolist() syndrome:
>> AFAIK, I don't have any machines on the buildbot... Is there another
>> Andrew with offline machines?
>
> Sorry, that would be Albert Strasheim.

We had to take those virtual machines offline, since they were
situated behind a very restrictive firewall.

The Sparcs should now be up and running properly.  I've notified Barry
about his client's problem (it happens from time to time).

Regards
St?fan


From david.huard at gmail.com  Mon Jun  9 16:50:12 2008
From: david.huard at gmail.com (David Huard)
Date: Mon, 9 Jun 2008 16:50:12 -0400
Subject: [Numpy-discussion] Bug in numpy.histogram?
In-Reply-To: <531B24CB-E945-4212-8955-AC8B30A060A3@mac.com>
References: <95E6A26E-8E38-4C11-A580-9414A890DFEC@mac.com>
	<1213024636.6490.71.camel@localhost>
	<531B24CB-E945-4212-8955-AC8B30A060A3@mac.com>
Message-ID: <91cf711d0806091350t7b25fa99l800fc3768b30db@mail.gmail.com>

2008/6/9 Tommy Grav <tgrav at mac.com>:

> I understand this and agree, but it still means that the API for
> histogram is
> broken since normed can only be used with the new=True parameter. I
> though
> the whole point of the future warning was to avoid this. It is not a
> big deal,
> just means that one is forced to use the new API somewhat quicker :)
>

Tommy,

you should be able to use normed=True as long as bins edges are not
specified explicitly.
That is, by setting bins=number_of_bins and range=[bin_min, bin_max], normed
should not raise any warning.

The case bins=edges_array and normed=True was simply too ugly too fix using
the old calling semantic due to this right edge at infinity problem. Also,
since there was a bug in histogram for this combination, we thought it just
as well to force the switch to the new behavior.

Sorry for the inconvenience,

David





>
> Cheers
>    Tommy
>
>
>
> On Jun 9, 2008, at 11:17 AM, Pauli Virtanen wrote:
>
> > ma, 2008-06-09 kello 11:11 -0400, Tommy Grav kirjoitti:
> >> With the most recent change in numpy 1.1 it seems that
> >> numpy.histogram
> >> was broken when wanting a normalized histogram. I thought the idea
> >> was
> >> to leave the functionality of histogram as it was in 1.1 and then
> >> break the api in 1.2?
> > [clip]
> >> data, bins = numpy.histogram(a,b,normed=True)
> >> Traceback (most recent call last):
> >>   File "<console>", line 0, in <module>
> >>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
> >> python2.5/site-packages/numpy/lib/function_base.py", line 189, in
> >> histogram
> >>     raise ValueError, 'Use new=True to pass bin edges explicitly.'
> >> ValueError: Use new=True to pass bin edges explicitly.
> >
> > I think the point in this specific change was that numpy.histogram
> > previously returned invalid results when normed=True and explicit bins
> > were given; the previous code always normalized the results assuming
> > the
> > bins were of equal size.
> >
> > Moreover, I think it was not obvious what "normalized" results should
> > mean when one of the bins is of infinite size.
> >
> >       Pauli
> >
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080609/d08c0d00/attachment.html>

From charlesr.harris at gmail.com  Mon Jun  9 16:55:18 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 9 Jun 2008 14:55:18 -0600
Subject: [Numpy-discussion] Buildbot dying horrible, slow death.
In-Reply-To: <9457e7c80806091337m10f64ec6rf6e65262a1cf0a60@mail.gmail.com>
References: <e06186140806090916yf6c559bqb4d8efe7a05bf6d4@mail.gmail.com>
	<484D73E1.4010709@astraw.com>
	<e06186140806091323h47c5fed3y41d64c516db5131a@mail.gmail.com>
	<9457e7c80806091337m10f64ec6rf6e65262a1cf0a60@mail.gmail.com>
Message-ID: <e06186140806091355m132853d0q4c771c73fd063d9d@mail.gmail.com>

On Mon, Jun 9, 2008 at 2:37 PM, St?fan van der Walt <stefan at sun.ac.za>
wrote:

> 2008/6/9 Charles R Harris <charlesr.harris at gmail.com>:
> >
> > On Mon, Jun 9, 2008 at 12:18 PM, Andrew Straw <strawman at astraw.com>
> wrote:
> >>
> >> Charles R Harris wrote:
> >> > Dear Friends,
> >> >
> >> > Our wonderful buildbot is in declining health. The Mac can't update
> >> > from svn, Andrew's machines are offline, the Sparcs have lost their
> >> > spark, and bsd_64 is suffering from tolist() syndrome:
> >> AFAIK, I don't have any machines on the buildbot... Is there another
> >> Andrew with offline machines?
> >
> > Sorry, that would be Albert Strasheim.
>
> We had to take those virtual machines offline, since they were
> situated behind a very restrictive firewall.
>

Sounds like we could set up single machines to run a number of different
OS's, at least for Intel architectures. Are there virtual machines on Sparc
and PPC that will do that also?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080609/55930a8a/attachment.html>

From ondrej at certik.cz  Mon Jun  9 17:18:42 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 9 Jun 2008 23:18:42 +0200
Subject: [Numpy-discussion] numpy 1.1: import numpy fails in Debian
In-Reply-To: <85b5c3130806090920t120922a0i70a9d6658c9143ba@mail.gmail.com>
References: <85b5c3130806090920t120922a0i70a9d6658c9143ba@mail.gmail.com>
Message-ID: <85b5c3130806091418j371494b8x5f2e6f7128b5ac55@mail.gmail.com>

On Mon, Jun 9, 2008 at 6:20 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> I tried to package numpy 1.1 and upload it to Debian, all worked fine,
> I installed the package and:
>
> In [1]: import numpy
> ---------------------------------------------------------------------------
> ImportError                               Traceback (most recent call last)
>
> /home/ondra/debian/packages/numpy/<ipython console> in <module>()
>
> /usr/lib/python2.5/site-packages/numpy/__init__.py in <module>()
>    105     import random
>    106     import ctypeslib
> --> 107     import ma
>        global ma = undefined
>    108
>    109     # Make these accessible from numpy name-space
>
> ImportError: No module named ma
>
>
>
> So I tried to investigate where the problem is, but I didn't figure it
> out. I am sending the build log. If anyone knows what's wrong, it'd be
> very helpful.
>
> The only thing I figured out so far is that the "ma" package gets
> installed into:
>
> debian/tmp/usr/lib/python2.5/site-packages/numpy/ma
>
> but it is not copied into the directory:
>
> debian/python-numpy/usr/share/pyshared/numpy/
>
> from which the package is made. So it's maybe a bug in some Debian
> package, rather than in numpy itself. But maybe some numpy developers
> know what is so different on the "ma" package (which as I understand
> was added in 1.1).

Andrew Straw fixed the problem, so the new numpy package is in incoming:

http://incoming.debian.org/

and should be on Debian mirrors tomorrow.

Ondrej


From peridot.faceted at gmail.com  Mon Jun  9 19:07:17 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 9 Jun 2008 19:07:17 -0400
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <1d36917a0806081618g5875fa3bt1bfe5f379e9a797a@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>
	<1d36917a0806081618g5875fa3bt1bfe5f379e9a797a@mail.gmail.com>
Message-ID: <ce557a360806091607v6a777a42id75102dd63b50e62@mail.gmail.com>

2008/6/8 Alan McIntyre <alan.mcintyre at gmail.com>:
> On Sun, Jun 8, 2008 at 6:49 PM, Nathan Bell <wnbell at gmail.com> wrote:
>> On Sun, Jun 8, 2008 at 6:15 AM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
>>> Right now there's "if __name__ == '__main__'" boilerplate at the end
>>> of every test module; nose can find and run tests without that being
>>> there.  I wanted to get the list's take on removing that--it just
>>> means that if you want to run the tests in one module, you have to do
>>> "nosetests numpy/blah/test_foo.py" instead of "python
>>> numpy/blah/test_foo.py".  Is there a reason not to remove that
>>> boilerplate since we've already decided to use nose?
>>
>> When making frequent changes to test_foo.py, it's often nice to run
>> test_foo.py directly, rather than installing the whole package and
>> then testing via nose.
>>
>> I would leave the decision up to the maintainers of the individual
>> submodules.  Personally, I find it useful to have.
>
> You can run the test scripts from the source tree without installing
> them by using nosetests, like this:
>
> nosetests nump/numpy/ilb/tests/test_getlimits.py
> or
> nosetests nump/numpy/ilb/tests

I am not enthusiastic about this change: it makes it harder for J.
Random Hacker to run a fileful of tests. She has to know about the
magical nose invocation - where should she find it out? If she's
fiddling with the code she can hardly avoid finding the boilerplate,
but will she know to go digging through the wiki (assuming it appears
there) to find the magical invocation required to run a particular
test?

Then again, it doesn't usually work for me to use the boilerplate to
run a single test without installing, since the non-installed test
doesn't necessarily find all the non-installed code, sometimes falling
back on the installed code and giving false passes or mysterious
failures. Usually I give up and just reinstall the whole beast and
rerun all tests every time I make a change. How does nose behave in
this situation?

Anne


From robert.kern at gmail.com  Mon Jun  9 19:15:53 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 9 Jun 2008 18:15:53 -0500
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <ce557a360806091607v6a777a42id75102dd63b50e62@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>
	<1d36917a0806081618g5875fa3bt1bfe5f379e9a797a@mail.gmail.com>
	<ce557a360806091607v6a777a42id75102dd63b50e62@mail.gmail.com>
Message-ID: <3d375d730806091615u1ab7dfd7i9e0cf937ffb09fdc@mail.gmail.com>

On Mon, Jun 9, 2008 at 18:07, Anne Archibald <peridot.faceted at gmail.com> wrote:

> Then again, it doesn't usually work for me to use the boilerplate to
> run a single test without installing, since the non-installed test
> doesn't necessarily find all the non-installed code, sometimes falling
> back on the installed code and giving false passes or mysterious
> failures. Usually I give up and just reinstall the whole beast and
> rerun all tests every time I make a change. How does nose behave in
> this situation?

nose doesn't affect importing; the test file imports work as normal.
If you want to test code in-place, you will need to do a build_ext
--inplace and set your sys.path accordingly (via PYTHONPATH, .pth
files, whatever). If you use setuptools/easy_install, this is all
combined: "python setupegg.py develop". In order to use an in-place
numpy to build packages like scipy, you will also need to symlink or
copy the generated header files into the appropriate places:

  python setup.py build_ext --inplace
  cd numpy/core/include/numpy
  ln -s ../../*.h .

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From kwgoodman at gmail.com  Mon Jun  9 19:34:08 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Mon, 9 Jun 2008 16:34:08 -0700
Subject: [Numpy-discussion] Coverting ranks to a Gaussian
Message-ID: <f4f93d420806091634g5991f285i66e8f304f291b0a5@mail.gmail.com>

Does anyone have a function that converts ranks into a Gaussian?

I have an array x:

>> import numpy as np
>> x = np.random.rand(5)

I rank it:

>> x = x.argsort().argsort()
>> x_ranked = x.argsort().argsort()
>> x_ranked
   array([3, 1, 4, 2, 0])

I would like to convert the ranks to a Gaussian without using scipy.
So instead of the equal distance between ranks in array x, I would
like the distance been them to follow a Gaussian distribution.

How far out in the tails of the Gaussian should 0 and N-1 (N=5 in the
example above) be? Ideally, or arbitrarily, the areas under the
Gaussian to the left of 0 (and the right of N-1) should be 1/N or
1/2N. Something like that. Or a fixed value is good too.


From robert.kern at gmail.com  Mon Jun  9 19:45:08 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 9 Jun 2008 18:45:08 -0500
Subject: [Numpy-discussion] Coverting ranks to a Gaussian
In-Reply-To: <f4f93d420806091634g5991f285i66e8f304f291b0a5@mail.gmail.com>
References: <f4f93d420806091634g5991f285i66e8f304f291b0a5@mail.gmail.com>
Message-ID: <3d375d730806091645p187ddd6fmf08aa0e0576e3e5d@mail.gmail.com>

On Mon, Jun 9, 2008 at 18:34, Keith Goodman <kwgoodman at gmail.com> wrote:
> Does anyone have a function that converts ranks into a Gaussian?
>
> I have an array x:
>
>>> import numpy as np
>>> x = np.random.rand(5)
>
> I rank it:
>
>>> x = x.argsort().argsort()
>>> x_ranked = x.argsort().argsort()
>>> x_ranked
>   array([3, 1, 4, 2, 0])

There are subtleties in computing ranks when ties are involved. Take a
look at the implementation of scipy.stats.rankdata().

> I would like to convert the ranks to a Gaussian without using scipy.

No dice. You are going to have to use scipy.special.ndtri somewhere. A
basic transformation (off the top of my head, I have no idea if this
is statistically meaningful):

  scipy.special.ndtri((ranks + 1.0) / (len(ranks) + 1.0))

Barring tied first or last items, this should give equal weight to
each of the tails outside of the range of your data.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From alan.mcintyre at gmail.com  Mon Jun  9 21:53:23 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 9 Jun 2008 21:53:23 -0400
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <ce557a360806091607v6a777a42id75102dd63b50e62@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>
	<1d36917a0806081618g5875fa3bt1bfe5f379e9a797a@mail.gmail.com>
	<ce557a360806091607v6a777a42id75102dd63b50e62@mail.gmail.com>
Message-ID: <1d36917a0806091853g3390a5e4m2ca1fa08b93f2857@mail.gmail.com>

On Mon, Jun 9, 2008 at 7:07 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> 2008/6/8 Alan McIntyre <alan.mcintyre at gmail.com>:
>> You can run the test scripts from the source tree without installing
>> them by using nosetests, like this: <snip>
>
> I am not enthusiastic about this change: it makes it harder for J.
> Random Hacker to run a fileful of tests. She has to know about the
> magical nose invocation - where should she find it out? If she's
> fiddling with the code she can hardly avoid finding the boilerplate,
> but will she know to go digging through the wiki (assuming it appears
> there) to find the magical invocation required to run a particular
> test?

Hmm you're right, it leaves no trail for someone to find out how to
run tests in a single module if they don't already know.  Now I'm
inclined to leave that little bit of code there. I'm glad I asked. :)

> Then again, it doesn't usually work for me to use the boilerplate to
> run a single test without installing, since the non-installed test
> doesn't necessarily find all the non-installed code, sometimes falling
> back on the installed code and giving false passes or mysterious
> failures. Usually I give up and just reinstall the whole beast and
> rerun all tests every time I make a change. How does nose behave in
> this situation?

Since I've been working mostly on the tests, I haven't really had to
cope with that yet.  The install only takes a short time (5-10 sec)
once everything is built, and I don't have to do it that often, but I
can see it would be a bit onerous if I was changing stuff that caused
recompiles, etc.

Is the stuff Robert pointed out on a wiki page somewhere? It would be
nice to have a "Welcome noob NumPy developer, here's how to do
NumPy-specific development things," page. There may be such a page,
but I just haven't stumbled across it yet.

Alan


From kwgoodman at gmail.com  Mon Jun  9 22:06:24 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Mon, 9 Jun 2008 19:06:24 -0700
Subject: [Numpy-discussion] Coverting ranks to a Gaussian
In-Reply-To: <3d375d730806091645p187ddd6fmf08aa0e0576e3e5d@mail.gmail.com>
References: <f4f93d420806091634g5991f285i66e8f304f291b0a5@mail.gmail.com>
	<3d375d730806091645p187ddd6fmf08aa0e0576e3e5d@mail.gmail.com>
Message-ID: <f4f93d420806091906j2b6608d4k3d1c702dfbd9b1da@mail.gmail.com>

On Mon, Jun 9, 2008 at 4:45 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Mon, Jun 9, 2008 at 18:34, Keith Goodman <kwgoodman at gmail.com> wrote:
>> Does anyone have a function that converts ranks into a Gaussian?
>>
>> I have an array x:
>>
>>>> import numpy as np
>>>> x = np.random.rand(5)
>>
>> I rank it:
>>
>>>> x = x.argsort().argsort()
>>>> x_ranked = x.argsort().argsort()
>>>> x_ranked
>>   array([3, 1, 4, 2, 0])
>
> There are subtleties in computing ranks when ties are involved. Take a
> look at the implementation of scipy.stats.rankdata().

Good point. I had to deal with ties and missing data. I bet
scipy.stats.rankdata() is faster than my implementation.


>> I would like to convert the ranks to a Gaussian without using scipy.
>
> No dice. You are going to have to use scipy.special.ndtri somewhere. A
> basic transformation (off the top of my head, I have no idea if this
> is statistically meaningful):
>
>  scipy.special.ndtri((ranks + 1.0) / (len(ranks) + 1.0))
>
> Barring tied first or last items, this should give equal weight to
> each of the tails outside of the range of your data.

Nice. Thank you. It passes the never wrong chi-by-eye test:

r = np.arange(1000)
g = special.ndtri((r + 1.0) / (len(r) + 1.0))
pylab.hist(g, 50)
pylab.show()

I wasn't able to use scipy.special.ndtri (after import scipy) like you
did. I had to do (but I'm new to scipy)

from scipy import special
special.ndtri

scipy.__version__
   '0.6.0'

from Debian Lenny.


From pgmdevlist at gmail.com  Mon Jun  9 22:02:46 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 9 Jun 2008 22:02:46 -0400
Subject: [Numpy-discussion] Coverting ranks to a Gaussian
In-Reply-To: <f4f93d420806091906j2b6608d4k3d1c702dfbd9b1da@mail.gmail.com>
References: <f4f93d420806091634g5991f285i66e8f304f291b0a5@mail.gmail.com>
	<3d375d730806091645p187ddd6fmf08aa0e0576e3e5d@mail.gmail.com>
	<f4f93d420806091906j2b6608d4k3d1c702dfbd9b1da@mail.gmail.com>
Message-ID: <200806092202.47577.pgmdevlist@gmail.com>

On Monday 09 June 2008 22:06:24 Keith Goodman wrote:
> On Mon, Jun 9, 2008 at 4:45 PM, Robert Kern <robert.kern at gmail.com> wrote:

> > There are subtleties in computing ranks when ties are involved. Take a
> > look at the implementation of scipy.stats.rankdata().
>
> Good point. I had to deal with ties and missing data. I bet
> scipy.stats.rankdata() is faster than my implementation.

There's a scipy.stats.mstats.rankdata() that take care of both ties and 
missing data. Missing data are allocated a rank of either 0 or the average 
rank, depending on some parameter.



From kwgoodman at gmail.com  Mon Jun  9 22:30:09 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Mon, 9 Jun 2008 19:30:09 -0700
Subject: [Numpy-discussion] Coverting ranks to a Gaussian
In-Reply-To: <200806092202.47577.pgmdevlist@gmail.com>
References: <f4f93d420806091634g5991f285i66e8f304f291b0a5@mail.gmail.com>
	<3d375d730806091645p187ddd6fmf08aa0e0576e3e5d@mail.gmail.com>
	<f4f93d420806091906j2b6608d4k3d1c702dfbd9b1da@mail.gmail.com>
	<200806092202.47577.pgmdevlist@gmail.com>
Message-ID: <f4f93d420806091930u7855e833la76ccd15031b88a4@mail.gmail.com>

On Mon, Jun 9, 2008 at 7:02 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
> On Monday 09 June 2008 22:06:24 Keith Goodman wrote:
>> On Mon, Jun 9, 2008 at 4:45 PM, Robert Kern <robert.kern at gmail.com> wrote:
>
>> > There are subtleties in computing ranks when ties are involved. Take a
>> > look at the implementation of scipy.stats.rankdata().
>>
>> Good point. I had to deal with ties and missing data. I bet
>> scipy.stats.rankdata() is faster than my implementation.
>
> There's a scipy.stats.mstats.rankdata() that take care of both ties and
> missing data. Missing data are allocated a rank of either 0 or the average
> rank, depending on some parameter.

That sounds interesting. But I can't find it:

>> import scipy
>> from scipy import stats
>> scipy.stats.m
scipy.stats.mannwhitneyu  scipy.stats.mean          scipy.stats.mielke
       scipy.stats.moment        scipy.stats.morestats
scipy.stats.maxwell       scipy.stats.median        scipy.stats.mode
       scipy.stats.mood          scipy.stats.mvn
>> scipy.stats.morestats.r
scipy.stats.morestats.r_     scipy.stats.morestats.ravel

In my implementation I leave the missing values as missing. I think
that would be a nice option for rankdata.


From robert.kern at gmail.com  Mon Jun  9 22:35:43 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 9 Jun 2008 21:35:43 -0500
Subject: [Numpy-discussion] Coverting ranks to a Gaussian
In-Reply-To: <f4f93d420806091906j2b6608d4k3d1c702dfbd9b1da@mail.gmail.com>
References: <f4f93d420806091634g5991f285i66e8f304f291b0a5@mail.gmail.com>
	<3d375d730806091645p187ddd6fmf08aa0e0576e3e5d@mail.gmail.com>
	<f4f93d420806091906j2b6608d4k3d1c702dfbd9b1da@mail.gmail.com>
Message-ID: <3d375d730806091935k5a27229cma03292e48d4f1be2@mail.gmail.com>

On Mon, Jun 9, 2008 at 21:06, Keith Goodman <kwgoodman at gmail.com> wrote:
> On Mon, Jun 9, 2008 at 4:45 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Mon, Jun 9, 2008 at 18:34, Keith Goodman <kwgoodman at gmail.com> wrote:
>>> Does anyone have a function that converts ranks into a Gaussian?
>>>
>>> I have an array x:
>>>
>>>>> import numpy as np
>>>>> x = np.random.rand(5)
>>>
>>> I rank it:
>>>
>>>>> x = x.argsort().argsort()
>>>>> x_ranked = x.argsort().argsort()
>>>>> x_ranked
>>>   array([3, 1, 4, 2, 0])
>>
>> There are subtleties in computing ranks when ties are involved. Take a
>> look at the implementation of scipy.stats.rankdata().
>
> Good point. I had to deal with ties and missing data. I bet
> scipy.stats.rankdata() is faster than my implementation.

Actually, it's pretty slow. I think there are opportunities to make it
faster, but I haven't explored them.

>>> I would like to convert the ranks to a Gaussian without using scipy.
>>
>> No dice. You are going to have to use scipy.special.ndtri somewhere. A
>> basic transformation (off the top of my head, I have no idea if this
>> is statistically meaningful):
>>
>>  scipy.special.ndtri((ranks + 1.0) / (len(ranks) + 1.0))
>>
>> Barring tied first or last items, this should give equal weight to
>> each of the tails outside of the range of your data.
>
> Nice. Thank you. It passes the never wrong chi-by-eye test:
>
> r = np.arange(1000)
> g = special.ndtri((r + 1.0) / (len(r) + 1.0))
> pylab.hist(g, 50)
> pylab.show()

BTW, what is your use case? If you are trying to compare your data to
the normal distribution, you might want to go the other way: find the
empirical quantiles of your data and compare them against the
hypothetical quantiles of the data on the distribution. This *is* an
established statistical technique; when you graph one versus the
other, you get what is known as a Q-Q plot.

> I wasn't able to use scipy.special.ndtri (after import scipy) like you
> did.

Actually, I didn't specify any import. The behavior you see is
correct. Importing scipy alone doesn't import any of the subpackages.
If you like, you can pretend there was an implicit "import
scipy.special" before the code I gave.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From pgmdevlist at gmail.com  Mon Jun  9 22:35:01 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 9 Jun 2008 22:35:01 -0400
Subject: [Numpy-discussion] Coverting ranks to a Gaussian
In-Reply-To: <f4f93d420806091930u7855e833la76ccd15031b88a4@mail.gmail.com>
References: <f4f93d420806091634g5991f285i66e8f304f291b0a5@mail.gmail.com>
	<200806092202.47577.pgmdevlist@gmail.com>
	<f4f93d420806091930u7855e833la76ccd15031b88a4@mail.gmail.com>
Message-ID: <200806092235.02938.pgmdevlist@gmail.com>

On Monday 09 June 2008 22:30:09 Keith Goodman wrote:
> On Mon, Jun 9, 2008 at 7:02 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
> > There's a scipy.stats.mstats.rankdata() that take care of both ties and
> > missing data. Missing data are allocated a rank of either 0 or the
> > average rank, depending on some parameter.
>
> That sounds interesting. But I can't find it:
> >> import scipy
> >> from scipy import stats

Yes, you should do
>>> import scipy.stats.mstats as mstats
>>> mstats.rankdata

> In my implementation I leave the missing values as missing. I think
> that would be a nice option for rankdata.

Handling missing data is why I needed a tailored rankdata. 
In mstats.rankdata, if you set the use_missing optional parameter to False 
(the default), they will have a rank of 0. As no other value will have a rank 
of zero, you can then remask with masked_values or masked_equal.


From kwgoodman at gmail.com  Mon Jun  9 23:09:26 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Mon, 9 Jun 2008 20:09:26 -0700
Subject: [Numpy-discussion] Coverting ranks to a Gaussian
In-Reply-To: <3d375d730806091935k5a27229cma03292e48d4f1be2@mail.gmail.com>
References: <f4f93d420806091634g5991f285i66e8f304f291b0a5@mail.gmail.com>
	<3d375d730806091645p187ddd6fmf08aa0e0576e3e5d@mail.gmail.com>
	<f4f93d420806091906j2b6608d4k3d1c702dfbd9b1da@mail.gmail.com>
	<3d375d730806091935k5a27229cma03292e48d4f1be2@mail.gmail.com>
Message-ID: <f4f93d420806092009k6cb38bccs9d3c516e0dfdc3d7@mail.gmail.com>

On Mon, Jun 9, 2008 at 7:35 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Mon, Jun 9, 2008 at 21:06, Keith Goodman <kwgoodman at gmail.com> wrote:
>> On Mon, Jun 9, 2008 at 4:45 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>> On Mon, Jun 9, 2008 at 18:34, Keith Goodman <kwgoodman at gmail.com> wrote:
>>>> Does anyone have a function that converts ranks into a Gaussian?
>>>>
>>>> I have an array x:
>>>>
>>>>>> import numpy as np
>>>>>> x = np.random.rand(5)
>>>>
>>>> I rank it:
>>>>
>>>>>> x = x.argsort().argsort()
>>>>>> x_ranked = x.argsort().argsort()
>>>>>> x_ranked
>>>>   array([3, 1, 4, 2, 0])
>>>
>>> There are subtleties in computing ranks when ties are involved. Take a
>>> look at the implementation of scipy.stats.rankdata().
>>
>> Good point. I had to deal with ties and missing data. I bet
>> scipy.stats.rankdata() is faster than my implementation.
>
> Actually, it's pretty slow. I think there are opportunities to make it
> faster, but I haven't explored them.
>
>>>> I would like to convert the ranks to a Gaussian without using scipy.
>>>
>>> No dice. You are going to have to use scipy.special.ndtri somewhere. A
>>> basic transformation (off the top of my head, I have no idea if this
>>> is statistically meaningful):
>>>
>>>  scipy.special.ndtri((ranks + 1.0) / (len(ranks) + 1.0))
>>>
>>> Barring tied first or last items, this should give equal weight to
>>> each of the tails outside of the range of your data.
>>
>> Nice. Thank you. It passes the never wrong chi-by-eye test:
>>
>> r = np.arange(1000)
>> g = special.ndtri((r + 1.0) / (len(r) + 1.0))
>> pylab.hist(g, 50)
>> pylab.show()
>
> BTW, what is your use case? If you are trying to compare your data to
> the normal distribution, you might want to go the other way: find the
> empirical quantiles of your data and compare them against the
> hypothetical quantiles of the data on the distribution. This *is* an
> established statistical technique; when you graph one versus the
> other, you get what is known as a Q-Q plot.

Some of the techniques I use are sensitive to noise. So to make things
more robust I sometimes transform my noisy, sort-of-normal data.
There's no theory to guide me, only empirical results.


From david at ar.media.kyoto-u.ac.jp  Tue Jun 10 02:34:46 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 10 Jun 2008 15:34:46 +0900
Subject: [Numpy-discussion] Visual Studio revamp: is anyone working on it ?
Message-ID: <484E2086.1050106@ar.media.kyoto-u.ac.jp>

Hi,

    IIRC, Gary suggested to cut the complications by using vcvarsall.bat 
instead of depending on the registry (which keeps changing between 
versions). I have started working on that for my own project (because 
scons does not work with VS 2008): it looks easy to at least prototype 
it, since distutils does this way for VS 2005 and 2008. If nobody is 
working on it, I could try to make it in shape for scons (are the 
licenses compatible between python and scons, e.g. ripping of some code 
from python into scons ?).

    thanks,

    David


From david at ar.media.kyoto-u.ac.jp  Tue Jun 10 02:38:54 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 10 Jun 2008 15:38:54 +0900
Subject: [Numpy-discussion] Visual Studio revamp: is anyone working on
 it ?
In-Reply-To: <484E2086.1050106@ar.media.kyoto-u.ac.jp>
References: <484E2086.1050106@ar.media.kyoto-u.ac.jp>
Message-ID: <484E217E.1050800@ar.media.kyoto-u.ac.jp>

David Cournapeau wrote:
> Hi,
>
>     IIRC, Gary suggested to cut the complications by using vcvarsall.bat 
> instead of depending on the registry (which keeps changing between 
> versions). I have started working on that for my own project (because 
> scons does not work with VS 2008): it looks easy to at least prototype 
> it, since distutils does this way for VS 2005 and 2008. If nobody is 
> working on it, I could try to make it in shape for scons (are the 
> licenses compatible between python and scons, e.g. ripping of some code 
> from python into scons ?).
>   

Sorry, wrong ML.

David


From peridot.faceted at gmail.com  Tue Jun 10 03:56:10 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 10 Jun 2008 01:56:10 -0600
Subject: [Numpy-discussion] Coverting ranks to a Gaussian
In-Reply-To: <f4f93d420806091634g5991f285i66e8f304f291b0a5@mail.gmail.com>
References: <f4f93d420806091634g5991f285i66e8f304f291b0a5@mail.gmail.com>
Message-ID: <ce557a360806100056ue79a6bcybdf34cbc1e8fcb6e@mail.gmail.com>

2008/6/9 Keith Goodman <kwgoodman at gmail.com>:
> Does anyone have a function that converts ranks into a Gaussian?
>
> I have an array x:
>
>>> import numpy as np
>>> x = np.random.rand(5)
>
> I rank it:
>
>>> x = x.argsort().argsort()
>>> x_ranked = x.argsort().argsort()
>>> x_ranked
>   array([3, 1, 4, 2, 0])
>
> I would like to convert the ranks to a Gaussian without using scipy.
> So instead of the equal distance between ranks in array x, I would
> like the distance been them to follow a Gaussian distribution.
>
> How far out in the tails of the Gaussian should 0 and N-1 (N=5 in the
> example above) be? Ideally, or arbitrarily, the areas under the
> Gaussian to the left of 0 (and the right of N-1) should be 1/N or
> 1/2N. Something like that. Or a fixed value is good too.

I'm actually not clear on what you need.

If what you need is for rank i of N to be the 100*i/N th percentile in
a Gaussian distribution, then you should indeed use scipy's functions
to accomplish that; I'd use scipy.stats.norm.ppf().

Of course, if your points were drawn from a Gaussian distribution,
they wouldn't be exactly 1/N apart, there would be some distribution.
Quite what the distribution of (say) the maximum or the median of N
points drawn from a Gaussian is, I can't say, though people have
looked at it. But if you want "typical" values, just generate N points
from a Gaussian and sort them:

V = np.random.randn(N)
V = np.sort(V)

return V[ranks]

Of course they will be different every time, but the distribution will be right.

Anne
P.S. why the "no scipy" restriction? it's a bit unreasonable. -A


From stefan at sun.ac.za  Tue Jun 10 05:57:41 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 10 Jun 2008 11:57:41 +0200
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <1d36917a0806091853g3390a5e4m2ca1fa08b93f2857@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>
	<1d36917a0806081618g5875fa3bt1bfe5f379e9a797a@mail.gmail.com>
	<ce557a360806091607v6a777a42id75102dd63b50e62@mail.gmail.com>
	<1d36917a0806091853g3390a5e4m2ca1fa08b93f2857@mail.gmail.com>
Message-ID: <9457e7c80806100257t1a6ffa87m121477814cbba17a@mail.gmail.com>

2008/6/10 Alan McIntyre <alan.mcintyre at gmail.com>:
> Is the stuff Robert pointed out on a wiki page somewhere? It would be
> nice to have a "Welcome noob NumPy developer, here's how to do
> NumPy-specific development things," page. There may be such a page,
> but I just haven't stumbled across it yet.

We could add this info the the `numpy` docstring; that way new users
will have it available without having to search the web.

http://sd-2116.dedibox.fr/pydocweb/doc/numpy/

Regards
St?fan


From kwgoodman at gmail.com  Tue Jun 10 09:17:01 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Tue, 10 Jun 2008 06:17:01 -0700
Subject: [Numpy-discussion] Coverting ranks to a Gaussian
In-Reply-To: <ce557a360806100056ue79a6bcybdf34cbc1e8fcb6e@mail.gmail.com>
References: <f4f93d420806091634g5991f285i66e8f304f291b0a5@mail.gmail.com>
	<ce557a360806100056ue79a6bcybdf34cbc1e8fcb6e@mail.gmail.com>
Message-ID: <f4f93d420806100617r1ce86b5bs9a5dd34b6b8c31fc@mail.gmail.com>

On Tue, Jun 10, 2008 at 12:56 AM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> 2008/6/9 Keith Goodman <kwgoodman at gmail.com>:
>> Does anyone have a function that converts ranks into a Gaussian?
>>
>> I have an array x:
>>
>>>> import numpy as np
>>>> x = np.random.rand(5)
>>
>> I rank it:
>>
>>>> x = x.argsort().argsort()
>>>> x_ranked = x.argsort().argsort()
>>>> x_ranked
>>   array([3, 1, 4, 2, 0])
>>
>> I would like to convert the ranks to a Gaussian without using scipy.
>> So instead of the equal distance between ranks in array x, I would
>> like the distance been them to follow a Gaussian distribution.
>>
>> How far out in the tails of the Gaussian should 0 and N-1 (N=5 in the
>> example above) be? Ideally, or arbitrarily, the areas under the
>> Gaussian to the left of 0 (and the right of N-1) should be 1/N or
>> 1/2N. Something like that. Or a fixed value is good too.
>
> I'm actually not clear on what you need.
>
> If what you need is for rank i of N to be the 100*i/N th percentile in
> a Gaussian distribution, then you should indeed use scipy's functions
> to accomplish that; I'd use scipy.stats.norm.ppf().
>
> Of course, if your points were drawn from a Gaussian distribution,
> they wouldn't be exactly 1/N apart, there would be some distribution.
> Quite what the distribution of (say) the maximum or the median of N
> points drawn from a Gaussian is, I can't say, though people have
> looked at it. But if you want "typical" values, just generate N points
> from a Gaussian and sort them:
>
> V = np.random.randn(N)
> V = np.sort(V)
>
> return V[ranks]
>
> Of course they will be different every time, but the distribution will be right.

I guess I botched the description of my problem.

I have data that contains outliers and other noise. I am trying
various transformations of the data to preprocess it before plugging
it into my prediction algorithm. One such transformation is to rank
the data and then convert that rank to a Gaussian. The particular
details of the transformation don't matter. I just want something
smooth and normal like.

> Anne
> P.S. why the "no scipy" restriction? it's a bit unreasonable. -A

I'd rather not pull in a scipy dependency for one function if there is
a numpy alternative. I think it is funny that you picked up on my
brief mention of scipy and called it unreasonable.


From kwgoodman at gmail.com  Tue Jun 10 10:08:31 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Tue, 10 Jun 2008 07:08:31 -0700
Subject: [Numpy-discussion] Inplace shift
In-Reply-To: <ce557a360806071848m59f56989yacc37f3549d1bd0c@mail.gmail.com>
References: <f4f93d420806061124s494eda58m405cbab2befc51d9@mail.gmail.com>
	<ce557a360806062246q6ad98541i882ed4fe2b2f0269@mail.gmail.com>
	<f4f93d420806070728m5b8d701dj6be7e96d7e7cc1e8@mail.gmail.com>
	<ce557a360806071848m59f56989yacc37f3549d1bd0c@mail.gmail.com>
Message-ID: <f4f93d420806100708u7a520891vf4b6128291c71188@mail.gmail.com>

On Sat, Jun 7, 2008 at 6:48 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> 2008/6/7 Keith Goodman <kwgoodman at gmail.com>:
>> On Fri, Jun 6, 2008 at 10:46 PM, Anne Archibald
>> <peridot.faceted at gmail.com> wrote:
>>> 2008/6/6 Keith Goodman <kwgoodman at gmail.com>:
>>>> I'd like to shift the columns of a 2d array one column to the right.
>>>> Is there a way to do that without making a copy?
>>>>
>>>> This doesn't work:
>>>>
>>>>>> import numpy as np
>>>>>> x = np.random.rand(2,3)
>>>>>> x[:,1:] = x[:,:-1]
>>>>>> x
>>>>
>>>> array([[ 0.44789223,  0.44789223,  0.44789223],
>>>>       [ 0.80600897,  0.80600897,  0.80600897]])
>>>
>>> As a workaround you can use backwards slices:
>>>worki
>>> In [40]: x = np.random.rand(2,3)
>>>
>>> In [41]: x[:,:0:-1] = x[:,-2::-1]
>>>
>>> In [42]: x
>>> Out[42]:
>>> array([[ 0.20183084,  0.20183084,  0.08156887],
>>>       [ 0.30611585,  0.30611585,  0.79001577]])
>>
>> Neat. It makes sense to go backwards. Thank you.
>>
>>> Less painful for numpy developers but more painful for users is to
>>> warn them about the status quo: operations on overlapping slices can
>>> happen in arbitrary order.
>>
>> Now I'm confused. Could some corner case of memory layout cause numpy
>> to work from right to left, breaking the workaround? Or can I depend
>> on the workaround working with numpy 1.0.4?
>
> I'm afraid so. And it's not such a corner case as that: if the array
> is laid out in "C contiguous" order, you have to go backwards, while
> if the array is laid out in "FORTRAN contiguous" order you have to go
> forwards.

I think I'll end up using a code snippet like shift2:

def shift(x):
    x2 = x.copy()
    x[:,1:] = x2[:,:-1]
    return x

def shift2(x):
    if x.flags.c_contiguous:
        x[:,:0:-1] = x[:,-2::-1]
    elif x.flags.f_contiguous:
        x[:,1:] = x[:,:-1]
    else:
        raise ValueError, 'x must be c_contiguous or f_contiguous'
    return x

>> x = np.random.rand(2,3)
>> timeit shift(x)
100000 loops, best of 3: 5.02 ?s per loop
>> timeit shift2(x)
100000 loops, best of 3: 4.75 ?s per loop

>> x = np.random.rand(500,500)
>> timeit shift(x)
100 loops, best of 3: 2.51 ms per loop
>> timeit shift2(x)
1000 loops, best of 3: 1.62 ms per loop
>>
>> x = x.T
>>
>> timeit shift(x)
100 loops, best of 3: 4.17 ms per loop
>> timeit shift2(x)
1000 loops, best of 3: 348 ?s per loop

f contiguous (x.T) is faster. How do I change an array from c
contiguous to f contiguous without changing the elements?


From dagss at student.matnat.uio.no  Tue Jun 10 14:13:22 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 10 Jun 2008 20:13:22 +0200
Subject: [Numpy-discussion] Coverting ranks to a Gaussian
In-Reply-To: <f4f93d420806100617r1ce86b5bs9a5dd34b6b8c31fc@mail.gmail.com>
References: <f4f93d420806091634g5991f285i66e8f304f291b0a5@mail.gmail.com>	<ce557a360806100056ue79a6bcybdf34cbc1e8fcb6e@mail.gmail.com>
	<f4f93d420806100617r1ce86b5bs9a5dd34b6b8c31fc@mail.gmail.com>
Message-ID: <484EC442.6060906@student.matnat.uio.no>

Keith Goodman wrote:
> I'd rather not pull in a scipy dependency for one function if there is
> a numpy alternative. I think it is funny that you picked up on my
> brief mention of scipy and called it unreasonable.
>   
(I didn't follow this exact discussion, arguing from general principles 
here about SciPy dependencies.) Try to look at it this way:

NumPy may solve almost all your needs, and you only need, say, 0.1% of 
SciPy.

Assume, then, that the same statement is true about n other people. The 
problem then is that the 0.1% that each person needs from SciPy does not 
typically overlap. So as n grows larger, and assuming that everyone use 
the same logic that you use, the amount of SciPy stuff that must be 
reimplemented on the NumPy discussion mailing list could become quite large.

(Besides, SciPy is open source, so you can always copy & paste the 
function from it if you only need one trivial bit of it. Not that I 
recommend doing that, but it's still better than reimplementing it 
yourself. Unless you're doing it for the learning experience of course.)

Dag Sverre


From christopher.e.kees at usace.army.mil  Tue Jun 10 14:37:47 2008
From: christopher.e.kees at usace.army.mil (Chris Kees)
Date: Tue, 10 Jun 2008 13:37:47 -0500
Subject: [Numpy-discussion] problems building in cygwin under vmware
In-Reply-To: <F5BF4B3D-66F8-4D4E-BA47-0E394AE7C16F@usace.army.mil>
References: <F5BF4B3D-66F8-4D4E-BA47-0E394AE7C16F@usace.army.mil>
Message-ID: <C6F8BBE3-929C-4BFE-91A4-C5FE104478ED@usace.army.mil>

The solution to this problem (roll back binutils to the previous  
cygwin version or fix numpy)  is here:

http://www.scipy.org/scipy/numpy/ticket/811

On Jun 9, 2008, at 2:41 PM, Chris Kees wrote:

> Hi,
>
> I'm getting an assembler error "Error: suffix or operands invalid for
> `fnstsw'" while trying to build numpy on cygwin (running under windows
> XP running on vmware on a mac pro). I've tried the last two releases
> of numpy and the svn version.  Has anybody ever seen this before?
>
> -Chris
>
> tail end of 'python setup.py build' output:
>
> compile options: '-g -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core/src
> -Inumpy/core/include -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core -
> Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -I/usr/
> include/python2.5 -c'
> gcc: build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/umathmodule.c
> In file included from numpy/core/src/umathmodule.c.src:2183:
> numpy/core/src/ufuncobject.c: In function `_extract_pyvals':
> numpy/core/src/ufuncobject.c:1164: warning: int format, long int arg
> (arg 4)
> numpy/core/src/ufuncobject.c:1164: warning: int format, long int arg
> (arg 5)
> /cygdrive/c/DOCUME~1/ADMINI~1/LOCALS~1/Temp/cclP4Hfs.s: Assembler
> messages:
> /cygdrive/c/DOCUME~1/ADMINI~1/LOCALS~1/Temp/cclP4Hfs.s:72160: Error:
> suffix or operands invalid for `fnstsw'


From fperez.net at gmail.com  Tue Jun 10 14:57:31 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 10 Jun 2008 11:57:31 -0700
Subject: [Numpy-discussion] Plans for Scipy Tutorials
Message-ID: <db6b5ecc0806101157q67713dfp8b6206249933714@mail.gmail.com>

Hi all,

I've now put up the near-final tutorial plans for SciPy 2008 here:

http://conference.scipy.org/tutorials

If your name is listed there and you disagree/can't make it, please
let me and Travis Oliphant know as soon as possible.

As the various presenters fine-tune their plan, we'll update the
details on each tutorial and provide links to pre-requisites,
installation pages, etc.  But the main topics are probably not going
to change now, barring any unforeseen event.

Regards,

f


From cburns at berkeley.edu  Tue Jun 10 18:23:10 2008
From: cburns at berkeley.edu (Christopher Burns)
Date: Tue, 10 Jun 2008 15:23:10 -0700
Subject: [Numpy-discussion] Cookbook/Documentation
In-Reply-To: <9457e7c80805191002v4659cd6g5d2ba6e6be4e0134@mail.gmail.com>
References: <200805191233.10380.pgmdevlist@gmail.com>
	<9457e7c80805191002v4659cd6g5d2ba6e6be4e0134@mail.gmail.com>
Message-ID: <764e38540806101523k7a1edffdm61caadc6cdef720e@mail.gmail.com>

Where is the "CookBookCategory"?  I'm afraid I don't understand that
reference below.  Are there plans to auto-generate the content in the
Cookbook Recipes (http://www.scipy.org/Cookbook) or is it still
reasonable for me to edit those pages?

Thanks,
Chris

On Mon, May 19, 2008 at 10:02 AM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi Pierre
>
> 2008/5/19 Pierre GM <pgmdevlist at gmail.com>:
>> * I've just noticed that the page describing RecordArrays
>> (http://www.scipy.org/RecordArrays) is not listed under the Cookbook: should
>> this be changed ? Shouldn't there be at least a link in the documentation
>> page ?
>
> How about we add those pages to a CookBookCategory and auto-generate
> the Cookbook (like I've done with ProposedEnhancements)?
>
>> * I was eventually considering writing down some basic docs for MaskedArrays:
>> should I create a page under the Cookbook ? Elsewhere ?
>
> That's a good place for now.  Use ReST (on the wiki, use {{{#!rst
> <your stuff here> }}}), then we can incorporate your work into the
> user guide later.
>
> Regards
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Christopher Burns
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From stefan at sun.ac.za  Tue Jun 10 18:31:11 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 11 Jun 2008 00:31:11 +0200
Subject: [Numpy-discussion] Cookbook/Documentation
In-Reply-To: <764e38540806101523k7a1edffdm61caadc6cdef720e@mail.gmail.com>
References: <200805191233.10380.pgmdevlist@gmail.com>
	<9457e7c80805191002v4659cd6g5d2ba6e6be4e0134@mail.gmail.com>
	<764e38540806101523k7a1edffdm61caadc6cdef720e@mail.gmail.com>
Message-ID: <9457e7c80806101531h604f548fle43451b857ae518e@mail.gmail.com>

Hi Chris

2008/6/11 Christopher Burns <cburns at berkeley.edu>:
> Where is the "CookBookCategory"?  I'm afraid I don't understand that
> reference below.  Are there plans to auto-generate the content in the
> Cookbook Recipes (http://www.scipy.org/Cookbook) or is it still
> reasonable for me to edit those pages?

The list is already generated at the bottom of Cookbook.  To make a
page show up there, add the following lines to the bottom (excluding
the triple-quotes):

"""
----
CategoryCookbook
"""

I suggest adding your page to the descriptive list at the top of Cookbook too.

Regards
St?fan


From cburns at berkeley.edu  Tue Jun 10 18:37:11 2008
From: cburns at berkeley.edu (Christopher Burns)
Date: Tue, 10 Jun 2008 15:37:11 -0700
Subject: [Numpy-discussion] Cookbook/Documentation
In-Reply-To: <9457e7c80806101531h604f548fle43451b857ae518e@mail.gmail.com>
References: <200805191233.10380.pgmdevlist@gmail.com>
	<9457e7c80805191002v4659cd6g5d2ba6e6be4e0134@mail.gmail.com>
	<764e38540806101523k7a1edffdm61caadc6cdef720e@mail.gmail.com>
	<9457e7c80806101531h604f548fle43451b857ae518e@mail.gmail.com>
Message-ID: <764e38540806101537l2b1cdbaehbbcb520575817e1@mail.gmail.com>

Excellent, thanks Stefan!

On Tue, Jun 10, 2008 at 3:31 PM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi Chris
>
> 2008/6/11 Christopher Burns <cburns at berkeley.edu>:
>> Where is the "CookBookCategory"?  I'm afraid I don't understand that
>> reference below.  Are there plans to auto-generate the content in the
>> Cookbook Recipes (http://www.scipy.org/Cookbook) or is it still
>> reasonable for me to edit those pages?
>
> The list is already generated at the bottom of Cookbook.  To make a
> page show up there, add the following lines to the bottom (excluding
> the triple-quotes):
>
> """
> ----
> CategoryCookbook
> """
>
> I suggest adding your page to the descriptive list at the top of Cookbook too.
>
> Regards
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From simon.palmer at gmail.com  Tue Jun 10 19:53:54 2008
From: simon.palmer at gmail.com (Simon Palmer)
Date: Wed, 11 Jun 2008 00:53:54 +0100
Subject: [Numpy-discussion] Large symmetrical matrix
Message-ID: <fe40ec880806101653x28dc425v100237d6f1ec30d4@mail.gmail.com>

Hi I have a problem which involves the creation of a large square matrix
which is zero across its diagonal and symmetrical about the diagonal i.e.
m[i,j] = m[j,i] and m[i,i] = 0.  So, in fact, it is a large triangular
matrix.  I was wondering whether there is any way of easily handling a
matrix of this shape without either incurring a memory penalty or a whole
whack of proprietary code?

To get through this I have implemented a 1D array which has ((n-1)^2)/2
elements inside a wrapper class which manpulates the arguments of array
accessors with some arithmetic to return the approriate value.  To be honest
I'd love to throw this away, but I haven't yet come across a feasible
alternative.

Any ideas?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080611/8d4b66ac/attachment.html>

From robert.kern at gmail.com  Tue Jun 10 20:05:20 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 10 Jun 2008 19:05:20 -0500
Subject: [Numpy-discussion] Large symmetrical matrix
In-Reply-To: <fe40ec880806101653x28dc425v100237d6f1ec30d4@mail.gmail.com>
References: <fe40ec880806101653x28dc425v100237d6f1ec30d4@mail.gmail.com>
Message-ID: <3d375d730806101705g5dcb23fdo3d7542b27a3cc16d@mail.gmail.com>

On Tue, Jun 10, 2008 at 18:53, Simon Palmer <simon.palmer at gmail.com> wrote:
> Hi I have a problem which involves the creation of a large square matrix
> which is zero across its diagonal and symmetrical about the diagonal i.e.
> m[i,j] = m[j,i] and m[i,i] = 0.  So, in fact, it is a large triangular
> matrix.  I was wondering whether there is any way of easily handling a
> matrix of this shape without either incurring a memory penalty or a whole
> whack of proprietary code?
>
> To get through this I have implemented a 1D array which has ((n-1)^2)/2
> elements inside a wrapper class which manpulates the arguments of array
> accessors with some arithmetic to return the approriate value.  To be honest
> I'd love to throw this away, but I haven't yet come across a feasible
> alternative.
>
> Any ideas?

What operations do you want to perform on this array?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From david.huard at gmail.com  Tue Jun 10 22:49:35 2008
From: david.huard at gmail.com (David Huard)
Date: Tue, 10 Jun 2008 22:49:35 -0400
Subject: [Numpy-discussion] NumpyTest problem
In-Reply-To: <e06186140806020842o31fcd333nefa8fbf39f92cfa3@mail.gmail.com>
References: <91cf711d0806020820o17343fdfua8fa2ee9ed91f081@mail.gmail.com>
	<e06186140806020842o31fcd333nefa8fbf39f92cfa3@mail.gmail.com>
Message-ID: <91cf711d0806101949u224345a9h4543d03e34ca8961@mail.gmail.com>

Charles,

This bug appeared after your change in r5217:

Index: numpytest.py
===================================================================
--- numpytest.py        (r?vision 5216)
+++ numpytest.py        (r?vision 5217)
@@ -527,7 +527,7 @@
         all_tests = unittest.TestSuite(suite_list)
         return all_tests

-    def test(self, level=1, verbosity=1, all=False, sys_argv=[],
+    def test(self, level=1, verbosity=1, all=True, sys_argv=[],
              testcase_pattern='.*'):
         """Run Numpy module test suite with level and verbosity.

running
NumpyTest().test(all=False) works, but
NumpyTest().test(all=True) doesn't, that is, it finds 0 test.

David


2008/6/2 Charles R Harris <charlesr.harris at gmail.com>:

>
>
> On Mon, Jun 2, 2008 at 9:20 AM, David Huard <david.huard at gmail.com> wrote:
>
>> Hi,
>>
>> There are 2 problems with NumpyTest
>>
>> 1. It fails if the command is given the file name only (without a
>> directory structure)
>>
>> E.g.:
>>
>> huardda at angus:~/repos/numpy/numpy/tests$ python test_ctypeslib.py
>> Traceback (most recent call last):
>>   File "test_ctypeslib.py", line 87, in <module>
>>     NumpyTest().run()
>>   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py",
>> line 655, in run
>>     testcase_pattern=options.testcase_pattern)
>>   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py",
>> line 575, in test
>>     level, verbosity)
>>   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py",
>> line 453, in _test_suite_from_all_tests
>>     importall(this_package)
>>   File "/usr/lib64/python2.5/site-packages/numpy/testing/numpytest.py",
>> line 681, in importall
>>     for subpackage_name in os.listdir(package_dir):
>> OSError: [Errno 2] No such file or directory: ''
>> huardda at angus:~/repos/numpy/numpy/tests$
>>
>>
>>
>> 2. It doesn't find tests it used to find:
>>
>> huardda at angus:~/repos/numpy/numpy$ python tests/test_ctypeslib.py
>>
>
>
> There haven't been many changes to the tests.  Could you fool with
> numpy.test(level=10,all=0) and such to see what happens? All=1 is now the
> default.
>
> I've also seen test run some tests twice. I don't know what was up with
> that.
>
> Chuck
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080610/ac3a0292/attachment.html>

From charlesr.harris at gmail.com  Tue Jun 10 23:20:48 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 10 Jun 2008 21:20:48 -0600
Subject: [Numpy-discussion] NumpyTest problem
In-Reply-To: <91cf711d0806101949u224345a9h4543d03e34ca8961@mail.gmail.com>
References: <91cf711d0806020820o17343fdfua8fa2ee9ed91f081@mail.gmail.com>
	<e06186140806020842o31fcd333nefa8fbf39f92cfa3@mail.gmail.com>
	<91cf711d0806101949u224345a9h4543d03e34ca8961@mail.gmail.com>
Message-ID: <e06186140806102020x489f0a78s2d7465d5446e83b0@mail.gmail.com>

On Tue, Jun 10, 2008 at 8:49 PM, David Huard <david.huard at gmail.com> wrote:

> Charles,
>
> This bug appeared after your change in r5217:
>
> Index: numpytest.py
> ===================================================================
> --- numpytest.py        (r?vision 5216)
> +++ numpytest.py        (r?vision 5217)
> @@ -527,7 +527,7 @@
>          all_tests = unittest.TestSuite(suite_list)
>          return all_tests
>
> -    def test(self, level=1, verbosity=1, all=False, sys_argv=[],
> +    def test(self, level=1, verbosity=1, all=True, sys_argv=[],
>               testcase_pattern='.*'):
>          """Run Numpy module test suite with level and verbosity.
>
> running
> NumpyTest().test(all=False) works, but
> NumpyTest().test(all=True) doesn't, that is, it finds 0 test.
>
> David
>

Yep, there seems to be a bug in test somewhere. Hmm, all is supposed to be
equivalent to level > 10 (but isn't), so I wonder if there is a conflict
with the level=1 default? But since we are moving to nose...

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080610/4eb87d62/attachment.html>

From simon.palmer at gmail.com  Wed Jun 11 11:33:26 2008
From: simon.palmer at gmail.com (Simon Palmer)
Date: Wed, 11 Jun 2008 16:33:26 +0100
Subject: [Numpy-discussion] Large symmetrical matrix
In-Reply-To: <3d375d730806101705g5dcb23fdo3d7542b27a3cc16d@mail.gmail.com>
References: <fe40ec880806101653x28dc425v100237d6f1ec30d4@mail.gmail.com>
	<3d375d730806101705g5dcb23fdo3d7542b27a3cc16d@mail.gmail.com>
Message-ID: <fe40ec880806110833o5d3fcccv18706ad252b03324@mail.gmail.com>

Pretty simple.  I don't do any transformations.  It is a euclidean distance
matrix between n vectors in my data space, so I use it for lookup of minima
and I recalculate portions of it during the processing of my algorithm.

It is the single largest limitation of my code, both in terms of performance
and scalability.  A fast and efficient solution to this issue would make a
huge difference to me.

On Wed, Jun 11, 2008 at 1:05 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On Tue, Jun 10, 2008 at 18:53, Simon Palmer <simon.palmer at gmail.com>
> wrote:
> > Hi I have a problem which involves the creation of a large square matrix
> > which is zero across its diagonal and symmetrical about the diagonal i.e.
> > m[i,j] = m[j,i] and m[i,i] = 0.  So, in fact, it is a large triangular
> > matrix.  I was wondering whether there is any way of easily handling a
> > matrix of this shape without either incurring a memory penalty or a whole
> > whack of proprietary code?
> >
> > To get through this I have implemented a 1D array which has ((n-1)^2)/2
> > elements inside a wrapper class which manpulates the arguments of array
> > accessors with some arithmetic to return the approriate value.  To be
> honest
> > I'd love to throw this away, but I haven't yet come across a feasible
> > alternative.
> >
> > Any ideas?
>
> What operations do you want to perform on this array?
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080611/88611d4a/attachment.html>

From charlesr.harris at gmail.com  Wed Jun 11 11:55:31 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 11 Jun 2008 09:55:31 -0600
Subject: [Numpy-discussion] Large symmetrical matrix
In-Reply-To: <fe40ec880806110833o5d3fcccv18706ad252b03324@mail.gmail.com>
References: <fe40ec880806101653x28dc425v100237d6f1ec30d4@mail.gmail.com>
	<3d375d730806101705g5dcb23fdo3d7542b27a3cc16d@mail.gmail.com>
	<fe40ec880806110833o5d3fcccv18706ad252b03324@mail.gmail.com>
Message-ID: <e06186140806110855r19760243m2d98923ab443842b@mail.gmail.com>

On Wed, Jun 11, 2008 at 9:33 AM, Simon Palmer <simon.palmer at gmail.com>
wrote:

> Pretty simple.  I don't do any transformations.  It is a euclidean distance
> matrix between n vectors in my data space, so I use it for lookup of minima
> and I recalculate portions of it during the processing of my algorithm.
>
> It is the single largest limitation of my code, both in terms of
> performance and scalability.  A fast and efficient solution to this issue
> would make a huge difference to me.
>

So are you doing hierarchical clustering?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080611/f92f016c/attachment.html>

From simon.palmer at gmail.com  Wed Jun 11 12:47:23 2008
From: simon.palmer at gmail.com (Simon Palmer)
Date: Wed, 11 Jun 2008 17:47:23 +0100
Subject: [Numpy-discussion] Large symmetrical matrix
In-Reply-To: <e06186140806110855r19760243m2d98923ab443842b@mail.gmail.com>
References: <fe40ec880806101653x28dc425v100237d6f1ec30d4@mail.gmail.com>
	<3d375d730806101705g5dcb23fdo3d7542b27a3cc16d@mail.gmail.com>
	<fe40ec880806110833o5d3fcccv18706ad252b03324@mail.gmail.com>
	<e06186140806110855r19760243m2d98923ab443842b@mail.gmail.com>
Message-ID: <fe40ec880806110947i51488203x2f074857bd88df06@mail.gmail.com>

clustering yes, hierarchical no.

On Wed, Jun 11, 2008 at 4:55 PM, Charles R Harris <charlesr.harris at gmail.com>
wrote:

>
>
> On Wed, Jun 11, 2008 at 9:33 AM, Simon Palmer <simon.palmer at gmail.com>
> wrote:
>
>> Pretty simple.  I don't do any transformations.  It is a euclidean
>> distance matrix between n vectors in my data space, so I use it for lookup
>> of minima and I recalculate portions of it during the processing of my
>> algorithm.
>>
>> It is the single largest limitation of my code, both in terms of
>> performance and scalability.  A fast and efficient solution to this issue
>> would make a huge difference to me.
>>
>
> So are you doing hierarchical clustering?
>
> Chuck
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080611/44774bdf/attachment.html>

From peter.skomoroch at gmail.com  Wed Jun 11 13:07:11 2008
From: peter.skomoroch at gmail.com (Peter Skomoroch)
Date: Wed, 11 Jun 2008 10:07:11 -0700
Subject: [Numpy-discussion] Large symmetrical matrix
In-Reply-To: <e06186140806110855r19760243m2d98923ab443842b@mail.gmail.com>
References: <fe40ec880806101653x28dc425v100237d6f1ec30d4@mail.gmail.com>
	<3d375d730806101705g5dcb23fdo3d7542b27a3cc16d@mail.gmail.com>
	<fe40ec880806110833o5d3fcccv18706ad252b03324@mail.gmail.com>
	<e06186140806110855r19760243m2d98923ab443842b@mail.gmail.com>
Message-ID: <e4fc0d2a0806111007mcc24396m3c1618faf3547177@mail.gmail.com>

You could do something like this, where MyDIstanceFunc is written with weave
and some intelligent caching.  This only computes the upper diagonal
elements in a sparse matrix:

self.NumDatapoints = len(self.DataPoints)
self.Distances = sparse.lil_matrix((self.NumDatapoints,self.NumDatapoints))

for index, x in enumerate(self.DataPoints):
    self.Distances[index,0:index] = array(map(lambda y:self.MyDistanceFunc(
x, y ), self.DataPoints[0:index] ))

If an approximation is ok, you can save space by only calculating entries
for random samples of the rows and doing a matrix factorization like NMF on
the matrix with missing elements.  Depending on the dataset and accuracy
needed, this can be a big space savings.





On Wed, Jun 11, 2008 at 8:55 AM, Charles R Harris <charlesr.harris at gmail.com>
wrote:

>
>
> On Wed, Jun 11, 2008 at 9:33 AM, Simon Palmer <simon.palmer at gmail.com>
> wrote:
>
>> Pretty simple.  I don't do any transformations.  It is a euclidean
>> distance matrix between n vectors in my data space, so I use it for lookup
>> of minima and I recalculate portions of it during the processing of my
>> algorithm.
>>
>> It is the single largest limitation of my code, both in terms of
>> performance and scalability.  A fast and efficient solution to this issue
>> would make a huge difference to me.
>>
>
> So are you doing hierarchical clustering?
>
> Chuck
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Peter N. Skomoroch
peter.skomoroch at gmail.com
http://www.datawrangling.com
http://del.icio.us/pskomoroch
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080611/f3ba554e/attachment.html>

From bsouthey at gmail.com  Wed Jun 11 13:12:23 2008
From: bsouthey at gmail.com (Bruce Southey)
Date: Wed, 11 Jun 2008 12:12:23 -0500
Subject: [Numpy-discussion] Large symmetrical matrix
In-Reply-To: <e06186140806110855r19760243m2d98923ab443842b@mail.gmail.com>
References: <fe40ec880806101653x28dc425v100237d6f1ec30d4@mail.gmail.com>	<3d375d730806101705g5dcb23fdo3d7542b27a3cc16d@mail.gmail.com>	<fe40ec880806110833o5d3fcccv18706ad252b03324@mail.gmail.com>
	<e06186140806110855r19760243m2d98923ab443842b@mail.gmail.com>
Message-ID: <48500777.5050906@gmail.com>

Charles R Harris wrote:
>
>
> On Wed, Jun 11, 2008 at 9:33 AM, Simon Palmer <simon.palmer at gmail.com 
> <mailto:simon.palmer at gmail.com>> wrote:
>
>     Pretty simple.  I don't do any transformations.  It is a euclidean
>     distance matrix between n vectors in my data space, so I use it
>     for lookup of minima and I recalculate portions of it during the
>     processing of my algorithm. 
>
>     It is the single largest limitation of my code, both in terms of
>     performance and scalability.  A fast and efficient solution to
>     this issue would make a huge difference to me.
>
>
> So are you doing hierarchical clustering?
>
> Chuck
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   
Lapack supports symmetric matrices (http://en.wikipedia.org/wiki/LAPACK) 
but, to my knowledge, these are not implemented in SciPy (and I am not 
sure if sparse will help).  So you have the memory penalty or 
performance penalty unless you create the bindings necessary that you need.

Bruce



From kwgoodman at gmail.com  Wed Jun 11 14:31:32 2008
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 11 Jun 2008 11:31:32 -0700
Subject: [Numpy-discussion] Large symmetrical matrix
In-Reply-To: <fe40ec880806110947i51488203x2f074857bd88df06@mail.gmail.com>
References: <fe40ec880806101653x28dc425v100237d6f1ec30d4@mail.gmail.com>
	<3d375d730806101705g5dcb23fdo3d7542b27a3cc16d@mail.gmail.com>
	<fe40ec880806110833o5d3fcccv18706ad252b03324@mail.gmail.com>
	<e06186140806110855r19760243m2d98923ab443842b@mail.gmail.com>
	<fe40ec880806110947i51488203x2f074857bd88df06@mail.gmail.com>
Message-ID: <f4f93d420806111131l2dfb3c32wa57ad7b4de0be4cc@mail.gmail.com>

On Wed, Jun 11, 2008 at 9:47 AM, Simon Palmer <simon.palmer at gmail.com> wrote:
> clustering yes, hierarchical no.

I think scipy-cluster [1] only stores the upper triangle of the
distance matrix.

[1] http://code.google.com/p/scipy-cluster/


From peridot.faceted at gmail.com  Wed Jun 11 14:34:50 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 11 Jun 2008 12:34:50 -0600
Subject: [Numpy-discussion] Large symmetrical matrix
In-Reply-To: <fe40ec880806101653x28dc425v100237d6f1ec30d4@mail.gmail.com>
References: <fe40ec880806101653x28dc425v100237d6f1ec30d4@mail.gmail.com>
Message-ID: <ce557a360806111134j3b7ebfdfm57fa6a8a0784a2fe@mail.gmail.com>

2008/6/10 Simon Palmer <simon.palmer at gmail.com>:
> Hi I have a problem which involves the creation of a large square matrix
> which is zero across its diagonal and symmetrical about the diagonal i.e.
> m[i,j] = m[j,i] and m[i,i] = 0.  So, in fact, it is a large triangular
> matrix.  I was wondering whether there is any way of easily handling a
> matrix of this shape without either incurring a memory penalty or a whole
> whack of proprietary code?
>
> To get through this I have implemented a 1D array which has ((n-1)^2)/2
> elements inside a wrapper class which manpulates the arguments of array
> accessors with some arithmetic to return the approriate value.  To be honest
> I'd love to throw this away, but I haven't yet come across a feasible
> alternative.
>
> Any ideas?

Well, there's a risk of shooting yourself in the foot, but here's a
way to get at the elements quickly and conveniently: store the
coefficients in a 1D array (as you do already). Then if your matrix is
M by M, element (i,j) is at position (M-2)*j+i-1, assuming i<j. Of
course, you can simply use this expression every time you want to look
an element up, as you do now, but that's no fun. So now create a
"view" A of this array having offset -1, shape (M,M), and strides
(1,(M-2)). (This kind of weird view can be created with the ndarray
constructor.) Then A[i,j] addresses the right element. A[j,i] is of
course totally wrong. But as long as you avoid i<j, you can use all
numpy's magical indexing tricks. Unfortunately you can't really use
ufuncs, since they'll waste time operating on the useless lower half.

Personally, I would live with the factor-of-two memory hit.

Anne


From simon.palmer at gmail.com  Wed Jun 11 16:14:38 2008
From: simon.palmer at gmail.com (Simon Palmer)
Date: Wed, 11 Jun 2008 21:14:38 +0100
Subject: [Numpy-discussion] Large symmetrical matrix
In-Reply-To: <ce557a360806111134j3b7ebfdfm57fa6a8a0784a2fe@mail.gmail.com>
References: <fe40ec880806101653x28dc425v100237d6f1ec30d4@mail.gmail.com>
	<ce557a360806111134j3b7ebfdfm57fa6a8a0784a2fe@mail.gmail.com>
Message-ID: <fe40ec880806111314u7956169ejcb0a659aafcf7872@mail.gmail.com>

Actually that is very close to what I currently do, which is why I want to
throw it away.

The factor of two memory hit is not really the problem, it is the scaling
O(N^2) which is the limitation

I suspect I may end up using the 1-d array plus arithmetic as it is at least
efficient for retrieval.  I have just started working on a stack which sits
between the vector array and a lookup for a minimum, I think that could
improve performance a great deal.  Next up is a disk based memory map of
some kind which might free me from the constraints of physical memory.
Anyone know of an implementation of array() which automatically reads/writes
to disk?

To answer another suggestion, I'm reluctant to sample not because it is not
a good approach but rather because the commercial application at the end of
this is much harder to explain, justify and sell if I have to include
reasoning for not using all the data.  As scientists this is obvious, as a
lay person, and particularly a business person, it appears to be madness.
The fewer leaps of faith necessary the better.

On Wed, Jun 11, 2008 at 7:34 PM, Anne Archibald <peridot.faceted at gmail.com>
wrote:

> 2008/6/10 Simon Palmer <simon.palmer at gmail.com>:
> > Hi I have a problem which involves the creation of a large square matrix
> > which is zero across its diagonal and symmetrical about the diagonal i.e.
> > m[i,j] = m[j,i] and m[i,i] = 0.  So, in fact, it is a large triangular
> > matrix.  I was wondering whether there is any way of easily handling a
> > matrix of this shape without either incurring a memory penalty or a whole
> > whack of proprietary code?
> >
> > To get through this I have implemented a 1D array which has ((n-1)^2)/2
> > elements inside a wrapper class which manpulates the arguments of array
> > accessors with some arithmetic to return the approriate value.  To be
> honest
> > I'd love to throw this away, but I haven't yet come across a feasible
> > alternative.
> >
> > Any ideas?
>
> Well, there's a risk of shooting yourself in the foot, but here's a
> way to get at the elements quickly and conveniently: store the
> coefficients in a 1D array (as you do already). Then if your matrix is
> M by M, element (i,j) is at position (M-2)*j+i-1, assuming i<j. Of
> course, you can simply use this expression every time you want to look
> an element up, as you do now, but that's no fun. So now create a
> "view" A of this array having offset -1, shape (M,M), and strides
> (1,(M-2)). (This kind of weird view can be created with the ndarray
> constructor.) Then A[i,j] addresses the right element. A[j,i] is of
> course totally wrong. But as long as you avoid i<j, you can use all
> numpy's magical indexing tricks. Unfortunately you can't really use
> ufuncs, since they'll waste time operating on the useless lower half.
>
> Personally, I would live with the factor-of-two memory hit.
>
> Anne
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080611/600f9f4f/attachment.html>

From hoytak at gmail.com  Wed Jun 11 16:26:34 2008
From: hoytak at gmail.com (Hoyt Koepke)
Date: Wed, 11 Jun 2008 13:26:34 -0700
Subject: [Numpy-discussion] Large symmetrical matrix
In-Reply-To: <fe40ec880806111314u7956169ejcb0a659aafcf7872@mail.gmail.com>
References: <fe40ec880806101653x28dc425v100237d6f1ec30d4@mail.gmail.com>
	<ce557a360806111134j3b7ebfdfm57fa6a8a0784a2fe@mail.gmail.com>
	<fe40ec880806111314u7956169ejcb0a659aafcf7872@mail.gmail.com>
Message-ID: <4db580fd0806111326l4d2382cfse58741f04e88e6c4@mail.gmail.com>

> The factor of two memory hit is not really the problem, it is the scaling
> O(N^2) which is the limitation

Have you thought about using kd-trees or similar structures?  If you
are only concerned about the minimum distances between two points,
that is the ideal data structure.  It would reduce your running time /
memory usage for generating / using the data from O(n^2) to O(n log
n), with a O(log n) lookup time for the closest distance to each
point.

Without doing something like that, looking at the distance between
each pair of points is \Omega(n^2).

--Hoyt

-- 
+++++++++++++++++++++++++++++++++++
Hoyt Koepke
UBC Department of Computer Science
http://www.cs.ubc.ca/~hoytak/
hoytak at gmail.com
+++++++++++++++++++++++++++++++++++


From robert.kern at gmail.com  Wed Jun 11 16:34:12 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 11 Jun 2008 15:34:12 -0500
Subject: [Numpy-discussion] Large symmetrical matrix
In-Reply-To: <fe40ec880806111314u7956169ejcb0a659aafcf7872@mail.gmail.com>
References: <fe40ec880806101653x28dc425v100237d6f1ec30d4@mail.gmail.com>
	<ce557a360806111134j3b7ebfdfm57fa6a8a0784a2fe@mail.gmail.com>
	<fe40ec880806111314u7956169ejcb0a659aafcf7872@mail.gmail.com>
Message-ID: <3d375d730806111334w710eb761of2e40598233feca5@mail.gmail.com>

On Wed, Jun 11, 2008 at 15:14, Simon Palmer <simon.palmer at gmail.com> wrote:
> Actually that is very close to what I currently do, which is why I want to
> throw it away.
>
> The factor of two memory hit is not really the problem, it is the scaling
> O(N^2) which is the limitation
>
> I suspect I may end up using the 1-d array plus arithmetic as it is at least
> efficient for retrieval.  I have just started working on a stack which sits
> between the vector array and a lookup for a minimum, I think that could
> improve performance a great deal.  Next up is a disk based memory map of
> some kind which might free me from the constraints of physical memory.
> Anyone know of an implementation of array() which automatically reads/writes
> to disk?

Well, if you have a decent OS, you should already be free of the
constraints of physical memory. The OS should swap data in and out as
necessary. But of course, we do have mmap support with numpy.memmap.
If you are using numpy 1.1.0, I would also recommend using
numpy.lib.format.open_memmap() to use the new NPY file format in order
to (hopefully) future-proof your data. Be careful with using mmap'ed
arrays, though. Disk seek time could kill your performance.

If this is the big hotspot in your program, you might want to consider
coding up exactly the behavior you need in C. It doesn't appea

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From barrywark at gmail.com  Wed Jun 11 18:00:33 2008
From: barrywark at gmail.com (Barry Wark)
Date: Wed, 11 Jun 2008 15:00:33 -0700
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <9457e7c80806100257t1a6ffa87m121477814cbba17a@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>
	<1d36917a0806081618g5875fa3bt1bfe5f379e9a797a@mail.gmail.com>
	<ce557a360806091607v6a777a42id75102dd63b50e62@mail.gmail.com>
	<1d36917a0806091853g3390a5e4m2ca1fa08b93f2857@mail.gmail.com>
	<9457e7c80806100257t1a6ffa87m121477814cbba17a@mail.gmail.com>
Message-ID: <cd7634ce0806111500k22c625a2u90e922faea529da@mail.gmail.com>

Could this also be added as a comment in the  module docstring for
test modules? It seems that anyone hacking on test code would be most
likely to see it there...?

Barry

On Tue, Jun 10, 2008 at 2:57 AM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2008/6/10 Alan McIntyre <alan.mcintyre at gmail.com>:
>> Is the stuff Robert pointed out on a wiki page somewhere? It would be
>> nice to have a "Welcome noob NumPy developer, here's how to do
>> NumPy-specific development things," page. There may be such a page,
>> but I just haven't stumbled across it yet.
>
> We could add this info the the `numpy` docstring; that way new users
> will have it available without having to search the web.
>
> http://sd-2116.dedibox.fr/pydocweb/doc/numpy/
>
> Regards
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From robert.kern at gmail.com  Wed Jun 11 18:09:20 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 11 Jun 2008 17:09:20 -0500
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <9457e7c80806100257t1a6ffa87m121477814cbba17a@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>
	<1d36917a0806081618g5875fa3bt1bfe5f379e9a797a@mail.gmail.com>
	<ce557a360806091607v6a777a42id75102dd63b50e62@mail.gmail.com>
	<1d36917a0806091853g3390a5e4m2ca1fa08b93f2857@mail.gmail.com>
	<9457e7c80806100257t1a6ffa87m121477814cbba17a@mail.gmail.com>
Message-ID: <3d375d730806111509v80d545asf473b6b44930cb38@mail.gmail.com>

On Tue, Jun 10, 2008 at 04:57, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2008/6/10 Alan McIntyre <alan.mcintyre at gmail.com>:
>> Is the stuff Robert pointed out on a wiki page somewhere? It would be
>> nice to have a "Welcome noob NumPy developer, here's how to do
>> NumPy-specific development things," page. There may be such a page,
>> but I just haven't stumbled across it yet.
>
> We could add this info the the `numpy` docstring; that way new users
> will have it available without having to search the web.

Build instructions don't really belong in docstrings. Put it in the
README.txt or DEV_README.txt.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco

From ondrej at certik.cz  Wed Jun 11 19:43:26 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 12 Jun 2008 01:43:26 +0200
Subject: [Numpy-discussion] array vs matrix
In-Reply-To: <ce557a360806071854m8767bfbkd27afe2128ef52f9@mail.gmail.com>
References: <85b5c3130806071237g16582e6cr2a122ce8c38886@mail.gmail.com>
	<3d375d730806071244o7d19bbc4yf7e8f2b1477e7e6b@mail.gmail.com>
	<ce557a360806071854m8767bfbkd27afe2128ef52f9@mail.gmail.com>
Message-ID: <85b5c3130806111643n144bb4ebs3b7c8856d042f740@mail.gmail.com>

On Sun, Jun 8, 2008 at 3:54 AM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> 2008/6/7 Robert Kern <robert.kern at gmail.com>:
>> On Sat, Jun 7, 2008 at 14:37, Ondrej Certik <ondrej at certik.cz> wrote:
>>> Hi,
>>>
>>> what is the current plan with array and matrix with regard of calculating
>>>
>>> sin(A)
>>>
>>> ? I.e. elementwise vs sin(A) = Q*sin(D)*Q^T? Is the current approach
>>> (elementwise for array and Q*sin(D)*Q^T for matrix) the way to go?
>>
>> I don't believe we intend to make numpy.matrix any more featureful. I
>> don't think it's a good idea for you to base sympy.Matrix's
>> capabilities in lock-step with numpy.matrix, though. There are very
>> different constraints at work. Please, do what you think is best for
>> sympy.Matrix.
>
> Let me elaborate somewhat:
>
> We recently ran across some painful quirks in numpy's handling of
> matrices, and they spawned massive discussion. As it stands now, there
> is significant interest in reimplementing the matrix object from
> scratch, with different behaviour. So emulating its current behaviour
> is not a win.
>
> For consistency, it makes a certain amount of sense to have sin(A)
> compute a matrix sine, since A**n computes a matrix power. But looking
> at the matrix exponential, I see that we have several implementations,
> none of which is satisfactory for all matrices. I would expect the
> matrix sine to be similar - particularly when faced with complex
> matrices - so perhaps needing an explicit matrix sine is a good thing?
>
> Also worth noting is that this problem can be evaded with namespaces;
> matrix sin could be scipy.matrixmath.sin, abbreviated perhaps to
> mm.sin, as opposed to np.sin.

I see. Thanks Robert and Anne for the replies. That's all I needed to know.

Ondrej


From alan.mcintyre at gmail.com  Wed Jun 11 19:51:17 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Wed, 11 Jun 2008 19:51:17 -0400
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <3d375d730806111509v80d545asf473b6b44930cb38@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>
	<1d36917a0806081618g5875fa3bt1bfe5f379e9a797a@mail.gmail.com>
	<ce557a360806091607v6a777a42id75102dd63b50e62@mail.gmail.com>
	<1d36917a0806091853g3390a5e4m2ca1fa08b93f2857@mail.gmail.com>
	<9457e7c80806100257t1a6ffa87m121477814cbba17a@mail.gmail.com>
	<3d375d730806111509v80d545asf473b6b44930cb38@mail.gmail.com>
Message-ID: <1d36917a0806111651l17f9e4e8pf121a109f0404e26@mail.gmail.com>

On Wed, Jun 11, 2008 at 6:09 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Tue, Jun 10, 2008 at 04:57, St?fan van der Walt <stefan at sun.ac.za> wrote:
>> 2008/6/10 Alan McIntyre <alan.mcintyre at gmail.com>:
>>> Is the stuff Robert pointed out on a wiki page somewhere? It would be
>>> nice to have a "Welcome noob NumPy developer, here's how to do
>>> NumPy-specific development things," page. There may be such a page,
>>> but I just haven't stumbled across it yet.
>>
>> We could add this info the the `numpy` docstring; that way new users
>> will have it available without having to search the web.
>
> Build instructions don't really belong in docstrings. Put it in the
> README.txt or DEV_README.txt.

I agree; I'm far more likely to look at the READMEs than in module
docstrings for build/test instructions.


From stefan at sun.ac.za  Wed Jun 11 20:51:51 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 12 Jun 2008 02:51:51 +0200
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <3d375d730806111509v80d545asf473b6b44930cb38@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>
	<1d36917a0806081618g5875fa3bt1bfe5f379e9a797a@mail.gmail.com>
	<ce557a360806091607v6a777a42id75102dd63b50e62@mail.gmail.com>
	<1d36917a0806091853g3390a5e4m2ca1fa08b93f2857@mail.gmail.com>
	<9457e7c80806100257t1a6ffa87m121477814cbba17a@mail.gmail.com>
	<3d375d730806111509v80d545asf473b6b44930cb38@mail.gmail.com>
Message-ID: <9457e7c80806111751h401ee5eegb9953b4b8b923390@mail.gmail.com>

2008/6/12 Robert Kern <robert.kern at gmail.com>:
> On Tue, Jun 10, 2008 at 04:57, St?fan van der Walt <stefan at sun.ac.za> wrote:
>> 2008/6/10 Alan McIntyre <alan.mcintyre at gmail.com>:
>>> Is the stuff Robert pointed out on a wiki page somewhere? It would be
>>> nice to have a "Welcome noob NumPy developer, here's how to do
>>> NumPy-specific development things," page. There may be such a page,
>>> but I just haven't stumbled across it yet.
>>
>> We could add this info the the `numpy` docstring; that way new users
>> will have it available without having to search the web.
>
> Build instructions don't really belong in docstrings. Put it in the
> README.txt or DEV_README.txt.

No, but running the tests is something every user should do before
using NumPy.  Unless you object, I'll add that bit to the docstring.

Cheers
St?fan


From robert.kern at gmail.com  Wed Jun 11 21:52:45 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 11 Jun 2008 20:52:45 -0500
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <9457e7c80806111751h401ee5eegb9953b4b8b923390@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>
	<1d36917a0806081618g5875fa3bt1bfe5f379e9a797a@mail.gmail.com>
	<ce557a360806091607v6a777a42id75102dd63b50e62@mail.gmail.com>
	<1d36917a0806091853g3390a5e4m2ca1fa08b93f2857@mail.gmail.com>
	<9457e7c80806100257t1a6ffa87m121477814cbba17a@mail.gmail.com>
	<3d375d730806111509v80d545asf473b6b44930cb38@mail.gmail.com>
	<9457e7c80806111751h401ee5eegb9953b4b8b923390@mail.gmail.com>
Message-ID: <3d375d730806111852n462dd080se588bc68a782bac@mail.gmail.com>

On Wed, Jun 11, 2008 at 19:51, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2008/6/12 Robert Kern <robert.kern at gmail.com>:
>> On Tue, Jun 10, 2008 at 04:57, St?fan van der Walt <stefan at sun.ac.za> wrote:
>>> 2008/6/10 Alan McIntyre <alan.mcintyre at gmail.com>:
>>>> Is the stuff Robert pointed out on a wiki page somewhere? It would be
>>>> nice to have a "Welcome noob NumPy developer, here's how to do
>>>> NumPy-specific development things," page. There may be such a page,
>>>> but I just haven't stumbled across it yet.
>>>
>>> We could add this info the the `numpy` docstring; that way new users
>>> will have it available without having to search the web.
>>
>> Build instructions don't really belong in docstrings. Put it in the
>> README.txt or DEV_README.txt.
>
> No, but running the tests is something every user should do before
> using NumPy.

Right, which means README.txt, not something you only see after you've
installed the thing. But what text exactly are you talking about?
Nothing I wrote about inplace builds (which is I *thought* was under
discussion) are relevant to "running the tests" in general but are
targeted specifically at developers.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco

From david at ar.media.kyoto-u.ac.jp  Thu Jun 12 00:35:45 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Jun 2008 13:35:45 +0900
Subject: [Numpy-discussion] [numscons] Would having two mandatory scons
 script per package be acceptable ?
Message-ID: <4850A7A1.2070404@ar.media.kyoto-u.ac.jp>

Hi,

    Short story: it would make my life simpler to have two mandatory 
scons scripts per package instead of only one. Long story: for some 
reasons linked to scons' notion of build directory, having two scons 
scripts files per package (instead of one) would simplify quite a bit 
some parts of numscons, as well as simplifying the addition of custom 
builders (for swig, cython, etc...) by other people. One of the two 
files would basically be exactly the same as the current one, and the 
other one independent of the package (always the same 3 lines). This is 
a bit ugly, but less ugly than the actual contortions I have to do to 
make scons and distutils play nice together.
    Is there any strong feeling against this ?

    cheers,

    David


From robert.kern at gmail.com  Thu Jun 12 01:02:06 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Jun 2008 00:02:06 -0500
Subject: [Numpy-discussion] [numscons] Would having two mandatory scons
	script per package be acceptable ?
In-Reply-To: <4850A7A1.2070404@ar.media.kyoto-u.ac.jp>
References: <4850A7A1.2070404@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730806112202t2770d89frb76d73314209a264@mail.gmail.com>

On Wed, Jun 11, 2008 at 23:35, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Hi,
>
>    Short story: it would make my life simpler to have two mandatory
> scons scripts per package instead of only one. Long story: for some
> reasons linked to scons' notion of build directory, having two scons
> scripts files per package (instead of one) would simplify quite a bit
> some parts of numscons, as well as simplifying the addition of custom
> builders (for swig, cython, etc...) by other people. One of the two
> files would basically be exactly the same as the current one, and the
> other one independent of the package (always the same 3 lines). This is
> a bit ugly, but less ugly than the actual contortions I have to do to
> make scons and distutils play nice together.
>    Is there any strong feeling against this ?

Can you give me a longer story? What are the three lines? Why are they
necessary in a separate, boilerplate file?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Thu Jun 12 01:08:32 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Jun 2008 14:08:32 +0900
Subject: [Numpy-discussion] [numscons] Would having two mandatory scons
 script per package be acceptable ?
In-Reply-To: <3d375d730806112202t2770d89frb76d73314209a264@mail.gmail.com>
References: <4850A7A1.2070404@ar.media.kyoto-u.ac.jp>
	<3d375d730806112202t2770d89frb76d73314209a264@mail.gmail.com>
Message-ID: <4850AF50.1020003@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
>
> Can you give me a longer story? What are the three lines? Why are they
> necessary in a separate, boilerplate file?

Sure: scons builds everything from the sources in the source directory, 
and build everything in the build directory.  By default, both 
directories are the same as the directory where the scons script is.

Now, this is of course unacceptable to use with distutils, and scons has 
this notion of build directory to change this: depending on where the 
scons script is called in the tree, I change the source and build 
directories accordingly. But for this to work, you have to change how to 
refer to source and targets in builders (scons objects to build things 
like object code and libraries). Again, unacceptable, so I did remove 
most of those in the scons scripts using severals 'tricks' in numscons. 
But the more I did those tricks, the more I realized it was fragile, and 
could break in obscure cases.

The recommended way to do it in scons is to use a second scons script: 
http://www.scons.org/wiki/UnderstandingBuildDir. The 'problem' is also 
linked to the fact that instead of building everything with one scons 
project, I have one scons project per package (because that's how 
distutils worked, in particular, and for other reasons I can develop if 
you want). I asked about doing the same thing as scons does without two 
files, but this seems difficult (I was told it is 'magic').

http://scons.tigris.org/servlets/ReadMsg?list=users&msgNo=13642

cheers,

David






From robert.kern at gmail.com  Thu Jun 12 01:39:46 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Jun 2008 00:39:46 -0500
Subject: [Numpy-discussion] [numscons] Would having two mandatory scons
	script per package be acceptable ?
In-Reply-To: <4850AF50.1020003@ar.media.kyoto-u.ac.jp>
References: <4850A7A1.2070404@ar.media.kyoto-u.ac.jp>
	<3d375d730806112202t2770d89frb76d73314209a264@mail.gmail.com>
	<4850AF50.1020003@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730806112239h6acbba18v850a30afd60c9b7b@mail.gmail.com>

On Thu, Jun 12, 2008 at 00:08, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Robert Kern wrote:
>>
>> Can you give me a longer story? What are the three lines? Why are they
>> necessary in a separate, boilerplate file?
>
> Sure: scons builds everything from the sources in the source directory,
> and build everything in the build directory.  By default, both
> directories are the same as the directory where the scons script is.
>
> Now, this is of course unacceptable to use with distutils, and scons has
> this notion of build directory to change this: depending on where the
> scons script is called in the tree, I change the source and build
> directories accordingly. But for this to work, you have to change how to
> refer to source and targets in builders (scons objects to build things
> like object code and libraries). Again, unacceptable, so I did remove
> most of those in the scons scripts using severals 'tricks' in numscons.
> But the more I did those tricks, the more I realized it was fragile, and
> could break in obscure cases.
>
> The recommended way to do it in scons is to use a second scons script:
> http://www.scons.org/wiki/UnderstandingBuildDir. The 'problem' is also
> linked to the fact that instead of building everything with one scons
> project, I have one scons project per package (because that's how
> distutils worked, in particular, and for other reasons I can develop if
> you want). I asked about doing the same thing as scons does without two
> files, but this seems difficult (I was told it is 'magic').
>
> http://scons.tigris.org/servlets/ReadMsg?list=users&msgNo=13642

Works for me.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Thu Jun 12 01:28:35 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Jun 2008 14:28:35 +0900
Subject: [Numpy-discussion] [numscons] Would having two mandatory scons
 script per package be acceptable ?
In-Reply-To: <3d375d730806112239h6acbba18v850a30afd60c9b7b@mail.gmail.com>
References: <4850A7A1.2070404@ar.media.kyoto-u.ac.jp>	<3d375d730806112202t2770d89frb76d73314209a264@mail.gmail.com>	<4850AF50.1020003@ar.media.kyoto-u.ac.jp>
	<3d375d730806112239h6acbba18v850a30afd60c9b7b@mail.gmail.com>
Message-ID: <4850B403.1000401@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
> Works for me.
>   

Do you mean having two files/packages is ok ?

cheers,

David



From robert.kern at gmail.com  Thu Jun 12 01:46:54 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Jun 2008 00:46:54 -0500
Subject: [Numpy-discussion] [numscons] Would having two mandatory scons
	script per package be acceptable ?
In-Reply-To: <4850B403.1000401@ar.media.kyoto-u.ac.jp>
References: <4850A7A1.2070404@ar.media.kyoto-u.ac.jp>
	<3d375d730806112202t2770d89frb76d73314209a264@mail.gmail.com>
	<4850AF50.1020003@ar.media.kyoto-u.ac.jp>
	<3d375d730806112239h6acbba18v850a30afd60c9b7b@mail.gmail.com>
	<4850B403.1000401@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730806112246h1944d7ckc2f6241bda3b3e7b@mail.gmail.com>

On Thu, Jun 12, 2008 at 00:28, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Robert Kern wrote:
>> Works for me.
>
> Do you mean having two files/packages is ok ?

Yes. You had me at "The recommended way to do it in scons ...."

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Thu Jun 12 01:35:50 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Jun 2008 14:35:50 +0900
Subject: [Numpy-discussion] [numscons] numscons build for numpy will be
	broken for a few hours
Message-ID: <4850B5B6.6070300@ar.media.kyoto-u.ac.jp>

Hi,

    Just to mention that building numpy and scipy with numscons will be 
broken for a few hours, the time to change some organizations.

    Sorry for the trouble,

    David


From matthieu.brucher at gmail.com  Thu Jun 12 02:00:23 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 12 Jun 2008 08:00:23 +0200
Subject: [Numpy-discussion] [numscons] Would having two mandatory scons
	script per package be acceptable ?
In-Reply-To: <4850AF50.1020003@ar.media.kyoto-u.ac.jp>
References: <4850A7A1.2070404@ar.media.kyoto-u.ac.jp>
	<3d375d730806112202t2770d89frb76d73314209a264@mail.gmail.com>
	<4850AF50.1020003@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0806112300h3c713a5ft135241f0c0caac1c@mail.gmail.com>

2008/6/12 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>
> Robert Kern wrote:
> >
> > Can you give me a longer story? What are the three lines? Why are they
> > necessary in a separate, boilerplate file?
>
> Sure: scons builds everything from the sources in the source directory,
> and build everything in the build directory.  By default, both
> directories are the same as the directory where the scons script is.
>
> Now, this is of course unacceptable to use with distutils, and scons has
> this notion of build directory to change this: depending on where the
> scons script is called in the tree, I change the source and build
> directories accordingly. But for this to work, you have to change how to
> refer to source and targets in builders (scons objects to build things
> like object code and libraries). Again, unacceptable, so I did remove
> most of those in the scons scripts using severals 'tricks' in numscons.
> But the more I did those tricks, the more I realized it was fragile, and
> could break in obscure cases.

Hi,

I don't understand your issues. We build everything in a variant dir
(what superseds a build dir in SCons 0.98), and we don't have troubles
indicating where are the sources and where are the targets as SCons
makes hard links between every source file and the build dir. Perhaps
I didn't understand exactly what you need, but it seems that just
calling the sub SConstruct scripts with :

env.SConscript( 'numpy/SConstruct',
   variant_dir = "$BUILDROOT",
                   )

should be enough so that you don't have to do something more with the BUILDROOT.

Matthieu
--
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher


From david at ar.media.kyoto-u.ac.jp  Thu Jun 12 05:23:50 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Jun 2008 18:23:50 +0900
Subject: [Numpy-discussion] [numscons] Would having two mandatory scons
 script per package be acceptable ?
In-Reply-To: <e76aa17f0806112300h3c713a5ft135241f0c0caac1c@mail.gmail.com>
References: <4850A7A1.2070404@ar.media.kyoto-u.ac.jp>	<3d375d730806112202t2770d89frb76d73314209a264@mail.gmail.com>	<4850AF50.1020003@ar.media.kyoto-u.ac.jp>
	<e76aa17f0806112300h3c713a5ft135241f0c0caac1c@mail.gmail.com>
Message-ID: <4850EB26.1040906@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
> I don't understand your issues. We build everything in a variant dir
> (what superseds a build dir in SCons 0.98), and we don't have troubles
> indicating where are the sources and where are the targets as SCons
> makes hard links between every source file and the build dir. Perhaps
> I didn't understand exactly what you need, but it seems that just
> calling the sub SConstruct scripts with :
>
> env.SConscript( 'numpy/SConstruct',
>    variant_dir = "$BUILDROOT",
>                    )
>
> should be enough so that you don't have to do something more with the BUILDROOT.
>   

Yes, that's basically what I am doing now (modulo some other warts 
related to the build_dir thing; I noticed a few bugs when you want to 
put everything, including sconsign and configuration files into the 
build dir). But you do need two scons scripts files for each package 
(don't forget that I am calling a new scons process for every 
subpackage, so I cannot have a top scons script to set up everything).

cheers,

David


From matthieu.brucher at gmail.com  Thu Jun 12 06:40:20 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 12 Jun 2008 12:40:20 +0200
Subject: [Numpy-discussion] [numscons] Would having two mandatory scons
	script per package be acceptable ?
In-Reply-To: <4850EB26.1040906@ar.media.kyoto-u.ac.jp>
References: <4850A7A1.2070404@ar.media.kyoto-u.ac.jp>
	<3d375d730806112202t2770d89frb76d73314209a264@mail.gmail.com>
	<4850AF50.1020003@ar.media.kyoto-u.ac.jp>
	<e76aa17f0806112300h3c713a5ft135241f0c0caac1c@mail.gmail.com>
	<4850EB26.1040906@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0806120340o6f9d5c69sc1753bfac2ddd110@mail.gmail.com>

>
> But you do need two scons scripts files for each package
> (don't forget that I am calling a new scons process for every
> subpackage, so I cannot have a top scons script to set up everything).
>

That was what I forgot about the differences between numpy and my project (I
also have two scons script at the top level) ;).

Matthieu
-- 
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080612/5c1e8060/attachment.html>

From travis at enthought.com  Fri Jun 13 08:35:58 2008
From: travis at enthought.com (Travis Vaught)
Date: Fri, 13 Jun 2008 07:35:58 -0500
Subject: [Numpy-discussion] EuroSciPy Early Registration Reminder
Message-ID: <C534D646-B858-41A2-B410-2168340F3AE8@enthought.com>

Greetings,

This is a friendly reminder that the Early Registration deadline for  
the EuroSciPy Conference is June 15th.  If you're interested in  
attending, but have not yet registered, please visit:

http://www.scipy.org/EuroSciPy2008

The talks schedule is also now available there.

Also, the keynote speaker this year will be Travis Oliphant, the  
primary author of the recent NumPy rewrite.  For those doing  
scientific computing using Python, this is a conference you'll not  
want to miss.

See you there.

Travis


From gael.varoquaux at normalesup.org  Fri Jun 13 09:21:11 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 13 Jun 2008 15:21:11 +0200
Subject: [Numpy-discussion] Switching to nose test framework
	(was:	NumpyTest problem)
In-Reply-To: <9457e7c80806111751h401ee5eegb9953b4b8b923390@mail.gmail.com>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>
	<1d36917a0806081618g5875fa3bt1bfe5f379e9a797a@mail.gmail.com>
	<ce557a360806091607v6a777a42id75102dd63b50e62@mail.gmail.com>
	<1d36917a0806091853g3390a5e4m2ca1fa08b93f2857@mail.gmail.com>
	<9457e7c80806100257t1a6ffa87m121477814cbba17a@mail.gmail.com>
	<3d375d730806111509v80d545asf473b6b44930cb38@mail.gmail.com>
	<9457e7c80806111751h401ee5eegb9953b4b8b923390@mail.gmail.com>
Message-ID: <20080613132111.GF3573@phare.normalesup.org>

On Thu, Jun 12, 2008 at 02:51:51AM +0200, St?fan van der Walt wrote:
> > Build instructions don't really belong in docstrings. Put it in the
> > README.txt or DEV_README.txt.

> No, but running the tests is something every user should do before
> using NumPy.  Unless you object, I'll add that bit to the docstring.

I don't agrre. My users have no clue what tests are. They have installed
a binary numpy, and can not remotely understand what the output of the
test suite would be. I don't think they should have to run test before
running numpy, they should have to if they run into strange problems, in
which case I expect them to be instructed to do so. It is true that good
documentation to run the tests when told so is useful, but it shouldn't
jump at your face.

Hell, hopefully we will soon have medical doctors (with no special
training) running numpy -- maybe we already do have!

Ga?l


From alan at ajackson.org  Fri Jun 13 15:34:35 2008
From: alan at ajackson.org (Alan Jackson)
Date: Fri, 13 Jun 2008 14:34:35 -0500
Subject: [Numpy-discussion] Switching to nose test framework (was:
 NumpyTest problem)
In-Reply-To: <20080613132111.GF3573@phare.normalesup.org>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>
	<1d36917a0806081618g5875fa3bt1bfe5f379e9a797a@mail.gmail.com>
	<ce557a360806091607v6a777a42id75102dd63b50e62@mail.gmail.com>
	<1d36917a0806091853g3390a5e4m2ca1fa08b93f2857@mail.gmail.com>
	<9457e7c80806100257t1a6ffa87m121477814cbba17a@mail.gmail.com>
	<3d375d730806111509v80d545asf473b6b44930cb38@mail.gmail.com>
	<9457e7c80806111751h401ee5eegb9953b4b8b923390@mail.gmail.com>
	<20080613132111.GF3573@phare.normalesup.org>
Message-ID: <20080613143435.177cc219@ajackson.org>

On Fri, 13 Jun 2008 15:21:11 +0200
Gael Varoquaux <gael.varoquaux at normalesup.org> wrote:

> On Thu, Jun 12, 2008 at 02:51:51AM +0200, St?fan van der Walt wrote:
> > > Build instructions don't really belong in docstrings. Put it in the
> > > README.txt or DEV_README.txt.
> 
> > No, but running the tests is something every user should do before
> > using NumPy.  Unless you object, I'll add that bit to the docstring.
> 
> I don't agrre. My users have no clue what tests are. They have installed
> a binary numpy, and can not remotely understand what the output of the
> test suite would be. I don't think they should have to run test before
> running numpy, they should have to if they run into strange problems, in
> which case I expect them to be instructed to do so. It is true that good
> documentation to run the tests when told so is useful, but it shouldn't
> jump at your face.
> 

Back in my perl and CPAN days, I really liked the paradigm where 
when doing an install, it was expected that you would run
make
make test
make install

Sometimes you couldn't do the install step without having run test. 

If it all works, great - don't even bother the installer. If a test fails, well
install time is a good time to know that something is broken. It could be
pretty ugly later on.


-- 
-----------------------------------------------------------------------
| Alan K. Jackson            | To see a World in a Grain of Sand      |
| alan at ajackson.org          | And a Heaven in a Wild Flower,         |
| www.ajackson.org           | Hold Infinity in the palm of your hand |
| Houston, Texas             | And Eternity in an hour. - Blake       |
-----------------------------------------------------------------------


From gael.varoquaux at normalesup.org  Sat Jun 14 13:20:44 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sat, 14 Jun 2008 19:20:44 +0200
Subject: [Numpy-discussion] Alex Martelli giving the SciPy 2008 Keynote
Message-ID: <20080614172043.GA31186@phare.normalesup.org>

On behalf of the SciPy2008 organizing committee, I am happy to announce
that the Keynote at the conference will be given by Alex Martelli.

It is a pleasure for us to receive Alex. He currently works as "Uber Tech
Leader" at Google and is the author of two of the Python classics: 
"Python in a nutshell" and the "Python CookBook". Alex graduated in
electronic engineering from the university of Bologna and worked in chip
design first for Texas Instrument, and later for IBM Research. During the
8 years he spent at IBM, he gradually shifted from hardware design to
software development while winning three Outstanding Technical
Achievement Awards. Then he joined think3 inc., and Italian CAD company,
as Senior Software Consultant where he developed libraries, network
protocols, GUI engines, event frameworks, and web access frontends. After
12 years at think3, he worked for 3 years as a freelance consultant,
mostly doing Python development, before joining Google.

Alex won the 2002 Activators' Choice Award, and the 2006 Frank Willison
award for outstanding contributions to the Python community.

Alex has also taught courses on programming, development methods,
object-oriented design, and numerical computing, at Ferrara University
(Italy) and other venues.  Alex's proudest achievement is the articles
that appeared in Bridge World (January/February 2000), which were hailed
as giant steps towards solving issues that had haunted contract-bridge
game theoreticians for decades.

This biography was loosely adapted from Alex's autobiography
(http://www.aleax.it/bio.txt), more information can be found on his
website http://www.aleax.it .

Ga?l



From millman at berkeley.edu  Sun Jun 15 01:16:37 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sat, 14 Jun 2008 22:16:37 -0700
Subject: [Numpy-discussion] Switching to nose test framework (was:
	NumpyTest problem)
In-Reply-To: <20080613143435.177cc219@ajackson.org>
References: <1d36917a0806080415r23c7144elfd6f71f2f196ac58@mail.gmail.com>
	<d05265cb0806081549i5fe017b0yf552d25bf51f239b@mail.gmail.com>
	<1d36917a0806081618g5875fa3bt1bfe5f379e9a797a@mail.gmail.com>
	<ce557a360806091607v6a777a42id75102dd63b50e62@mail.gmail.com>
	<1d36917a0806091853g3390a5e4m2ca1fa08b93f2857@mail.gmail.com>
	<9457e7c80806100257t1a6ffa87m121477814cbba17a@mail.gmail.com>
	<3d375d730806111509v80d545asf473b6b44930cb38@mail.gmail.com>
	<9457e7c80806111751h401ee5eegb9953b4b8b923390@mail.gmail.com>
	<20080613132111.GF3573@phare.normalesup.org>
	<20080613143435.177cc219@ajackson.org>
Message-ID: <c7009a550806142216g75df778dge0c12a1a0eb61424@mail.gmail.com>

Hey Alan,

Please go ahead and commit your changes to use nose as our testing
framework.  We have all ready decided to go with nose.  SciPy has all
ready been converted.  I don't think anyone raised any concerns that
we should use another testing framework at this point.  Once we switch
over to using nose, we can improve documentation and refine how we run
tests.  But the sooner we make the switch the more time we will have
to do this before releasing 1.2.0.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From gruben at bigpond.net.au  Mon Jun 16 12:30:24 2008
From: gruben at bigpond.net.au (gruben at bigpond.net.au)
Date: Tue, 17 Jun 2008 2:30:24 +1000
Subject: [Numpy-discussion] Detecting phase windings
Message-ID: <10495791.1213633824742.JavaMail.root@nschwwebs02p>

I have a speed problem with the approach I'm using to detect phase wrappings in a 3D data set. In my application, phaseField is a 3D array containing the phase values of a field. In order to detect the vortices/phase windings at each point, I check for windings on each of 3 faces of a 2x2 cube with the following code:

def HasVortex(cube):
    ''' cube is a 2x2x2 array containing the phase values
        returns True if any of 3 orthogonal faces contains a phase wrapping
    '''
    # extract 3 cube faces - flatten to make indexing easier
    c1 = cube[0,:,:].flatten()
    c3 = cube[:,0,:].flatten()
    c5 = cube[:,:,0].flatten()

    # now order the phases in a circuit around each face, finishing in the same corner as we began
    # Use numpy's phase unwrapping code, which has a default threshold of pi
    u1 = unwrap(c1[cwi])
    u3 = unwrap(c3[cwi])
    u5 = unwrap(c5[cwi])

    # check whether the final unwrapped value coincides with the value in the cell we started at
    return not allclose(r_[u1[0],u3[0],u5[0]], r_[u1[4],u3[4],u5[4]])

vortexArray = array([int(HasVortex(phaseField[x:x+2,y:y+2,z:z+2]))
                    for x in range(phaseField.shape[0]-1)
                    for y in range(phaseField.shape[1]-1)
                    for z in range(phaseField.shape[2]-1)]\
                   ).reshape((xs-1,ys-1,zs-1))

Whilst this does what I want, it's incredibly slow. I'm wondering whether anyone has done this before, or can think of a better approach.

My thoughts about a better approach are to stack the values along 3 new dimensions making a 6D array and use unwrap along the 3 new dimensions. Something like the following may give you an idea of what I mean, but it's a toy example trying to extend 2D into 3D, rather than 3D into 6D, because I haven't come to grips with enough of numpy's axis reshaping and stacking tricks to avoid getting a severe headache when trying to generalize it:

a = array([[1.,3.], [6.,5.]])
b = np.dstack((a, roll(a,-1,axis=1), roll(roll(a,-1,axis=0),-1,axis=1), roll(a,-1,axis=0), a))
print np.unwrap(b, axis=2)

A problem with this approach is that the memory requirements for the stacked version will be much greater, so some version using views would be preferable.

Any ideas?

Gary Ruben


From deathicrow at gmail.com  Mon Jun 16 15:30:13 2008
From: deathicrow at gmail.com (Chandler Latour)
Date: Mon, 16 Jun 2008 14:30:13 -0500
Subject: [Numpy-discussion] Polyfit
Message-ID: <2A20157F-4E6E-4D06-B750-8376BEF76E0B@gmail.com>

Hello,

I'm new to the whole numpy scene, but I've been wanting to run a  
regression on some data.  I belive that polyfit is the way to go, but  
I was wondering if there exists a way to force the intercept to be 0.   
Any help would be much appreciated.

Thanks


From charlesr.harris at gmail.com  Mon Jun 16 15:39:12 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 16 Jun 2008 13:39:12 -0600
Subject: [Numpy-discussion] Polyfit
In-Reply-To: <2A20157F-4E6E-4D06-B750-8376BEF76E0B@gmail.com>
References: <2A20157F-4E6E-4D06-B750-8376BEF76E0B@gmail.com>
Message-ID: <e06186140806161239t7317f587td2bb63f79f0ae709@mail.gmail.com>

On Mon, Jun 16, 2008 at 1:30 PM, Chandler Latour <deathicrow at gmail.com>
wrote:

> Hello,
>
> I'm new to the whole numpy scene, but I've been wanting to run a
> regression on some data.  I belive that polyfit is the way to go, but
> I was wondering if there exists a way to force the intercept to be 0.
> Any help would be much appreciated.
>

You mean fit a line passing through the origin?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080616/24654f74/attachment.html>

From deathicrow at gmail.com  Mon Jun 16 15:47:11 2008
From: deathicrow at gmail.com (Chandler Latour)
Date: Mon, 16 Jun 2008 14:47:11 -0500
Subject: [Numpy-discussion] Polyfit
In-Reply-To: <e06186140806161239t7317f587td2bb63f79f0ae709@mail.gmail.com>
References: <2A20157F-4E6E-4D06-B750-8376BEF76E0B@gmail.com>
	<e06186140806161239t7317f587td2bb63f79f0ae709@mail.gmail.com>
Message-ID: <EA294D6E-5CD8-407A-B001-0579C36643EA@gmail.com>

Yes, exactly what I meant.

On Jun 16, 2008, at 2:39 PM, Charles R Harris wrote:

>
>
> On Mon, Jun 16, 2008 at 1:30 PM, Chandler Latour  
> <deathicrow at gmail.com> wrote:
> Hello,
>
> I'm new to the whole numpy scene, but I've been wanting to run a
> regression on some data.  I belive that polyfit is the way to go, but
> I was wondering if there exists a way to force the intercept to be 0.
> Any help would be much appreciated.
>
> You mean fit a line passing through the origin?
>
> Chuck
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080616/494428e1/attachment.html>

From charlesr.harris at gmail.com  Mon Jun 16 16:25:18 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 16 Jun 2008 14:25:18 -0600
Subject: [Numpy-discussion] Polyfit
In-Reply-To: <EA294D6E-5CD8-407A-B001-0579C36643EA@gmail.com>
References: <2A20157F-4E6E-4D06-B750-8376BEF76E0B@gmail.com>
	<e06186140806161239t7317f587td2bb63f79f0ae709@mail.gmail.com>
	<EA294D6E-5CD8-407A-B001-0579C36643EA@gmail.com>
Message-ID: <e06186140806161325h5cde901cu1f760cd7fb4737c0@mail.gmail.com>

On Mon, Jun 16, 2008 at 1:47 PM, Chandler Latour <deathicrow at gmail.com>
wrote:

> Yes, exactly what I meant.
>

Polyfit just fits polynomials, there is no way of fixing the constant to
zero. Your best bet is to use linalg.lstsq directly to fit the function you
want.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080616/d68b9cc0/attachment.html>

From simon.palmer at gmail.com  Mon Jun 16 17:33:53 2008
From: simon.palmer at gmail.com (Simon Palmer)
Date: Mon, 16 Jun 2008 22:33:53 +0100
Subject: [Numpy-discussion] Polyfit
In-Reply-To: <e06186140806161325h5cde901cu1f760cd7fb4737c0@mail.gmail.com>
References: <2A20157F-4E6E-4D06-B750-8376BEF76E0B@gmail.com>
	<e06186140806161239t7317f587td2bb63f79f0ae709@mail.gmail.com>
	<EA294D6E-5CD8-407A-B001-0579C36643EA@gmail.com>
	<e06186140806161325h5cde901cu1f760cd7fb4737c0@mail.gmail.com>
Message-ID: <fe40ec880806161433rcb51c85qac65661a21c3110b@mail.gmail.com>

At the risk of uttering a heresy, are you bound to Python for this?  I bet
you could find a C library that will work well, plus it is not a hard
algorithm to code yourself.  I am pretty sure I have used a numerical
recipes algorithm for regression in my distant past.

Also I can't help thinking the idea of forcing your regression fit through
the origin is a of a bit strange thing to do.  Do you want it to pass
through the origin for visualisation purposes?  What if the origin is not a
statistically valid place for the regression fit to pass through?

On Mon, Jun 16, 2008 at 9:25 PM, Charles R Harris <charlesr.harris at gmail.com>
wrote:

>
>
> On Mon, Jun 16, 2008 at 1:47 PM, Chandler Latour <deathicrow at gmail.com>
> wrote:
>
>> Yes, exactly what I meant.
>>
>
> Polyfit just fits polynomials, there is no way of fixing the constant to
> zero. Your best bet is to use linalg.lstsq directly to fit the function you
> want.
>
> Chuck
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080616/29218562/attachment.html>

From deathicrow at gmail.com  Mon Jun 16 18:02:01 2008
From: deathicrow at gmail.com (Chandler Latour)
Date: Mon, 16 Jun 2008 17:02:01 -0500
Subject: [Numpy-discussion] Polyfit
In-Reply-To: <fe40ec880806161433rcb51c85qac65661a21c3110b@mail.gmail.com>
References: <2A20157F-4E6E-4D06-B750-8376BEF76E0B@gmail.com>
	<e06186140806161239t7317f587td2bb63f79f0ae709@mail.gmail.com>
	<EA294D6E-5CD8-407A-B001-0579C36643EA@gmail.com>
	<e06186140806161325h5cde901cu1f760cd7fb4737c0@mail.gmail.com>
	<fe40ec880806161433rcb51c85qac65661a21c3110b@mail.gmail.com>
Message-ID: <7854F25E-FB06-4CEB-BAF8-1041AB232368@gmail.com>

I believe I'm bound to python.

In terms of forcing the regression through the origin, the purpose is  
partly for visualization but it also should fit the data.  It would  
not make sense to model the data with an initial value other than 0.


On Jun 16, 2008, at 4:33 PM, Simon Palmer wrote:

> At the risk of uttering a heresy, are you bound to Python for this?   
> I bet you could find a C library that will work well, plus it is not  
> a hard algorithm to code yourself.  I am pretty sure I have used a  
> numerical recipes algorithm for regression in my distant past.
>
> Also I can't help thinking the idea of forcing your regression fit  
> through the origin is a of a bit strange thing to do.  Do you want  
> it to pass through the origin for visualisation purposes?  What if  
> the origin is not a statistically valid place for the regression fit  
> to pass through?
>
> On Mon, Jun 16, 2008 at 9:25 PM, Charles R Harris <charlesr.harris at gmail.com 
> > wrote:
>
>
> On Mon, Jun 16, 2008 at 1:47 PM, Chandler Latour  
> <deathicrow at gmail.com> wrote:
> Yes, exactly what I meant.
>
> Polyfit just fits polynomials, there is no way of fixing the  
> constant to zero. Your best bet is to use linalg.lstsq directly to  
> fit the function you want.
>
> Chuck
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080616/8f9a2bed/attachment.html>

From alan.mcintyre at gmail.com  Mon Jun 16 20:28:16 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 16 Jun 2008 20:28:16 -0400
Subject: [Numpy-discussion] nose changes checked in
Message-ID: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>

Hi all,

I just checked in the switch to use nose to run unit tests.  Please
let me know if you experience any difficulties as a result.

Thanks,
Alan


From robert.kern at gmail.com  Mon Jun 16 20:39:03 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 16 Jun 2008 19:39:03 -0500
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
Message-ID: <3d375d730806161739l18685535y5f87f8049debc79d@mail.gmail.com>

On Mon, Jun 16, 2008 at 19:28, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> Hi all,
>
> I just checked in the switch to use nose to run unit tests.  Please
> let me know if you experience any difficulties as a result.

Please port over the changes I made to scipy.testing in scipy's r4424
in order to avoid importing nose until the actual execution of tests.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From charlesr.harris at gmail.com  Mon Jun 16 21:04:23 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 16 Jun 2008 19:04:23 -0600
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
Message-ID: <e06186140806161804p5ca64dcbx81c45dc41ae193cf@mail.gmail.com>

On Mon, Jun 16, 2008 at 6:28 PM, Alan McIntyre <alan.mcintyre at gmail.com>
wrote:

> Hi all,
>
> I just checked in the switch to use nose to run unit tests.  Please
> let me know if you experience any difficulties as a result.
>

Did you add any documentation to the readme? Any particular version
dependency for Nose?

Hmm, plenty of output, 1 failure, and it looks like a future warning in
histogram needs to be disabled.

In [1]: numpy.test()
Not implemented: Defined_Binary_Op
Not implemented: Defined_Binary_Op
Defined_Operator not defined used by Generic_Spec
Needs match implementation: Allocate_Stmt
Needs match implementation: Associate_Construct
Needs match implementation: Backspace_Stmt
Needs match implementation: Block_Data
Needs match implementation: Case_Construct
Needs match implementation: Case_Selector
Needs match implementation: Case_Stmt
Needs match implementation: Control_Edit_Desc
Needs match implementation: Data_Component_Def_Stmt
Needs match implementation: Data_Implied_Do
Needs match implementation: Data_Stmt
Needs match implementation: Data_Stmt_Set
Needs match implementation: Deallocate_Stmt
Needs match implementation: Derived_Type_Def
Needs match implementation: Endfile_Stmt
Needs match implementation: Entry_Stmt
Needs match implementation: Enum_Def
Needs match implementation: Flush_Stmt
Needs match implementation: Forall_Construct
Needs match implementation: Forall_Header
Needs match implementation: Forall_Triplet_Spec
Needs match implementation: Format_Item
Needs match implementation: Function_Stmt
Needs match implementation: Generic_Binding
Needs match implementation: Generic_Spec
Needs match implementation: Implicit_Part
Needs match implementation: Inquire_Stmt
Needs match implementation: Interface_Block
Needs match implementation: Interface_Body
Needs match implementation: Interface_Stmt
Needs match implementation: Internal_Subprogram_Part
Needs match implementation: Io_Implied_Do
Needs match implementation: Io_Implied_Do_Control
Needs match implementation: Main_Program
Needs match implementation: Module
Needs match implementation: Module_Subprogram_Part
Needs match implementation: Namelist_Stmt
Needs match implementation: Pointer_Assignment_Stmt
Needs match implementation: Position_Edit_Desc
Needs match implementation: Proc_Attr_Spec
Needs match implementation: Proc_Component_Def_Stmt
Needs match implementation: Procedure_Declaration_Stmt
Needs match implementation: Procedure_Stmt
Needs match implementation: Read_Stmt
Needs match implementation: Rewind_Stmt
Needs match implementation: Select_Type_Construct
Needs match implementation: Select_Type_Stmt
Needs match implementation: Specific_Binding
Needs match implementation: Target_Stmt
Needs match implementation: Type_Bound_Procedure_Part
Needs match implementation: Where_Construct
-----
Nof match implementation needs: 51 out of 224
Nof tests needs: 224 out of 224
Total number of classes: 529
-----
...................................................................................................................................................................................................................................................................../usr/lib/python2.5/site-packages/numpy/lib/function_base.py:166:
FutureWarning:
        The semantics of histogram will be modified in
        release 1.2 to improve outlier handling. The new behavior can be
        obtained using new=True. Note that the new version accepts/returns
        the bin edges instead of the left bin edges.
        Please read the docstring for more information.
  Please read the docstring for more information.""", FutureWarning)
/usr/lib/python2.5/site-packages/numpy/lib/function_base.py:193:
FutureWarning:
            The semantic for bins will change in version 1.2.
            The bins will become the bin edges, instead of the left bin
edges.

  """, FutureWarning)
...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................E......................................................................................................................................................................................................................................................................................................
======================================================================
ERROR: Failure: <type 'exceptions.ImportError'>(No module named test_linalg)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/nose/loader.py", line 361, in
loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python2.5/site-packages/nose/importer.py", line 39, in
importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python2.5/site-packages/nose/importer.py", line 71, in
importFromDir
    fh, filename, desc = find_module(part, path)
ImportError: No module named test_linalg

----------------------------------------------------------------------
Ran 1659 tests in 8.347s

FAILED (errors=1)


Looks slower than the old testing framework, but not enough to matter.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080616/987e880d/attachment.html>

From peridot.faceted at gmail.com  Mon Jun 16 21:33:11 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 16 Jun 2008 19:33:11 -0600
Subject: [Numpy-discussion] Polyfit
In-Reply-To: <7854F25E-FB06-4CEB-BAF8-1041AB232368@gmail.com>
References: <2A20157F-4E6E-4D06-B750-8376BEF76E0B@gmail.com>
	<e06186140806161239t7317f587td2bb63f79f0ae709@mail.gmail.com>
	<EA294D6E-5CD8-407A-B001-0579C36643EA@gmail.com>
	<e06186140806161325h5cde901cu1f760cd7fb4737c0@mail.gmail.com>
	<fe40ec880806161433rcb51c85qac65661a21c3110b@mail.gmail.com>
	<7854F25E-FB06-4CEB-BAF8-1041AB232368@gmail.com>
Message-ID: <ce557a360806161833lc96708bgc5b3fceca3b6fd30@mail.gmail.com>

2008/6/16 Chandler Latour <deathicrow at gmail.com>:
> I believe I'm bound to python.
> In terms of forcing the regression through the origin, the purpose is partly
> for visualization but it also should fit the data.  It would not make sense
> to model the data with an initial value other than 0.

Polyfit is just a thin wrapper around numpy.linalg.lstsq. You can do
something like:
A = np.power.outer(xs,np.arange(1,n))
outputs = np.linalg.lstsq(A,ys)

Anne


From alan.mcintyre at gmail.com  Mon Jun 16 22:17:52 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 16 Jun 2008 22:17:52 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <3d375d730806161739l18685535y5f87f8049debc79d@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<3d375d730806161739l18685535y5f87f8049debc79d@mail.gmail.com>
Message-ID: <1d36917a0806161917s501396ebr63b0119a914bfc8@mail.gmail.com>

On Mon, Jun 16, 2008 at 8:39 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Mon, Jun 16, 2008 at 19:28, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> Please port over the changes I made to scipy.testing in scipy's r4424
> in order to avoid importing nose until the actual execution of tests.

Done in r5289.


From alan.mcintyre at gmail.com  Mon Jun 16 22:18:36 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 16 Jun 2008 22:18:36 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <e06186140806161804p5ca64dcbx81c45dc41ae193cf@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<e06186140806161804p5ca64dcbx81c45dc41ae193cf@mail.gmail.com>
Message-ID: <1d36917a0806161918u94751bdu8f2630d775256ad9@mail.gmail.com>

On Mon, Jun 16, 2008 at 9:04 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> Did you add any documentation to the readme? Any particular version
> dependency for Nose?

I did update README.txt, but I forgot to put the minimum Nose version
in there; thanks for mentioning that.  I just checked in an update
that includes it.

> Hmm, plenty of output, 1 failure, and it looks like a future warning in
> histogram needs to be disabled.
>
> In [1]: numpy.test()
> Not implemented: Defined_Binary_Op
> Not implemented: Defined_Binary_Op
> Defined_Operator not defined used by Generic_Spec
> Needs match implementation: Allocate_Stmt
> Needs match implementation: Associate_Construct
<snip>

That stream of "Not implemented" and "Needs match implementation"
stuff comes from numpy/f2py/lib/parser/test_Fortran2003.py and
Fortran2003.py.  I can silence most of that output by disabling those
module-level "if 1:" blocks in those two files, but there's still
complaints about Defined_Binary_Op not being implemented.  If someone
more knowledgeable about that module could comment on that I'd
appreciate it.

I'll also look into that test_linalg failure first thing in the
morning; I don' get that here.

> Looks slower than the old testing framework, but not enough to matter.

There were at least a couple of tests that didn't get run under the
old framework, but the ones I can recall didn't look complicated
enough to make a difference.

Looks like the buildbots need nose installed or upgraded as well.

Thanks,
Alan


From charlesr.harris at gmail.com  Mon Jun 16 22:47:42 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 16 Jun 2008 20:47:42 -0600
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <1d36917a0806161918u94751bdu8f2630d775256ad9@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<e06186140806161804p5ca64dcbx81c45dc41ae193cf@mail.gmail.com>
	<1d36917a0806161918u94751bdu8f2630d775256ad9@mail.gmail.com>
Message-ID: <e06186140806161947v401fda35nae849afa25096e63@mail.gmail.com>

On Mon, Jun 16, 2008 at 8:18 PM, Alan McIntyre <alan.mcintyre at gmail.com>
wrote:

> On Mon, Jun 16, 2008 at 9:04 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> > Did you add any documentation to the readme? Any particular version
> > dependency for Nose?
>
> I did update README.txt, but I forgot to put the minimum Nose version
> in there; thanks for mentioning that.  I just checked in an update
> that includes it.
>
> > Hmm, plenty of output, 1 failure, and it looks like a future warning in
> > histogram needs to be disabled.
> >
> > In [1]: numpy.test()
> > Not implemented: Defined_Binary_Op
> > Not implemented: Defined_Binary_Op
> > Defined_Operator not defined used by Generic_Spec
> > Needs match implementation: Allocate_Stmt
> > Needs match implementation: Associate_Construct
> <snip>
>
> That stream of "Not implemented" and "Needs match implementation"
> stuff comes from numpy/f2py/lib/parser/test_Fortran2003.py and
> Fortran2003.py.  I can silence most of that output by disabling those
> module-level "if 1:" blocks in those two files, but there's still
> complaints about Defined_Binary_Op not being implemented.  If someone
> more knowledgeable about that module could comment on that I'd
> appreciate it.
>
> I'll also look into that test_linalg failure first thing in the
> morning; I don' get that here.
>
> > Looks slower than the old testing framework, but not enough to matter.
>
> There were at least a couple of tests that didn't get run under the
> old framework, but the ones I can recall didn't look complicated
> enough to make a difference.
>
> Looks like the buildbots need nose installed or upgraded as well.
>

Thanks for doing this. It looks like the transition to nose is going to be
pretty smooth.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080616/8b7163a1/attachment.html>

From robert.kern at gmail.com  Mon Jun 16 23:17:43 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 16 Jun 2008 22:17:43 -0500
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <1d36917a0806161918u94751bdu8f2630d775256ad9@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<e06186140806161804p5ca64dcbx81c45dc41ae193cf@mail.gmail.com>
	<1d36917a0806161918u94751bdu8f2630d775256ad9@mail.gmail.com>
Message-ID: <3d375d730806162017saff24afjbbe2b5a3fb87b11f@mail.gmail.com>

On Mon, Jun 16, 2008 at 21:18, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> On Mon, Jun 16, 2008 at 9:04 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:

>> In [1]: numpy.test()
>> Not implemented: Defined_Binary_Op
>> Not implemented: Defined_Binary_Op
>> Defined_Operator not defined used by Generic_Spec
>> Needs match implementation: Allocate_Stmt
>> Needs match implementation: Associate_Construct
> <snip>
>
> That stream of "Not implemented" and "Needs match implementation"
> stuff comes from numpy/f2py/lib/parser/test_Fortran2003.py and
> Fortran2003.py.  I can silence most of that output by disabling those
> module-level "if 1:" blocks in those two files, but there's still
> complaints about Defined_Binary_Op not being implemented.  If someone
> more knowledgeable about that module could comment on that I'd
> appreciate it.

These files were for the in-development g3 version of f2py. That
development has moved outside of numpy, so I think they can be removed
wholesale. Some of them seem to require a Fortran 90 compiler, so I
have had them fail for me. Removing these is a high priority.

Pearu, can you confirm? Can we give you the task of removing the
unused g3 portions of f2py for the 1.2 release?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Tue Jun 17 03:24:22 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 17 Jun 2008 16:24:22 +0900
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
Message-ID: <485766A6.8090308@ar.media.kyoto-u.ac.jp>

Alan McIntyre wrote:
> Hi all,
>
> I just checked in the switch to use nose to run unit tests.  Please
> let me know if you experience any difficulties as a result.
>   

It does not work with python2.6a3, but it is a nose problem, apparently 
(I have the exact same error)

http://bzimmer.ziclix.com/2008/05/16/pysmug-on-python26a3/

Just thought I would mention it if other people try things with python2.6

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Tue Jun 17 03:26:47 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 17 Jun 2008 16:26:47 +0900
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <485766A6.8090308@ar.media.kyoto-u.ac.jp>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
Message-ID: <48576737.3040908@ar.media.kyoto-u.ac.jp>

David Cournapeau wrote:
> It does not work with python2.6a3, but it is a nose problem, apparently 
> (I have the exact same error)
>   

Sorry, it is a python26 problem, not nose.

cheers,

David


From alan.mcintyre at gmail.com  Tue Jun 17 07:38:40 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Tue, 17 Jun 2008 07:38:40 -0400
Subject: [Numpy-discussion] Buildbot config
Message-ID: <1d36917a0806170438h6f33694cr1e6cdfc7c8a01496@mail.gmail.com>

To whomever knows,

Where do we keep the buildbot configuration? The arguments for
numpy.test have changed with the switch to nose, so the test step
fails.  It also appears nose needs to be installed/updated on the
build slaves.

Thanks,
Alan


From D.Hendriks at tue.nl  Tue Jun 17 09:12:55 2008
From: D.Hendriks at tue.nl (D.Hendriks (Dennis))
Date: Tue, 17 Jun 2008 15:12:55 +0200
Subject: [Numpy-discussion] Numpy 1.1.0 on Python 2.3?
Message-ID: <4857B857.4030206@tue.nl>

Hello,

After I just upgraded my Numpy from 1.0.3.1 to 1.1.0 and executed (using 
Python 2.3.4):

  python -c 'import numpy; numpy.test()' 

I got:

Traceback (most recent call last):
  File "<string>", line 1, in ?
  File "#######/lib/python2.3/site-packages/numpy/__init__.py", line 
107, in ?
    import ma
  File "#######/lib/python2.3/site-packages/numpy/ma/__init__.py", line 
14, in ?
    import core
  File "#######/lib/python2.3/site-packages/numpy/ma/core.py", line 114, 
in ?
    max_filler.update([(k,-numpy.inf) for k in [numpy.float32, 
numpy.float64]])
AttributeError: keys

It seems Python 2.3 doesn't allow updating dictionaries using a list of 
key-value-pair tuples; it only allows a dictionary as parameter to update().

I could not find anything about Python 2.3 no longer being supported by 
Numpy. Did I miss anything?

Dennis



From david.huard at gmail.com  Tue Jun 17 09:26:05 2008
From: david.huard at gmail.com (David Huard)
Date: Tue, 17 Jun 2008 09:26:05 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <48576737.3040908@ar.media.kyoto-u.ac.jp>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
Message-ID: <91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>

I noticed that NumpyTest and NumpyTestCase disappeared, and now I am
wondering whether these classes part of the public interface or were they
reserved for internal usage ?

In the former, it might be well to deprecate them before removing them.

Cheers,

David

2008/6/17 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:

> David Cournapeau wrote:
> > It does not work with python2.6a3, but it is a nose problem, apparently
> > (I have the exact same error)
> >
>
> Sorry, it is a python26 problem, not nose.
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080617/fd8a553b/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Tue Jun 17 09:22:49 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 17 Jun 2008 22:22:49 +0900
Subject: [Numpy-discussion] Numpy 1.1.0 on Python 2.3?
In-Reply-To: <4857B857.4030206@tue.nl>
References: <4857B857.4030206@tue.nl>
Message-ID: <4857BAA9.1030504@ar.media.kyoto-u.ac.jp>

D.Hendriks (Dennis) wrote:
>
> I could not find anything about Python 2.3 no longer being supported by 
> Numpy. Did I miss anything?
>   

If you're right (the operation not being possible under python 2.3), 
then this is a bug. Numpy 1.1.* still supports python 2.3. Starting from 
1.2, we will only support 2.4 (and above).

May I suggest you to report this bug to our bug tracking system ? 
http://projects.scipy.org/scipy/numpy

cheers,

David


From alan.mcintyre at gmail.com  Tue Jun 17 10:49:05 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Tue, 17 Jun 2008 10:49:05 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <3d375d730806162017saff24afjbbe2b5a3fb87b11f@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<e06186140806161804p5ca64dcbx81c45dc41ae193cf@mail.gmail.com>
	<1d36917a0806161918u94751bdu8f2630d775256ad9@mail.gmail.com>
	<3d375d730806162017saff24afjbbe2b5a3fb87b11f@mail.gmail.com>
Message-ID: <1d36917a0806170749x64e15ff7ybb228677df27166a@mail.gmail.com>

On Mon, Jun 16, 2008 at 11:17 PM, Robert Kern <robert.kern at gmail.com> wrote:
> These files were for the in-development g3 version of f2py. That
> development has moved outside of numpy, so I think they can be removed
> wholesale. Some of them seem to require a Fortran 90 compiler, so I
> have had them fail for me. Removing these is a high priority.

Nice, thanks.  For what it's worth, removing both of those files
doesn't seem to cause any problems for me locally with NumPy or SciPy.


From alan.mcintyre at gmail.com  Tue Jun 17 10:53:27 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Tue, 17 Jun 2008 10:53:27 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
Message-ID: <1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>

On Tue, Jun 17, 2008 at 9:26 AM, David Huard <david.huard at gmail.com> wrote:
> I noticed that NumpyTest and NumpyTestCase disappeared, and now I am
> wondering whether these classes part of the public interface or were they
> reserved for internal usage ?
>
> In the former, it might be well to deprecate them before removing them.

ParametricTestCase is gone too.  There was at least one person using
it that said he didn't mind porting to the nose equivalent, but I
expect that's an indication there's more people out there using them.
If there's a consensus that they need to go back in and get marked as
deprecated, I'll put them back.

Alan


From pearu at cens.ioc.ee  Tue Jun 17 11:14:49 2008
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Tue, 17 Jun 2008 18:14:49 +0300 (EEST)
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <3d375d730806162017saff24afjbbe2b5a3fb87b11f@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<e06186140806161804p5ca64dcbx81c45dc41ae193cf@mail.gmail.com>
	<1d36917a0806161918u94751bdu8f2630d775256ad9@mail.gmail.com>
	<3d375d730806162017saff24afjbbe2b5a3fb87b11f@mail.gmail.com>
Message-ID: <61340.85.164.145.1.1213715689.squirrel@cens.ioc.ee>

On Tue, June 17, 2008 6:17 am, Robert Kern wrote:
> On Mon, Jun 16, 2008 at 21:18, Alan McIntyre <alan.mcintyre at gmail.com>
> wrote:
>> On Mon, Jun 16, 2008 at 9:04 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>
>>> In [1]: numpy.test()
>>> Not implemented: Defined_Binary_Op
>>> Not implemented: Defined_Binary_Op
>>> Defined_Operator not defined used by Generic_Spec
>>> Needs match implementation: Allocate_Stmt
>>> Needs match implementation: Associate_Construct
>> <snip>
>>
>> That stream of "Not implemented" and "Needs match implementation"
>> stuff comes from numpy/f2py/lib/parser/test_Fortran2003.py and
>> Fortran2003.py.  I can silence most of that output by disabling those
>> module-level "if 1:" blocks in those two files, but there's still
>> complaints about Defined_Binary_Op not being implemented.  If someone
>> more knowledgeable about that module could comment on that I'd
>> appreciate it.
>
> These files were for the in-development g3 version of f2py. That
> development has moved outside of numpy, so I think they can be removed
> wholesale. Some of them seem to require a Fortran 90 compiler, so I
> have had them fail for me. Removing these is a high priority.
>
> Pearu, can you confirm? Can we give you the task of removing the
> unused g3 portions of f2py for the 1.2 release?

Yes, I have created the corresponding ticket some time ago:
  http://projects.scipy.org/scipy/numpy/ticket/758

Pearu



From nwagner at iam.uni-stuttgart.de  Tue Jun 17 11:53:53 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Tue, 17 Jun 2008 17:53:53 +0200
Subject: [Numpy-discussion] ImportError: No module named numpy.distutils.core
Message-ID: <web-103442223@uni-stuttgart.de>

Hi all,

How do I install numpy from svn ?
  
With latest svn I get

/usr/bin/python setup.py install
Traceback (most recent call last):
   File "setup.py", line 96, in ?
     setup_package()
   File "setup.py", line 68, in setup_package
     from numpy.distutils.core import setup
ImportError: No module named numpy.distutils.core

Cheers,

            Nils



From charlesr.harris at gmail.com  Tue Jun 17 11:59:04 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 17 Jun 2008 09:59:04 -0600
Subject: [Numpy-discussion] ImportError: No module named
	numpy.distutils.core
In-Reply-To: <web-103442223@uni-stuttgart.de>
References: <web-103442223@uni-stuttgart.de>
Message-ID: <e06186140806170859y49166d91la9daa73e77886d1c@mail.gmail.com>

On Tue, Jun 17, 2008 at 9:53 AM, Nils Wagner <nwagner at iam.uni-stuttgart.de>
wrote:

> Hi all,
>
> How do I install numpy from svn ?
>
> With latest svn I get
>
> /usr/bin/python setup.py install
> Traceback (most recent call last):
>   File "setup.py", line 96, in ?
>     setup_package()
>   File "setup.py", line 68, in setup_package
>     from numpy.distutils.core import setup
> ImportError: No module named numpy.distutils.core
>

I don't have that problem. What happens if you delete the build directory?
What if you build separately from install?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080617/3d10d858/attachment.html>

From nwagner at iam.uni-stuttgart.de  Tue Jun 17 12:04:19 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Tue, 17 Jun 2008 18:04:19 +0200
Subject: [Numpy-discussion] ImportError: No module named
 numpy.distutils.core
In-Reply-To: <web-103442223@uni-stuttgart.de>
References: <web-103442223@uni-stuttgart.de>
Message-ID: <web-103442696@uni-stuttgart.de>

On Tue, 17 Jun 2008 17:53:53 +0200
  "Nils Wagner" <nwagner at iam.uni-stuttgart.de> wrote:
> Hi all,
> 
> How do I install numpy from svn ?
>  
> With latest svn I get
> 
> /usr/bin/python setup.py install
> Traceback (most recent call last):
>   File "setup.py", line 96, in ?
>     setup_package()
>   File "setup.py", line 68, in setup_package
>     from numpy.distutils.core import setup
> ImportError: No module named numpy.distutils.core
> 
> Cheers,
> 
>            Nils
> 

Please ignore the message. I have removed my numpy 
directory by accident.

Sorry for the noise.


Anyway, numpy.test() now results in

  ======================================================================
FAIL: 
test_format.test_memmap_roundtrip('\x93NUMPY\x01\x00F\x00{\'shape\': 
(3, 3), \'descr\': \'>c16\', \'fortran_order\': False} 
          \n@$\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@(\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@\x14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@\x1c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 
'\x93NUMPY\x01\x00F\x00{\'shape\': (3, 3), 
\'fortran_order\': False, \'descr\': \'>c16\'} 
          \n@$\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@(\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@\x14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@\x1c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib/python2.4/site-packages/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/lib/python2.4/site-packages/numpy/lib/tests/test_format.py", 
line 414, in assert_equal
     assert o1 == o2
AssertionError

----------------------------------------------------------------------
Ran 1655 tests in 29.608s

FAILED (failures=144)


From charlesr.harris at gmail.com  Tue Jun 17 12:13:31 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 17 Jun 2008 10:13:31 -0600
Subject: [Numpy-discussion] ImportError: No module named
	numpy.distutils.core
In-Reply-To: <web-103442696@uni-stuttgart.de>
References: <web-103442223@uni-stuttgart.de> <web-103442696@uni-stuttgart.de>
Message-ID: <e06186140806170913o2184b856pb78da4fcc66bdf69@mail.gmail.com>

On Tue, Jun 17, 2008 at 10:04 AM, Nils Wagner <nwagner at iam.uni-stuttgart.de>
wrote:

> On Tue, 17 Jun 2008 17:53:53 +0200
>   "Nils Wagner" <nwagner at iam.uni-stuttgart.de> wrote:
> > Hi all,
> >
> > How do I install numpy from svn ?
> >
> > With latest svn I get
> >
> > /usr/bin/python setup.py install
> > Traceback (most recent call last):
> >   File "setup.py", line 96, in ?
> >     setup_package()
> >   File "setup.py", line 68, in setup_package
> >     from numpy.distutils.core import setup
> > ImportError: No module named numpy.distutils.core
> >
> > Cheers,
> >
> >            Nils
> >
>
> Please ignore the message. I have removed my numpy
> directory by accident.
>
> Sorry for the noise.
>
>
> Anyway, numpy.test() now results in
>
>  ======================================================================
> FAIL:
> test_format.test_memmap_roundtrip('\x93NUMPY\x01\x00F\x00{\'shape\':
> (3, 3), \'descr\': \'>c16\', \'fortran_order\': False}
>          \n@$\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
> (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
> ,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
> \x14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
> \x1c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
> "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
> \x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
> '\x93NUMPY\x01\x00F\x00{\'shape\': (3, 3),
> \'fortran_order\': False, \'descr\': \'>c16\'}
>          \n@$\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
> (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
> ,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
> \x14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
> \x1c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
> "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
> \x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File "/usr/lib/python2.4/site-packages/nose/case.py",
> line 203, in runTest
>     self.test(*self.arg)
>   File
> "/usr/lib/python2.4/site-packages/numpy/lib/tests/test_format.py",
> line 414, in assert_equal
>     assert o1 == o2
> AssertionError
>
> ----------------------------------------------------------------------
> Ran 1655 tests in 29.608s
>
> FAILED (failures=144)
>

OS? Architecture? Compiler? Nose version?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080617/d1923c63/attachment.html>

From nwagner at iam.uni-stuttgart.de  Tue Jun 17 12:31:31 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Tue, 17 Jun 2008 18:31:31 +0200
Subject: [Numpy-discussion] ImportError: No module
	named	numpy.distutils.core
In-Reply-To: <e06186140806170913o2184b856pb78da4fcc66bdf69@mail.gmail.com>
References: <web-103442223@uni-stuttgart.de> <web-103442696@uni-stuttgart.de>
	<e06186140806170913o2184b856pb78da4fcc66bdf69@mail.gmail.com>
Message-ID: <web-103443894@uni-stuttgart.de>

On Tue, 17 Jun 2008 10:13:31 -0600
  "Charles R Harris" <charlesr.harris at gmail.com> wrote:
> On Tue, Jun 17, 2008 at 10:04 AM, Nils Wagner 
><nwagner at iam.uni-stuttgart.de>
> wrote:
> 
>> On Tue, 17 Jun 2008 17:53:53 +0200
>>   "Nils Wagner" <nwagner at iam.uni-stuttgart.de> wrote:
>> > Hi all,
>> >
>> > How do I install numpy from svn ?
>> >
>> > With latest svn I get
>> >
>> > /usr/bin/python setup.py install
>> > Traceback (most recent call last):
>> >   File "setup.py", line 96, in ?
>> >     setup_package()
>> >   File "setup.py", line 68, in setup_package
>> >     from numpy.distutils.core import setup
>> > ImportError: No module named numpy.distutils.core
>> >
>> > Cheers,
>> >
>> >            Nils
>> >
>>
>> Please ignore the message. I have removed my numpy
>> directory by accident.
>>
>> Sorry for the noise.
>>
>>
>> Anyway, numpy.test() now results in
>>
>>  ======================================================================
>> FAIL:
>> test_format.test_memmap_roundtrip('\x93NUMPY\x01\x00F\x00{\'shape\':
>> (3, 3), \'descr\': \'>c16\', \'fortran_order\': False}
>>          \n@$\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>> (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>> ,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>> \x14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>> \x1c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>> "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>> \x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
>> '\x93NUMPY\x01\x00F\x00{\'shape\': (3, 3),
>> \'fortran_order\': False, \'descr\': \'>c16\'}
>>          \n@$\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>> (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>> ,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>> \x14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>> \x1c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>> "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>> \x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>    File "/usr/lib/python2.4/site-packages/nose/case.py",
>> line 203, in runTest
>>     self.test(*self.arg)
>>   File
>> "/usr/lib/python2.4/site-packages/numpy/lib/tests/test_format.py",
>> line 414, in assert_equal
>>     assert o1 == o2
>> AssertionError
>>
>> ----------------------------------------------------------------------
>> Ran 1655 tests in 29.608s
>>
>> FAILED (failures=144)
>>
> 
> OS? Architecture? Compiler? Nose version?
> 
>>> nose.__version__
'0.10.1'
uname -a
Linux linux 2.6.11.4-21.17-default #1 Fri Apr 6 08:42:34 
UTC 2007 i686 athlon i386 GNU/Linux
Python 2.4 (#1, Oct 13 2006, 17:13:31)
[GCC 3.3.5 20050117 (prerelease) (SUSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more 
information.
  
Nils


From nwagner at iam.uni-stuttgart.de  Tue Jun 17 12:49:49 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Tue, 17 Jun 2008 18:49:49 +0200
Subject: [Numpy-discussion] ImportError: No
	module	named	numpy.distutils.core
In-Reply-To: <web-103443894@uni-stuttgart.de>
References: <web-103442223@uni-stuttgart.de> <web-103442696@uni-stuttgart.de>
	<e06186140806170913o2184b856pb78da4fcc66bdf69@mail.gmail.com>
	<web-103443894@uni-stuttgart.de>
Message-ID: <web-103444734@uni-stuttgart.de>

On Tue, 17 Jun 2008 18:31:31 +0200
  "Nils Wagner" <nwagner at iam.uni-stuttgart.de> wrote:
> On Tue, 17 Jun 2008 10:13:31 -0600
>  "Charles R Harris" <charlesr.harris at gmail.com> wrote:
>> On Tue, Jun 17, 2008 at 10:04 AM, Nils Wagner 
>><nwagner at iam.uni-stuttgart.de>
>> wrote:
>> 
>>> On Tue, 17 Jun 2008 17:53:53 +0200
>>>   "Nils Wagner" <nwagner at iam.uni-stuttgart.de> wrote:
>>> > Hi all,
>>> >
>>> > How do I install numpy from svn ?
>>> >
>>> > With latest svn I get
>>> >
>>> > /usr/bin/python setup.py install
>>> > Traceback (most recent call last):
>>> >   File "setup.py", line 96, in ?
>>> >     setup_package()
>>> >   File "setup.py", line 68, in setup_package
>>> >     from numpy.distutils.core import setup
>>> > ImportError: No module named numpy.distutils.core
>>> >
>>> > Cheers,
>>> >
>>> >            Nils
>>> >
>>>
>>> Please ignore the message. I have removed my numpy
>>> directory by accident.
>>>
>>> Sorry for the noise.
>>>
>>>
>>> Anyway, numpy.test() now results in
>>>
>>>  ======================================================================
>>> FAIL:
>>> test_format.test_memmap_roundtrip('\x93NUMPY\x01\x00F\x00{\'shape\':
>>> (3, 3), \'descr\': \'>c16\', \'fortran_order\': False}
>>>          \n@$\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>>> (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>>> ,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>>> \x14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>>> \x1c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>>> "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>>> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>>> \x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
>>> '\x93NUMPY\x01\x00F\x00{\'shape\': (3, 3),
>>> \'fortran_order\': False, \'descr\': \'>c16\'}
>>>          \n@$\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>>> (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>>> ,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>>> \x14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>>> \x1c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>>> "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>>> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@
>>> \x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>    File "/usr/lib/python2.4/site-packages/nose/case.py",
>>> line 203, in runTest
>>>     self.test(*self.arg)
>>>   File
>>> "/usr/lib/python2.4/site-packages/numpy/lib/tests/test_format.py",
>>> line 414, in assert_equal
>>>     assert o1 == o2
>>> AssertionError
>>>
>>> ----------------------------------------------------------------------
>>> Ran 1655 tests in 29.608s
>>>
>>> FAILED (failures=144)
>>>
>> 
>> OS? Architecture? Compiler? Nose version?
>> 
>>>> nose.__version__
> '0.10.1'
> uname -a
> Linux linux 2.6.11.4-21.17-default #1 Fri Apr 6 08:42:34 
> UTC 2007 i686 athlon i386 GNU/Linux
> Python 2.4 (#1, Oct 13 2006, 17:13:31)
> [GCC 3.3.5 20050117 (prerelease) (SUSE Linux)] on linux2
> Type "help", "copyright", "credits" or "license" for 
>more 
> information.
>  
> Nils
  
BTW, the failures persist after an upgrade of nose.
>>> import nose
>>> nose.__version__
'0.10.3'

Cheers,
             Nils


From robert.kern at gmail.com  Tue Jun 17 13:18:30 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 17 Jun 2008 12:18:30 -0500
Subject: [Numpy-discussion] ImportError: No module named
	numpy.distutils.core
In-Reply-To: <web-103442696@uni-stuttgart.de>
References: <web-103442223@uni-stuttgart.de> <web-103442696@uni-stuttgart.de>
Message-ID: <3d375d730806171018l3a3e775dg6fdeb36e548a0fb9@mail.gmail.com>

On Tue, Jun 17, 2008 at 11:04, Nils Wagner <nwagner at iam.uni-stuttgart.de> wrote:
> On Tue, 17 Jun 2008 17:53:53 +0200
>  "Nils Wagner" <nwagner at iam.uni-stuttgart.de> wrote:
> Anyway, numpy.test() now results in
>
>  ======================================================================
> FAIL:
> test_format.test_memmap_roundtrip('\x93NUMPY\x01\x00F\x00{\'shape\':
> (3, 3), \'descr\': \'>c16\', \'fortran_order\': False}

> '\x93NUMPY\x01\x00F\x00{\'shape\': (3, 3),
> \'fortran_order\': False, \'descr\': \'>c16\'}

Okay, it looks like pprint.pformat() doesn't sort the keys reliably
like I thought it did. Your installation is fine; the testis buggy as
it relies on something it (apparently) shouldn't rely on.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From doutriaux1 at llnl.gov  Tue Jun 17 17:40:00 2008
From: doutriaux1 at llnl.gov (Charles Doutriaux)
Date: Tue, 17 Jun 2008 14:40:00 -0700
Subject: [Numpy-discussion] building 1.1.0 on 64bit,
	seems to be stuck on infinite loop or something
In-Reply-To: <4857B857.4030206@tue.nl>
References: <4857B857.4030206@tue.nl>
Message-ID: <48582F30.6050002@llnl.gov>

Hello,

I'm trying to build 1.1.0 on a:
Linux thunder0 2.6.9-74chaos #1 SMP Wed Oct 24 08:41:12 PDT 2007 ia64 
ia64 ia64 GNU/Linux

using python 2.5 (latest stable)

It seems to be stuck forever at the end of that output:

anybody ran into this?

~/CDAT/5.0.0.beta1/bin/python setup.py install
Running from numpy source directory.
F2PY Version 2_5237
blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in /g/g90/cdoutrix/CDAT/Externals/lib
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in 
/g/g90/cdoutrix/CDAT/Externals/lib
  NOT AVAILABLE

atlas_blas_info:
  libraries f77blas,cblas,atlas not found in 
/g/g90/cdoutrix/CDAT/Externals/lib
  NOT AVAILABLE

/g/g90/cdoutrix/svn/cdat/branches/devel/build/numpy-1.1.0/numpy/distutils/system_info.py:1340: 
UserWarning:
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)
blas_info:
  libraries blas not found in /g/g90/cdoutrix/CDAT/Externals/lib
  NOT AVAILABLE

/g/g90/cdoutrix/svn/cdat/branches/devel/build/numpy-1.1.0/numpy/distutils/system_info.py:1349: 
UserWarning:
    Blas (http://www.netlib.org/blas/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [blas]) or by setting
    the BLAS environment variable.
  warnings.warn(BlasNotFoundError.__doc__)
blas_src_info:
  NOT AVAILABLE

/g/g90/cdoutrix/svn/cdat/branches/devel/build/numpy-1.1.0/numpy/distutils/system_info.py:1352: 
UserWarning:
    Blas (http://www.netlib.org/blas/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [blas_src]) or by setting
    the BLAS_SRC environment variable.
  warnings.warn(BlasSrcNotFoundError.__doc__)
  NOT AVAILABLE

lapack_opt_info:
lapack_mkl_info:
mkl_info:
  libraries mkl,vml,guide not found in /g/g90/cdoutrix/CDAT/Externals/lib
  NOT AVAILABLE

  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in 
/g/g90/cdoutrix/CDAT/Externals/lib
  libraries lapack_atlas not found in /g/g90/cdoutrix/CDAT/Externals/lib
numpy.distutils.system_info.atlas_threads_info
  NOT AVAILABLE

atlas_info:
  libraries f77blas,cblas,atlas not found in 
/g/g90/cdoutrix/CDAT/Externals/lib
  libraries lapack_atlas not found in /g/g90/cdoutrix/CDAT/Externals/lib
numpy.distutils.system_info.atlas_info
  NOT AVAILABLE

/g/g90/cdoutrix/svn/cdat/branches/devel/build/numpy-1.1.0/numpy/distutils/system_info.py:1247: 
UserWarning:
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)
lapack_info:
  libraries lapack not found in /g/g90/cdoutrix/CDAT/Externals/lib
  NOT AVAILABLE

/g/g90/cdoutrix/svn/cdat/branches/devel/build/numpy-1.1.0/numpy/distutils/system_info.py:1258: 
UserWarning:
    Lapack (http://www.netlib.org/lapack/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [lapack]) or by setting
    the LAPACK environment variable.
  warnings.warn(LapackNotFoundError.__doc__)
lapack_src_info:
  NOT AVAILABLE

/g/g90/cdoutrix/svn/cdat/branches/devel/build/numpy-1.1.0/numpy/distutils/system_info.py:1261: 
UserWarning:
    Lapack (http://www.netlib.org/lapack/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [lapack_src]) or by setting
    the LAPACK_SRC environment variable.
  warnings.warn(LapackSrcNotFoundError.__doc__)
  NOT AVAILABLE

running install
running build
running scons
customize UnixCCompiler
Found executable /usr/local/bin/gcc
customize GnuFCompiler
Found executable /usr/local/bin/g77
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
customize GnuFCompiler
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
customize UnixCCompiler
customize UnixCCompiler using scons
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands 
--compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands 
--fcompiler options
running build_src
building py_modules sources
building extension "numpy.core.multiarray" sources
  adding 'build/src.linux-ia64-2.5/numpy/core/config.h' to sources.
  adding 'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h' to sources.
executing numpy/core/code_generators/generate_array_api.py
  adding 'build/src.linux-ia64-2.5/numpy/core/__multiarray_api.h' to 
sources.
  adding 'build/src.linux-ia64-2.5/numpy/core/src' to include_dirs.
numpy.core - nothing done with h_files = 
['build/src.linux-ia64-2.5/numpy/core/src/scalartypes.inc', 
'build/src.linux-ia64-2.5/numpy/core/src/arraytypes.inc', 
'build/src.linux-ia64-2.5/numpy/core/config.h', 
'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h', 
'build/src.linux-ia64-2.5/numpy/core/__multiarray_api.h']
building extension "numpy.core.umath" sources
  adding 'build/src.linux-ia64-2.5/numpy/core/config.h' to sources.
  adding 'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h' to sources.
executing numpy/core/code_generators/generate_ufunc_api.py
  adding 'build/src.linux-ia64-2.5/numpy/core/__ufunc_api.h' to sources.
  adding 'build/src.linux-ia64-2.5/numpy/core/src' to include_dirs.
numpy.core - nothing done with h_files = 
['build/src.linux-ia64-2.5/numpy/core/src/scalartypes.inc', 
'build/src.linux-ia64-2.5/numpy/core/src/arraytypes.inc', 
'build/src.linux-ia64-2.5/numpy/core/config.h', 
'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h', 
'build/src.linux-ia64-2.5/numpy/core/__ufunc_api.h']
building extension "numpy.core._sort" sources
  adding 'build/src.linux-ia64-2.5/numpy/core/config.h' to sources.
  adding 'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h' to sources.
executing numpy/core/code_generators/generate_array_api.py
  adding 'build/src.linux-ia64-2.5/numpy/core/__multiarray_api.h' to 
sources.
numpy.core - nothing done with h_files = 
['build/src.linux-ia64-2.5/numpy/core/config.h', 
'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h','build/src.linux-ia64-2.5/numpy/core/__multiarray_api.h']
building extension "numpy.core.scalarmath" sources
  adding 'build/src.linux-ia64-2.5/numpy/core/config.h' to sources.
  adding 'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h' to sources.
executing numpy/core/code_generators/generate_array_api.py
  adding 'build/src.linux-ia64-2.5/numpy/core/__multiarray_api.h' to 
sources.
executing numpy/core/code_generators/generate_ufunc_api.py
  adding 'build/src.linux-ia64-2.5/numpy/core/__ufunc_api.h' to sources.
numpy.core - nothing done with h_files = 
['build/src.linux-ia64-2.5/numpy/core/config.h', 
'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h','build/src.linux-ia64-2.5/numpy/core/__multiarray_api.h', 
'build/src.linux-ia64-2.5/numpy/core/__ufunc_api.h']
building extension "numpy.core._dotblas" sources
building extension "numpy.lib._compiled_base" sources
building extension "numpy.numarray._capi" sources
building extension "numpy.fft.fftpack_lite" sources
building extension "numpy.linalg.lapack_lite" sources
### Warning:  Using unoptimized lapack ###
  adding 'numpy/linalg/lapack_litemodule.c' to sources.
  adding 'numpy/linalg/python_xerbla.c' to sources.
  adding 'numpy/linalg/zlapack_lite.c' to sources.
  adding 'numpy/linalg/dlapack_lite.c' to sources.
  adding 'numpy/linalg/blas_lite.c' to sources.
  adding 'numpy/linalg/dlamch.c' to sources.
  adding 'numpy/linalg/f2c_lite.c' to sources.
building extension "numpy.random.mtrand" sources
customize GnuFCompiler
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
customize GnuFCompiler
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
customize GnuFCompiler using config
C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall 
-Wstrict-prototypes -fPIC

compile options: '-Inumpy/core/src -Inumpy/core/include 
-I/g/g90/cdoutrix/CDAT/5.0.0.beta1/include/python2.5 
-I/g/g90/cdoutrix/CDAT/5.0.0.beta1/include/python2.5 -c'
gcc: _configtest.c
gcc _configtest.o -o _configtest
_configtest
failure.
removing: _configtest.c _configtest.o _configtest
building data_files sources
running build_py
copying build/src.linux-ia64-2.5/numpy/__config__.py -> 
build/lib.linux-ia64-2.5/numpy
copying build/src.linux-ia64-2.5/numpy/distutils/__config__.py -> 
build/lib.linux-ia64-2.5/numpy/distutils
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
building 'numpy.random.mtrand' extension
compiling C sources
C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall 
-Wstrict-prototypes -fPIC

compile options: '-Inumpy/core/include 
-Ibuild/src.linux-ia64-2.5/numpy/core -Inumpy/core/src 
-Inumpy/core/include 
-I/g/g90/cdoutrix/CDAT/5.0.0.beta1/include/python2.5 
-I/g/g90/cdoutrix/CDAT/5.0.0.beta1/include/python2.5 -c'
gcc: numpy/random/mtrand/mtrand.c
                                                                          




From doutriaux1 at llnl.gov  Tue Jun 17 17:44:48 2008
From: doutriaux1 at llnl.gov (Charles Doutriaux)
Date: Tue, 17 Jun 2008 14:44:48 -0700
Subject: [Numpy-discussion] building 1.1.0 on 64bit,
 seems to be stuck on infinite loop or something
In-Reply-To: <48582F30.6050002@llnl.gov>
References: <4857B857.4030206@tue.nl> <48582F30.6050002@llnl.gov>
Message-ID: <48583050.6010503@llnl.gov>

Just so it might help debuging, when i kill it i get:
_exec_command_posix failed (status=2)
C.


Charles Doutriaux wrote:
> Hello,
>
> I'm trying to build 1.1.0 on a:
> Linux thunder0 2.6.9-74chaos #1 SMP Wed Oct 24 08:41:12 PDT 2007 ia64 
> ia64 ia64 GNU/Linux
>
> using python 2.5 (latest stable)
>
> It seems to be stuck forever at the end of that output:
>
> anybody ran into this?
>
> ~/CDAT/5.0.0.beta1/bin/python setup.py install
> Running from numpy source directory.
> F2PY Version 2_5237
> blas_opt_info:
> blas_mkl_info:
>   libraries mkl,vml,guide not found in /g/g90/cdoutrix/CDAT/Externals/lib
>   NOT AVAILABLE
>
> atlas_blas_threads_info:
> Setting PTATLAS=ATLAS
>   libraries ptf77blas,ptcblas,atlas not found in 
> /g/g90/cdoutrix/CDAT/Externals/lib
>   NOT AVAILABLE
>
> atlas_blas_info:
>   libraries f77blas,cblas,atlas not found in 
> /g/g90/cdoutrix/CDAT/Externals/lib
>   NOT AVAILABLE
>
> /g/g90/cdoutrix/svn/cdat/branches/devel/build/numpy-1.1.0/numpy/distutils/system_info.py:1340: 
> UserWarning:
>     Atlas (http://math-atlas.sourceforge.net/) libraries not found.
>     Directories to search for the libraries can be specified in the
>     numpy/distutils/site.cfg file (section [atlas]) or by setting
>     the ATLAS environment variable.
>   warnings.warn(AtlasNotFoundError.__doc__)
> blas_info:
>   libraries blas not found in /g/g90/cdoutrix/CDAT/Externals/lib
>   NOT AVAILABLE
>
> /g/g90/cdoutrix/svn/cdat/branches/devel/build/numpy-1.1.0/numpy/distutils/system_info.py:1349: 
> UserWarning:
>     Blas (http://www.netlib.org/blas/) libraries not found.
>     Directories to search for the libraries can be specified in the
>     numpy/distutils/site.cfg file (section [blas]) or by setting
>     the BLAS environment variable.
>   warnings.warn(BlasNotFoundError.__doc__)
> blas_src_info:
>   NOT AVAILABLE
>
> /g/g90/cdoutrix/svn/cdat/branches/devel/build/numpy-1.1.0/numpy/distutils/system_info.py:1352: 
> UserWarning:
>     Blas (http://www.netlib.org/blas/) sources not found.
>     Directories to search for the sources can be specified in the
>     numpy/distutils/site.cfg file (section [blas_src]) or by setting
>     the BLAS_SRC environment variable.
>   warnings.warn(BlasSrcNotFoundError.__doc__)
>   NOT AVAILABLE
>
> lapack_opt_info:
> lapack_mkl_info:
> mkl_info:
>   libraries mkl,vml,guide not found in /g/g90/cdoutrix/CDAT/Externals/lib
>   NOT AVAILABLE
>
>   NOT AVAILABLE
>
> atlas_threads_info:
> Setting PTATLAS=ATLAS
>   libraries ptf77blas,ptcblas,atlas not found in 
> /g/g90/cdoutrix/CDAT/Externals/lib
>   libraries lapack_atlas not found in /g/g90/cdoutrix/CDAT/Externals/lib
> numpy.distutils.system_info.atlas_threads_info
>   NOT AVAILABLE
>
> atlas_info:
>   libraries f77blas,cblas,atlas not found in 
> /g/g90/cdoutrix/CDAT/Externals/lib
>   libraries lapack_atlas not found in /g/g90/cdoutrix/CDAT/Externals/lib
> numpy.distutils.system_info.atlas_info
>   NOT AVAILABLE
>
> /g/g90/cdoutrix/svn/cdat/branches/devel/build/numpy-1.1.0/numpy/distutils/system_info.py:1247: 
> UserWarning:
>     Atlas (http://math-atlas.sourceforge.net/) libraries not found.
>     Directories to search for the libraries can be specified in the
>     numpy/distutils/site.cfg file (section [atlas]) or by setting
>     the ATLAS environment variable.
>   warnings.warn(AtlasNotFoundError.__doc__)
> lapack_info:
>   libraries lapack not found in /g/g90/cdoutrix/CDAT/Externals/lib
>   NOT AVAILABLE
>
> /g/g90/cdoutrix/svn/cdat/branches/devel/build/numpy-1.1.0/numpy/distutils/system_info.py:1258: 
> UserWarning:
>     Lapack (http://www.netlib.org/lapack/) libraries not found.
>     Directories to search for the libraries can be specified in the
>     numpy/distutils/site.cfg file (section [lapack]) or by setting
>     the LAPACK environment variable.
>   warnings.warn(LapackNotFoundError.__doc__)
> lapack_src_info:
>   NOT AVAILABLE
>
> /g/g90/cdoutrix/svn/cdat/branches/devel/build/numpy-1.1.0/numpy/distutils/system_info.py:1261: 
> UserWarning:
>     Lapack (http://www.netlib.org/lapack/) sources not found.
>     Directories to search for the sources can be specified in the
>     numpy/distutils/site.cfg file (section [lapack_src]) or by setting
>     the LAPACK_SRC environment variable.
>   warnings.warn(LapackSrcNotFoundError.__doc__)
>   NOT AVAILABLE
>
> running install
> running build
> running scons
> customize UnixCCompiler
> Found executable /usr/local/bin/gcc
> customize GnuFCompiler
> Found executable /usr/local/bin/g77
> gnu: no Fortran 90 compiler found
> gnu: no Fortran 90 compiler found
> customize GnuFCompiler
> gnu: no Fortran 90 compiler found
> gnu: no Fortran 90 compiler found
> customize UnixCCompiler
> customize UnixCCompiler using scons
> running config_cc
> unifing config_cc, config, build_clib, build_ext, build commands 
> --compiler options
> running config_fc
> unifing config_fc, config, build_clib, build_ext, build commands 
> --fcompiler options
> running build_src
> building py_modules sources
> building extension "numpy.core.multiarray" sources
>   adding 'build/src.linux-ia64-2.5/numpy/core/config.h' to sources.
>   adding 'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h' to sources.
> executing numpy/core/code_generators/generate_array_api.py
>   adding 'build/src.linux-ia64-2.5/numpy/core/__multiarray_api.h' to 
> sources.
>   adding 'build/src.linux-ia64-2.5/numpy/core/src' to include_dirs.
> numpy.core - nothing done with h_files = 
> ['build/src.linux-ia64-2.5/numpy/core/src/scalartypes.inc', 
> 'build/src.linux-ia64-2.5/numpy/core/src/arraytypes.inc', 
> 'build/src.linux-ia64-2.5/numpy/core/config.h', 
> 'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h', 
> 'build/src.linux-ia64-2.5/numpy/core/__multiarray_api.h']
> building extension "numpy.core.umath" sources
>   adding 'build/src.linux-ia64-2.5/numpy/core/config.h' to sources.
>   adding 'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h' to sources.
> executing numpy/core/code_generators/generate_ufunc_api.py
>   adding 'build/src.linux-ia64-2.5/numpy/core/__ufunc_api.h' to sources.
>   adding 'build/src.linux-ia64-2.5/numpy/core/src' to include_dirs.
> numpy.core - nothing done with h_files = 
> ['build/src.linux-ia64-2.5/numpy/core/src/scalartypes.inc', 
> 'build/src.linux-ia64-2.5/numpy/core/src/arraytypes.inc', 
> 'build/src.linux-ia64-2.5/numpy/core/config.h', 
> 'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h', 
> 'build/src.linux-ia64-2.5/numpy/core/__ufunc_api.h']
> building extension "numpy.core._sort" sources
>   adding 'build/src.linux-ia64-2.5/numpy/core/config.h' to sources.
>   adding 'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h' to sources.
> executing numpy/core/code_generators/generate_array_api.py
>   adding 'build/src.linux-ia64-2.5/numpy/core/__multiarray_api.h' to 
> sources.
> numpy.core - nothing done with h_files = 
> ['build/src.linux-ia64-2.5/numpy/core/config.h', 
> 'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h','build/src.linux-ia64-2.5/numpy/core/__multiarray_api.h']
> building extension "numpy.core.scalarmath" sources
>   adding 'build/src.linux-ia64-2.5/numpy/core/config.h' to sources.
>   adding 'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h' to sources.
> executing numpy/core/code_generators/generate_array_api.py
>   adding 'build/src.linux-ia64-2.5/numpy/core/__multiarray_api.h' to 
> sources.
> executing numpy/core/code_generators/generate_ufunc_api.py
>   adding 'build/src.linux-ia64-2.5/numpy/core/__ufunc_api.h' to sources.
> numpy.core - nothing done with h_files = 
> ['build/src.linux-ia64-2.5/numpy/core/config.h', 
> 'build/src.linux-ia64-2.5/numpy/core/numpyconfig.h','build/src.linux-ia64-2.5/numpy/core/__multiarray_api.h', 
> 'build/src.linux-ia64-2.5/numpy/core/__ufunc_api.h']
> building extension "numpy.core._dotblas" sources
> building extension "numpy.lib._compiled_base" sources
> building extension "numpy.numarray._capi" sources
> building extension "numpy.fft.fftpack_lite" sources
> building extension "numpy.linalg.lapack_lite" sources
> ### Warning:  Using unoptimized lapack ###
>   adding 'numpy/linalg/lapack_litemodule.c' to sources.
>   adding 'numpy/linalg/python_xerbla.c' to sources.
>   adding 'numpy/linalg/zlapack_lite.c' to sources.
>   adding 'numpy/linalg/dlapack_lite.c' to sources.
>   adding 'numpy/linalg/blas_lite.c' to sources.
>   adding 'numpy/linalg/dlamch.c' to sources.
>   adding 'numpy/linalg/f2c_lite.c' to sources.
> building extension "numpy.random.mtrand" sources
> customize GnuFCompiler
> gnu: no Fortran 90 compiler found
> gnu: no Fortran 90 compiler found
> customize GnuFCompiler
> gnu: no Fortran 90 compiler found
> gnu: no Fortran 90 compiler found
> customize GnuFCompiler using config
> C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall 
> -Wstrict-prototypes -fPIC
>
> compile options: '-Inumpy/core/src -Inumpy/core/include 
> -I/g/g90/cdoutrix/CDAT/5.0.0.beta1/include/python2.5 
> -I/g/g90/cdoutrix/CDAT/5.0.0.beta1/include/python2.5 -c'
> gcc: _configtest.c
> gcc _configtest.o -o _configtest
> _configtest
> failure.
> removing: _configtest.c _configtest.o _configtest
> building data_files sources
> running build_py
> copying build/src.linux-ia64-2.5/numpy/__config__.py -> 
> build/lib.linux-ia64-2.5/numpy
> copying build/src.linux-ia64-2.5/numpy/distutils/__config__.py -> 
> build/lib.linux-ia64-2.5/numpy/distutils
> running build_ext
> customize UnixCCompiler
> customize UnixCCompiler using build_ext
> building 'numpy.random.mtrand' extension
> compiling C sources
> C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall 
> -Wstrict-prototypes -fPIC
>
> compile options: '-Inumpy/core/include 
> -Ibuild/src.linux-ia64-2.5/numpy/core -Inumpy/core/src 
> -Inumpy/core/include 
> -I/g/g90/cdoutrix/CDAT/5.0.0.beta1/include/python2.5 
> -I/g/g90/cdoutrix/CDAT/5.0.0.beta1/include/python2.5 -c'
> gcc: numpy/random/mtrand/mtrand.c
>                                                                           
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   



From charlesr.harris at gmail.com  Tue Jun 17 18:37:10 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 17 Jun 2008 16:37:10 -0600
Subject: [Numpy-discussion] building 1.1.0 on 64bit,
	seems to be stuck on infinite loop or something
In-Reply-To: <48582F30.6050002@llnl.gov>
References: <4857B857.4030206@tue.nl> <48582F30.6050002@llnl.gov>
Message-ID: <e06186140806171537o40cfe90cn1d45fddcb404446@mail.gmail.com>

On Tue, Jun 17, 2008 at 3:40 PM, Charles Doutriaux <doutriaux1 at llnl.gov>
wrote:

> Hello,
>
> I'm trying to build 1.1.0 on a:
> Linux thunder0 2.6.9-74chaos #1 SMP Wed Oct 24 08:41:12 PDT 2007 ia64
> ia64 ia64 GNU/Linux
>
> using python 2.5 (latest stable)
>
> It seems to be stuck forever at the end of that output:
>
> anybody ran into this?
>
> ~/CDAT/5.0.0.beta1/bin/python setup.py install
> Running from numpy source directory.
> F2PY Version 2_5237
> blas_opt_info:
> blas_mkl_info:
>  libraries mkl,vml,guide not found in /g/g90/cdoutrix/CDAT/Externals/lib
>  NOT AVAILABLE
>

I've had mixed success trying to set up CDAT5.0.0.beta1 and wasn't motivated
to pursue it. I'll give it another shot, but probably not until the weekend.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080617/abc8be17/attachment.html>

From peridot.faceted at gmail.com  Tue Jun 17 20:15:11 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 17 Jun 2008 18:15:11 -0600
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
Message-ID: <ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>

2008/6/17 Alan McIntyre <alan.mcintyre at gmail.com>:
> On Tue, Jun 17, 2008 at 9:26 AM, David Huard <david.huard at gmail.com> wrote:
>> I noticed that NumpyTest and NumpyTestCase disappeared, and now I am
>> wondering whether these classes part of the public interface or were they
>> reserved for internal usage ?
>>
>> In the former, it might be well to deprecate them before removing them.
>
> ParametricTestCase is gone too.  There was at least one person using
> it that said he didn't mind porting to the nose equivalent, but I
> expect that's an indication there's more people out there using them.
> If there's a consensus that they need to go back in and get marked as
> deprecated, I'll put them back.

Uh, I assumed NumpyTestCase was public and used it. I'm presumably not
alone, so perhaps a deprecation warning would be good. What
backward-compatible class should I use? unittest.TestCase?

Thanks,
Anne


From robert.kern at gmail.com  Tue Jun 17 20:26:32 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 17 Jun 2008 19:26:32 -0500
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
Message-ID: <3d375d730806171726l2eae780ck6549a6c9bb387fe5@mail.gmail.com>

On Tue, Jun 17, 2008 at 09:53, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> On Tue, Jun 17, 2008 at 9:26 AM, David Huard <david.huard at gmail.com> wrote:
>> I noticed that NumpyTest and NumpyTestCase disappeared, and now I am
>> wondering whether these classes part of the public interface or were they
>> reserved for internal usage ?
>>
>> In the former, it might be well to deprecate them before removing them.
>
> ParametricTestCase is gone too.  There was at least one person using
> it that said he didn't mind porting to the nose equivalent, but I
> expect that's an indication there's more people out there using them.
> If there's a consensus that they need to go back in and get marked as
> deprecated, I'll put them back.

Yes, please do this.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From david.huard at gmail.com  Tue Jun 17 20:59:49 2008
From: david.huard at gmail.com (David Huard)
Date: Tue, 17 Jun 2008 20:59:49 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
Message-ID: <91cf711d0806171759r594e7ccdjca722a5d1e8415f0@mail.gmail.com>

2008/6/17 Anne Archibald <peridot.faceted at gmail.com>:

> 2008/6/17 Alan McIntyre <alan.mcintyre at gmail.com>:
> > On Tue, Jun 17, 2008 at 9:26 AM, David Huard <david.huard at gmail.com>
> wrote:
> >> I noticed that NumpyTest and NumpyTestCase disappeared, and now I am
> >> wondering whether these classes part of the public interface or were
> they
> >> reserved for internal usage ?
> >>
> >> In the former, it might be well to deprecate them before removing them.
> >
> > ParametricTestCase is gone too.  There was at least one person using
> > it that said he didn't mind porting to the nose equivalent, but I
> > expect that's an indication there's more people out there using them.
> > If there's a consensus that they need to go back in and get marked as
> > deprecated, I'll put them back.
>
> Uh, I assumed NumpyTestCase was public and used it. I'm presumably not
> alone, so perhaps a deprecation warning would be good. What
> backward-compatible class should I use? unittest.TestCase?
>

Yes. You'll also have to replace the NumpyTest().run() by something else.
Either the new nose method or the old fashioned
if __name__ == '__main__':
     unittest.main()

Also, note that unittest.TestCase is more restrictive than NumpyTestCase
regarding the method names that are considered tests. While NumpyTestCase
accepted method names starting with check_, TestCase won't recognize those
as tests.

David



> Thanks,
> Anne
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080617/4f20ea42/attachment.html>

From dineshbvadhia at hotmail.com  Wed Jun 18 21:08:32 2008
From: dineshbvadhia at hotmail.com (Dinesh B Vadhia)
Date: Wed, 18 Jun 2008 18:08:32 -0700
Subject: [Numpy-discussion] Array and vector initialization good practice
Message-ID: <BAY109-DS293BE0D420E20BF492C12A3AB0@phx.gbl>

Say, there are two initialized numpy arrays:

A = numpy.blah
B = numpy.blah.blah

With, C = A*d + B*e which doesn't have to be initialized beforehand (as A and B are and so are d and e)

Now, place this in a function ie.

def havingFun(d, e)
    C = A*d + B*e
    return C

The main program calls havingFun(d, e) continuously with new vectors d and e each time (A and B do not change).

Assume that A, B, C, d, e are all very large (>100,000 rows and/or columns).  

In the first call to havingFun, C is created.  In the next call, does Python/Numpy/Scipy have C available to overwrite it or does it create a new C (having deleted the previous C)?

Wrt to efficiency and performance does it matter if C is initialized beforehand or not?

Cheers!

Dinesh

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080618/788cd647/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Tue Jun 17 21:16:21 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 18 Jun 2008 10:16:21 +0900
Subject: [Numpy-discussion] building 1.1.0 on 64bit,
 seems to be stuck on infinite loop or something
In-Reply-To: <48582F30.6050002@llnl.gov>
References: <4857B857.4030206@tue.nl> <48582F30.6050002@llnl.gov>
Message-ID: <485861E5.2010205@ar.media.kyoto-u.ac.jp>

Charles Doutriaux wrote:
> Hello,
>
> I'm trying to build 1.1.0 on a:
> Linux thunder0 2.6.9-74chaos #1 SMP Wed Oct 24 08:41:12 PDT 2007 ia64 
> ia64 ia64 GNU/Linux
>
> using python 2.5 (latest stable)
>
> It seems to be stuck forever at the end of that output:
>   

Is it stuck with 100 % Cpu usage ? May well be a compiler bug ? Did you 
check gcc buglist for your gcc version ?

cheers,

David


From alan.mcintyre at gmail.com  Tue Jun 17 23:44:58 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Tue, 17 Jun 2008 23:44:58 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
Message-ID: <1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>

On Tue, Jun 17, 2008 at 8:15 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> Uh, I assumed NumpyTestCase was public and used it. I'm presumably not
> alone, so perhaps a deprecation warning would be good. What
> backward-compatible class should I use? unittest.TestCase?

Yes, unittest.TestCase seemed to be completely adequate for all the
existing tests in NumPy.  Unless you need some functionality
explicitly implemented in NumpyTestCase (like the 'measure' or
'rundocs' methods), you can just replace it with TestCase.   TestCase
can be imported from numpy.testing, although (now that I think about
it) it's probably less cryptic to just import it from unittest.

For module-level doctests, you can place something like this in the module:

from numpy.testing import *
def test():
    return rundocs()

You can replace ParametricTest with generators, as described here:
     http://scipy.org/scipy/scipy/wiki/TestingGuidelines

I will document all this more completely once the test setup isn't
changing on a daily basis, honest.

I'm assuming this experience should tell me that any item "bar" that I
can get by issuing "from numpy.foo import bar" should be considered
the public API and therefore deprecated instead of removed?  ;)

Cheers,
Alan


From alan.mcintyre at gmail.com  Tue Jun 17 23:49:38 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Tue, 17 Jun 2008 23:49:38 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <3d375d730806171726l2eae780ck6549a6c9bb387fe5@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<3d375d730806171726l2eae780ck6549a6c9bb387fe5@mail.gmail.com>
Message-ID: <1d36917a0806172049r6b2424f3l3b223508faaa0e38@mail.gmail.com>

On Tue, Jun 17, 2008 at 8:26 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Tue, Jun 17, 2008 at 09:53, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
>> If there's a consensus that they need to go back in and get marked as
>> deprecated, I'll put them back.
>
> Yes, please do this.

Using numpy.deprecate on __init__ seems to work for deprecating a
class, although it says that __init__ is deprecated instead of the
class.  Is there a better way to deprecate a class?


From alan.mcintyre at gmail.com  Wed Jun 18 00:02:49 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Wed, 18 Jun 2008 00:02:49 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <3d375d730806161739l18685535y5f87f8049debc79d@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<3d375d730806161739l18685535y5f87f8049debc79d@mail.gmail.com>
Message-ID: <1d36917a0806172102k831ea6el41c2e034b08a2de2@mail.gmail.com>

On Mon, Jun 16, 2008 at 8:39 PM, Robert Kern <robert.kern at gmail.com> wrote:
> Please port over the changes I made to scipy.testing in scipy's r4424
> in order to avoid importing nose until the actual execution of tests.

By the way, I notice that making those changes broke the ability to
run a single test module with the command "python
numpy/numpy/foo/tests/test_bar.py".  So for both NumPy and SciPy, the
boilerplate "if __name__ == '__main__'" stuff needs to be changed if
we're going to retain that capability.

I think I'd prefer to see a convenience function in numpy.testing that
could be used to run the module instead of sprinkling "import nose" in
all that boilerplate, especially since most of the test modules are
already doing a "from numpy.testing import *" anyway.


From robert.kern at gmail.com  Wed Jun 18 01:02:45 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 Jun 2008 00:02:45 -0500
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
	<1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
Message-ID: <3d375d730806172202i40e7ee4asb129f054aeebe8cb@mail.gmail.com>

On Tue, Jun 17, 2008 at 22:44, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> I'm assuming this experience should tell me that any item "bar" that I
> can get by issuing "from numpy.foo import bar" should be considered
> the public API and therefore deprecated instead of removed?  ;)

It's usually a good guess. But asking is usually better than guessing.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From tom.duck at dal.ca  Wed Jun 18 09:55:19 2008
From: tom.duck at dal.ca (Thomas J. Duck)
Date: Wed, 18 Jun 2008 10:55:19 -0300
Subject: [Numpy-discussion] Strange behavior for argsort() and take()
Message-ID: <908F6F2E-C070-4DD2-AFDF-B1029A5A9336@dal.ca>

Hi,

      I have found what I think is some strange behavior for argsort 
() and take().  First, here is an example that works as expected:

      >>> x = numpy.array([1,0,3,2])
      >>> x.argsort()
      array([1, 0, 3, 2])

argsort() returns the original array, which is self-indexing for  
numbers 0 through 3...

      >>> x.take(x.argsort())
      array([0, 1, 2, 3])

...and take() provides the correct sort with ascending order.

Now check out what happens when we swap the middle two numbers:

      >>> x = numpy.array([1,3,0,2])
      >>> x.argsort()
      array([2, 0, 3, 1])

argsort() appears to have indexed the numbers in descending order  
(which was not expected)....

      >>> x.take(x.argsort())
      array([0, 1, 2, 3])

... but the take() operation puts them in ascending order anyway  
(which was really not expected)!

      Can anyone shed some light on what is happening here for me?  I  
have tested this on both my OS X/Fink and Debian/Lenny systems with  
the same result.

Thanks,
Tom

--

Thomas J. Duck <tom.duck at dal.ca>

Associate Professor,
Department of Physics and Atmospheric Science, Dalhousie University,
Halifax, Nova Scotia, Canada, B3H 3J5.
Tel: (902)494-1456 | Fax: (902)494-5191 | Lab: (902)494-3813
Web: http://aolab.phys.dal.ca/




From doutriaux1 at llnl.gov  Wed Jun 18 10:24:52 2008
From: doutriaux1 at llnl.gov (Charles Doutriaux)
Date: Wed, 18 Jun 2008 07:24:52 -0700
Subject: [Numpy-discussion] building 1.1.0 on 64bit,
 seems to be stuck on infinite loop or something
In-Reply-To: <e06186140806171537o40cfe90cn1d45fddcb404446@mail.gmail.com>
References: <4857B857.4030206@tue.nl> <48582F30.6050002@llnl.gov>
	<e06186140806171537o40cfe90cn1d45fddcb404446@mail.gmail.com>
Message-ID: <48591AB4.403@llnl.gov>

Well this got nothing to do with cdat 5.0 yet. At this point it just 
built python from sources, and is putting numpy 1.1.0 in it.

I'll check the compiler bug option next... What if it is indeed a 
compiler bug? What to do next?

C.

Charles R Harris wrote:
>
>
> On Tue, Jun 17, 2008 at 3:40 PM, Charles Doutriaux 
> <doutriaux1 at llnl.gov <mailto:doutriaux1 at llnl.gov>> wrote:
>
>     Hello,
>
>     I'm trying to build 1.1.0 on a:
>     Linux thunder0 2.6.9-74chaos #1 SMP Wed Oct 24 08:41:12 PDT 2007 ia64
>     ia64 ia64 GNU/Linux
>
>     using python 2.5 (latest stable)
>
>     It seems to be stuck forever at the end of that output:
>
>     anybody ran into this?
>
>     ~/CDAT/5.0.0.beta1/bin/python setup.py install
>     Running from numpy source directory.
>     F2PY Version 2_5237
>     blas_opt_info:
>     blas_mkl_info:
>      libraries mkl,vml,guide not found in
>     /g/g90/cdoutrix/CDAT/Externals/lib
>      NOT AVAILABLE
>
>
> I've had mixed success trying to set up CDAT5.0.0.beta1 and wasn't 
> motivated to pursue it. I'll give it another shot, but probably not 
> until the weekend.
>
> Chuck
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From doutriaux1 at llnl.gov  Wed Jun 18 10:51:00 2008
From: doutriaux1 at llnl.gov (Charles Doutriaux)
Date: Wed, 18 Jun 2008 07:51:00 -0700
Subject: [Numpy-discussion] building 1.1.0 on 64bit,
 seems to be stuck on infinite loop or something
In-Reply-To: <485861E5.2010205@ar.media.kyoto-u.ac.jp>
References: <4857B857.4030206@tue.nl> <48582F30.6050002@llnl.gov>
	<485861E5.2010205@ar.media.kyoto-u.ac.jp>
Message-ID: <485920D4.4040809@llnl.gov>

Hi there,

looks like it, moving from gcc3.3.4 to gcc4.1.2 seems to have fix it.

Thx,

C.

David Cournapeau wrote:
> Charles Doutriaux wrote:
>   
>> Hello,
>>
>> I'm trying to build 1.1.0 on a:
>> Linux thunder0 2.6.9-74chaos #1 SMP Wed Oct 24 08:41:12 PDT 2007 ia64 
>> ia64 ia64 GNU/Linux
>>
>> using python 2.5 (latest stable)
>>
>> It seems to be stuck forever at the end of that output:
>>   
>>     
>
> Is it stuck with 100 % Cpu usage ? May well be a compiler bug ? Did you 
> check gcc buglist for your gcc version ?
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   



From peridot.faceted at gmail.com  Wed Jun 18 11:10:01 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 18 Jun 2008 09:10:01 -0600
Subject: [Numpy-discussion] Strange behavior for argsort() and take()
In-Reply-To: <908F6F2E-C070-4DD2-AFDF-B1029A5A9336@dal.ca>
References: <908F6F2E-C070-4DD2-AFDF-B1029A5A9336@dal.ca>
Message-ID: <ce557a360806180810h382208f1pda1136362d050f08@mail.gmail.com>

2008/6/18 Thomas J. Duck <tom.duck at dal.ca>:

>      I have found what I think is some strange behavior for argsort
> () and take().  First, here is an example that works as expected:
>
>      >>> x = numpy.array([1,0,3,2])
>      >>> x.argsort()
>      array([1, 0, 3, 2])
>
> argsort() returns the original array, which is self-indexing for
> numbers 0 through 3...
>
>      >>> x.take(x.argsort())
>      array([0, 1, 2, 3])
>
> ...and take() provides the correct sort with ascending order.
>
> Now check out what happens when we swap the middle two numbers:
>
>      >>> x = numpy.array([1,3,0,2])
>      >>> x.argsort()
>      array([2, 0, 3, 1])
>
> argsort() appears to have indexed the numbers in descending order
> (which was not expected)....
>
>      >>> x.take(x.argsort())
>      array([0, 1, 2, 3])
>
> ... but the take() operation puts them in ascending order anyway
> (which was really not expected)!
>
>      Can anyone shed some light on what is happening here for me?  I
> have tested this on both my OS X/Fink and Debian/Lenny systems with
> the same result.

This is the intended behaviour. The array returned by np.argsort() is
designed to be used as an index into the original array (take() does
almost the same thing as indexing). In particular, when x.argsort()
returns array([2,0,3,1]), that means that the sorted array is x[2],
x[0], x[3], x[1]. It might be clearer to sort a different array:

In [3]: x = np.array([0.1,0.3,0.0,0.2])

In [5]: x.argsort()
Out[5]: array([2, 0, 3, 1])

In [7]: x.take(x.argsort())
Out[7]: array([ 0. ,  0.1,  0.2,  0.3])

If you would like to think of it more mathematically, when you feed
np.argsort() an array that represents a permutation of the numbers
0,1,...,n-1, you get back the inverse permutation. When you pass a
permutation as the argument to x.take(), you apply the permutation to
x. (You can also invert a permutation by feeding it as indices to
arange(n)).

I have been tempted to write some support functions for manipulating
permutations, but I'm not sure how generally useful they would be.

Anne


From peridot.faceted at gmail.com  Wed Jun 18 11:34:50 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 18 Jun 2008 09:34:50 -0600
Subject: [Numpy-discussion] Array and vector initialization good practice
In-Reply-To: <BAY109-DS293BE0D420E20BF492C12A3AB0@phx.gbl>
References: <BAY109-DS293BE0D420E20BF492C12A3AB0@phx.gbl>
Message-ID: <ce557a360806180834t30d378f3yb650b29de7596fff@mail.gmail.com>

2008/6/18 Dinesh B Vadhia <dineshbvadhia at hotmail.com>:
> Say, there are two initialized numpy arrays:

It is somewhat misleading to think of numpy arrays as "initialized" or
"uninitialized". All numpy arrays are dynamically allocated, and under
many circumstances they are given their values on creation.

> A = numpy.blah
> B = numpy.blah.blah
>
> With, C = A*d + B*e which doesn't have to be initialized beforehand (as A
> and B are and so are d and e)

What actually happens in the evaluation of this expression is that a
temporary array (say, T1) is created containing A*d, another (say T2)
is created containing B*e, and then a new array is created containing
T1+T2. The name "C" is set to  point to this freshly-allocated array.
Since nothing references T1 and T2 anymore, they are freed.

> Now, place this in a function ie.
>
> def havingFun(d, e)
>     C = A*d + B*e
>     return C
>
> The main program calls havingFun(d, e) continuously with new vectors d and e
> each time (A and B do not change).
>
> Assume that A, B, C, d, e are all very large (>100,000 rows and/or
> columns).
>
> In the first call to havingFun, C is created.  In the next call, does
> Python/Numpy/Scipy have C available to overwrite it or does it create a new
> C (having deleted the previous C)?

The short answer is that yes, it allocates a new C.

> Wrt to efficiency and performance does it matter if C is initialized
> beforehand or not?

If you had first done
C = np.zeros(100000)
then when you write
C = A*d + B*e
the name "C" is made to point to the freshly-allocated array rather
than the old array (which is still full of zeros). If nothing else
points to the array full of zeros, it is deleted. So "initializing" C
in this way does nothing but waste time.

Memory allocation is a very fast operation - just a handful of CPU
cycles under normal circumstances. Generally it's better to write your
program without worrying about the creation of temporary arrays; only
if
(a) the program is too slow, and
(b) you have determined that it is the creation of temporary arrays is
what is making it slow
is it worth rewriting your program to reduce their use.

There *is* a way to reduce the generation of temporary arrays, but it
does not always make your program faster. It certainly makes it harder
to read. The last time I went through and did this to my program -
which allocated several hundred megabytes worth of arrays - it didn't
make the slightest difference to the runtime. But in case if helps,
here it is:

Most numpy element-wise calculations are carried out by so-called
"ufuncs", which I think stands for "universal functions". These come
with a certain amount of machinery to handle broadcasting, and they
are invoked both directly, as in "np.sin(x)" and "np.atan2(x,y)" and
implicitly as in "-x" and "x+y", which get translated to
"np.negative(x)" and "np.add(x,y)" under the hood. Among the machinery
they support is _output arguments_:

In [6]: x = np.linspace(0,2*np.pi,9)

In [7]: y = np.zeros(9)

In [8]: np.sin(x,y)
Out[8]:
array([  0.00000000e+00,   7.07106781e-01,   1.00000000e+00,
         7.07106781e-01,   1.22460635e-16,  -7.07106781e-01,
        -1.00000000e+00,  -7.07106781e-01,  -2.44921271e-16])

In [9]: y
Out[9]:
array([  0.00000000e+00,   7.07106781e-01,   1.00000000e+00,
         7.07106781e-01,   1.22460635e-16,  -7.07106781e-01,
        -1.00000000e+00,  -7.07106781e-01,  -2.44921271e-16])

Regrettably, they do not accept keyword arguments ("out=y") but if you
give a ufunc more arguments than it normally takes, the last one is an
output argument, indicating that the results are stored in the
existing array rather than a newly-allocated one. Some other numpy
functions (but not all of them) can accept an output argument as well.
Thus if for some reason you want to reduce memory allocation, you can
rewrite your program, so that
C = A*d+B*e
becomes
C = np.empty(n)
T = np.empty(n)
np.multiply(B,e,T)
np.multiply(A,d,C)
np.add(C,T,C)

You should be careful with this, because I have found that reducing
temporaries can *increase* the memory usage of my programs, since it
involves keeping around arrays that would normally be deleted and
reallocated only when needed. But once in a while, it may speed things
up.

Anne

P.S. There are other approaches to reducing temporaries; numpexpr was
created for the explicit purpose, so you might look into them if
you're interested. -A


From peridot.faceted at gmail.com  Wed Jun 18 11:42:57 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 18 Jun 2008 09:42:57 -0600
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
	<1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
Message-ID: <ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>

2008/6/17 Alan McIntyre <alan.mcintyre at gmail.com>:
> On Tue, Jun 17, 2008 at 8:15 PM, Anne Archibald
> <peridot.faceted at gmail.com> wrote:
>> Uh, I assumed NumpyTestCase was public and used it. I'm presumably not
>> alone, so perhaps a deprecation warning would be good. What
>> backward-compatible class should I use? unittest.TestCase?
>
> Yes, unittest.TestCase seemed to be completely adequate for all the
> existing tests in NumPy.  Unless you need some functionality
> explicitly implemented in NumpyTestCase (like the 'measure' or
> 'rundocs' methods), you can just replace it with TestCase.   TestCase
> can be imported from numpy.testing, although (now that I think about
> it) it's probably less cryptic to just import it from unittest.
>
> For module-level doctests, you can place something like this in the module:
>
> from numpy.testing import *
> def test():
>    return rundocs()
>
> You can replace ParametricTest with generators, as described here:
>     http://scipy.org/scipy/scipy/wiki/TestingGuidelines

Hmm. This won't work with the current version of numpy, will it? That
is, it needs nose. (I run much code on the work machines, which I
(thankfully) do not administer, and which are running a variety of
ancient versions of numpy (some even have only Numeric, to support
horrible quasi-in-house C extensions).) So I'd like to write my tests
in a way that they can run on both new and old versions of numpy.

> I will document all this more completely once the test setup isn't
> changing on a daily basis, honest.
>
> I'm assuming this experience should tell me that any item "bar" that I
> can get by issuing "from numpy.foo import bar" should be considered
> the public API and therefore deprecated instead of removed?  ;)

Well, probably. But more so for those that are used widely throughout
numpy itself, since many of us learn how to write code using numpy by
reading numpy source. (Yes, this means that "internal" conventions
like "numpy.core.whatever" get used by people who aren't writing
numpy.)

Anne


From garg1 at ualberta.ca  Wed Jun 18 11:47:47 2008
From: garg1 at ualberta.ca (Rahul Garg)
Date: Wed, 18 Jun 2008 09:47:47 -0600
Subject: [Numpy-discussion] parallel loop annotations
Message-ID: <20080618094747.93p5popyoskgggok@webmail.ualberta.ca>

Hi.

I have been working on a Python to C compiler called unPython (updates  
coming tomorow). One of the features I want to work on is parallel  
loop annotations similar to OpenMP for C.  Given the multicore  
revolution, and given the lack of real concurrency in CPython, such a  
feature will be desirable.

One pseudo-C example of this :
#pragma openmp parallel for
for(i=0;i<m;i++) a[i] = b[i]*c[i];

The additional annotation tells the compiler that the loop is  
parallel. The compiler and runtime takes care of spawning threads etc.

I want to add a similar annotation to Python
example usage :

"pragma parallel for"
for i in xrange(m): a[i] = b[i]*c[i]

The interpreter ignores such strings and the code will of course  
execute serially but such strings can potentially be used by compilers.
Any thoughts on this? What form of annotations would you like to see?
Is it pythonic?

thanks,
rahul


From stefan at sun.ac.za  Wed Jun 18 12:05:27 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 18 Jun 2008 18:05:27 +0200
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
	<1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
	<ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>
Message-ID: <9457e7c80806180905g3058536ah16da7da057b237a2@mail.gmail.com>

2008/6/18 Anne Archibald <peridot.faceted at gmail.com>:
> Well, probably. But more so for those that are used widely throughout
> numpy itself, since many of us learn how to write code using numpy by
> reading numpy source. (Yes, this means that "internal" conventions
> like "numpy.core.whatever" get used by people who aren't writing
> numpy.)

People shouldn't be using code from `numpy.core` directly.  When we
refactor code behind the scenes, we need a workspace to do it in, and
if people start putting their hands in the engine we can't protect
them from getting hurt.  A bigger warning sign may be appropriate (or,
to take it to the extreme, rename `core` to `_core`).

The testing code in NumPy is going to be replaced by Nose.  This means
that eventually all internal test scanning capabilities in NumPy will
disappear.  It isn't necessary to remove NumpyTestCase entirely.
Rather

a) Warn that Nose is becoming a dependency (next release).
b) Replace NumpyTestCase with a Nose-dependent equivalent in the
release thereafter.

Regards
St?fan


From chanley at stsci.edu  Wed Jun 18 12:21:08 2008
From: chanley at stsci.edu (Christopher Hanley)
Date: Wed, 18 Jun 2008 12:21:08 -0400
Subject: [Numpy-discussion] nose test return values
Message-ID: <485935F4.5020006@stsci.edu>

Hi,

I have received the following message from our system guru here at STScI 
regarding the recent changes to the way testing is done with nose.  The 
automated scripts he was using to monitor the success or failure of 
numpy builds is now broken.  Below is his message:

The new numpy test interface returns None instead of 
unittest._TextTestResult

Even if you don't want to return unittest objects, the test should 
return some indication of which tests failed, which could not be run due 
to errors (a test that failed is different from a test that couldn't 
even run), and whether the test suite as a whole can be considered passing.

For my purposes, the wasSuccessful() method of the returned object is 
absolutely essential.  If wasSuccessful() returns true, I am not going 
to look at the numpy tests; if it returns false, somebody has to find 
which tests failed and decide whether that is going to affect the other 
developers.



OLD:

 >>> n=numpy.test(level=10)
...lots out output...
 >>> n
<unittest._TextTestResult run=693 errors=0 failures=0>
 >>> n.errors
[]
 >>> n.failures
[]
 >>> n.wasSuccessful()
True
 >>>



NEW:

 >>> n = numpy.test(label="full", verbose=0)
 >>> n
 >>> print n
None
 >>> numpy.__version__
'1.2.0.dev5292'



My application:

import sys
import numpy

n = numpy.test(level=10)

print "errors: "
for x in n.errors:
    print " ",x

print "failures: "
for x in n.failures:
    print " ",x

if n.wasSuccessful() :
    sys.exit(0)
else :
    sys.exit(1)



Of course, these same comments will apply to scipy when I next install it.

--

I believe that he makes a very good point.  Is there any way that some 
form of test report object can be returned?

Thanks,
Chris



-- 
Christopher Hanley
Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338


From stefan at sun.ac.za  Wed Jun 18 12:53:23 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 18 Jun 2008 18:53:23 +0200
Subject: [Numpy-discussion] Strange behavior for argsort() and take()
In-Reply-To: <ce557a360806180810h382208f1pda1136362d050f08@mail.gmail.com>
References: <908F6F2E-C070-4DD2-AFDF-B1029A5A9336@dal.ca>
	<ce557a360806180810h382208f1pda1136362d050f08@mail.gmail.com>
Message-ID: <9457e7c80806180953l3542792bh2e5122763615ad64@mail.gmail.com>

2008/6/18 Anne Archibald <peridot.faceted at gmail.com>:
> In [7]: x.take(x.argsort())
> Out[7]: array([ 0. ,  0.1,  0.2,  0.3])
>
> If you would like to think of it more mathematically, when you feed
> np.argsort() an array that represents a permutation of the numbers
> 0,1,...,n-1, you get back the inverse permutation. When you pass a
> permutation as the argument to x.take(), you apply the permutation to
> x. (You can also invert a permutation by feeding it as indices to
> arange(n)).
>
> I have been tempted to write some support functions for manipulating
> permutations, but I'm not sure how generally useful they would be.

Do we have an easy way of grabbing the elements out of an array that
correspond to

argmax(x, axis) ?

`take` won't do the job; there `axis` has a different meaning.

St?fan


From tom.duck at dal.ca  Wed Jun 18 12:57:56 2008
From: tom.duck at dal.ca (Thomas J. Duck)
Date: Wed, 18 Jun 2008 13:57:56 -0300
Subject: [Numpy-discussion] Strange behavior for argsort() and take()
In-Reply-To: <ce557a360806180810h382208f1pda1136362d050f08@mail.gmail.com>
References: <908F6F2E-C070-4DD2-AFDF-B1029A5A9336@dal.ca>
	<ce557a360806180810h382208f1pda1136362d050f08@mail.gmail.com>
Message-ID: <76F51024-582B-4360-9F82-5929243543E6@dal.ca>



      Thanks, Anne.  I misinterpreted what argsort() provides.  I was  
thinking about it in terms of the kind of behaviour exhibited by  
searchsorted().  --Tom



--

On 18-Jun-08, at 12:10 PM, Anne Archibald wrote:

> 2008/6/18 Thomas J. Duck <tom.duck at dal.ca>:
>
>>      I have found what I think is some strange behavior for argsort
>> () and take().  First, here is an example that works as expected:
>>
>>>>> x = numpy.array([1,0,3,2])
>>>>> x.argsort()
>>      array([1, 0, 3, 2])
>>
>> argsort() returns the original array, which is self-indexing for
>> numbers 0 through 3...
>>
>>>>> x.take(x.argsort())
>>      array([0, 1, 2, 3])
>>
>> ...and take() provides the correct sort with ascending order.
>>
>> Now check out what happens when we swap the middle two numbers:
>>
>>>>> x = numpy.array([1,3,0,2])
>>>>> x.argsort()
>>      array([2, 0, 3, 1])
>>
>> argsort() appears to have indexed the numbers in descending order
>> (which was not expected)....
>>
>>>>> x.take(x.argsort())
>>      array([0, 1, 2, 3])
>>
>> ... but the take() operation puts them in ascending order anyway
>> (which was really not expected)!
>>
>>      Can anyone shed some light on what is happening here for me?  I
>> have tested this on both my OS X/Fink and Debian/Lenny systems with
>> the same result.
>
> This is the intended behaviour. The array returned by np.argsort() is
> designed to be used as an index into the original array (take() does
> almost the same thing as indexing). In particular, when x.argsort()
> returns array([2,0,3,1]), that means that the sorted array is x[2],
> x[0], x[3], x[1]. It might be clearer to sort a different array:
>
> In [3]: x = np.array([0.1,0.3,0.0,0.2])
>
> In [5]: x.argsort()
> Out[5]: array([2, 0, 3, 1])
>
> In [7]: x.take(x.argsort())
> Out[7]: array([ 0. ,  0.1,  0.2,  0.3])
>
> If you would like to think of it more mathematically, when you feed
> np.argsort() an array that represents a permutation of the numbers
> 0,1,...,n-1, you get back the inverse permutation. When you pass a
> permutation as the argument to x.take(), you apply the permutation to
> x. (You can also invert a permutation by feeding it as indices to
> arange(n)).
>
> I have been tempted to write some support functions for manipulating
> permutations, but I'm not sure how generally useful they would be.
>
> Anne
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

--

Thomas J. Duck <tom.duck at dal.ca>

Associate Professor,
Department of Physics and Atmospheric Science, Dalhousie University,
Halifax, Nova Scotia, Canada, B3H 3J5.
Tel: (902)494-1456 | Fax: (902)494-5191 | Lab: (902)494-3813
Web: http://aolab.phys.dal.ca/




From alan.mcintyre at gmail.com  Wed Jun 18 12:59:55 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Wed, 18 Jun 2008 12:59:55 -0400
Subject: [Numpy-discussion] nose test return values
In-Reply-To: <485935F4.5020006@stsci.edu>
References: <485935F4.5020006@stsci.edu>
Message-ID: <1d36917a0806180959ib4710d4u9d9ba5623f7053b5@mail.gmail.com>

On Wed, Jun 18, 2008 at 12:21 PM, Christopher Hanley <chanley at stsci.edu> wrote:
> I have received the following message from our system guru here at STScI
> regarding the recent changes to the way testing is done with nose.  The
> automated scripts he was using to monitor the success or failure of
> numpy builds is now broken.  Below is his message:
>
> The new numpy test interface returns None instead of
> unittest._TextTestResult
<snip>
> I believe that he makes a very good point.  Is there any way that some
> form of test report object can be returned?

I'll see if I can make that work correctly. The buildbots currently
depend on this behavior as well.


From chanley at stsci.edu  Wed Jun 18 13:07:27 2008
From: chanley at stsci.edu (Christopher Hanley)
Date: Wed, 18 Jun 2008 13:07:27 -0400
Subject: [Numpy-discussion] nose test return values
In-Reply-To: <1d36917a0806180959ib4710d4u9d9ba5623f7053b5@mail.gmail.com>
References: <485935F4.5020006@stsci.edu>
	<1d36917a0806180959ib4710d4u9d9ba5623f7053b5@mail.gmail.com>
Message-ID: <485940CF.5090206@stsci.edu>

Alan McIntyre wrote:
> On Wed, Jun 18, 2008 at 12:21 PM, Christopher Hanley <chanley at stsci.edu> wrote:
>> I have received the following message from our system guru here at STScI
>> regarding the recent changes to the way testing is done with nose.  The
>> automated scripts he was using to monitor the success or failure of
>> numpy builds is now broken.  Below is his message:
>>
>> The new numpy test interface returns None instead of
>> unittest._TextTestResult
> <snip>
>> I believe that he makes a very good point.  Is there any way that some
>> form of test report object can be returned?
> 
> I'll see if I can make that work correctly. The buildbots currently
> depend on this behavior as well.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

Thanks!

Chris



-- 
Christopher Hanley
Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338


From charlesr.harris at gmail.com  Wed Jun 18 13:10:40 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 18 Jun 2008 11:10:40 -0600
Subject: [Numpy-discussion] Strange behavior for argsort() and take()
In-Reply-To: <9457e7c80806180953l3542792bh2e5122763615ad64@mail.gmail.com>
References: <908F6F2E-C070-4DD2-AFDF-B1029A5A9336@dal.ca>
	<ce557a360806180810h382208f1pda1136362d050f08@mail.gmail.com>
	<9457e7c80806180953l3542792bh2e5122763615ad64@mail.gmail.com>
Message-ID: <e06186140806181010x79a387efyb585f145c59dfd72@mail.gmail.com>

On Wed, Jun 18, 2008 at 10:53 AM, St?fan van der Walt <stefan at sun.ac.za>
wrote:

> 2008/6/18 Anne Archibald <peridot.faceted at gmail.com>:
> > In [7]: x.take(x.argsort())
> > Out[7]: array([ 0. ,  0.1,  0.2,  0.3])
> >
> > If you would like to think of it more mathematically, when you feed
> > np.argsort() an array that represents a permutation of the numbers
> > 0,1,...,n-1, you get back the inverse permutation. When you pass a
> > permutation as the argument to x.take(), you apply the permutation to
> > x. (You can also invert a permutation by feeding it as indices to
> > arange(n)).
> >
> > I have been tempted to write some support functions for manipulating
> > permutations, but I'm not sure how generally useful they would be.
>
> Do we have an easy way of grabbing the elements out of an array that
> correspond to
>
> argmax(x, axis) ?
>
> `take` won't do the job; there `axis` has a different meaning.
>

No. I've been thinking of making such a function for a while, called
argtake, for using the results of argsort. I'd also like it too work with
argmax and such that reduce the rank of the matrices, but that is not quite
the same problem. Adding a keep rank flag to argmax and such would help with
that.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080618/76fc57f1/attachment.html>

From alan.mcintyre at gmail.com  Wed Jun 18 13:16:10 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Wed, 18 Jun 2008 13:16:10 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
	<1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
	<ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>
Message-ID: <1d36917a0806181016o8db0c69x142d7cd2975957ad@mail.gmail.com>

On Wed, Jun 18, 2008 at 11:42 AM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
>> You can replace ParametricTest with generators, as described here:
>>     http://scipy.org/scipy/scipy/wiki/TestingGuidelines
>
> Hmm. This won't work with the current version of numpy, will it? That
> is, it needs nose. (I run much code on the work machines, which I
> (thankfully) do not administer, and which are running a variety of
> ancient versions of numpy (some even have only Numeric, to support
> horrible quasi-in-house C extensions).) So I'd like to write my tests
> in a way that they can run on both new and old versions of numpy.

Yes, right now the generator test method needs nose to work correctly.
 Once the old test framework stuff is added back, you should be able
to run 1.1 tests on 1.2.  When I'm adding that stuff back I'll also
add tests for numpy.testing to make sure the old test styles keep
working.


From alan.mcintyre at gmail.com  Wed Jun 18 13:47:30 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Wed, 18 Jun 2008 13:47:30 -0400
Subject: [Numpy-discussion] Buildbot config
In-Reply-To: <1d36917a0806170438h6f33694cr1e6cdfc7c8a01496@mail.gmail.com>
References: <1d36917a0806170438h6f33694cr1e6cdfc7c8a01496@mail.gmail.com>
Message-ID: <1d36917a0806181047g60f1d22sdc0336102b956aa9@mail.gmail.com>

On Tue, Jun 17, 2008 at 7:38 AM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> Where do we keep the buildbot configuration? The arguments for
> numpy.test have changed with the switch to nose, so the test step
> fails.  It also appears nose needs to be installed/updated on the
> build slaves.

Eh, please disregard this request. Unless somebody objects, I'm just
going to revert numpy.test to its original signature and (as close as
possible) behavior, so the buildbot script (and people's existing code
that uses numpy.test) shouldn't need updating.


From peridot.faceted at gmail.com  Wed Jun 18 13:48:13 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 18 Jun 2008 11:48:13 -0600
Subject: [Numpy-discussion] Strange behavior for argsort() and take()
In-Reply-To: <9457e7c80806180953l3542792bh2e5122763615ad64@mail.gmail.com>
References: <908F6F2E-C070-4DD2-AFDF-B1029A5A9336@dal.ca>
	<ce557a360806180810h382208f1pda1136362d050f08@mail.gmail.com>
	<9457e7c80806180953l3542792bh2e5122763615ad64@mail.gmail.com>
Message-ID: <ce557a360806181048x4272a8d3ub592c166feb647d@mail.gmail.com>

2008/6/18 St?fan van der Walt <stefan at sun.ac.za>:
> 2008/6/18 Anne Archibald <peridot.faceted at gmail.com>:
>> In [7]: x.take(x.argsort())
>> Out[7]: array([ 0. ,  0.1,  0.2,  0.3])
>>
>> If you would like to think of it more mathematically, when you feed
>> np.argsort() an array that represents a permutation of the numbers
>> 0,1,...,n-1, you get back the inverse permutation. When you pass a
>> permutation as the argument to x.take(), you apply the permutation to
>> x. (You can also invert a permutation by feeding it as indices to
>> arange(n)).
>>
>> I have been tempted to write some support functions for manipulating
>> permutations, but I'm not sure how generally useful they would be.
>
> Do we have an easy way of grabbing the elements out of an array that
> correspond to
>
> argmax(x, axis) ?
>
> `take` won't do the job; there `axis` has a different meaning.

Well, here's a quick hack:

def take_axis(X, ix, axis):
    XX = np.rollaxis(X,axis)
    s = XX.shape
    return XX[(ix,)+tuple(np.indices(s[1:]))]

And a version that works for argsort()ing:

def take_axis_argsort(X, ix, axis):
    XX = np.rollaxis(X,axis)
    ixix = np.rollaxis(ix,axis)
    s = XX.shape
    return np.rollaxis(XX[(ixix,)+tuple(np.indices(s)[1:])],0,axis+1)

I'm always a little bit leery of using indices(), since it can produce
very large arrays:

In [60]: np.indices((2,3,4)).shape
Out[60]: (3, 2, 3, 4)

Anne

P.S. rollaxis() saves so very much pain; I do wonder if "unrollaxis"
would be useful, even though it's just
def unrollaxis(X, axis):
    return rollaxis(X, 0, axis+1)
-A

From peridot.faceted at gmail.com  Wed Jun 18 13:51:37 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 18 Jun 2008 11:51:37 -0600
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <9457e7c80806180905g3058536ah16da7da057b237a2@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
	<1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
	<ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>
	<9457e7c80806180905g3058536ah16da7da057b237a2@mail.gmail.com>
Message-ID: <ce557a360806181051u5a984e77od5baf45e3dee4a4a@mail.gmail.com>

2008/6/18 St?fan van der Walt <stefan at sun.ac.za>:
> 2008/6/18 Anne Archibald <peridot.faceted at gmail.com>:
>> Well, probably. But more so for those that are used widely throughout
>> numpy itself, since many of us learn how to write code using numpy by
>> reading numpy source. (Yes, this means that "internal" conventions
>> like "numpy.core.whatever" get used by people who aren't writing
>> numpy.)
>
> People shouldn't be using code from `numpy.core` directly.  When we
> refactor code behind the scenes, we need a workspace to do it in, and
> if people start putting their hands in the engine we can't protect
> them from getting hurt.  A bigger warning sign may be appropriate (or,
> to take it to the extreme, rename `core` to `_core`).

I'm not arguing users should be using numpy.core (and I don't use it),
but given the nature of numpy and its documentation, many users look
into the code, see "numpy.core" and copy that. I think unless we want
to actually expose sub-namespaces - which was actively discussed - we
probably should put an underscore.

Not that we can without breaking the code of all those people who are
already using numpy.core.

Anne

From charlesr.harris at gmail.com  Wed Jun 18 14:34:36 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 18 Jun 2008 12:34:36 -0600
Subject: [Numpy-discussion] Strange behavior for argsort() and take()
In-Reply-To: <ce557a360806181048x4272a8d3ub592c166feb647d@mail.gmail.com>
References: <908F6F2E-C070-4DD2-AFDF-B1029A5A9336@dal.ca>
	<ce557a360806180810h382208f1pda1136362d050f08@mail.gmail.com>
	<9457e7c80806180953l3542792bh2e5122763615ad64@mail.gmail.com>
	<ce557a360806181048x4272a8d3ub592c166feb647d@mail.gmail.com>
Message-ID: <e06186140806181134l1cce08d7kab0af9416baa7f77@mail.gmail.com>

On Wed, Jun 18, 2008 at 11:48 AM, Anne Archibald <peridot.faceted at gmail.com>
wrote:

> 2008/6/18 St?fan van der Walt <stefan at sun.ac.za>:
> > 2008/6/18 Anne Archibald <peridot.faceted at gmail.com>:
> >> In [7]: x.take(x.argsort())
> >> Out[7]: array([ 0. ,  0.1,  0.2,  0.3])
> >>
> >> If you would like to think of it more mathematically, when you feed
> >> np.argsort() an array that represents a permutation of the numbers
> >> 0,1,...,n-1, you get back the inverse permutation. When you pass a
> >> permutation as the argument to x.take(), you apply the permutation to
> >> x. (You can also invert a permutation by feeding it as indices to
> >> arange(n)).
> >>
> >> I have been tempted to write some support functions for manipulating
> >> permutations, but I'm not sure how generally useful they would be.
> >
> > Do we have an easy way of grabbing the elements out of an array that
> > correspond to
> >
> > argmax(x, axis) ?
> >
> > `take` won't do the job; there `axis` has a different meaning.
>
> Well, here's a quick hack:
>
> def take_axis(X, ix, axis):
>    XX = np.rollaxis(X,axis)
>    s = XX.shape
>    return XX[(ix,)+tuple(np.indices(s[1:]))]
>
> And a version that works for argsort()ing:
>
> def take_axis_argsort(X, ix, axis):
>    XX = np.rollaxis(X,axis)
>    ixix = np.rollaxis(ix,axis)
>    s = XX.shape
>    return np.rollaxis(XX[(ixix,)+tuple(np.indices(s)[1:])],0,axis+1)
>
> I'm always a little bit leery of using indices(), since it can produce
> very large arrays:
>

And fancy indexing too, as it tends to be slow. The axis=None case also
needs to be handled. I think this is best done in C where the code stacks
for argsort and arg{max,min} are good starting points. The possibility of
different integer types for the index array adds a bit of complication.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080618/4cef0552/attachment.html>

From alan.mcintyre at gmail.com  Wed Jun 18 14:42:22 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Wed, 18 Jun 2008 14:42:22 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <9457e7c80806180905g3058536ah16da7da057b237a2@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
	<1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
	<ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>
	<9457e7c80806180905g3058536ah16da7da057b237a2@mail.gmail.com>
Message-ID: <1d36917a0806181142y4438d165pe604b2607a117bf6@mail.gmail.com>

On Wed, Jun 18, 2008 at 12:05 PM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> a) Warn that Nose is becoming a dependency (next release).

Is "next release" referring to 1.2 or the release after that?  If it's
the release after 1.2, then I assume that 1.2 must still be able to
run all its tests without nose.


From stefan at sun.ac.za  Wed Jun 18 16:04:45 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 18 Jun 2008 22:04:45 +0200
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <1d36917a0806181142y4438d165pe604b2607a117bf6@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
	<1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
	<ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>
	<9457e7c80806180905g3058536ah16da7da057b237a2@mail.gmail.com>
	<1d36917a0806181142y4438d165pe604b2607a117bf6@mail.gmail.com>
Message-ID: <9457e7c80806181304x75364a91yee65a596dd870d1e@mail.gmail.com>

2008/6/18 Alan McIntyre <alan.mcintyre at gmail.com>:
> Is "next release" referring to 1.2 or the release after that?  If it's
> the release after 1.2, then I assume that 1.2 must still be able to
> run all its tests without nose.

Alternatively, we could distribute Nose inside of NumPy for one
release?  I suppose the Debian guys would shoot us :)

Cheers
St?fan


From robert.kern at gmail.com  Wed Jun 18 16:12:39 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 Jun 2008 15:12:39 -0500
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
	<1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
	<ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>
Message-ID: <3d375d730806181312q54a4a1a8k2ee39c6120e331b2@mail.gmail.com>

On Wed, Jun 18, 2008 at 10:42, Anne Archibald <peridot.faceted at gmail.com> wrote:
> 2008/6/17 Alan McIntyre <alan.mcintyre at gmail.com>:

>> You can replace ParametricTest with generators, as described here:
>>     http://scipy.org/scipy/scipy/wiki/TestingGuidelines
>
> Hmm. This won't work with the current version of numpy, will it? That
> is, it needs nose. (I run much code on the work machines, which I
> (thankfully) do not administer, and which are running a variety of
> ancient versions of numpy (some even have only Numeric, to support
> horrible quasi-in-house C extensions).) So I'd like to write my tests
> in a way that they can run on both new and old versions of numpy.

Changing your test suites to use nose does not mean that you have to
upgrade to a new numpy. The SVN trunk of numpy needs nose, but nose
does not need the SVN trunk of numpy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From robert.kern at gmail.com  Wed Jun 18 16:15:32 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 Jun 2008 15:15:32 -0500
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <1d36917a0806181142y4438d165pe604b2607a117bf6@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<485766A6.8090308@ar.media.kyoto-u.ac.jp>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
	<1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
	<ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>
	<9457e7c80806180905g3058536ah16da7da057b237a2@mail.gmail.com>
	<1d36917a0806181142y4438d165pe604b2607a117bf6@mail.gmail.com>
Message-ID: <3d375d730806181315p7ee53cf4v1bada478d87c362f@mail.gmail.com>

On Wed, Jun 18, 2008 at 13:42, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> On Wed, Jun 18, 2008 at 12:05 PM, St?fan van der Walt <stefan at sun.ac.za> wrote:
>> a) Warn that Nose is becoming a dependency (next release).
>
> Is "next release" referring to 1.2 or the release after that?  If it's
> the release after 1.2, then I assume that 1.2 must still be able to
> run all its tests without nose.

We've been already been making that warning for some time now, in the
proper venues. warning.warn() is good for DeprecationWarnings, but not
this. We are good to go for nose being used in 1.2.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco

From strawman at astraw.com  Wed Jun 18 17:02:51 2008
From: strawman at astraw.com (Andrew Straw)
Date: Wed, 18 Jun 2008 14:02:51 -0700
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <9457e7c80806181304x75364a91yee65a596dd870d1e@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>	<485766A6.8090308@ar.media.kyoto-u.ac.jp>	<48576737.3040908@ar.media.kyoto-u.ac.jp>	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>	<1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>	<ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>	<9457e7c80806180905g3058536ah16da7da057b237a2@mail.gmail.com>	<1d36917a0806181142y4438d165pe604b2607a117bf6@mail.gmail.com>
	<9457e7c80806181304x75364a91yee65a596dd870d1e@mail.gmail.com>
Message-ID: <485977FB.30009@astraw.com>

St?fan van der Walt wrote:
> 2008/6/18 Alan McIntyre <alan.mcintyre at gmail.com>:
>   
>> Is "next release" referring to 1.2 or the release after that?  If it's
>> the release after 1.2, then I assume that 1.2 must still be able to
>> run all its tests without nose.
>>     
>
> Alternatively, we could distribute Nose inside of NumPy for one
> release?  I suppose the Debian guys would shoot us :)
No, I don't think they'll shoot anyone. :) It can just be disabled with
a Debian specific patch to numpy. And since nose is LGPL (and therefore
DFSG kosher), I don't think there'd be any reason to even bother
re-making the source tarball. I'm quite sure the Debian packagers have
dealt with worse.


From alan.mcintyre at gmail.com  Wed Jun 18 17:12:58 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Wed, 18 Jun 2008 17:12:58 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <3d375d730806181315p7ee53cf4v1bada478d87c362f@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<48576737.3040908@ar.media.kyoto-u.ac.jp>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
	<1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
	<ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>
	<9457e7c80806180905g3058536ah16da7da057b237a2@mail.gmail.com>
	<1d36917a0806181142y4438d165pe604b2607a117bf6@mail.gmail.com>
	<3d375d730806181315p7ee53cf4v1bada478d87c362f@mail.gmail.com>
Message-ID: <1d36917a0806181412k1e906bf7hc42e9b1dd4c92ecd@mail.gmail.com>

On Wed, Jun 18, 2008 at 4:15 PM, Robert Kern <robert.kern at gmail.com> wrote:
> We've been already been making that warning for some time now, in the
> proper venues. warning.warn() is good for DeprecationWarnings, but not
> this. We are good to go for nose being used in 1.2.

Ok, so somebody tell me if I've got anything wrong here:

- All the tests included in NumPy 1.2 will be run using nose (whether
it's included in NumPy or not).  None of the tests will use any part
of the old test framework or assume that old test framework rules are
still valid.
- All the old test classes must be retained, with deprecation
warnings.  Third party tests that use them must still work when run
with Numpy 1.2.
- The signature of numpy.test in 1.2 will be backward compatible with
1.1, and it will at least return some indication of failure (if not
the same object as in 1.1).  This will, by the way, make it different
from the signature and behavior of scipy.test.
- The output (to stdout/stderr) of numpy.test in 1.2 will be different
from 1.1, since nose isn't displaying the total number of tests it
finds in each subpackage.


From robert.kern at gmail.com  Wed Jun 18 17:24:16 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 Jun 2008 16:24:16 -0500
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <1d36917a0806181412k1e906bf7hc42e9b1dd4c92ecd@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<91cf711d0806170626y3fc98ebq33664f4330828b51@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
	<1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
	<ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>
	<9457e7c80806180905g3058536ah16da7da057b237a2@mail.gmail.com>
	<1d36917a0806181142y4438d165pe604b2607a117bf6@mail.gmail.com>
	<3d375d730806181315p7ee53cf4v1bada478d87c362f@mail.gmail.com>
	<1d36917a0806181412k1e906bf7hc42e9b1dd4c92ecd@mail.gmail.com>
Message-ID: <3d375d730806181424s78755312o35dbfa58fa24b107@mail.gmail.com>

On Wed, Jun 18, 2008 at 16:12, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> On Wed, Jun 18, 2008 at 4:15 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> We've been already been making that warning for some time now, in the
>> proper venues. warning.warn() is good for DeprecationWarnings, but not
>> this. We are good to go for nose being used in 1.2.
>
> Ok, so somebody tell me if I've got anything wrong here:
>
> - All the tests included in NumPy 1.2 will be run using nose (whether
> it's included in NumPy or not).  None of the tests will use any part
> of the old test framework or assume that old test framework rules are
> still valid.

Right.

> - All the old test classes must be retained, with deprecation
> warnings.  Third party tests that use them must still work when run
> with Numpy 1.2.

Yes.

> - The signature of numpy.test in 1.2 will be backward compatible with
> 1.1, and it will at least return some indication of failure (if not
> the same object as in 1.1).  This will, by the way, make it different
> from the signature and behavior of scipy.test.

scipy.test() should be made to match numpy.test(). scipy.testing was a
staging ground for the nose changes in numpy, not a separate branch of
development.

For my preference, we should accept the old argument signature with a
deprecation warning but prefer the current signature. This is a little
hairy, but such is life with deprecations.

> - The output (to stdout/stderr) of numpy.test in 1.2 will be different
> from 1.1, since nose isn't displaying the total number of tests it
> finds in each subpackage.

That's fine.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From discerptor at gmail.com  Wed Jun 18 18:15:00 2008
From: discerptor at gmail.com (Joshua Lippai)
Date: Wed, 18 Jun 2008 18:15:00 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
Message-ID: <9911419a0806181515l5ccadf45ya6158cd4f8fc1c1f@mail.gmail.com>

The new testing system works well over here, built on Mac OS X 10.5.2
with GCC 4.2. No errors/failures, but there is that warning Charles
mentioned as well as the noticeable difference in speed between this
and the old tests.

Josh

On Mon, Jun 16, 2008 at 8:28 PM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> Hi all,
>
> I just checked in the switch to use nose to run unit tests.  Please
> let me know if you experience any difficulties as a result.
>
> Thanks,
> Alan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From stefan at sun.ac.za  Wed Jun 18 18:35:14 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 19 Jun 2008 00:35:14 +0200
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <9911419a0806181515l5ccadf45ya6158cd4f8fc1c1f@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<9911419a0806181515l5ccadf45ya6158cd4f8fc1c1f@mail.gmail.com>
Message-ID: <9457e7c80806181535k764e987dva3583fc9336bd839@mail.gmail.com>

2008/6/19 Joshua Lippai <discerptor at gmail.com>:
> The new testing system works well over here, built on Mac OS X 10.5.2
> with GCC 4.2. No errors/failures, but there is that warning Charles
> mentioned as well as the noticeable difference in speed between this
> and the old tests.

Nose does a more thorough job of finding tests (which takes some time
in itself).  It executes 1343 tests, compared to just over 1000
before.

Regards
St?fan


From stefan at sun.ac.za  Wed Jun 18 18:54:46 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 19 Jun 2008 00:54:46 +0200
Subject: [Numpy-discussion] parallel loop annotations
In-Reply-To: <20080618094747.93p5popyoskgggok@webmail.ualberta.ca>
References: <20080618094747.93p5popyoskgggok@webmail.ualberta.ca>
Message-ID: <9457e7c80806181554h701bf5e5xf34863bdb6be14b9@mail.gmail.com>

2008/6/18 Rahul Garg <garg1 at ualberta.ca>:
> I want to add a similar annotation to Python
> example usage :
>
> "pragma parallel for"
> for i in xrange(m): a[i] = b[i]*c[i]
>
> The interpreter ignores such strings and the code will of course
> execute serially but such strings can potentially be used by compilers.
> Any thoughts on this? What form of annotations would you like to see?
> Is it pythonic?

Is the idea to have the annotations completely separated from the
executed source?  Otherwise, I'd just as well make it implicit:

with parallel(i):
    for i in range(m):
        a[i] = b[i] * c[i]

That is also roughly the form that Fernando implemented for IPython1,
and is very intuitive.

What have you been using for other annotations so far?  Python itself
supports at least one other annotation:

# encoding: utf-8

so maybe

# pragma: parallel for

could work.

Regards
St?fan


From bevan07 at gmail.com  Wed Jun 18 19:41:01 2008
From: bevan07 at gmail.com (bevan)
Date: Wed, 18 Jun 2008 23:41:01 +0000 (UTC)
Subject: [Numpy-discussion] Find groups of true values or sequence of values
Message-ID: <loom.20080618T232816-826@post.gmane.org>

Hello,

I am looking for some pointers that will hopefully get me a round an issue I 
have hit.

I have a timeseries of river flow and would like to carry out some analysis on 
the recession periods.  That is anytime the values are decreasing.  I would 
like to restrict (mask) my data to any values that are in a continuous sequence 
of 3 or more (in the example below), that are decreasing in value.

Hopefully this example helps:

import numpy as np

Flow = np.array([15.4,20.5,19.4,18.7,18.6,35.5,34.8,25.1,26.7])
FlowDiff = np.diff(Flow)
boolFlowdiff = FlowDiff>0
MaskFlow = np.ma.masked_array(Flow[1:],boolFlowdiff)

print MaskFlow
[-- 19.4 18.7 18.6 -- 34.8 25.1 --]

The output I would like is
[-- 19.4 18.7 18.6 -- -- -- --]
Where the second groups is blanked because the sequence only has 2 members.


thanks for your time,

bevan



From garg1 at ualberta.ca  Wed Jun 18 19:43:49 2008
From: garg1 at ualberta.ca (Rahul Garg)
Date: Wed, 18 Jun 2008 17:43:49 -0600
Subject: [Numpy-discussion] parallel loop annotations
In-Reply-To: <9457e7c80806181554h701bf5e5xf34863bdb6be14b9@mail.gmail.com>
References: <20080618094747.93p5popyoskgggok@webmail.ualberta.ca>
	<9457e7c80806181554h701bf5e5xf34863bdb6be14b9@mail.gmail.com>
Message-ID: <20080618174349.myuvne1tco8c48cc@webmail.ualberta.ca>

Thanks for pointing them out. I will look into both of them. I like  
the "with" statement. From an implementation perspective, the "with"  
statement looks simpler since the Python parser seems to discard  
comments and I use the Python parser as front end for compiler.

For other annotations, I have been using either decorators or strings.
Decorators are used for type annotating functions. Strings just before
class declarations are currently necessary for telling the compiler
about fields of a class.

Strings can get a little ugly. Its better to use language features  
whenever possible. I will look into ipython's notation.

More comments welcome :)

thanks,
rahul


Quoting St?fan van der Walt <stefan at sun.ac.za>:

> 2008/6/18 Rahul Garg <garg1 at ualberta.ca>:
>> I want to add a similar annotation to Python
>> example usage :
>>
>> "pragma parallel for"
>> for i in xrange(m): a[i] = b[i]*c[i]
>>
>> The interpreter ignores such strings and the code will of course
>> execute serially but such strings can potentially be used by compilers.
>> Any thoughts on this? What form of annotations would you like to see?
>> Is it pythonic?
>
> Is the idea to have the annotations completely separated from the
> executed source?  Otherwise, I'd just as well make it implicit:
>
> with parallel(i):
>     for i in range(m):
>         a[i] = b[i] * c[i]
>
> That is also roughly the form that Fernando implemented for IPython1,
> and is very intuitive.
>
> What have you been using for other annotations so far?  Python itself
> supports at least one other annotation:
>
> # encoding: utf-8
>
> so maybe
>
> # pragma: parallel for
>
> could work.
>
> Regards
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>




From fperez.net at gmail.com  Thu Jun 19 00:48:59 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 18 Jun 2008 21:48:59 -0700
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
Message-ID: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>

Hi all,

after I updated the pyrex code to use cython, a bit of confusion led
me to using .pxi includes instead of .pxd files and cimport.  That
turned out to be mistaken, as later clarified by the Cython team:

http://codespeak.net/pipermail/cython-dev/2008-April/000628.html

Would anyone object to my fixing the code to the new convention?  The
changes are minimal, the .pxi files can just be renamed and then a few
trivial modifications are done.  But I figured since this gets shipped
as canonical example code, vetting from the core developers is a good
idea.

In particular, I used:

# Import the pieces of the Python C API we need to use (from c_python.pxd):
cimport c_python as py

# Import the NumPy C API (from c_numpy.pxd)
cimport c_numpy as cnp

# Import the NumPy module for access to its usual Python API
import numpy as np

I don't know if those are the conventions people want to use.  I went
with 'cnp' for symmetry, and plain 'py' for simplicity.

The patch is otherwise an inconsequential adding of those prefixes,
since the old include mechanism was equivalent to an 'import *' while
this gives us a namespace.  I've attached the diff for the numpyx file
here for review (the actual diff is lager because of the renames, but
there's no code changes there).

Stefan, core team?

Cheers,

f
-------------- next part --------------
A non-text attachment was scrubbed...
Name: numpyx.diff
Type: text/x-diff
Size: 3285 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080618/d2ba2727/attachment.diff>

From robert.kern at gmail.com  Thu Jun 19 00:54:26 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 Jun 2008 23:54:26 -0500
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
Message-ID: <3d375d730806182154je3294afp34e523d79bd6a779@mail.gmail.com>

On Wed, Jun 18, 2008 at 23:48, Fernando Perez <fperez.net at gmail.com> wrote:
> Stefan, core team?

Knock yourself out.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From fperez.net at gmail.com  Thu Jun 19 02:17:26 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 18 Jun 2008 23:17:26 -0700
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <3d375d730806182154je3294afp34e523d79bd6a779@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<3d375d730806182154je3294afp34e523d79bd6a779@mail.gmail.com>
Message-ID: <db6b5ecc0806182317m6095eb5bs49a0653fdc086ade@mail.gmail.com>

On Wed, Jun 18, 2008 at 9:54 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Wed, Jun 18, 2008 at 23:48, Fernando Perez <fperez.net at gmail.com> wrote:
>> Stefan, core team?
>
> Knock yourself out.

Thanks.  Committed in r5298.

Cheers,

f


From stefan at sun.ac.za  Thu Jun 19 05:31:32 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 19 Jun 2008 11:31:32 +0200
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
Message-ID: <9457e7c80806190231wca4d7ds653a943e24a5077a@mail.gmail.com>

2008/6/19 Fernando Perez <fperez.net at gmail.com>:
> after I updated the pyrex code to use cython, a bit of confusion led
> me to using .pxi includes instead of .pxd files and cimport.  That
> turned out to be mistaken, as later clarified by the Cython team:
>
> http://codespeak.net/pipermail/cython-dev/2008-April/000628.html

You weren't the only one confused.  In fact, reading the FAQ didn't help me:

http://wiki.cython.org/FAQ#head-a3d09805a03c014080feff45fe8e22024d473d62

The NumPy definitions are all external, so why shouldn't they go in a .pxi?

Some other questions:

- There's a bug in the latest Cython annotation, preventing the
example from compiling.  Workaround: replace "cython -a" with
"cython".

- We could also add a scons buildfile, to show how extensions are
built with and without numscons.

- import_array().  Can we put it at the bottom of c_numpy.pxd, so that
users don't have to import it every time?

- Author names: should we remove author names from module headers and
add them in THANKS.txt or some other contributor list?

Regards
St?fan


From peridot.faceted at gmail.com  Thu Jun 19 10:26:43 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 19 Jun 2008 10:26:43 -0400
Subject: [Numpy-discussion] Strange behavior for argsort() and take()
In-Reply-To: <e06186140806181134l1cce08d7kab0af9416baa7f77@mail.gmail.com>
References: <908F6F2E-C070-4DD2-AFDF-B1029A5A9336@dal.ca>
	<ce557a360806180810h382208f1pda1136362d050f08@mail.gmail.com>
	<9457e7c80806180953l3542792bh2e5122763615ad64@mail.gmail.com>
	<ce557a360806181048x4272a8d3ub592c166feb647d@mail.gmail.com>
	<e06186140806181134l1cce08d7kab0af9416baa7f77@mail.gmail.com>
Message-ID: <ce557a360806190726j2c1de89al879b7dde48f7dc31@mail.gmail.com>

2008/6/18 Charles R Harris <charlesr.harris at gmail.com>:
>
>
> On Wed, Jun 18, 2008 at 11:48 AM, Anne Archibald <peridot.faceted at gmail.com>
> wrote:
>>
>> 2008/6/18 St?fan van der Walt <stefan at sun.ac.za>:
>> > 2008/6/18 Anne Archibald <peridot.faceted at gmail.com>:
>> >> In [7]: x.take(x.argsort())
>> >> Out[7]: array([ 0. ,  0.1,  0.2,  0.3])
>> >>
>> >> If you would like to think of it more mathematically, when you feed
>> >> np.argsort() an array that represents a permutation of the numbers
>> >> 0,1,...,n-1, you get back the inverse permutation. When you pass a
>> >> permutation as the argument to x.take(), you apply the permutation to
>> >> x. (You can also invert a permutation by feeding it as indices to
>> >> arange(n)).
>> >>
>> >> I have been tempted to write some support functions for manipulating
>> >> permutations, but I'm not sure how generally useful they would be.
>> >
>> > Do we have an easy way of grabbing the elements out of an array that
>> > correspond to
>> >
>> > argmax(x, axis) ?
>> >
>> > `take` won't do the job; there `axis` has a different meaning.
>>
>> Well, here's a quick hack:
>>
>> def take_axis(X, ix, axis):
>>    XX = np.rollaxis(X,axis)
>>    s = XX.shape
>>    return XX[(ix,)+tuple(np.indices(s[1:]))]
>>
>> And a version that works for argsort()ing:
>>
>> def take_axis_argsort(X, ix, axis):
>>    XX = np.rollaxis(X,axis)
>>    ixix = np.rollaxis(ix,axis)
>>    s = XX.shape
>>    return np.rollaxis(XX[(ixix,)+tuple(np.indices(s)[1:])],0,axis+1)
>>
>> I'm always a little bit leery of using indices(), since it can produce
>> very large arrays:
>
> And fancy indexing too, as it tends to be slow. The axis=None case also
> needs to be handled. I think this is best done in C where the code stacks
> for argsort and arg{max,min} are good starting points. The possibility of
> different integer types for the index array adds a bit of complication.

Actually, apart from the sizes of the index arrays, I don't really
think fancy indexing can be beat. I suspect that one could write
"tuple_indices" that used broadcasting to get the index arrays, if you
were willing to return a tuple. That would bring these up to nearly C
speed, I think.

Of course they're not general routines, but I think with a bit of work
they might serve.

Anne

From alan.mcintyre at gmail.com  Thu Jun 19 12:59:48 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Thu, 19 Jun 2008 12:59:48 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <3d375d730806181424s78755312o35dbfa58fa24b107@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<1d36917a0806170753j27d76569jf022d1f482ced678@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
	<1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
	<ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>
	<9457e7c80806180905g3058536ah16da7da057b237a2@mail.gmail.com>
	<1d36917a0806181142y4438d165pe604b2607a117bf6@mail.gmail.com>
	<3d375d730806181315p7ee53cf4v1bada478d87c362f@mail.gmail.com>
	<1d36917a0806181412k1e906bf7hc42e9b1dd4c92ecd@mail.gmail.com>
	<3d375d730806181424s78755312o35dbfa58fa24b107@mail.gmail.com>
Message-ID: <1d36917a0806190959y72e4caa9i44d4e03f836753d4@mail.gmail.com>

On Wed, Jun 18, 2008 at 5:24 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> - The signature of numpy.test in 1.2 will be backward compatible with
>> 1.1, and it will at least return some indication of failure (if not
>> the same object as in 1.1).  This will, by the way, make it different
>> from the signature and behavior of scipy.test.
>
> scipy.test() should be made to match numpy.test(). scipy.testing was a
> staging ground for the nose changes in numpy, not a separate branch of
> development.

Ok. Jarrod mentioned that the intent is to change SciPy over to use
numpy.testing (and remove scipy.testing) once NumPy's nose transition
is complete.  Is that something that something that will happen
simultaneously in the next release, or will SciPy lag behind one
cycle?  Either way, at some point in the next few weeks I'll try
making that change locally just to see if it turns up any surprising
requirements in the NumPy side.

> For my preference, we should accept the old argument signature with a
> deprecation warning but prefer the current signature. This is a little
> hairy, but such is life with deprecations.

Do you want a blanket warning, or only a warning if test() gets
keyword arguments from the old signature?

Also, since the numpy.testing.ScipyTestCase and ScipyTest classes
already had deprecation warnings in 1.1, I'll leave them out in 1.2,
if that's ok.

Thanks,
Alan


From charlesr.harris at gmail.com  Thu Jun 19 13:54:32 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 19 Jun 2008 11:54:32 -0600
Subject: [Numpy-discussion] CFLAGS= causes bad build
Message-ID: <e06186140806191054m6e39a32agd84ddfec5ba1ab1f@mail.gmail.com>

Having an empty CFLAGS environment variable and running

$ python setup.py build

emits tons of these messages:

numpy/core/src/scalartypes.inc.src: In function 'scalar_value':
numpy/core/src/scalartypes.inc.src:77: warning: dereferencing type-punned
pointer will break strict-aliasing rules

The compile flags are missing -fno-strict-aliasing. I don't know if this is
a problem with the python distutils or what, but it would be nice if it
didn't occur.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080619/c2fd3dc9/attachment.html>

From lists at informa.tiker.net  Wed Jun 18 21:44:55 2008
From: lists at informa.tiker.net (Andreas =?utf-8?q?Kl=C3=B6ckner?=)
Date: Wed, 18 Jun 2008 21:44:55 -0400
Subject: [Numpy-discussion] Fancy index assign ignores extra assignees
Message-ID: <200806182144.58066.lists@informa.tiker.net>

Hi all,

Is this supposed to be like that, i.e. is the fancy __setitem__ supposed to 
not complain about unused assignees?

>>> v = zeros((10,))
>>> z = [1,2,5]
>>> v[z] = [1,2,4,5]
>>> v
array([ 0.,  1.,  2.,  0.,  0.,  4.,  0.,  0.,  0.,  0.])

Contrast with:

>>> v[1:3] = [1,2,3,4]
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ValueError: shape mismatch: objects cannot be broadcast to a single shape

Andreas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080618/6c890b61/attachment.sig>

From robert.kern at gmail.com  Thu Jun 19 14:36:12 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 19 Jun 2008 13:36:12 -0500
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <1d36917a0806190959y72e4caa9i44d4e03f836753d4@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<ce557a360806171715y3b967982ya22542047eefca27@mail.gmail.com>
	<1d36917a0806172044o4f58c19ehec6fb9b8df16a1df@mail.gmail.com>
	<ce557a360806180842k2246428dmdaee99768e96140d@mail.gmail.com>
	<9457e7c80806180905g3058536ah16da7da057b237a2@mail.gmail.com>
	<1d36917a0806181142y4438d165pe604b2607a117bf6@mail.gmail.com>
	<3d375d730806181315p7ee53cf4v1bada478d87c362f@mail.gmail.com>
	<1d36917a0806181412k1e906bf7hc42e9b1dd4c92ecd@mail.gmail.com>
	<3d375d730806181424s78755312o35dbfa58fa24b107@mail.gmail.com>
	<1d36917a0806190959y72e4caa9i44d4e03f836753d4@mail.gmail.com>
Message-ID: <3d375d730806191136ic39bf23x22182f3d8758141d@mail.gmail.com>

On Thu, Jun 19, 2008 at 11:59, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> On Wed, Jun 18, 2008 at 5:24 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>> - The signature of numpy.test in 1.2 will be backward compatible with
>>> 1.1, and it will at least return some indication of failure (if not
>>> the same object as in 1.1).  This will, by the way, make it different
>>> from the signature and behavior of scipy.test.
>>
>> scipy.test() should be made to match numpy.test(). scipy.testing was a
>> staging ground for the nose changes in numpy, not a separate branch of
>> development.
>
> Ok. Jarrod mentioned that the intent is to change SciPy over to use
> numpy.testing (and remove scipy.testing) once NumPy's nose transition
> is complete.  Is that something that something that will happen
> simultaneously in the next release, or will SciPy lag behind one
> cycle?

No scipy was released with the new scipy.testing stuff, so scipy
itself is the only customer. We can transition ourselves.

> Either way, at some point in the next few weeks I'll try
> making that change locally just to see if it turns up any surprising
> requirements in the NumPy side.
>
>> For my preference, we should accept the old argument signature with a
>> deprecation warning but prefer the current signature. This is a little
>> hairy, but such is life with deprecations.
>
> Do you want a blanket warning, or only a warning if test() gets
> keyword arguments from the old signature?

The latter, please. I know it's tricky, but it is doable.

> Also, since the numpy.testing.ScipyTestCase and ScipyTest classes
> already had deprecation warnings in 1.1, I'll leave them out in 1.2,
> if that's ok.

Yes.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From fperez.net at gmail.com  Thu Jun 19 16:16:38 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 19 Jun 2008 13:16:38 -0700
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <9457e7c80806190231wca4d7ds653a943e24a5077a@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<9457e7c80806190231wca4d7ds653a943e24a5077a@mail.gmail.com>
Message-ID: <db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>

On Thu, Jun 19, 2008 at 2:31 AM, St?fan van der Walt <stefan at sun.ac.za> wrote:

> You weren't the only one confused.  In fact, reading the FAQ didn't help me:
>
> http://wiki.cython.org/FAQ#head-a3d09805a03c014080feff45fe8e22024d473d62
>
> The NumPy definitions are all external, so why shouldn't they go in a .pxi?

I'm not up to speed on all the details, but the cython guys were
pretty clear on the final piece of advice, so I'm just taking it at
face value here.  I haven't followed the discussion on pxi vs pxd in
detail (there have been several on their list), but the '.pxi files
are mostly a historical artifact' argument led me to stop caring and
start using pxds for good.

> Some other questions:
>
> - There's a bug in the latest Cython annotation, preventing the
> example from compiling.  Workaround: replace "cython -a" with
> "cython".

Mmh, I have 0.9.8 (their last release) and it works for me...

> - We could also add a scons buildfile, to show how extensions are
> built with and without numscons.

Yup.

> - import_array().  Can we put it at the bottom of c_numpy.pxd, so that
> users don't have to import it every time?

Sounds fine to me, but I don't know if there could be a subtle problem
with pxds used by multiple pyx files.  I'm not comfortable enough with
cython to answer the question.

> - Author names: should we remove author names from module headers and
> add them in THANKS.txt or some other contributor list?

I left them as they were.  If we want to revisit the discussion on
per-file author tags vs per-project files that's fine, but it's a
separate discussion from the cython technical questions.

Cheers,

f


From matthew.brett at gmail.com  Thu Jun 19 16:19:43 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 19 Jun 2008 20:19:43 +0000
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<9457e7c80806190231wca4d7ds653a943e24a5077a@mail.gmail.com>
	<db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>
Message-ID: <1e2af89e0806191319v6333da18g95ce4eafad8b5a46@mail.gmail.com>

Hi,

>> - There's a bug in the latest Cython annotation, preventing the
>> example from compiling.  Workaround: replace "cython -a" with
>> "cython".
>
> Mmh, I have 0.9.8 (their last release) and it works for me...

I've got the same version, works for me too...

Matthew


From stefan at sun.ac.za  Thu Jun 19 17:08:15 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 19 Jun 2008 23:08:15 +0200
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <1e2af89e0806191319v6333da18g95ce4eafad8b5a46@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<9457e7c80806190231wca4d7ds653a943e24a5077a@mail.gmail.com>
	<db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>
	<1e2af89e0806191319v6333da18g95ce4eafad8b5a46@mail.gmail.com>
Message-ID: <9457e7c80806191408x24278fc1ua1a3da35c070dac@mail.gmail.com>

2008/6/19 Matthew Brett <matthew.brett at gmail.com>:
>>> - There's a bug in the latest Cython annotation, preventing the
>>> example from compiling.  Workaround: replace "cython -a" with
>>> "cython".
>>
>> Mmh, I have 0.9.8 (their last release) and it works for me...
>
> I've got the same version, works for me too...

I'm referring to the latest version from hg.

Regards
St?fan


From thrabe at burnham.org  Thu Jun 19 17:47:07 2008
From: thrabe at burnham.org (Thomas Hrabe)
Date: Thu, 19 Jun 2008 14:47:07 -0700
Subject: [Numpy-discussion] embedded arrays - continued
References: <A7E1AE993C63414B8B2F762E13BF4D898E4E9F@MAIL.burnham.org>
	<3d375d730806061527w42fccb9dt8e5cc857934d4196@mail.gmail.com>
Message-ID: <A7E1AE993C63414B8B2F762E13BF4D898E4EA7@MAIL.burnham.org>

Hi,

I am still tracking down the memory bug in my C module and I have found a odd inconsistency.

I call import_array in the init function of the module and I want to check in the later code if numpy has been initialised before by checking  the 
PyArray_API
value if it is NULL.

I have found out that PyArray_API in the init function is in a different memory location than for my later call. Thus I assume that PyArray_API is not static. Could someone please explain me the code snipplet of the PyArray_API definition in __multiarray_api.h? What do the constants represent? 

#if defined(NO_IMPORT) || defined(NO_IMPORT_ARRAY)
extern void **PyArray_API;
#else
#if defined(PY_ARRAY_UNIQUE_SYMBOL)
void **PyArray_API;
#else
static void **PyArray_API=NULL;
#endif
#endif


Best,
Thomas

-----Urspr?ngliche Nachricht-----
Von: numpy-discussion-bounces at scipy.org im Auftrag von Robert Kern
Gesendet: Fr 06.06.2008 15:27
An: Discussion of Numerical Python
Betreff: Re: [Numpy-discussion] embedded arrays
 
On Fri, Jun 6, 2008 at 17:10, Thomas Hrabe <thrabe at burnham.org> wrote:
> Hi all,
>
> while writing a extension module in C++ for python & numpy, I find a strange
> error.
>
> I can send and retrieve numpy arrays to and from my module.
> But python stops if I do the following:
>
> a = numpy.array([[1.1,2,3],[4,5,6]])
> PM.put(a,'a')   //send a to the module
> b = PM.get('a') //get a identical copy from the module
> print b
> array([[ 1.1,  2. ,  3. ],
>        [ 4. ,  5. ,  6. ]])
>
> b.__class__
> Out[36]: <type 'numpy.ndarray'>
>
>
> perfect, until
> a == b
>
> the interpreter does not continue from here...
> I can add values to to b, everything, but a == b simply crashes ...?
>
> Does anybody have a clue for this problem?

Not really. It probably depends on some details with your interfacing.
Since we don't have access to your code, we don't have much to go on.
You might have buggy reference counting or perhaps you gave the numpy
ndarray ownership of the array's memory when it actually shouldn't.
Memory issues can be a bit finicky where everything will work for a
while, then crash.

Try running your program under a C debugger like gdb so we can get a
backtrace. That might give us some more ideas about exactly where
problems are occurring.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 4275 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080619/7a1b41af/attachment.bin>

From matthew.brett at gmail.com  Thu Jun 19 18:46:31 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 19 Jun 2008 22:46:31 +0000
Subject: [Numpy-discussion] Cython headers in numpy include
In-Reply-To: <1e2af89e0806191454v455471ect7fd430b5871cc00d@mail.gmail.com>
References: <1e2af89e0806191454v455471ect7fd430b5871cc00d@mail.gmail.com>
Message-ID: <1e2af89e0806191546u7315d922v9a4e3275a5f9112c@mail.gmail.com>

Hi,

Following on from Fernando's post about his Cython example, I would
like to suggest adding his .pxd files to the standard numpy include
directory, fetched with np.get_include()

Why:
Because anyone writing a Cython extension for numpy will need these
files.  At the moment, this means that everyone will have need a copy
of these files
in their package directory or similar, and it will be less likely that
people will share any improvements, fixes or extensions.

Alternatively, or additionally, add np.get_pxd_include()

Any thoughts?

Matthew


From vinjvinj at gmail.com  Thu Jun 19 18:48:28 2008
From: vinjvinj at gmail.com (Vineet Jain (gmail))
Date: Thu, 19 Jun 2008 18:48:28 -0400
Subject: [Numpy-discussion] Is there a function to calculate
	ecnomic	beta coefficient in numpy given two time series data.
References: <000c01c8c6a4$9f0fad00$dd2f0700$@com>
	<f4f93d420806041804n5c0302b3le70c093673140a79@mail.gmail.com> 
Message-ID: <03c801c8d25e$9821f460$c865dd20$@com>

I took the following code and applied it to aapl and qqqq time series (see
attached file):

import numpy as np
lstsq = np.linalg.lstsq
from numpy import float64, extract

aapl_array = np.array([row[0] for row in stock_and_market_values])
qqqq_array = np.array([row[1] for row in stock_and_market_values])

A = np.ones((len(qqqq_array), 2), dtype=float64)
A[:,0] = aapl_array
result = lstsq(A, qqqq_array)
print result

The result is:

(array([  0.13851625,  29.57888955]), array([ 144.23291488]), 2, array([
639.591
08529,    0.94451427]))

And the beta comes out to be 0.138 which is a low. It should be closer to 2.
Any idea on what I'm doing wrong.

Vineet


-----Original Message-----
From: Vineet Jain (gmail) [mailto:vinjvinj at gmail.com] 
Sent: Wednesday, June 04, 2008 9:24 PM
To: 'Discussion of Numerical Python'
Subject: RE: [Numpy-discussion] Is there a function to calculate ecnomic
beta coefficient in numpy given two time series data.

Thanks Keith!

-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Keith Goodman
Sent: Wednesday, June 04, 2008 9:04 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Is there a function to calculate ecnomic
beta coefficient in numpy given two time series data.

On Wed, Jun 4, 2008 at 5:39 PM, Vineet Jain (gmail) <vinjvinj at gmail.com>
wrote:
> Timeseries1 = daily or weekly close of stock a
>
> Timeseries2 = daily or weekly close of market index (spx, qqqq, etc)
>
>
>
> Beta of stock a is what I would like to compute as explained in this
article
> on Wikipedia:
>
>
>
> http://en.wikipedia.org/wiki/Beta_coefficient
>
>
>
> I'm trying to compute the beta of entire stock market (about 15,000
> instruments) one stock at a time and would like to use the spiders and
qqqq
> to represent the overall market.

Unless you run out of memory (or if you want to handle missing returns
which may occur on different dates in each series) there is no need to
do it one stock at a time:

>> import numpy.matlib as mp
>> mrkt = mp.randn(250,1)  # <----- 250 days of returns
>> stocks = mp.randn(250, 4)  # <---- 4 stocks
>> beta, resids, rank, s = mp.linalg.lstsq(mrkt, stocks)
>> beta
   matrix([[-0.01701467,  0.11242168,  0.00207398,  0.03920687]])

And you can use mp.log to convert the price ratios to log returns.

It might also be useful to shuffle (mp.random.shuffle) the market
returns and repeat the beta calculation many times to estimate the
noise level of your beta estimates.
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: beta.py
Type: application/octet-stream
Size: 9365 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080619/7a8ffed1/attachment.obj>

From reggie at merfinllc.com  Thu Jun 19 18:55:59 2008
From: reggie at merfinllc.com (Reggie Dugard)
Date: Thu, 19 Jun 2008 15:55:59 -0700
Subject: [Numpy-discussion] ma's maximum_fill_value function in 1.1.0
Message-ID: <1213916159.6827.25.camel@zorro.merfinllc.com>

This is just a minor question/problem with the new numpy.ma in version
1.1.0.

Because maximum and minimum in ma lack an accumulate attribute, I've
duplicated the functionality using the
maximum_fill_value/minimum_fill_value functions and doing something
like:

np.ma.masked_array(np.maximum.accumulate(np.ma.filled(x, np.ma.maximum_fill_value(x))), x.mask)

In version 1.0.4 of numpy, maximum_fill_value/minimum_fill_value were
located in ma's namespace so the above line would work.  In the latest
version,  the ma module has an __all__ which does not include
maximum_fill_value/minimum_fill_value and I'm forced to get them from
numpy.ma.core (which doesn't seem like a very clean way of doing it).

So I guess my question is:  was this done intentionally and if so why,
or can these functions be added to __all__ to get the old behavior
back. 

Thanks for your help with this.
-- 
Reggie Dugard <reggie at merfinllc.com>
Merfin, LLC



From robert.kern at gmail.com  Thu Jun 19 18:59:55 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 19 Jun 2008 17:59:55 -0500
Subject: [Numpy-discussion] Cython headers in numpy include
In-Reply-To: <1e2af89e0806191546u7315d922v9a4e3275a5f9112c@mail.gmail.com>
References: <1e2af89e0806191454v455471ect7fd430b5871cc00d@mail.gmail.com>
	<1e2af89e0806191546u7315d922v9a4e3275a5f9112c@mail.gmail.com>
Message-ID: <3d375d730806191559w1460188ao758302a82a9dfc87@mail.gmail.com>

On Thu, Jun 19, 2008 at 17:46, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> Following on from Fernando's post about his Cython example, I would
> like to suggest adding his .pxd files to the standard numpy include
> directory, fetched with np.get_include()

I am a bit hesitant to add them to a standard "public" path until they
are close to complete.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From matthew.brett at gmail.com  Thu Jun 19 19:02:10 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 19 Jun 2008 23:02:10 +0000
Subject: [Numpy-discussion] Cython headers in numpy include
In-Reply-To: <3d375d730806191559w1460188ao758302a82a9dfc87@mail.gmail.com>
References: <1e2af89e0806191454v455471ect7fd430b5871cc00d@mail.gmail.com>
	<1e2af89e0806191546u7315d922v9a4e3275a5f9112c@mail.gmail.com>
	<3d375d730806191559w1460188ao758302a82a9dfc87@mail.gmail.com>
Message-ID: <1e2af89e0806191602v213f376eu3d97ef8181afd37c@mail.gmail.com>

>> Following on from Fernando's post about his Cython example, I would
>> like to suggest adding his .pxd files to the standard numpy include
>> directory, fetched with np.get_include()
>
> I am a bit hesitant to add them to a standard "public" path until they
> are close to complete.

As usual, fair comment.  Do we then have the goal before us of
inclusion in a public path when they are close to complete?

Matthew


From robert.kern at gmail.com  Thu Jun 19 19:08:22 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 19 Jun 2008 18:08:22 -0500
Subject: [Numpy-discussion] Is there a function to calculate ecnomic
	beta coefficient in numpy given two time series data.
In-Reply-To: <03c801c8d25e$9821f460$c865dd20$@com>
References: <000c01c8c6a4$9f0fad00$dd2f0700$@com>
	<f4f93d420806041804n5c0302b3le70c093673140a79@mail.gmail.com>
	<03c801c8d25e$9821f460$c865dd20$@com>
Message-ID: <3d375d730806191608l540f259je8327bbc4dd71de0@mail.gmail.com>

On Thu, Jun 19, 2008 at 17:48, Vineet Jain (gmail) <vinjvinj at gmail.com> wrote:
> I took the following code and applied it to aapl and qqqq time series (see
> attached file):
>
> import numpy as np
> lstsq = np.linalg.lstsq
> from numpy import float64, extract
>
> aapl_array = np.array([row[0] for row in stock_and_market_values])
> qqqq_array = np.array([row[1] for row in stock_and_market_values])
>
> A = np.ones((len(qqqq_array), 2), dtype=float64)
> A[:,0] = aapl_array
> result = lstsq(A, qqqq_array)
> print result
>
> The result is:
>
> (array([  0.13851625,  29.57888955]), array([ 144.23291488]), 2, array([
> 639.591
> 08529,    0.94451427]))
>
> And the beta comes out to be 0.138 which is a low. It should be closer to 2.
> Any idea on what I'm doing wrong.

Beta is supposed to be calculated on returns, not prices.

aapl_ret = np.log(aapl_array[1:] / aapl_array[:-1])
qqqq_ret = np.log(qqqq_array[1:] / qqqq_array[:-1])

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From robert.kern at gmail.com  Thu Jun 19 19:12:53 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 19 Jun 2008 18:12:53 -0500
Subject: [Numpy-discussion] Cython headers in numpy include
In-Reply-To: <1e2af89e0806191602v213f376eu3d97ef8181afd37c@mail.gmail.com>
References: <1e2af89e0806191454v455471ect7fd430b5871cc00d@mail.gmail.com>
	<1e2af89e0806191546u7315d922v9a4e3275a5f9112c@mail.gmail.com>
	<3d375d730806191559w1460188ao758302a82a9dfc87@mail.gmail.com>
	<1e2af89e0806191602v213f376eu3d97ef8181afd37c@mail.gmail.com>
Message-ID: <3d375d730806191612o14917851wddf33f46a01ec7c6@mail.gmail.com>

On Thu, Jun 19, 2008 at 18:02, Matthew Brett <matthew.brett at gmail.com> wrote:
>>> Following on from Fernando's post about his Cython example, I would
>>> like to suggest adding his .pxd files to the standard numpy include
>>> directory, fetched with np.get_include()
>>
>> I am a bit hesitant to add them to a standard "public" path until they
>> are close to complete.
>
> As usual, fair comment.  Do we then have the goal before us of
> inclusion in a public path when they are close to complete?

Cython looks in the include_dirs for .pxd files, right? Then yes, I
would support putting them alongside arrayobject.h. If possible, we
should be careful to make sure that the .pxds work with Pyrex, too. I
got burned recently by a change to .pxd semantics in Pyrex 0.9.8, so
this could be tricky.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From matthew.brett at gmail.com  Thu Jun 19 19:18:19 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 19 Jun 2008 23:18:19 +0000
Subject: [Numpy-discussion] Cython headers in numpy include
In-Reply-To: <3d375d730806191612o14917851wddf33f46a01ec7c6@mail.gmail.com>
References: <1e2af89e0806191454v455471ect7fd430b5871cc00d@mail.gmail.com>
	<1e2af89e0806191546u7315d922v9a4e3275a5f9112c@mail.gmail.com>
	<3d375d730806191559w1460188ao758302a82a9dfc87@mail.gmail.com>
	<1e2af89e0806191602v213f376eu3d97ef8181afd37c@mail.gmail.com>
	<3d375d730806191612o14917851wddf33f46a01ec7c6@mail.gmail.com>
Message-ID: <1e2af89e0806191618o5885660dm3fbcf5e757eabb7b@mail.gmail.com>

Hi,

> Cython looks in the include_dirs for .pxd files, right? Then yes, I
> would support putting them alongside arrayobject.h.

Yes, right - Fernando just pointed me to the reference - quoting him:

"As indicated here:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/sharing.html#mozTocId559554

the search path for 'cimport foo' to find foo.pxd is the -I search
path of the compiler"

> If possible, we
> should be careful to make sure that the .pxds work with Pyrex, too. I
> got burned recently by a change to .pxd semantics in Pyrex 0.9.8, so
> this could be tricky.

I don't have deep enough understanding of the differences to give a
reliable answer as to whether this is practical.  If it wasn't
practical, we could either standardize to Cython, or (less likely
surely) Pyrex, or have two sets of pxd files,

Matthew


From vinjvinj at gmail.com  Thu Jun 19 19:18:50 2008
From: vinjvinj at gmail.com (Vineet Jain (gmail))
Date: Thu, 19 Jun 2008 19:18:50 -0400
Subject: [Numpy-discussion] Is there a function to calculate
	ecnomic	beta coefficient in numpy given two time series data.
In-Reply-To: <3d375d730806191608l540f259je8327bbc4dd71de0@mail.gmail.com>
References: <000c01c8c6a4$9f0fad00$dd2f0700$@com>	<f4f93d420806041804n5c0302b3le70c093673140a79@mail.gmail.com>	<03c801c8d25e$9821f460$c865dd20$@com>
	<3d375d730806191608l540f259je8327bbc4dd71de0@mail.gmail.com>
Message-ID: <03d201c8d262$d5facab0$81f06010$@com>

Thanks. 

With your changes, I'm getting a beta of 0.23 which I think is still wrong.
I would have expected a value closer to 2. 

Vineet


-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Robert Kern
Sent: Thursday, June 19, 2008 7:08 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Is there a function to calculate ecnomic
beta coefficient in numpy given two time series data.

On Thu, Jun 19, 2008 at 17:48, Vineet Jain (gmail) <vinjvinj at gmail.com>
wrote:
> I took the following code and applied it to aapl and qqqq time series (see
> attached file):
>
> import numpy as np
> lstsq = np.linalg.lstsq
> from numpy import float64, extract
>
> aapl_array = np.array([row[0] for row in stock_and_market_values])
> qqqq_array = np.array([row[1] for row in stock_and_market_values])
>
> A = np.ones((len(qqqq_array), 2), dtype=float64)
> A[:,0] = aapl_array
> result = lstsq(A, qqqq_array)
> print result
>
> The result is:
>
> (array([  0.13851625,  29.57888955]), array([ 144.23291488]), 2, array([
> 639.591
> 08529,    0.94451427]))
>
> And the beta comes out to be 0.138 which is a low. It should be closer to
2.
> Any idea on what I'm doing wrong.

Beta is supposed to be calculated on returns, not prices.

aapl_ret = np.log(aapl_array[1:] / aapl_array[:-1])
qqqq_ret = np.log(qqqq_array[1:] / qqqq_array[:-1])

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From robert.kern at gmail.com  Thu Jun 19 19:32:14 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 19 Jun 2008 18:32:14 -0500
Subject: [Numpy-discussion] Is there a function to calculate ecnomic
	beta coefficient in numpy given two time series data.
In-Reply-To: <03d201c8d262$d5facab0$81f06010$@com>
References: <000c01c8c6a4$9f0fad00$dd2f0700$@com>
	<f4f93d420806041804n5c0302b3le70c093673140a79@mail.gmail.com>
	<03c801c8d25e$9821f460$c865dd20$@com>
	<3d375d730806191608l540f259je8327bbc4dd71de0@mail.gmail.com>
	<03d201c8d262$d5facab0$81f06010$@com>
Message-ID: <3d375d730806191632k4dd8920au87bfed410e5365a8@mail.gmail.com>

On Thu, Jun 19, 2008 at 18:18, Vineet Jain (gmail) <vinjvinj at gmail.com> wrote:
> Thanks.
>
> With your changes, I'm getting a beta of 0.23 which I think is still wrong.
> I would have expected a value closer to 2.

To get the beta of AAPL versus the index QQQQ, you need to swap
aapl_ret and qqqq_ret when you construct the arguments to lstsq().
That gives a beta of 1.0-1.2 depending on whether or not you omit the
outlier. Google Finance reports a beta of 2.4 for AAPL recently, but
this is probably against the S&P 500 rather than a tech index like
QQQQ. Try computing the betas of AAPL and QQQQ against the S&P 500.


import numpy as np
lstsq = np.linalg.lstsq
from numpy import float64, extract

aapl_array = np.array([row[0] for row in stock_and_market_values])
aapl_ret = np.log(aapl_array[1:] / aapl_array[:-1])
qqqq_array = np.array([row[1] for row in stock_and_market_values])
qqqq_ret = np.log(qqqq_array[1:] / qqqq_array[:-1])

# There's a big outlier in the AAPL returns. AAPL did really well one day.
#m = aapl_ret < 0.1
#aapl_ret = aapl_ret[m]
#qqqq_ret = qqqq_ret[m]

A = np.ones((len(qqqq_ret), 2), dtype=float64)
A[:,0] = qqqq_ret
result = lstsq(A, aapl_ret)
print result

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From stefan at sun.ac.za  Thu Jun 19 20:02:25 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 20 Jun 2008 02:02:25 +0200
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<9457e7c80806190231wca4d7ds653a943e24a5077a@mail.gmail.com>
	<db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>
Message-ID: <9457e7c80806191702y35c7348di78c2e236c02bcf9d@mail.gmail.com>

2008/6/19 Fernando Perez <fperez.net at gmail.com>:
>> - import_array().  Can we put it at the bottom of c_numpy.pxd, so that
>> users don't have to import it every time?
>
> Sounds fine to me, but I don't know if there could be a subtle problem
> with pxds used by multiple pyx files.  I'm not comfortable enough with
> cython to answer the question.

>From what I understand, a .pxd file is loaded only once -- is that
correct?  Either way, I don't think executing import_array more than
once could hurt (or could it -- Robert?).

Regards
St?fan


From robert.kern at gmail.com  Thu Jun 19 20:06:01 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 19 Jun 2008 19:06:01 -0500
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <9457e7c80806191702y35c7348di78c2e236c02bcf9d@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<9457e7c80806190231wca4d7ds653a943e24a5077a@mail.gmail.com>
	<db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>
	<9457e7c80806191702y35c7348di78c2e236c02bcf9d@mail.gmail.com>
Message-ID: <3d375d730806191706h441912ebmba3ae981e32d4eb@mail.gmail.com>

On Thu, Jun 19, 2008 at 19:02, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2008/6/19 Fernando Perez <fperez.net at gmail.com>:
>>> - import_array().  Can we put it at the bottom of c_numpy.pxd, so that
>>> users don't have to import it every time?
>>
>> Sounds fine to me, but I don't know if there could be a subtle problem
>> with pxds used by multiple pyx files.  I'm not comfortable enough with
>> cython to answer the question.
>
> >From what I understand, a .pxd file is loaded only once -- is that
> correct?  Either way, I don't think executing import_array more than
> once could hurt (or could it -- Robert?).

I don't think it would cause bad things to happen. It is slightly
suboptimal, but only just.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco

From pgmdevlist at gmail.com  Thu Jun 19 20:15:30 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 19 Jun 2008 20:15:30 -0400
Subject: [Numpy-discussion] ma's maximum_fill_value function in 1.1.0
In-Reply-To: <1213916159.6827.25.camel@zorro.merfinllc.com>
References: <1213916159.6827.25.camel@zorro.merfinllc.com>
Message-ID: <200806192015.32791.pgmdevlist@gmail.com>

Reggie,
Good to hear from you.

There are no particular reason why maximum/minimum/default_fill_value 
functions should stay in limbo, I'll put them in __all__. I'll also try to 
implement the accumulate/reduceat functions for maximum/minimum, using the 
work you've done on _extrema_functions.

Your question raises a good point: is there any consensus on using __all__ 
instead of the module namespace ?


P.

On Thursday 19 June 2008 18:55:59 Reggie Dugard wrote:
> This is just a minor question/problem with the new numpy.ma in version
> 1.1.0.
>
> Because maximum and minimum in ma lack an accumulate attribute, I've
> duplicated the functionality using the
> maximum_fill_value/minimum_fill_value functions and doing something
> like:
>
> np.ma.masked_array(np.maximum.accumulate(np.ma.filled(x,
> np.ma.maximum_fill_value(x))), x.mask)
>
> In version 1.0.4 of numpy, maximum_fill_value/minimum_fill_value were
> located in ma's namespace so the above line would work.  In the latest
> version,  the ma module has an __all__ which does not include
> maximum_fill_value/minimum_fill_value and I'm forced to get them from
> numpy.ma.core (which doesn't seem like a very clean way of doing it).
>
> So I guess my question is:  was this done intentionally and if so why,
> or can these functions be added to __all__ to get the old behavior
> back.
>
> Thanks for your help with this.




From fperez.net at gmail.com  Thu Jun 19 20:18:20 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 19 Jun 2008 17:18:20 -0700
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <3d375d730806191706h441912ebmba3ae981e32d4eb@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<9457e7c80806190231wca4d7ds653a943e24a5077a@mail.gmail.com>
	<db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>
	<9457e7c80806191702y35c7348di78c2e236c02bcf9d@mail.gmail.com>
	<3d375d730806191706h441912ebmba3ae981e32d4eb@mail.gmail.com>
Message-ID: <db6b5ecc0806191718x5dfea216v8d2c94d331f4227a@mail.gmail.com>

On Thu, Jun 19, 2008 at 5:06 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Thu, Jun 19, 2008 at 19:02, St?fan van der Walt <stefan at sun.ac.za> wrote:
>> 2008/6/19 Fernando Perez <fperez.net at gmail.com>:
>>>> - import_array().  Can we put it at the bottom of c_numpy.pxd, so that
>>>> users don't have to import it every time?
>>>
>>> Sounds fine to me, but I don't know if there could be a subtle problem
>>> with pxds used by multiple pyx files.  I'm not comfortable enough with
>>> cython to answer the question.
>>
>> >From what I understand, a .pxd file is loaded only once -- is that
>> correct?  Either way, I don't think executing import_array more than
>> once could hurt (or could it -- Robert?).
>
> I don't think it would cause bad things to happen. It is slightly
> suboptimal, but only just.

As long as it doesn't cause problems, then I'd vote for doing it.
This only happens at module import time, so even if several modules
using cython call it, it's only when they get imported.  I think the
API simplification is worth it (minor, but still nice).

I just tested the changes and there seem to be no ill effects.  Should
I  go ahead and commit it? I'll do it in a single commit with no other
changes so it's easy to revert should it prove to be a bad idea.

Cheers,

f


From fperez.net at gmail.com  Thu Jun 19 20:22:59 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 19 Jun 2008 17:22:59 -0700
Subject: [Numpy-discussion] Nose, doctests and extension modules
Message-ID: <db6b5ecc0806191722r2ba58092rbecee2eac7067efd@mail.gmail.com>

Hi all,

as we transition over to nose in full (which I'm very happy to see),
we'll want to have a good solution to finding doctests in extension
modules.  My initial efforts haven't been very fruitful on this front,
and if any of you has any insights, I'd appreciate to hear them.  I
started the conversation on the testing in python mailing list:

http://lists.idyll.org/pipermail/testing-in-python/2008-June/000701.html

but I'm convinced (for the reasons given there) that the solution I
got won't work in general, even if it did work for that one little
case.

Any insights appreciated.

Cheers,

f


From robert.kern at gmail.com  Thu Jun 19 20:28:46 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 19 Jun 2008 19:28:46 -0500
Subject: [Numpy-discussion] ma's maximum_fill_value function in 1.1.0
In-Reply-To: <200806192015.32791.pgmdevlist@gmail.com>
References: <1213916159.6827.25.camel@zorro.merfinllc.com>
	<200806192015.32791.pgmdevlist@gmail.com>
Message-ID: <3d375d730806191728p1129d4d5sf44f53d1c1207541@mail.gmail.com>

On Thu, Jun 19, 2008 at 19:15, Pierre GM <pgmdevlist at gmail.com> wrote:

> Your question raises a good point: is there any consensus on using __all__
> instead of the module namespace ?

I'm not sure what you mean. Can you clarify?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From robert.kern at gmail.com  Thu Jun 19 20:29:28 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 19 Jun 2008 19:29:28 -0500
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <db6b5ecc0806191718x5dfea216v8d2c94d331f4227a@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<9457e7c80806190231wca4d7ds653a943e24a5077a@mail.gmail.com>
	<db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>
	<9457e7c80806191702y35c7348di78c2e236c02bcf9d@mail.gmail.com>
	<3d375d730806191706h441912ebmba3ae981e32d4eb@mail.gmail.com>
	<db6b5ecc0806191718x5dfea216v8d2c94d331f4227a@mail.gmail.com>
Message-ID: <3d375d730806191729h1d3d5953m586a5fb3f4fcacdc@mail.gmail.com>

On Thu, Jun 19, 2008 at 19:18, Fernando Perez <fperez.net at gmail.com> wrote:
> On Thu, Jun 19, 2008 at 5:06 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Thu, Jun 19, 2008 at 19:02, St?fan van der Walt <stefan at sun.ac.za> wrote:
>>> 2008/6/19 Fernando Perez <fperez.net at gmail.com>:
>>>>> - import_array().  Can we put it at the bottom of c_numpy.pxd, so that
>>>>> users don't have to import it every time?
>>>>
>>>> Sounds fine to me, but I don't know if there could be a subtle problem
>>>> with pxds used by multiple pyx files.  I'm not comfortable enough with
>>>> cython to answer the question.
>>>
>>> >From what I understand, a .pxd file is loaded only once -- is that
>>> correct?  Either way, I don't think executing import_array more than
>>> once could hurt (or could it -- Robert?).
>>
>> I don't think it would cause bad things to happen. It is slightly
>> suboptimal, but only just.
>
> As long as it doesn't cause problems, then I'd vote for doing it.
> This only happens at module import time, so even if several modules
> using cython call it, it's only when they get imported.  I think the
> API simplification is worth it (minor, but still nice).
>
> I just tested the changes and there seem to be no ill effects.  Should
> I  go ahead and commit it? I'll do it in a single commit with no other
> changes so it's easy to revert should it prove to be a bad idea.

Sure.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco

From pgmdevlist at gmail.com  Thu Jun 19 20:38:19 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 19 Jun 2008 20:38:19 -0400
Subject: [Numpy-discussion] ma's maximum_fill_value function in 1.1.0
In-Reply-To: <3d375d730806191728p1129d4d5sf44f53d1c1207541@mail.gmail.com>
References: <1213916159.6827.25.camel@zorro.merfinllc.com>
	<200806192015.32791.pgmdevlist@gmail.com>
	<3d375d730806191728p1129d4d5sf44f53d1c1207541@mail.gmail.com>
Message-ID: <200806192038.20475.pgmdevlist@gmail.com>

On Thursday 19 June 2008 20:28:46 Robert Kern wrote:
> On Thu, Jun 19, 2008 at 19:15, Pierre GM <pgmdevlist at gmail.com> wrote:
> > Your question raises a good point: is there any consensus on using
> > __all__ instead of the module namespace ?
>
> I'm not sure what you mean. Can you clarify?

 __all__ is used on a regular basis in numpy instead of using modules 
namespace: why ?


From robert.kern at gmail.com  Thu Jun 19 20:45:40 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 19 Jun 2008 19:45:40 -0500
Subject: [Numpy-discussion] ma's maximum_fill_value function in 1.1.0
In-Reply-To: <200806192038.20475.pgmdevlist@gmail.com>
References: <1213916159.6827.25.camel@zorro.merfinllc.com>
	<200806192015.32791.pgmdevlist@gmail.com>
	<3d375d730806191728p1129d4d5sf44f53d1c1207541@mail.gmail.com>
	<200806192038.20475.pgmdevlist@gmail.com>
Message-ID: <3d375d730806191745h1fae179ao5759b6ca8e60593a@mail.gmail.com>

On Thu, Jun 19, 2008 at 19:38, Pierre GM <pgmdevlist at gmail.com> wrote:
> On Thursday 19 June 2008 20:28:46 Robert Kern wrote:
>> On Thu, Jun 19, 2008 at 19:15, Pierre GM <pgmdevlist at gmail.com> wrote:
>> > Your question raises a good point: is there any consensus on using
>> > __all__ instead of the module namespace ?
>>
>> I'm not sure what you mean. Can you clarify?
>
>  __all__ is used on a regular basis in numpy instead of using modules
> namespace: why ?

The contents of numpy.core.foo, etc., are typically exposed at a
higher level, like in numpy.core.__init__ and numpy.__init__. Defining
numpy.core.foo.__all__ lets the __init__.py files just "from
numpy.core.foo import *" and have a nearly-sensible namespace.

Even if you aren't catering to the "from ... import *" use case,
including a name in __all__ is a strong and useful signal about what
is part of the public API for your module.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From matthew.brett at gmail.com  Thu Jun 19 21:10:28 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Fri, 20 Jun 2008 01:10:28 +0000
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <3d375d730806191729h1d3d5953m586a5fb3f4fcacdc@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<9457e7c80806190231wca4d7ds653a943e24a5077a@mail.gmail.com>
	<db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>
	<9457e7c80806191702y35c7348di78c2e236c02bcf9d@mail.gmail.com>
	<3d375d730806191706h441912ebmba3ae981e32d4eb@mail.gmail.com>
	<db6b5ecc0806191718x5dfea216v8d2c94d331f4227a@mail.gmail.com>
	<3d375d730806191729h1d3d5953m586a5fb3f4fcacdc@mail.gmail.com>
Message-ID: <1e2af89e0806191810l452b15f4h9f790c17f4f38fe9@mail.gmail.com>

Hi,

>> As long as it doesn't cause problems, then I'd vote for doing it.
>> This only happens at module import time, so even if several modules
>> using cython call it, it's only when they get imported.  I think the
>> API simplification is worth it (minor, but still nice).
>>
>> I just tested the changes and there seem to be no ill effects.  Should
>> I  go ahead and commit it? I'll do it in a single commit with no other
>> changes so it's easy to revert should it prove to be a bad idea.

I may be missing something, but given cython / pyrex are generating c
files, as long as you don't call import_array in your pyx file as well
as your pxd file, I don't think it could make any difference that
several cython modules use the same pxd file.

Matthew


From peridot.faceted at gmail.com  Thu Jun 19 23:42:08 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 19 Jun 2008 23:42:08 -0400
Subject: [Numpy-discussion] Find groups of true values or sequence of
	values
In-Reply-To: <loom.20080618T232816-826@post.gmane.org>
References: <loom.20080618T232816-826@post.gmane.org>
Message-ID: <ce557a360806192042r491cebf3xbe3499579199415@mail.gmail.com>

2008/6/18 bevan <bevan07 at gmail.com>:
> Hello,
>
> I am looking for some pointers that will hopefully get me a round an issue I
> have hit.
>
> I have a timeseries of river flow and would like to carry out some analysis on
> the recession periods.  That is anytime the values are decreasing.  I would
> like to restrict (mask) my data to any values that are in a continuous sequence
> of 3 or more (in the example below), that are decreasing in value.
>
> Hopefully this example helps:
>
> import numpy as np
>
> Flow = np.array([15.4,20.5,19.4,18.7,18.6,35.5,34.8,25.1,26.7])
> FlowDiff = np.diff(Flow)
> boolFlowdiff = FlowDiff>0
> MaskFlow = np.ma.masked_array(Flow[1:],boolFlowdiff)
>
> print MaskFlow
> [-- 19.4 18.7 18.6 -- 34.8 25.1 --]
>
> The output I would like is
> [-- 19.4 18.7 18.6 -- -- -- --]
> Where the second groups is blanked because the sequence only has 2 members.

I would tackle this in steps: find the decreasing pairs, then find
places where two of them occur in a row, then construct your mask.

Finding decreases:
d = diff(X)<0
finding two decreases in a row:
t = d[1:] & d[:-1]
creating the right mask:
m = np.zeros(n,dtype=np.bool)
m[:n-2] |= t
m[1:n-1] |= t
m[2:n] |= t

For finding longer runs you would want other tricks.

Anne


From david at ar.media.kyoto-u.ac.jp  Thu Jun 19 23:39:05 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 20 Jun 2008 12:39:05 +0900
Subject: [Numpy-discussion] CFLAGS= causes bad build
In-Reply-To: <e06186140806191054m6e39a32agd84ddfec5ba1ab1f@mail.gmail.com>
References: <e06186140806191054m6e39a32agd84ddfec5ba1ab1f@mail.gmail.com>
Message-ID: <485B2659.6090501@ar.media.kyoto-u.ac.jp>

Charles R Harris wrote:
> The compile flags are missing -fno-strict-aliasing. I don't know if 
> this is a problem with the python distutils or what, but it would be 
> nice if it didn't occur.

Yes. That's one of the reasons why I started numscons. The current 
behavior in numpy.distutils is for the same reason as for LDFLAGS: 
sometimes, this behavior is necessary for unsupported compiler/platform 
configurations.

cheers,

David


From peridot.faceted at gmail.com  Fri Jun 20 00:13:07 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Fri, 20 Jun 2008 00:13:07 -0400
Subject: [Numpy-discussion] Detecting phase windings
In-Reply-To: <10495791.1213633824742.JavaMail.root@nschwwebs02p>
References: <10495791.1213633824742.JavaMail.root@nschwwebs02p>
Message-ID: <ce557a360806192113q61a8f893t89b8a074a653e8ea@mail.gmail.com>

2008/6/16 gruben at bigpond.net.au <gruben at bigpond.net.au>:
> I have a speed problem with the approach I'm using to detect phase wrappings in a 3D data set. In my application, phaseField is a 3D array containing the phase values of a field. In order to detect the vortices/phase windings at each point, I check for windings on each of 3 faces of a 2x2 cube with the following code:
>
> def HasVortex(cube):
>    ''' cube is a 2x2x2 array containing the phase values
>        returns True if any of 3 orthogonal faces contains a phase wrapping
>    '''
>    # extract 3 cube faces - flatten to make indexing easier
>    c1 = cube[0,:,:].flatten()
>    c3 = cube[:,0,:].flatten()
>    c5 = cube[:,:,0].flatten()
>
>    # now order the phases in a circuit around each face, finishing in the same corner as we began
>    # Use numpy's phase unwrapping code, which has a default threshold of pi
>    u1 = unwrap(c1[cwi])
>    u3 = unwrap(c3[cwi])
>    u5 = unwrap(c5[cwi])
>
>    # check whether the final unwrapped value coincides with the value in the cell we started at
>    return not allclose(r_[u1[0],u3[0],u5[0]], r_[u1[4],u3[4],u5[4]])
>
> vortexArray = array([int(HasVortex(phaseField[x:x+2,y:y+2,z:z+2]))
>                    for x in range(phaseField.shape[0]-1)
>                    for y in range(phaseField.shape[1]-1)
>                    for z in range(phaseField.shape[2]-1)]\
>                   ).reshape((xs-1,ys-1,zs-1))
>
> Whilst this does what I want, it's incredibly slow. I'm wondering whether anyone has done this before, or can think of a better approach.
>
> My thoughts about a better approach are to stack the values along 3 new dimensions making a 6D array and use unwrap along the 3 new dimensions. Something like the following may give you an idea of what I mean, but it's a toy example trying to extend 2D into 3D, rather than 3D into 6D, because I haven't come to grips with enough of numpy's axis reshaping and stacking tricks to avoid getting a severe headache when trying to generalize it:
>
> a = array([[1.,3.], [6.,5.]])
> b = np.dstack((a, roll(a,-1,axis=1), roll(roll(a,-1,axis=0),-1,axis=1), roll(a,-1,axis=0), a))
> print np.unwrap(b, axis=2)
>
> A problem with this approach is that the memory requirements for the stacked version will be much greater, so some version using views would be preferable.
>
> Any ideas?

I'd start by looking at the problem one face at a time:
def find_vortices(X, axis=0):
    XX = np.rollaxis(X,axis)
    loop = np.concatenate((XX[np.newaxis,:-1,:-1,...],
                                      XX[np.newaxis,1:,:-1,...],
                                      XX[np.newaxis,1:,1:,...],
                                      XX[np.newaxis,:-1,1:,...],
                                      XX[np.newaxis,:-1,:-1,...]),axis=0)
    loop = np.unwrap(loop)
    r = np.abs(loop[0,...]-loop[-1,...])<np.pi/2
    return np.rollaxis(r,0,axis+1)
This does require temporary space some five or ten times your original
space required, which is not good. You might be able to do better by
reimplementing the phase wrapping code; your condition is equivalent
to "the number of upward steps of more than pi is different from the
number of downward steps of more than pi". But every operation on the
whole array at once is going to produce a temporary roughly the size
of the array, as in fact is the result boolean array (though you will
presumably use nonzero() to convert it to a list of locations, which
will presumably be fairly small until you hit a totally decorrelated
image).

If your array is large, as sounds like the case, you can try a
standard trick for cutting down on temporary sizes: use a for loop
along the smallest dimension and a vector along the rest. In fact in
this case I'd use a 2D vortex finder and iterate along the remaining
axis; do this three times and you're done. (This can bit you if you
have an extremely long skinny array, but otherwise the python overhead
will be pretty small.)

Anne


From fperez.net at gmail.com  Fri Jun 20 00:18:34 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 19 Jun 2008 21:18:34 -0700
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <3d375d730806191729h1d3d5953m586a5fb3f4fcacdc@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<9457e7c80806190231wca4d7ds653a943e24a5077a@mail.gmail.com>
	<db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>
	<9457e7c80806191702y35c7348di78c2e236c02bcf9d@mail.gmail.com>
	<3d375d730806191706h441912ebmba3ae981e32d4eb@mail.gmail.com>
	<db6b5ecc0806191718x5dfea216v8d2c94d331f4227a@mail.gmail.com>
	<3d375d730806191729h1d3d5953m586a5fb3f4fcacdc@mail.gmail.com>
Message-ID: <db6b5ecc0806192118w25a6b530j4135c4e6cda90b8a@mail.gmail.com>

On Thu, Jun 19, 2008 at 5:29 PM, Robert Kern <robert.kern at gmail.com> wrote:

>> I just tested the changes and there seem to be no ill effects.  Should
>> I  go ahead and commit it? I'll do it in a single commit with no other
>> changes so it's easy to revert should it prove to be a bad idea.
>
> Sure.

Thanks.  Done in r5301.

Cheers,

f


From david at ar.media.kyoto-u.ac.jp  Fri Jun 20 02:29:20 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 20 Jun 2008 15:29:20 +0900
Subject: [Numpy-discussion] Adding a version system to the deprecation
 functions in numpy/lib/utils ?
Message-ID: <485B4E40.5070504@ar.media.kyoto-u.ac.jp>

Hi,

    I was thinking about adding a versioning into the deprecation 
related functions. E.g., you could say that one function is deprecated 
in version 1.2, but adding a in_version in deprecate. Does anyone have 
strong feeling against it ? It should be a transparant change. I would 
also like to add a deprecation function to handle arguments and default 
values, but this will be more work, obviously.

cheers,

David


From robert.kern at gmail.com  Fri Jun 20 02:53:40 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Jun 2008 01:53:40 -0500
Subject: [Numpy-discussion] Adding a version system to the deprecation
	functions in numpy/lib/utils ?
In-Reply-To: <485B4E40.5070504@ar.media.kyoto-u.ac.jp>
References: <485B4E40.5070504@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730806192353n3720a21fked67e3125048df63@mail.gmail.com>

On Fri, Jun 20, 2008 at 01:29, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Hi,
>
>    I was thinking about adding a versioning into the deprecation
> related functions. E.g., you could say that one function is deprecated
> in version 1.2, but adding a in_version in deprecate. Does anyone have
> strong feeling against it ?

As long as we keep the policy that DeprecationWarnings last precisely
one minor version, I don't think it really adds any information.

> It should be a transparant change. I would
> also like to add a deprecation function to handle arguments and default
> values, but this will be more work, obviously.

It will be hard to write something generic. The individual functions
will still have to write code that handles the old arguments anyways,
so all it would do is move the warning message from a warnings.warn()
into the decorator. It's not worth it.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Fri Jun 20 02:54:07 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 20 Jun 2008 15:54:07 +0900
Subject: [Numpy-discussion] Adding a version system to the deprecation
 functions in numpy/lib/utils ?
In-Reply-To: <3d375d730806192353n3720a21fked67e3125048df63@mail.gmail.com>
References: <485B4E40.5070504@ar.media.kyoto-u.ac.jp>
	<3d375d730806192353n3720a21fked67e3125048df63@mail.gmail.com>
Message-ID: <485B540F.7060209@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
>
> As long as we keep the policy that DeprecationWarnings last precisely
> one minor version, I don't think it really adds any information.
>   

I was not aware of this policy. Obviously, it has no use in that case.

>
> It will be hard to write something generic. The individual functions
> will still have to write code that handles the old arguments anyways,
> so all it would do is move the warning message from a warnings.warn()
> into the decorator. It's not worth it.
>   

If we change default values of some functions, you don't think it would 
be useful to raise a warning if people do not use the argument (that is 
use the default value) ? I remember a long time ago, when some functions 
got their axis argument changed (pre 1.0 numpy), it took me a long time 
to realize why my code broke.

By saying the function has to handle the old argument, you are implying 
that we don't allow API changes, right ? If we enforce this, again, 
deprecating argument/default value has no value at all. But up to now, 
it has happened fairly regularly.

cheers,

David



From robert.kern at gmail.com  Fri Jun 20 03:12:13 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Jun 2008 02:12:13 -0500
Subject: [Numpy-discussion] Adding a version system to the deprecation
	functions in numpy/lib/utils ?
In-Reply-To: <485B540F.7060209@ar.media.kyoto-u.ac.jp>
References: <485B4E40.5070504@ar.media.kyoto-u.ac.jp>
	<3d375d730806192353n3720a21fked67e3125048df63@mail.gmail.com>
	<485B540F.7060209@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730806200012q36bc7338k7a73ee1cfca18ccf@mail.gmail.com>

On Fri, Jun 20, 2008 at 01:54, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Robert Kern wrote:

>> It will be hard to write something generic. The individual functions
>> will still have to write code that handles the old arguments anyways,
>> so all it would do is move the warning message from a warnings.warn()
>> into the decorator. It's not worth it.
>
> If we change default values of some functions, you don't think it would
> be useful to raise a warning if people do not use the argument (that is
> use the default value) ? I remember a long time ago, when some functions
> got their axis argument changed (pre 1.0 numpy), it took me a long time
> to realize why my code broke.

I don't see a reason to wrap it in a decorator. It saves no significant effort.

> By saying the function has to handle the old argument, you are implying
> that we don't allow API changes, right ? If we enforce this, again,
> deprecating argument/default value has no value at all. But up to now,
> it has happened fairly regularly.

It should not be happening regularly in the future.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Fri Jun 20 03:15:38 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 20 Jun 2008 16:15:38 +0900
Subject: [Numpy-discussion] Adding a version system to the deprecation
 functions in numpy/lib/utils ?
In-Reply-To: <3d375d730806200012q36bc7338k7a73ee1cfca18ccf@mail.gmail.com>
References: <485B4E40.5070504@ar.media.kyoto-u.ac.jp>	<3d375d730806192353n3720a21fked67e3125048df63@mail.gmail.com>	<485B540F.7060209@ar.media.kyoto-u.ac.jp>
	<3d375d730806200012q36bc7338k7a73ee1cfca18ccf@mail.gmail.com>
Message-ID: <485B591A.80902@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
>
> I don't see a reason to wrap it in a decorator. It saves no significant effort.
>   

If the warning is spit out in any case (whether the user uses the 
default value or not), and if we don't need versioning, I understand it 
is not that useful.

I guess my concern is what do we do with deprecated functions: do we 
keep them, for how long ? There are some deprecate in current numpy 
which dates back to the pre 1.0 release. Maybe that's not really important.

>
> It should not be happening regularly in the future.

Ok.

David



From robert.kern at gmail.com  Fri Jun 20 03:41:01 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Jun 2008 02:41:01 -0500
Subject: [Numpy-discussion] Adding a version system to the deprecation
	functions in numpy/lib/utils ?
In-Reply-To: <485B591A.80902@ar.media.kyoto-u.ac.jp>
References: <485B4E40.5070504@ar.media.kyoto-u.ac.jp>
	<3d375d730806192353n3720a21fked67e3125048df63@mail.gmail.com>
	<485B540F.7060209@ar.media.kyoto-u.ac.jp>
	<3d375d730806200012q36bc7338k7a73ee1cfca18ccf@mail.gmail.com>
	<485B591A.80902@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730806200041s523ce775vc6f27c4964f50e00@mail.gmail.com>

On Fri, Jun 20, 2008 at 02:15, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> I guess my concern is what do we do with deprecated functions: do we
> keep them, for how long ? There are some deprecate in current numpy
> which dates back to the pre 1.0 release. Maybe that's not really important.

They should deprecated for one minor version and then deleted.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Fri Jun 20 03:37:50 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 20 Jun 2008 16:37:50 +0900
Subject: [Numpy-discussion] Adding a version system to the deprecation
 functions in numpy/lib/utils ?
In-Reply-To: <3d375d730806200041s523ce775vc6f27c4964f50e00@mail.gmail.com>
References: <485B4E40.5070504@ar.media.kyoto-u.ac.jp>	<3d375d730806192353n3720a21fked67e3125048df63@mail.gmail.com>	<485B540F.7060209@ar.media.kyoto-u.ac.jp>	<3d375d730806200012q36bc7338k7a73ee1cfca18ccf@mail.gmail.com>	<485B591A.80902@ar.media.kyoto-u.ac.jp>
	<3d375d730806200041s523ce775vc6f27c4964f50e00@mail.gmail.com>
Message-ID: <485B5E4E.90305@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
>
> They should deprecated for one minor version and then deleted.

Ok, thanks for the clarification. Since I was not aware of this, and did 
not see this information on the wiki, I added it in 
http://scipy.org/scipy/numpy/wiki/ApiDeprecation. Let me know if there 
is something I misunderstood from your remarks,

cheers,

David


From gruben at bigpond.net.au  Fri Jun 20 05:09:49 2008
From: gruben at bigpond.net.au (Gary Ruben)
Date: Fri, 20 Jun 2008 10:09:49 +0100
Subject: [Numpy-discussion] Detecting phase windings
In-Reply-To: <ce557a360806192113q61a8f893t89b8a074a653e8ea@mail.gmail.com>
References: <10495791.1213633824742.JavaMail.root@nschwwebs02p>
	<ce557a360806192113q61a8f893t89b8a074a653e8ea@mail.gmail.com>
Message-ID: <485B73DD.8090100@bigpond.net.au>

Hi Anne,
Thanks for the approach ideas - I'll take a look at this soon to try to 
understand it. Currently I'm visiting a LabView-based lab who already 
have something that works, and works fast, so I'm being encouraged to 
use LabView, but I'd like to show them more of the joys of Python. The 
memory requirements aren't yet an issue with the data sets I'm using, 
but they could be later.

Anne Archibald wrote:
> 2008/6/16 gruben at bigpond.net.au <gruben at bigpond.net.au>:
>> I have a speed problem with the approach I'm using to detect phase
>> wrappings in a 3D data set.
<big snip>
> I'd start by looking at the problem one face at a time:
> def find_vortices(X, axis=0):
>     XX = np.rollaxis(X,axis)
>     loop = np.concatenate((XX[np.newaxis,:-1,:-1,...],
>                                       XX[np.newaxis,1:,:-1,...],
>                                       XX[np.newaxis,1:,1:,...],
>                                       XX[np.newaxis,:-1,1:,...],
>                                       XX[np.newaxis,:-1,:-1,...]),axis=0)
>     loop = np.unwrap(loop)
>     r = np.abs(loop[0,...]-loop[-1,...])<np.pi/2
>     return np.rollaxis(r,0,axis+1)
<snip>
> standard trick for cutting down on temporary sizes: use a for loop 
> along the smallest dimension and a vector along the rest. In fact in 
> this case I'd use a 2D vortex finder and iterate along the remaining 
> axis; do this three times and you're done.

I might also try three 1D finders, keeping three temporary boolean 
result arrays, then logically OR them.

thanks,
Gary R.


From izakmarais at yahoo.com  Fri Jun 20 05:28:20 2008
From: izakmarais at yahoo.com (izak marais)
Date: Fri, 20 Jun 2008 02:28:20 -0700 (PDT)
Subject: [Numpy-discussion] Sintax differences
Message-ID: <681848.94393.qm@web50906.mail.re2.yahoo.com>

Hi

Is there a reason why the syntax is rand(n,n), but zeros((n,n)) to create an n*n array? If not, perhaps this could be cleaned up?

Regards,
Izak Marais



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080620/bd55673c/attachment.html>

From wnbell at gmail.com  Fri Jun 20 05:59:57 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Fri, 20 Jun 2008 04:59:57 -0500
Subject: [Numpy-discussion] Sintax differences
In-Reply-To: <681848.94393.qm@web50906.mail.re2.yahoo.com>
References: <681848.94393.qm@web50906.mail.re2.yahoo.com>
Message-ID: <d05265cb0806200259n27b5adf5q57a1d90d520ee216@mail.gmail.com>

On Fri, Jun 20, 2008 at 4:28 AM, izak marais <izakmarais at yahoo.com> wrote:
> Hi
>
> Is there a reason why the syntax is rand(n,n), but zeros((n,n)) to create an
> n*n array? If not, perhaps this could be cleaned up?
>

SciPy's rand() is just a convenience wrapper:


>>> help(rand)
Help on built-in function rand:

rand(...)
    Return an array of the given dimensions which is initialized to
    random numbers from a uniform distribution in the range [0,1).

    rand(d0, d1, ..., dn) -> random values

    Note:  This is a convenience function. If you want an
                interface that takes a tuple as the first argument
                use numpy.random.random_sample(shape_tuple).


-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From izakmarais at yahoo.com  Fri Jun 20 06:01:44 2008
From: izakmarais at yahoo.com (izak marais)
Date: Fri, 20 Jun 2008 03:01:44 -0700 (PDT)
Subject: [Numpy-discussion] Sintax differences
Message-ID: <713913.52813.qm@web50912.mail.re2.yahoo.com>

I apologise for the horrible spelling mistake in the mail title!



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080620/3de67320/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Fri Jun 20 06:32:12 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 20 Jun 2008 19:32:12 +0900
Subject: [Numpy-discussion] Weird error with nose tests on windows
Message-ID: <485B872C.5060402@ar.media.kyoto-u.ac.jp>

Hi,

    While trying to reproduce some bugs on windows with recent numpy, I 
got major failures for the whole test suite, with something related to 
numeric not found in numpy\core\__init__.py (__all__ += numeric.__all__ 
line). Nose crashes because at that line, meaning most of the tests 
fail; note that this is nose specific, importing numpy works fine... 
which is strange. I don't understand how importing __all__ += 
numeric.__all__ is supposed to work since there was no import numeric 
before. IOW, I understand why it fails, but I don't understand why it 
does not fail when importing (or with tests on linux either; maybe the 
different nose version). Does it make sense to put an import numeric 
before in the __init__.py file ?

cheers,

David


From lbrooks at MIT.EDU  Fri Jun 20 08:51:37 2008
From: lbrooks at MIT.EDU (Lane Brooks)
Date: Fri, 20 Jun 2008 06:51:37 -0600
Subject: [Numpy-discussion] CAPI segfault running python embedded
Message-ID: <485BA7D9.7030802@mit.edu>

I have successfully written several extension modules that use the numpy 
CAPI to manipulate numpy arrays.  I have to say numpy is great.

I am now having a problem, however, when calling numpy capi functions 
when running python embedded in a third party, closed source, 
application.  It segfaults whenever I make any PyArray* function call.  
Any ideas on why that might be happening?

Thanks,
Lane




From david at ar.media.kyoto-u.ac.jp  Fri Jun 20 09:08:45 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 20 Jun 2008 22:08:45 +0900
Subject: [Numpy-discussion] CAPI segfault running python embedded
In-Reply-To: <485BA7D9.7030802@mit.edu>
References: <485BA7D9.7030802@mit.edu>
Message-ID: <485BABDD.109@ar.media.kyoto-u.ac.jp>

Lane Brooks wrote:
> I have successfully written several extension modules that use the numpy 
> CAPI to manipulate numpy arrays.  I have to say numpy is great.
>
> I am now having a problem, however, when calling numpy capi functions 
> when running python embedded in a third party, closed source, 
> application.  It segfaults whenever I make any PyArray* function call.  
> Any ideas on why that might be happening?
>   

You most likely forgot to call import_array functions when initializing 
your extension (the PyMODULE_INIT function): with python, extensions 
which define new C-API does it through an array of function pointers, 
which is garbage before you call import_array. I have never used 
embedded python, so I don't know how initialization works there, but 
that should be similar I guess.

cheers,

David


From lbrooks at MIT.EDU  Fri Jun 20 09:30:33 2008
From: lbrooks at MIT.EDU (Lane Brooks)
Date: Fri, 20 Jun 2008 07:30:33 -0600
Subject: [Numpy-discussion] CAPI segfault running python embedded
In-Reply-To: <485BABDD.109@ar.media.kyoto-u.ac.jp>
References: <485BA7D9.7030802@mit.edu> <485BABDD.109@ar.media.kyoto-u.ac.jp>
Message-ID: <485BB0F9.7080706@mit.edu>

David Cournapeau wrote:
> Lane Brooks wrote:
>   
>> I have successfully written several extension modules that use the numpy 
>> CAPI to manipulate numpy arrays.  I have to say numpy is great.
>>
>> I am now having a problem, however, when calling numpy capi functions 
>> when running python embedded in a third party, closed source, 
>> application.  It segfaults whenever I make any PyArray* function call.  
>> Any ideas on why that might be happening?
>>   
>>     
>
> You most likely forgot to call import_array functions when initializing 
> your extension (the PyMODULE_INIT function): with python, extensions 
> which define new C-API does it through an array of function pointers, 
> which is garbage before you call import_array. I have never used 
> embedded python, so I don't know how initialization works there, but 
> that should be similar I guess.
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   
Your exactly right.  I was side-tracked by the fact that this was 
embedded python (which is not really that different) and thought was 
thinking it was an environmental condition.  Adding the import_array() 
cleared the problem up immediately.  Thanks for the help.

Lane


From gael.varoquaux at normalesup.org  Fri Jun 20 10:17:32 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 20 Jun 2008 16:17:32 +0200
Subject: [Numpy-discussion] Right way of looping throught ndarrays using
	Cython
Message-ID: <20080620141732.GA10745@phare.normalesup.org>

Hi,

I am trying to figure the right way of looping throught ndarrays using
Cython, currently. Things seem to have evolved a bit compared to what
some documents on the web claim (eg "for i from 0<=i<n" does not seem
faster than "for i in range(n)"). I know there is integration of pex with
cython on the horizon, but it is not there yet, and I don't want to
commit to a moving target.

Does somebody have a example of fast looping through ndarrays using
modern Cython idioms?

Cheers,

Ga?l


From fperez.net at gmail.com  Fri Jun 20 13:51:53 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 20 Jun 2008 10:51:53 -0700
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <db6b5ecc0806192118w25a6b530j4135c4e6cda90b8a@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<9457e7c80806190231wca4d7ds653a943e24a5077a@mail.gmail.com>
	<db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>
	<9457e7c80806191702y35c7348di78c2e236c02bcf9d@mail.gmail.com>
	<3d375d730806191706h441912ebmba3ae981e32d4eb@mail.gmail.com>
	<db6b5ecc0806191718x5dfea216v8d2c94d331f4227a@mail.gmail.com>
	<3d375d730806191729h1d3d5953m586a5fb3f4fcacdc@mail.gmail.com>
	<db6b5ecc0806192118w25a6b530j4135c4e6cda90b8a@mail.gmail.com>
Message-ID: <db6b5ecc0806201051s2b87de69k925b4d998c2f974d@mail.gmail.com>

On Thu, Jun 19, 2008 at 9:18 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Thu, Jun 19, 2008 at 5:29 PM, Robert Kern <robert.kern at gmail.com> wrote:
>
>>> I just tested the changes and there seem to be no ill effects.  Should
>>> I  go ahead and commit it? I'll do it in a single commit with no other
>>> changes so it's easy to revert should it prove to be a bad idea.
>>
>> Sure.
>
> Thanks.  Done in r5301.

Well, the simplistic test script we had in didn't show any problems,
but Matthew Brett was more careful than me and went looking into the
generated code:

On Fri, Jun 20, 2008 at 1:32 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
>>>> I just tested the changes and there seem to be no ill effects.  Should
>>>> I  go ahead and commit it? I'll do it in a single commit with no other
>>>> changes so it's easy to revert should it prove to be a bad idea.
>>>
>>> Sure.
>>
>> Thanks.  Done in r5301.
>
> Hmm - but does import_array() get called when it's in the pxd file?  I
> just updated svn and generated the numpyx.c file, and can't see
> import_array() there.  Isn't the pxd file just definitions?

And indeed, it was my fault for not RTFM:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/sharing.html#mozTocId411233

which says:

It [the .pxd file] cannot contain the implementations of any C or
Python functions, or any Python class definitions, or **any executable
statements**. (emphasis mine).


I verified further by putting the import_array() back into  the .pyx
file and indeed:

- i_a() in .pxd -> missing from .c file.
- i_a() in .pyx -> OK in .c file.

It thus seems that  we must keep the import_array call out of the .pxd
and users still need to remember to make it themselves.

I'll go ahead and clean up, since the mess was my fault.

Many thanks to Matthew for catching it!

Cheers,

f


From strawman at astraw.com  Fri Jun 20 14:11:32 2008
From: strawman at astraw.com (Andrew Straw)
Date: Fri, 20 Jun 2008 11:11:32 -0700
Subject: [Numpy-discussion] Updating cython directory to pxd
	usage:	objections?
In-Reply-To: <db6b5ecc0806201051s2b87de69k925b4d998c2f974d@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>	<9457e7c80806190231wca4d7ds653a943e24a5077a@mail.gmail.com>	<db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>	<9457e7c80806191702y35c7348di78c2e236c02bcf9d@mail.gmail.com>	<3d375d730806191706h441912ebmba3ae981e32d4eb@mail.gmail.com>	<db6b5ecc0806191718x5dfea216v8d2c94d331f4227a@mail.gmail.com>	<3d375d730806191729h1d3d5953m586a5fb3f4fcacdc@mail.gmail.com>	<db6b5ecc0806192118w25a6b530j4135c4e6cda90b8a@mail.gmail.com>
	<db6b5ecc0806201051s2b87de69k925b4d998c2f974d@mail.gmail.com>
Message-ID: <485BF2D4.4080600@astraw.com>

Fernando Perez wrote:
> I verified further by putting the import_array() back into  the .pyx
> file and indeed:
>
> - i_a() in .pxd -> missing from .c file.
> - i_a() in .pyx -> OK in .c file.
>
> It thus seems that  we must keep the import_array call out of the .pxd
> and users still need to remember to make it themselves.
>   
It's not the worst thing in the world, either -- sometimes one wants a
bit of the C structures layout from the .pxd file to use the array
interface without actually calling any of the numpy C API. Thus,
occasionally, calling import_array() would be a (probably very minor)
waste. However, this appears to be a completely moot point now...

-Andrew


From fperez.net at gmail.com  Fri Jun 20 14:25:00 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 20 Jun 2008 11:25:00 -0700
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <485BF2D4.4080600@astraw.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<9457e7c80806190231wca4d7ds653a943e24a5077a@mail.gmail.com>
	<db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>
	<9457e7c80806191702y35c7348di78c2e236c02bcf9d@mail.gmail.com>
	<3d375d730806191706h441912ebmba3ae981e32d4eb@mail.gmail.com>
	<db6b5ecc0806191718x5dfea216v8d2c94d331f4227a@mail.gmail.com>
	<3d375d730806191729h1d3d5953m586a5fb3f4fcacdc@mail.gmail.com>
	<db6b5ecc0806192118w25a6b530j4135c4e6cda90b8a@mail.gmail.com>
	<db6b5ecc0806201051s2b87de69k925b4d998c2f974d@mail.gmail.com>
	<485BF2D4.4080600@astraw.com>
Message-ID: <db6b5ecc0806201125l42614bf2ga7d47f79c67cf753@mail.gmail.com>

On Fri, Jun 20, 2008 at 11:11 AM, Andrew Straw <strawman at astraw.com> wrote:
> Fernando Perez wrote:
>> I verified further by putting the import_array() back into  the .pyx
>> file and indeed:
>>
>> - i_a() in .pxd -> missing from .c file.
>> - i_a() in .pyx -> OK in .c file.
>>
>> It thus seems that  we must keep the import_array call out of the .pxd
>> and users still need to remember to make it themselves.
>>
> It's not the worst thing in the world, either -- sometimes one wants a
> bit of the C structures layout from the .pxd file to use the array
> interface without actually calling any of the numpy C API. Thus,
> occasionally, calling import_array() would be a (probably very minor)
> waste. However, this appears to be a completely moot point now...

Yup, fixed in r5303.  Sorry for the (small) mess.

Cheers,

f


From robert.kern at gmail.com  Fri Jun 20 15:07:08 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Jun 2008 14:07:08 -0500
Subject: [Numpy-discussion] Right way of looping throught ndarrays using
	Cython
In-Reply-To: <20080620141732.GA10745@phare.normalesup.org>
References: <20080620141732.GA10745@phare.normalesup.org>
Message-ID: <3d375d730806201207v227e5a7fne4e09020caa98c2d@mail.gmail.com>

On Fri, Jun 20, 2008 at 09:17, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> Hi,
>
> I am trying to figure the right way of looping throught ndarrays using
> Cython, currently. Things seem to have evolved a bit compared to what
> some documents on the web claim (eg "for i from 0<=i<n" does not seem
> faster than "for i in range(n)"). I know there is integration of pex with
> cython on the horizon, but it is not there yet, and I don't want to
> commit to a moving target.
>
> Does somebody have a example of fast looping through ndarrays using
> modern Cython idioms?

If you're using normal Python indexing, then that's where all your
time is being spent. You need to grab the actual .data pointer and do
C indexing to get speed. Can you show us the code you are timing?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From matthew.brett at gmail.com  Fri Jun 20 15:11:40 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Fri, 20 Jun 2008 19:11:40 +0000
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <db6b5ecc0806201125l42614bf2ga7d47f79c67cf753@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<db6b5ecc0806191316u7183d75eja224d10927ad4799@mail.gmail.com>
	<9457e7c80806191702y35c7348di78c2e236c02bcf9d@mail.gmail.com>
	<3d375d730806191706h441912ebmba3ae981e32d4eb@mail.gmail.com>
	<db6b5ecc0806191718x5dfea216v8d2c94d331f4227a@mail.gmail.com>
	<3d375d730806191729h1d3d5953m586a5fb3f4fcacdc@mail.gmail.com>
	<db6b5ecc0806192118w25a6b530j4135c4e6cda90b8a@mail.gmail.com>
	<db6b5ecc0806201051s2b87de69k925b4d998c2f974d@mail.gmail.com>
	<485BF2D4.4080600@astraw.com>
	<db6b5ecc0806201125l42614bf2ga7d47f79c67cf753@mail.gmail.com>
Message-ID: <1e2af89e0806201211g134c2c73m58b6583efaa6056b@mail.gmail.com>

Hi,

As a matter of interest, what is the relationship, if any, between (in Cython)

import numpy

and

cnp.import_array()

Are they initializing different copies of the same thing?

Best,

Matthew


From gael.varoquaux at normalesup.org  Fri Jun 20 15:13:35 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 20 Jun 2008 21:13:35 +0200
Subject: [Numpy-discussion] Right way of looping throught ndarrays	using
	Cython
In-Reply-To: <3d375d730806201207v227e5a7fne4e09020caa98c2d@mail.gmail.com>
References: <20080620141732.GA10745@phare.normalesup.org>
	<3d375d730806201207v227e5a7fne4e09020caa98c2d@mail.gmail.com>
Message-ID: <20080620191334.GE4646@phare.normalesup.org>

On Fri, Jun 20, 2008 at 02:07:08PM -0500, Robert Kern wrote:
> > Does somebody have a example of fast looping through ndarrays using
> > modern Cython idioms?

> If you're using normal Python indexing, then that's where all your
> time is being spent. You need to grab the actual .data pointer and do
> C indexing to get speed. Can you show us the code you are timing?

That is indeed what I was thinking. There is no way of doing this appart
by using the point-style indexing? I guess I am trying to find the best
(as in most readable) way of doing this. This is for teaching, not
production, so I am very much interested in having something simple.

I am attaching my test file.

Cheers,

Ga?l


From gael.varoquaux at normalesup.org  Fri Jun 20 15:13:56 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 20 Jun 2008 21:13:56 +0200
Subject: [Numpy-discussion] Right way of looping throught ndarrays	using
	Cython
In-Reply-To: <20080620191334.GE4646@phare.normalesup.org>
References: <20080620141732.GA10745@phare.normalesup.org>
	<3d375d730806201207v227e5a7fne4e09020caa98c2d@mail.gmail.com>
	<20080620191334.GE4646@phare.normalesup.org>
Message-ID: <20080620191356.GF4646@phare.normalesup.org>

Oups, forgot the attachement.

On Fri, Jun 20, 2008 at 09:13:34PM +0200, Gael Varoquaux wrote:
> On Fri, Jun 20, 2008 at 02:07:08PM -0500, Robert Kern wrote:
> > > Does somebody have a example of fast looping through ndarrays using
> > > modern Cython idioms?

> > If you're using normal Python indexing, then that's where all your
> > time is being spent. You need to grab the actual .data pointer and do
> > C indexing to get speed. Can you show us the code you are timing?

> That is indeed what I was thinking. There is no way of doing this appart
> by using the point-style indexing? I guess I am trying to find the best
> (as in most readable) way of doing this. This is for teaching, not
> production, so I am very much interested in having something simple.

> I am attaching my test file.

> Cheers,

> Ga?l
-------------- next part --------------
from numpy import zeros, mgrid

# Make sure numpy is initialized.
include "c_numpy.pxd"

##############################################################################
cdef int inner_loop(float c_x, float c_y):
    cdef float x, y, x_buffer 
    x = 0; y = 0
    cdef int i
    for i in range(50):
        x_buffer = x*x - y*y + c_x
        y = 2*x*y + c_y
        x = x_buffer
        if (x*x + x*y > 100):
            return 50 - i

def do_Mandelbrot_cython():
    x, y = mgrid[-1.5:0.5:500j, -1:1:500j]
    threshold_time = zeros((500, 500))
    cdef int i, j
    for i in range(500):
        for j in range(500):
            threshold_time[i, j] = inner_loop(x[i, j], y[i, j])
    return threshold_time

# %timeit on do_Mandelbrot_cython, on epsilon, gives 761ms per loop

def main():
    threshold_time = do_Mandelbrot_cython()
    from pylab import imshow, cm, clf, show
    clf()
    imshow(threshold_time, cmap=cm.spectral, extent=(-1, 1, -1, 1))
    show()




From robert.kern at gmail.com  Fri Jun 20 15:19:41 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Jun 2008 14:19:41 -0500
Subject: [Numpy-discussion] Right way of looping throught ndarrays using
	Cython
In-Reply-To: <20080620191334.GE4646@phare.normalesup.org>
References: <20080620141732.GA10745@phare.normalesup.org>
	<3d375d730806201207v227e5a7fne4e09020caa98c2d@mail.gmail.com>
	<20080620191334.GE4646@phare.normalesup.org>
Message-ID: <3d375d730806201219g51a8891ex8c76a1198056324@mail.gmail.com>

On Fri, Jun 20, 2008 at 14:13, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Fri, Jun 20, 2008 at 02:07:08PM -0500, Robert Kern wrote:
>> > Does somebody have a example of fast looping through ndarrays using
>> > modern Cython idioms?
>
>> If you're using normal Python indexing, then that's where all your
>> time is being spent. You need to grab the actual .data pointer and do
>> C indexing to get speed. Can you show us the code you are timing?
>
> That is indeed what I was thinking. There is no way of doing this appart
> by using the point-style indexing? I guess I am trying to find the best
> (as in most readable) way of doing this. This is for teaching, not
> production, so I am very much interested in having something simple.

I believe the numpy-Cython SoC project intended to address just that.
I haven't kept up since the initial proposal, though.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From robert.kern at gmail.com  Fri Jun 20 15:23:37 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Jun 2008 14:23:37 -0500
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <1e2af89e0806201211g134c2c73m58b6583efaa6056b@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<9457e7c80806191702y35c7348di78c2e236c02bcf9d@mail.gmail.com>
	<3d375d730806191706h441912ebmba3ae981e32d4eb@mail.gmail.com>
	<db6b5ecc0806191718x5dfea216v8d2c94d331f4227a@mail.gmail.com>
	<3d375d730806191729h1d3d5953m586a5fb3f4fcacdc@mail.gmail.com>
	<db6b5ecc0806192118w25a6b530j4135c4e6cda90b8a@mail.gmail.com>
	<db6b5ecc0806201051s2b87de69k925b4d998c2f974d@mail.gmail.com>
	<485BF2D4.4080600@astraw.com>
	<db6b5ecc0806201125l42614bf2ga7d47f79c67cf753@mail.gmail.com>
	<1e2af89e0806201211g134c2c73m58b6583efaa6056b@mail.gmail.com>
Message-ID: <3d375d730806201223q7965e15bof26a9365376e9210@mail.gmail.com>

On Fri, Jun 20, 2008 at 14:11, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> As a matter of interest, what is the relationship, if any, between (in Cython)
>
> import numpy
>
> and
>
> cnp.import_array()
>
> Are they initializing different copies of the same thing?

No. "import numpy" is essentially the same as the pure Python
equivalent; it loads the module and puts it into the namespace.
cnp.import_array() loads the module (or reuses the already loaded
one), does not put it into any namespace, but then, most importantly,
grabs the pointer to the table of function pointers and assigns it to
the local void** variable. All of the numpy API is made available to
third-party extension modules by #define macros which look up into
this table.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From discerptor at gmail.com  Fri Jun 20 15:27:47 2008
From: discerptor at gmail.com (Joshua Lippai)
Date: Fri, 20 Jun 2008 15:27:47 -0400
Subject: [Numpy-discussion] nose changes checked in
In-Reply-To: <9457e7c80806181535k764e987dva3583fc9336bd839@mail.gmail.com>
References: <1d36917a0806161728v2266a50dj212ffcee451cd533@mail.gmail.com>
	<9911419a0806181515l5ccadf45ya6158cd4f8fc1c1f@mail.gmail.com>
	<9457e7c80806181535k764e987dva3583fc9336bd839@mail.gmail.com>
Message-ID: <9911419a0806201227u6a9ca280m6cab15038d645c46@mail.gmail.com>

I realise this, but 1659 tests in 8.739s with Nose as opposed to just
over 1000 tests in right around 1 second with the previous test system
means there is some kind of slowdown involved besides just the number
of tests being found. Not that I mind: I'm not looking for blazing
speed when running tests, and this Nose system should make it easier
to manage the tests.

Josh

On Wed, Jun 18, 2008 at 6:35 PM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2008/6/19 Joshua Lippai <discerptor at gmail.com>:
>> The new testing system works well over here, built on Mac OS X 10.5.2
>> with GCC 4.2. No errors/failures, but there is that warning Charles
>> mentioned as well as the noticeable difference in speed between this
>> and the old tests.
>
> Nose does a more thorough job of finding tests (which takes some time
> in itself).  It executes 1343 tests, compared to just over 1000
> before.
>
> Regards
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From haase at msg.ucsf.edu  Fri Jun 20 16:17:00 2008
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Fri, 20 Jun 2008 22:17:00 +0200
Subject: [Numpy-discussion] Adding a version system to the deprecation
	functions in numpy/lib/utils ?
In-Reply-To: <485B5E4E.90305@ar.media.kyoto-u.ac.jp>
References: <485B4E40.5070504@ar.media.kyoto-u.ac.jp>
	<3d375d730806192353n3720a21fked67e3125048df63@mail.gmail.com>
	<485B540F.7060209@ar.media.kyoto-u.ac.jp>
	<3d375d730806200012q36bc7338k7a73ee1cfca18ccf@mail.gmail.com>
	<485B591A.80902@ar.media.kyoto-u.ac.jp>
	<3d375d730806200041s523ce775vc6f27c4964f50e00@mail.gmail.com>
	<485B5E4E.90305@ar.media.kyoto-u.ac.jp>
Message-ID: <bc657ead0806201317u7ec65829w74ac5c3ee6993a73@mail.gmail.com>

On Fri, Jun 20, 2008 at 9:37 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Robert Kern wrote:
>>
>> They should deprecated for one minor version and then deleted.
>
> Ok, thanks for the clarification. Since I was not aware of this, and did
> not see this information on the wiki, I added it in
> http://scipy.org/scipy/numpy/wiki/ApiDeprecation. Let me know if there
> is something I misunderstood from your remarks,
>
Hi David,
you say:
"""For example, if a function is deprecated in numpy 1.1.0, it will
remain so in all 1.1.x releases, but it can and should be removed in
1.2.0. """

But what if, say, with version 1.1.2 we decide to change a given
default argument. So you get the deprecation warning starting with
1.1.2.
But now, if this happens to be the last version of 1.1.x. -- you say
that already with the next version ( being 1.2.0 ) it would change
without a deprecation warning given anymore !?

Thanks,
Sebastian Haase


From robert.kern at gmail.com  Fri Jun 20 16:25:14 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Jun 2008 15:25:14 -0500
Subject: [Numpy-discussion] Adding a version system to the deprecation
	functions in numpy/lib/utils ?
In-Reply-To: <bc657ead0806201317u7ec65829w74ac5c3ee6993a73@mail.gmail.com>
References: <485B4E40.5070504@ar.media.kyoto-u.ac.jp>
	<3d375d730806192353n3720a21fked67e3125048df63@mail.gmail.com>
	<485B540F.7060209@ar.media.kyoto-u.ac.jp>
	<3d375d730806200012q36bc7338k7a73ee1cfca18ccf@mail.gmail.com>
	<485B591A.80902@ar.media.kyoto-u.ac.jp>
	<3d375d730806200041s523ce775vc6f27c4964f50e00@mail.gmail.com>
	<485B5E4E.90305@ar.media.kyoto-u.ac.jp>
	<bc657ead0806201317u7ec65829w74ac5c3ee6993a73@mail.gmail.com>
Message-ID: <3d375d730806201325j19b01fe9i9fd00f5490d20f7f@mail.gmail.com>

On Fri, Jun 20, 2008 at 15:17, Sebastian Haase <haase at msg.ucsf.edu> wrote:
> On Fri, Jun 20, 2008 at 9:37 AM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
>> Robert Kern wrote:
>>>
>>> They should deprecated for one minor version and then deleted.
>>
>> Ok, thanks for the clarification. Since I was not aware of this, and did
>> not see this information on the wiki, I added it in
>> http://scipy.org/scipy/numpy/wiki/ApiDeprecation. Let me know if there
>> is something I misunderstood from your remarks,
>>
> Hi David,
> you say:
> """For example, if a function is deprecated in numpy 1.1.0, it will
> remain so in all 1.1.x releases, but it can and should be removed in
> 1.2.0. """
>
> But what if, say, with version 1.1.2 we decide to change a given
> default argument. So you get the deprecation warning starting with
> 1.1.2.
> But now, if this happens to be the last version of 1.1.x. -- you say
> that already with the next version ( being 1.2.0 ) it would change
> without a deprecation warning given anymore !?

You missed this part (emphasis added): "Deprecation warnings should
exist for *one*full*minor*release*cycle* before the deprecated
features are removed."

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From alan.mcintyre at gmail.com  Fri Jun 20 17:01:54 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Fri, 20 Jun 2008 17:01:54 -0400
Subject: [Numpy-discussion] Test framework changes
Message-ID: <1d36917a0806201401y65187588s5bae0f6c79061269@mail.gmail.com>

Hi all,

Just wanted to get feedback about the following changes before I make
them.  Please speak up if any of this seems objectionable to you.

- The old test framework classes will be restored, but will not be
used anywhere in NumPy's tests.  If your old tests still don't work
with the restored classes, please let me know and I'll fix it.

- Add a function to numpy.testing to execute a module's tests via the
"if __name__ == '__main__'" hook.  It takes one optional argument, the
name of the file to run (if not present, it uses the same method as
NoseTester to look it up from the stack).  The intent is to make the
boilerplate as simple as possible.

  if __name__ == '__main__':
      run_module_suite()

  If somebody has a suggestion for a better name, I'd love to hear it.
I didn't want "test" in the name, because then I have to explicitly
tell nose to ignore it when it's looking for test functions.

- Remove numpy/testing/pkgtester.py since it now just has one line of
code that imports NoseTester (as Tester) from nosetester.py (it used
to create a null tester if nose wasn't present, but this was removed
by porting of r4424 from SciPy).  NoseTester will still be made
available as numpy.testing.Tester.

- numpy.test (and all the other test functions in subpackages) will
take the following positional arguments: label, verbose, extra_argv,
doctests, coverage (the same arguments as SciPy.test except for the
new coverage option).  The old arguments can be passed in as keyword
arguments (which seems to be how they were passed in in all the
examples I could find), and they will be emulated as much as possible
by the new test suite, but a deprecation warning will be issued.

- numpy.test now returns an object with a wasSuccessful method; under
the old test framework it returned a unittest._TextTestResult, but
since nose.run only returns success/failure, I'm just wrapping this
result in a dummy object to match the old behavior until I can figure
out how to get a real _TextTestResult from nose.

(The two changes to numpy.test should allow the buildbots to run the
tests properly again)

Thanks,
Alan


From dalcinl at gmail.com  Fri Jun 20 17:26:40 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 20 Jun 2008 18:26:40 -0300
Subject: [Numpy-discussion] Right way of looping throught ndarrays using
	Cython
In-Reply-To: <20080620141732.GA10745@phare.normalesup.org>
References: <20080620141732.GA10745@phare.normalesup.org>
Message-ID: <e7ba66e40806201426o53b3cc00l56d2e7c495c2510@mail.gmail.com>

On 6/20/08, Gael Varoquaux <gael.varoquaux at normalesup.org> wrote:
>  I am trying to figure the right way of looping throught ndarrays using
>  Cython, currently. Things seem to have evolved a bit compared to what
>  some documents on the web claim (eg "for i from 0<=i<n" does not seem
>  faster than "for i in range(n)").

Regarding for loops, Cython recently gained an optimization (perhaps
you already know this, but just in case). If the 'i' variable is a
'cdef'  one, then Cython does not actually use 'range', but a plain C
for loop.  See yourself:

cdef void foo():
    cdef int i, j=0
    for i in range(5,20,3):
        j += i

The generated C code is (comments stripped):

static  void __pyx_f_7fortest_foo(void) {
  int __pyx_v_i;
  int __pyx_v_j;
  __pyx_v_j = 0;
  for (__pyx_v_i = 5; __pyx_v_i < 20; __pyx_v_i+=3) {
    __pyx_v_j += __pyx_v_i;
  }
}


Really, really nice!!!, doesn't it? You can just forget about the 'for
i from ...' form (at least for the 99% of the cases, I think).

Additionally, 'for' loops can now also be written like this (without 'from'):

for 0<= i < n:
    do_stuff()

but IMHO, the 'range' optimization is just a big win!

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594


From ramercer at gmail.com  Fri Jun 20 17:27:33 2008
From: ramercer at gmail.com (Adam Mercer)
Date: Fri, 20 Jun 2008 16:27:33 -0500
Subject: [Numpy-discussion] Specifying a fortran compiler on numpy build.
Message-ID: <799406d60806201427i586805b2p6208b14cfe782dfc@mail.gmail.com>

Hi

If I specify a fortran compiler when building numpy, does that have
any effect on what is installed? In other words, must I build numpy
against a fortran compiler in order to successfully build and use
extension written in fortran - such as scipy?

Cheers

Adam


From robert.kern at gmail.com  Fri Jun 20 17:35:19 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Jun 2008 16:35:19 -0500
Subject: [Numpy-discussion] Test framework changes
In-Reply-To: <1d36917a0806201401y65187588s5bae0f6c79061269@mail.gmail.com>
References: <1d36917a0806201401y65187588s5bae0f6c79061269@mail.gmail.com>
Message-ID: <3d375d730806201435x23d14c1fr696c82a417700bb@mail.gmail.com>

On Fri, Jun 20, 2008 at 16:01, Alan McIntyre <alan.mcintyre at gmail.com> wrote:

> - numpy.test now returns an object with a wasSuccessful method; under
> the old test framework it returned a unittest._TextTestResult, but
> since nose.run only returns success/failure, I'm just wrapping this
> result in a dummy object to match the old behavior until I can figure
> out how to get a real _TextTestResult from nose.

So NoseTester.run() basically just calls nose.run(). That basically
just instantiates nose.core.TestProgram and returns the .success
attribute of it. Unfortunately, the TextTestResults object (a nose
subclass of unittest._TextTestResults) gets created and discarded
inside the nose.core.TestProgram.runTests() method. However, if you
were to subclass it and override that method to store the
TextTestResults to an attribute, you could return it from
NoseTester.run().

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From alan.mcintyre at gmail.com  Fri Jun 20 17:35:58 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Fri, 20 Jun 2008 17:35:58 -0400
Subject: [Numpy-discussion] NumPy coverage via nose
Message-ID: <1d36917a0806201435u25668edbs3f1a106587d93a9d@mail.gmail.com>

Hi all,

Nose has a plugin (included in the current version of nose) that uses
Ned Batchelder's coverage module
(http://nedbatchelder.com/code/modules/coverage.html) to provide
coverage information.  Is it acceptable to use this capability in nose
to provide coverage reporting for NumPy?  The coverage module is only
needed if you actually want to check coverage.

Thanks,
Alan


From robert.kern at gmail.com  Fri Jun 20 17:38:16 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Jun 2008 16:38:16 -0500
Subject: [Numpy-discussion] Specifying a fortran compiler on numpy build.
In-Reply-To: <799406d60806201427i586805b2p6208b14cfe782dfc@mail.gmail.com>
References: <799406d60806201427i586805b2p6208b14cfe782dfc@mail.gmail.com>
Message-ID: <3d375d730806201438va8469dfoa31f1d1944e650fe@mail.gmail.com>

On Fri, Jun 20, 2008 at 16:27, Adam Mercer <ramercer at gmail.com> wrote:
> Hi
>
> If I specify a fortran compiler when building numpy, does that have
> any effect on what is installed? In other words, must I build numpy
> against a fortran compiler in order to successfully build and use
> extension written in fortran - such as scipy?

No. It just affects the Fortran compiler (if any) used to build numpy.
The only place this might affect you is if you use a LAPACK or BLAS
that needs to be linked with a Fortran compiler. Generally, you don't
have to specify anything.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From robert.kern at gmail.com  Fri Jun 20 17:44:47 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Jun 2008 16:44:47 -0500
Subject: [Numpy-discussion] NumPy coverage via nose
In-Reply-To: <1d36917a0806201435u25668edbs3f1a106587d93a9d@mail.gmail.com>
References: <1d36917a0806201435u25668edbs3f1a106587d93a9d@mail.gmail.com>
Message-ID: <3d375d730806201444h37f0b3ayd8531ec2c33e27b4@mail.gmail.com>

On Fri, Jun 20, 2008 at 16:35, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> Hi all,
>
> Nose has a plugin (included in the current version of nose) that uses
> Ned Batchelder's coverage module
> (http://nedbatchelder.com/code/modules/coverage.html) to provide
> coverage information.  Is it acceptable to use this capability in nose
> to provide coverage reporting for NumPy?  The coverage module is only
> needed if you actually want to check coverage.

Specifically, what changes are you thinking of making?
numpy.test(coverage=True) already triggers nose's coverage feature.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From ramercer at gmail.com  Fri Jun 20 17:45:34 2008
From: ramercer at gmail.com (Adam Mercer)
Date: Fri, 20 Jun 2008 16:45:34 -0500
Subject: [Numpy-discussion] Specifying a fortran compiler on numpy build.
In-Reply-To: <3d375d730806201438va8469dfoa31f1d1944e650fe@mail.gmail.com>
References: <799406d60806201427i586805b2p6208b14cfe782dfc@mail.gmail.com>
	<3d375d730806201438va8469dfoa31f1d1944e650fe@mail.gmail.com>
Message-ID: <799406d60806201445i6e983580iffbe6d967b6f573@mail.gmail.com>

On Fri, Jun 20, 2008 at 4:38 PM, Robert Kern <robert.kern at gmail.com> wrote:

> No. It just affects the Fortran compiler (if any) used to build numpy.
> The only place this might affect you is if you use a LAPACK or BLAS
> that needs to be linked with a Fortran compiler. Generally, you don't
> have to specify anything.

Thanks for the clarification.

Cheers

Adam


From alan.mcintyre at gmail.com  Fri Jun 20 17:52:41 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Fri, 20 Jun 2008 17:52:41 -0400
Subject: [Numpy-discussion] NumPy coverage via nose
In-Reply-To: <3d375d730806201444h37f0b3ayd8531ec2c33e27b4@mail.gmail.com>
References: <1d36917a0806201435u25668edbs3f1a106587d93a9d@mail.gmail.com>
	<3d375d730806201444h37f0b3ayd8531ec2c33e27b4@mail.gmail.com>
Message-ID: <1d36917a0806201452w433930fdneaec90d5dbb2ca25@mail.gmail.com>

On Fri, Jun 20, 2008 at 5:44 PM, Robert Kern <robert.kern at gmail.com> wrote:
> Specifically, what changes are you thinking of making?
> numpy.test(coverage=True) already triggers nose's coverage feature.

Actually, that wasn't supposed to get added in my last checkin; I was
experimenting with it and forgot to take it out. :/


From alan.mcintyre at gmail.com  Fri Jun 20 18:04:09 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Fri, 20 Jun 2008 18:04:09 -0400
Subject: [Numpy-discussion] Test framework changes
In-Reply-To: <3d375d730806201435x23d14c1fr696c82a417700bb@mail.gmail.com>
References: <1d36917a0806201401y65187588s5bae0f6c79061269@mail.gmail.com>
	<3d375d730806201435x23d14c1fr696c82a417700bb@mail.gmail.com>
Message-ID: <1d36917a0806201504x3b7dd98aqae05ffa2f5f4a82f@mail.gmail.com>

On Fri, Jun 20, 2008 at 5:35 PM, Robert Kern <robert.kern at gmail.com> wrote:
> So NoseTester.run() basically just calls nose.run(). That basically
> just instantiates nose.core.TestProgram and returns the .success
> attribute of it. Unfortunately, the TextTestResults object (a nose
> subclass of unittest._TextTestResults) gets created and discarded
> inside the nose.core.TestProgram.runTests() method. However, if you
> were to subclass it and override that method to store the
> TextTestResults to an attribute, you could return it from
> NoseTester.run().

Yep. I was hoping there was some built-in way to get to the details of
the results via the nose API, but that doesn't appear to be something
the nose developers considered. I'll probably go ahead and do as you
suggested instead of making a temporary class to hold the result.


From robert.kern at gmail.com  Fri Jun 20 18:37:17 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Jun 2008 17:37:17 -0500
Subject: [Numpy-discussion] NumPy coverage via nose
In-Reply-To: <1d36917a0806201452w433930fdneaec90d5dbb2ca25@mail.gmail.com>
References: <1d36917a0806201435u25668edbs3f1a106587d93a9d@mail.gmail.com>
	<3d375d730806201444h37f0b3ayd8531ec2c33e27b4@mail.gmail.com>
	<1d36917a0806201452w433930fdneaec90d5dbb2ca25@mail.gmail.com>
Message-ID: <3d375d730806201537l7a0da804yc4d1034f74813318@mail.gmail.com>

On Fri, Jun 20, 2008 at 16:52, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> On Fri, Jun 20, 2008 at 5:44 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> Specifically, what changes are you thinking of making?
>> numpy.test(coverage=True) already triggers nose's coverage feature.
>
> Actually, that wasn't supposed to get added in my last checkin; I was
> experimenting with it and forgot to take it out. :/

It needs a little bit of work. For example, you have
--cover-package=numpy hard-coded, but NumpyTest.test() will also be
exposed as scipy.test(). But otherwise, yes, I think the functionality
is useful.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From fperez.net at gmail.com  Fri Jun 20 19:22:51 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 20 Jun 2008 16:22:51 -0700
Subject: [Numpy-discussion] Test framework changes
In-Reply-To: <1d36917a0806201504x3b7dd98aqae05ffa2f5f4a82f@mail.gmail.com>
References: <1d36917a0806201401y65187588s5bae0f6c79061269@mail.gmail.com>
	<3d375d730806201435x23d14c1fr696c82a417700bb@mail.gmail.com>
	<1d36917a0806201504x3b7dd98aqae05ffa2f5f4a82f@mail.gmail.com>
Message-ID: <db6b5ecc0806201622q331e8e84m8838f609b3f913bb@mail.gmail.com>

On Fri, Jun 20, 2008 at 3:04 PM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> On Fri, Jun 20, 2008 at 5:35 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> So NoseTester.run() basically just calls nose.run(). That basically
>> just instantiates nose.core.TestProgram and returns the .success
>> attribute of it. Unfortunately, the TextTestResults object (a nose
>> subclass of unittest._TextTestResults) gets created and discarded
>> inside the nose.core.TestProgram.runTests() method. However, if you
>> were to subclass it and override that method to store the
>> TextTestResults to an attribute, you could return it from
>> NoseTester.run().
>
> Yep. I was hoping there was some built-in way to get to the details of
> the results via the nose API, but that doesn't appear to be something
> the nose developers considered. I'll probably go ahead and do as you
> suggested instead of making a temporary class to hold the result.

It may be worth bringing it up wtih the nose guys here:

http://lists.idyll.org/listinfo/testing-in-python

The nose author seems very responsive, and Titus Brown is on the list
and cares a lot about numpy/scipy, and he may offer suggestions as
well.

Cheers,

f


From alan.mcintyre at gmail.com  Fri Jun 20 20:22:29 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Fri, 20 Jun 2008 20:22:29 -0400
Subject: [Numpy-discussion] Test framework changes
In-Reply-To: <db6b5ecc0806201622q331e8e84m8838f609b3f913bb@mail.gmail.com>
References: <1d36917a0806201401y65187588s5bae0f6c79061269@mail.gmail.com>
	<3d375d730806201435x23d14c1fr696c82a417700bb@mail.gmail.com>
	<1d36917a0806201504x3b7dd98aqae05ffa2f5f4a82f@mail.gmail.com>
	<db6b5ecc0806201622q331e8e84m8838f609b3f913bb@mail.gmail.com>
Message-ID: <1d36917a0806201722p2b2ed4f4y613435efc42a6fba@mail.gmail.com>

On Fri, Jun 20, 2008 at 7:22 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> It may be worth bringing it up wtih the nose guys here:
>
> http://lists.idyll.org/listinfo/testing-in-python
>
> The nose author seems very responsive, and Titus Brown is on the list
> and cares a lot about numpy/scipy, and he may offer suggestions as
> well.

Thanks, I'll do that. :)


From fperez.net at gmail.com  Fri Jun 20 23:00:59 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 20 Jun 2008 20:00:59 -0700
Subject: [Numpy-discussion] Nose, doctests and extension modules.
Message-ID: <db6b5ecc0806202000m3d9178d0y7b74b06fd56ccad6@mail.gmail.com>

Howdy,

I made some progress on the issue I mentioned earlier:

http://lists.idyll.org/pipermail/testing-in-python/2008-June/000709.html

As indicated there, a bug report has been filed with Python itself:

http://bugs.python.org/issue3158

But in the meantime, Alan may want to apply the monkeypatch solution
until the whole thing is correctly fixed upstream.  I'm attaching the
little cython example that contains the monkeypatch.  If anyone starts
writing cython code with docstrings, or putting proper doctests in
hand-written extension modules (which is more annoying but can be done
just as well), this fix will mean that the testing system can find
them.

Cheers,

f
-------------- next part --------------
A non-text attachment was scrubbed...
Name: primes.tgz
Type: application/x-gzip
Size: 2451 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080620/905c125f/attachment.bin>

From charlesr.harris at gmail.com  Fri Jun 20 23:21:04 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 20 Jun 2008 21:21:04 -0600
Subject: [Numpy-discussion] Deprecated ufuncs
Message-ID: <e06186140806202021w361f75a3n5abbfa4ea619e63@mail.gmail.com>

What should we do about this:

DeprecationWarning: complex divmod(), // and % are deprecated


Note that these functions are not defined in numpy, rather complex numbers
are promoted to python objects and the appropriate method/pyfunction is
called. For fmod this returns an attribute error because this method is not
defined for complex objects. For remainder (which has the same doc string as
fmod), we get the deprecation warning when PyNumber_Remainder is called on
the object. We also get occasional errors and segfaults depending on the
arguments and their order.

I think we have two options: define these functions for ourselves, or remove
them. The latter requires removing objects from the available signatures
because otherwise complex are automatically promoted to the next highest
available type, objects -- a fact that partly accounts for the variety of
promotion rules for ufuncs. The other option is to define them for
ourselves. There is actually a third option, which is not to automatically
promote up the chain of available types until a function is found. I rather
like that myself because it makes explicit the operations available for the
various types and, if we should happen to implement another type, the
function signatures won't change.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080620/0eaa6b37/attachment.html>

From charlesr.harris at gmail.com  Sat Jun 21 01:21:56 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 20 Jun 2008 23:21:56 -0600
Subject: [Numpy-discussion] NotImplementedType should be an exception
Message-ID: <e06186140806202221n5ab6d587re987232dfe9dedcd@mail.gmail.com>

Shouldn't this raise an NotImplementedError exception?

In [7]: type(remainder(complex192(1), complex192(1)))
Out[7]: <type 'NotImplementedType'>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080620/c08c906d/attachment.html>

From charlesr.harris at gmail.com  Sat Jun 21 02:07:34 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 21 Jun 2008 00:07:34 -0600
Subject: [Numpy-discussion] NotImplementedType should be an exception
In-Reply-To: <e06186140806202221n5ab6d587re987232dfe9dedcd@mail.gmail.com>
References: <e06186140806202221n5ab6d587re987232dfe9dedcd@mail.gmail.com>
Message-ID: <e06186140806202307i3844001cicb2a0387da17ab8d@mail.gmail.com>

On Fri, Jun 20, 2008 at 11:21 PM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

> Shouldn't this raise an NotImplementedError exception?
>
> In [7]: type(remainder(complex192(1), complex192(1)))
> Out[7]: <type 'NotImplementedType'>
>

I suspect these types should both have Py_TPFLAGS_CHECKTYPES bit set in the
tp_flag so that python will raise a TypeError.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080621/105ae161/attachment.html>

From dagss at student.matnat.uio.no  Sat Jun 21 02:50:27 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Jun 2008 08:50:27 +0200 (CEST)
Subject: [Numpy-discussion] On my Cython/NumPy project
Message-ID: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>

Since there's been a lot of Cython discussion lately I thought I'd speak
up and start a thread specifically for my project.

We've just recently set a clear direction for my Google Summer of Code
project for integrating NumPy in a better way with Cython. What we ended
up with is that I will over the summer add Cython support for the Python 3
buffer interface:

http://wiki.cython.org/enhancements/buffer

(We aim to support NumPy back to Python 2.3, perhaps as a special case,
but the buffer interface will guide in what way we support it.)

General feedback is welcome; in particular, I need more opinions about
what syntax people would like. We seem unable to find something that we
really like; this is the current best candidate (cdef is the way you
declare types on variables in Cython):

cdef int i = 4, j = 6
cdef np.ndarray[np.float64, 2] arr = np.zeros((10, 10), dtype=np.float64)
arr[i, j] = 1
...

The code above the under the hood acquires a buffer and uses it for
efficient access. arr[5], arr[5,6,7], arr[2:4,...] and in general anything
but two simple indices will be passed to Python, while arr[i, j] will be
passed to the buffer.

This is a natural first step and I'll focus on this exclusively in the
beginning.

(That said, once this is in place, there are of course lots of
opportunities of doing an even better job in supporting the numerical
community with Cython: ndenumerate, more efficient buffer acquisition and
array creation, more efficient calling of NumPy/SciPy functions without
going through the Python "calling convention", etc..)

Dag Sverre



From dagss at student.matnat.uio.no  Sat Jun 21 02:59:25 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Jun 2008 08:59:25 +0200 (CEST)
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
Message-ID: <1834.65.102.174.34.1214031565.squirrel@webmail.uio.no>

Dag wrote:
> General feedback is welcome; in particular, I need more opinions about
> what syntax people would like. We seem unable to find something that we
> really like; this is the current best candidate (cdef is the way you
> declare types on variables in Cython):
>
> cdef int i = 4, j = 6
> cdef np.ndarray[np.float64, 2] arr = np.zeros((10, 10), dtype=np.float64)
> arr[i, j] = 1
> ...
>

Some more important points:
- There will likely be a compiler flag (and perhaps per-block or
per-variable pragmas) on whether bounds-checking is done or not)
- It doesn't look like negative indices will be supported in this mode, as
it adds another branch in potentially very tight loops. Opinions on this
are welcome though. In safe mode, bounds checking will catch it, while in
unsafe mode it will at best segfault and at worst corrupt data.

The negative indices thing is potentially confusing to new users. Pex uses
a different syntax (arr{i, j} for efficient array lookups) partly to make
this fact very explicit. Thoughts?

Dag Sverre



From h.rathgen at utwente.nl  Sat Jun 21 08:28:58 2008
From: h.rathgen at utwente.nl (Helmut Rathgen)
Date: Sat, 21 Jun 2008 14:28:58 +0200
Subject: [Numpy-discussion] seeking help with f2py_options
Message-ID: <485CF40A.7010607@utwente.nl>

Dear all,

I am trying to write a setp.py based on numpy.distutils for a mixed 
python/fortran90 package.

I'd like to specify the fortran compiler, such as the path to the 
compiler, compiler flags, etc. in setup.py.

I seemed to understand that this should be done by passing 'f2py_options 
= [ "...", "..." ]' to numpy.distutils.core.Extension()

However, I get errors such as

helmut at ncv77:~/src/testdistutils$ python setup.py build
running build
running scons
customize UnixCCompiler
Found executable /usr/bin/gcc
customize GnuFCompiler
Could not locate executable g77
Could not locate executable f77
customize IntelFCompiler
Found executable /opt/intel/fc/10.0.026/bin/ifort
customize IntelFCompiler
customize UnixCCompiler
customize UnixCCompiler using scons
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands 
--compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands 
--fcompiler options
running build_src
building extension "mrcwaf" sources
f2py options: ['--fcompiler=intel', 
'--f90exec=/opt/intel/fc/10.0.026/bin/ifort', '--opt="-O3 -xW -ipo"', 
'--noarch']
f2py: src/mrcwaf/mrcwaf.pyf
Unknown option '--fcompiler=intel'


How to use f2py_options - or should flags be passed in a different way?

Any help is greatly appreciated!

Thank you in advance!

Kind Regards,

Helmut Rathgen


-- 
Dipl. Phys. Helmut Rathgen
Physics of Complex Fluids | Faculty of Science and Technology
University of Twente | Postbus 217 | 7500 AE Enschede | The Netherlands
office +31 (0)53 489 3093                       home +31 (0)53 478 2346
fax    +31 (0)53 489 1096                       cell +31 (0)61 645 8938
sip    sip:helmut.rathgen at ekiga.net
http://pcf.tnw.utwente.nl/people/phd_students/h_rathgen_helmut.doc/


From Joris.DeRidder at ster.kuleuven.be  Sat Jun 21 08:57:40 2008
From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder)
Date: Sat, 21 Jun 2008 14:57:40 +0200
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
Message-ID: <783042A0-02F8-4E0C-9512-BE09CCDA5F05@ster.kuleuven.be>

Hi Dag,

> General feedback is welcome; in particular, I need more opinions about
> what syntax people would like. We seem unable to find something that  
> we
> really like; this is the current best candidate (cdef is the way you
> declare types on variables in Cython):
>
> cdef int i = 4, j = 6
> cdef np.ndarray[np.float64, 2] arr = np.zeros((10, 10),  
> dtype=np.float64)
> arr[i, j] = 1
> ...
>
> The code above the under the hood acquires a buffer and uses it for
> efficient access. arr[5], arr[5,6,7], arr[2:4,...] and in general  
> anything
> but two simple indices will be passed to Python, while arr[i, j]  
> will be
> passed to the buffer.

The syntax looks quite good, I think.
Some questions though:
- I guess there are some technical reasons why np.float64 in your  
example has to be repeated twice?
- When you say "negative indices are not supported" you mean that they  
are passed to python, or
   won't they work at all?

I'm looking forward to the results of your Cython project!

Cheers,
Joris


Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm



From pearu at cens.ioc.ee  Sat Jun 21 09:04:23 2008
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Sat, 21 Jun 2008 16:04:23 +0300 (EEST)
Subject: [Numpy-discussion] seeking help with f2py_options
In-Reply-To: <485CF40A.7010607@utwente.nl>
References: <485CF40A.7010607@utwente.nl>
Message-ID: <60427.88.89.32.166.1214053463.squirrel@cens.ioc.ee>

On Sat, June 21, 2008 3:28 pm, Helmut Rathgen wrote:
> Dear all,
>
> I am trying to write a setp.py based on numpy.distutils for a mixed
> python/fortran90 package.
>
> I'd like to specify the fortran compiler, such as the path to the
> compiler, compiler flags, etc. in setup.py.
>
> I seemed to understand that this should be done by passing 'f2py_options
> = [ "...", "..." ]' to numpy.distutils.core.Extension()
>
> However, I get errors such as
>
> helmut at ncv77:~/src/testdistutils$ python setup.py build
> running build
> running scons
> customize UnixCCompiler
> Found executable /usr/bin/gcc
> customize GnuFCompiler
> Could not locate executable g77
> Could not locate executable f77
> customize IntelFCompiler
> Found executable /opt/intel/fc/10.0.026/bin/ifort
> customize IntelFCompiler
> customize UnixCCompiler
> customize UnixCCompiler using scons
> running config_cc
> unifing config_cc, config, build_clib, build_ext, build commands
> --compiler options
> running config_fc
> unifing config_fc, config, build_clib, build_ext, build commands
> --fcompiler options
> running build_src
> building extension "mrcwaf" sources
> f2py options: ['--fcompiler=intel',
> '--f90exec=/opt/intel/fc/10.0.026/bin/ifort', '--opt="-O3 -xW -ipo"',
> '--noarch']
> f2py: src/mrcwaf/mrcwaf.pyf
> Unknown option '--fcompiler=intel'
>
>
> How to use f2py_options - or should flags be passed in a different way?
>

Note that --fcompiler= and other such options are actually options
to numpy.distutils (f2py script would just pass these options
forward to numpy.distutils). f2py_options can contain only f2py
specific options.
Hence you should try to modify sys.path
in the beggining of the setup.py file to specify the fortran
compiler options. For example, in setup.py file, insert:

import sys
sys.path.extend('config_fc --fcompiler=intel ....'.split())

See
  python setup.py config_fc --help
  python setup.py build_ext --help
for more information about possible options.

HTH,
Pearu



From oliphant at enthought.com  Sat Jun 21 09:39:15 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Sat, 21 Jun 2008 08:39:15 -0500
Subject: [Numpy-discussion] NotImplementedType should be an exception
In-Reply-To: <e06186140806202221n5ab6d587re987232dfe9dedcd@mail.gmail.com>
References: <e06186140806202221n5ab6d587re987232dfe9dedcd@mail.gmail.com>
Message-ID: <485D0483.3050309@enthought.com>

Charles R Harris wrote:
> Shouldn't this raise an NotImplementedError exception?
>
> In [7]: type(remainder(complex192(1), complex192(1)))
> Out[7]: <type 'NotImplementedType'>
I'm not sure if it is relevant in this case but we do need to keep in 
mind that Python uses the NotImplementedType as a signal to hand off the 
calculation to the other object in a binary operation.

If we change this here we may affect that behavior, unintentionally.

-Travis



From charlesr.harris at gmail.com  Sat Jun 21 10:57:12 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 21 Jun 2008 08:57:12 -0600
Subject: [Numpy-discussion] NotImplementedType should be an exception
In-Reply-To: <485D0483.3050309@enthought.com>
References: <e06186140806202221n5ab6d587re987232dfe9dedcd@mail.gmail.com>
	<485D0483.3050309@enthought.com>
Message-ID: <e06186140806210757o40fac9bbnc831a7360f403ab1@mail.gmail.com>

On Sat, Jun 21, 2008 at 7:39 AM, Travis E. Oliphant <oliphant at enthought.com>
wrote:

> Charles R Harris wrote:
> > Shouldn't this raise an NotImplementedError exception?
> >
> > In [7]: type(remainder(complex192(1), complex192(1)))
> > Out[7]: <type 'NotImplementedType'>
> I'm not sure if it is relevant in this case but we do need to keep in
> mind that Python uses the NotImplementedType as a signal to hand off the
> calculation to the other object in a binary operation.
>
> If we change this here we may affect that behavior, unintentionally.
>

But Python only does that if the Py_TPFLAGS_CHECKTYPES bit in the tp_flag is
set. With that flag it should try with the other variable, then raise a Type
error if that fails also. That's why I think the flag isn't set for these
variables; we should never see the NotImplementedType. And at first glance
at the code, I don't think that flag *is* set for the type.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080621/ce3b65dc/attachment.html>

From alan.mcintyre at gmail.com  Sat Jun 21 12:12:12 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Sat, 21 Jun 2008 12:12:12 -0400
Subject: [Numpy-discussion] Win64 buildbot stuck?
Message-ID: <1d36917a0806210912i6ffe6a8ag5a6554f04e116161@mail.gmail.com>

It looks like the Windows_XP_x86_64_MSVC buildbot is stuck in the
build phase for some reason.


From gael.varoquaux at normalesup.org  Sat Jun 21 12:22:24 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sat, 21 Jun 2008 18:22:24 +0200
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <1834.65.102.174.34.1214031565.squirrel@webmail.uio.no>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<1834.65.102.174.34.1214031565.squirrel@webmail.uio.no>
Message-ID: <20080621162224.GA30359@phare.normalesup.org>

On Sat, Jun 21, 2008 at 08:59:25AM +0200, Dag Sverre Seljebotn wrote:
> The negative indices thing is potentially confusing to new users. Pex uses
> a different syntax (arr{i, j} for efficient array lookups) partly to make
> this fact very explicit. Thoughts?

I don't like the different syntax. I would really be happy if Cython code
could stay as close as possible to python code. In particular we want to
make it as easy as possible to prototype code in Python and move it to
Cython.

My 2 cents,

Ga?l



From matthew.brett at gmail.com  Sat Jun 21 12:32:36 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 21 Jun 2008 16:32:36 +0000
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
Message-ID: <1e2af89e0806210932u744eea87x15efd3e00da530f3@mail.gmail.com>

Hi,

Thanks a lot for the email - it's an exciting project.

> cdef int i = 4, j = 6
> cdef np.ndarray[np.float64, 2] arr = np.zeros((10, 10), dtype=np.float64)
> arr[i, j] = 1

I'm afraid I'm going to pitch into an area I'm ignorant of, and I'm
sorry for that, but do you think there is any way of allowing fast
indexing and operations on arrays that do not have their type declared
at compile time?

Cheers,

Matthew


From charlesr.harris at gmail.com  Sat Jun 21 14:01:24 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 21 Jun 2008 12:01:24 -0600
Subject: [Numpy-discussion] NotImplementedType should be an exception
In-Reply-To: <e06186140806210757o40fac9bbnc831a7360f403ab1@mail.gmail.com>
References: <e06186140806202221n5ab6d587re987232dfe9dedcd@mail.gmail.com>
	<485D0483.3050309@enthought.com>
	<e06186140806210757o40fac9bbnc831a7360f403ab1@mail.gmail.com>
Message-ID: <e06186140806211101u62816e2cl690720ba4d33121e@mail.gmail.com>

On Sat, Jun 21, 2008 at 8:57 AM, Charles R Harris <charlesr.harris at gmail.com>
wrote:

>
>
> On Sat, Jun 21, 2008 at 7:39 AM, Travis E. Oliphant <
> oliphant at enthought.com> wrote:
>
>> Charles R Harris wrote:
>> > Shouldn't this raise an NotImplementedError exception?
>> >
>> > In [7]: type(remainder(complex192(1), complex192(1)))
>> > Out[7]: <type 'NotImplementedType'>
>> I'm not sure if it is relevant in this case but we do need to keep in
>> mind that Python uses the NotImplementedType as a signal to hand off the
>> calculation to the other object in a binary operation.
>>
>> If we change this here we may affect that behavior, unintentionally.
>>
>
> But Python only does that if the Py_TPFLAGS_CHECKTYPES bit in the tp_flag
> is set. With that flag it should try with the other variable, then raise a
> Type error if that fails also. That's why I think the flag isn't set for
> these variables; we should never see the NotImplementedType. And at first
> glance at the code, I don't think that flag *is* set for the type.
>

This particular return comes from ufunc_generic_call and I'm not sure what
Python should or can do in this case. Thoughts?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080621/e8545b4c/attachment.html>

From dagss at student.matnat.uio.no  Sat Jun 21 14:55:54 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Jun 2008 20:55:54 +0200 (CEST)
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <1e2af89e0806210932u744eea87x15efd3e00da530f3@mail.gmail.com>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<1e2af89e0806210932u744eea87x15efd3e00da530f3@mail.gmail.com>
Message-ID: <2219.65.102.174.35.1214074554.squirrel@webmail.uio.no>

Matthew Brett wrote:
> Hi,
>
> Thanks a lot for the email - it's an exciting project.
>
>> cdef int i = 4, j = 6
>> cdef np.ndarray[np.float64, 2] arr = np.zeros((10, 10),
>> dtype=np.float64)
>> arr[i, j] = 1
>
> I'm afraid I'm going to pitch into an area I'm ignorant of, and I'm
> sorry for that, but do you think there is any way of allowing fast
> indexing and operations on arrays that do not have their type declared
> at compile time?

Well... the Cython compiler definitely needs to know about which type it
is -- the array simply has a byte buffer, and one needs to know how to
interpret that (how many bytes per element etc.).

However, I could make it so that if you left out the type, it would be
auto-detected. I.e.:

cdef np.ndarray[2] arr = ...
cdef np.float64 x = arr[3,4]

However, if you then did something like this on a 32-bit-system:

cdef np.ndarray[2] arr = ...
print arr[3,4]

Then it would assume arr is an object, and if arr is really float64 you
would segfault.

So the real answer is: Yes, the type is needed.

Dag Sverre



From fperez.net at gmail.com  Sat Jun 21 15:02:35 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Sat, 21 Jun 2008 12:02:35 -0700
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
Message-ID: <db6b5ecc0806211202u4fe19241yd0ea12d17c4f994e@mail.gmail.com>

On Fri, Jun 20, 2008 at 11:50 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Since there's been a lot of Cython discussion lately I thought I'd speak
> up and start a thread specifically for my project.

Thanks for coming over for the discussion!

> The code above the under the hood acquires a buffer and uses it for
> efficient access. arr[5], arr[5,6,7], arr[2:4,...] and in general anything
> but two simple indices will be passed to Python, while arr[i, j] will be
> passed to the buffer.

Just so I understand correctly: do you mean that in general only 2
indices are supported 'natively', or that for an n-dim array, only
*exactly n* indices are supported natively and other approaches are
delegated to pure python?

Cheers,

f


From dagss at student.matnat.uio.no  Sat Jun 21 15:03:45 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Jun 2008 21:03:45 +0200 (CEST)
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <783042A0-02F8-4E0C-9512-BE09CCDA5F05@ster.kuleuven.be>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<783042A0-02F8-4E0C-9512-BE09CCDA5F05@ster.kuleuven.be>
Message-ID: <2256.65.102.174.35.1214075025.squirrel@webmail.uio.no>

Joris De Ridder wrote:
> Hi Dag,
>
>> General feedback is welcome; in particular, I need more opinions about
>> what syntax people would like. We seem unable to find something that
>> we
>> really like; this is the current best candidate (cdef is the way you
>> declare types on variables in Cython):
>>
>> cdef int i = 4, j = 6
>> cdef np.ndarray[np.float64, 2] arr = np.zeros((10, 10),
>> dtype=np.float64)
>> arr[i, j] = 1
>> ...
>>
>> The code above the under the hood acquires a buffer and uses it for
>> efficient access. arr[5], arr[5,6,7], arr[2:4,...] and in general
>> anything
>> but two simple indices will be passed to Python, while arr[i, j]
>> will be
>> passed to the buffer.
>
> The syntax looks quite good, I think.
> Some questions though:
> - I guess there are some technical reasons why np.float64 in your
> example has to be repeated twice?

There are all sorts of reasons. The right hand side is disconnected from
the left hand side. It could just as well say

cdef np.ndarray[float64, 2] arr = my_func()
or
cdef np.ndarray[float64, 2] arr = x
or
cdef np.ndarray[float64, 2] arr = np.zeros((3, 3),
dtype=np.int).astype(np.float64)

(If you assign a Python object that is not an ndarray of right dimensions
and type, a runtime exception is raised.)

So: If you can figure out a nice rule for not having to repeat the type
I'm all ears, but, really, the repeating of the type was only incidental.

> - When you say "negative indices are not supported" you mean that they
> are passed to python, or
>    won't they work at all?

In unsafe mode: If you are really lucky they will segfault your
application, however usually you will not be that fortunate and they will
corrupt your data instead.

In safe mode, they will raise an IndexError.

(Passing them to Python would require to know that they are negative in
the first place. And knowing that they are negative would require an extra
if-test in tight, tight inner loops.)

(Well, of course, if you actually do "arr[-3, -2]" I suppose we can raise
a compiler error, the point is if the negative values are stored in
variables -- how do you know if they are negative without checking?)

Dag Sverre



From dagss at student.matnat.uio.no  Sat Jun 21 15:09:45 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Jun 2008 21:09:45 +0200 (CEST)
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <db6b5ecc0806211202u4fe19241yd0ea12d17c4f994e@mail.gmail.com>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<db6b5ecc0806211202u4fe19241yd0ea12d17c4f994e@mail.gmail.com>
Message-ID: <2287.65.102.174.35.1214075385.squirrel@webmail.uio.no>

Fernando Perez wrote:
> On Fri, Jun 20, 2008 at 11:50 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Since there's been a lot of Cython discussion lately I thought I'd speak
>> up and start a thread specifically for my project.
>
> Thanks for coming over for the discussion!
>
>> The code above the under the hood acquires a buffer and uses it for
>> efficient access. arr[5], arr[5,6,7], arr[2:4,...] and in general
>> anything
>> but two simple indices will be passed to Python, while arr[i, j] will be
>> passed to the buffer.
>
> Just so I understand correctly: do you mean that in general only 2
> indices are supported 'natively', or that for an n-dim array, only
> *exactly n* indices are supported natively and other approaches are
> delegated to pure python?

That is the idea. At least for NumPy, all the other cases will generate
new arrays, so they tend to be a different kind of operations. One can
look into improving slicing efficiency etc. later.

Dag Sverre



From charlesr.harris at gmail.com  Sat Jun 21 16:00:48 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 21 Jun 2008 14:00:48 -0600
Subject: [Numpy-discussion] NotImplementedType should be an exception
In-Reply-To: <e06186140806211101u62816e2cl690720ba4d33121e@mail.gmail.com>
References: <e06186140806202221n5ab6d587re987232dfe9dedcd@mail.gmail.com>
	<485D0483.3050309@enthought.com>
	<e06186140806210757o40fac9bbnc831a7360f403ab1@mail.gmail.com>
	<e06186140806211101u62816e2cl690720ba4d33121e@mail.gmail.com>
Message-ID: <e06186140806211300va1a59b8ke84c5fed157cdd2c@mail.gmail.com>

On Sat, Jun 21, 2008 at 12:01 PM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

>
>
> On Sat, Jun 21, 2008 at 8:57 AM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>>
>>
>> On Sat, Jun 21, 2008 at 7:39 AM, Travis E. Oliphant <
>> oliphant at enthought.com> wrote:
>>
>>> Charles R Harris wrote:
>>> > Shouldn't this raise an NotImplementedError exception?
>>> >
>>> > In [7]: type(remainder(complex192(1), complex192(1)))
>>> > Out[7]: <type 'NotImplementedType'>
>>> I'm not sure if it is relevant in this case but we do need to keep in
>>> mind that Python uses the NotImplementedType as a signal to hand off the
>>> calculation to the other object in a binary operation.
>>>
>>> If we change this here we may affect that behavior, unintentionally.
>>>
>>
>> But Python only does that if the Py_TPFLAGS_CHECKTYPES bit in the tp_flag
>> is set. With that flag it should try with the other variable, then raise a
>> Type error if that fails also. That's why I think the flag isn't set for
>> these variables; we should never see the NotImplementedType. And at first
>> glance at the code, I don't think that flag *is* set for the type.
>>
>
> This particular return comes from ufunc_generic_call and I'm not sure what
> Python should or can do in this case. Thoughts?
>

It looks like Py_NotImplemented is a message to the python interpreter and
should not be visible to the user. So in this case I think we should raise
an exception.

PEP 208: New Coercion Model

How numeric coercion is done at the C level was significantly modified. This
will only affect the authors of C extensions to Python, allowing them more
flexibility in writing extension types that support numeric operations.

Extension types can now set the type flag Py_TPFLAGS_CHECKTYPES in their
PyTypeObject structure to indicate that they support the new coercion model.
In such extension types, the numeric slot functions can no longer assume
that they'll be passed two arguments of the same type; instead they may be
passed two arguments of differing types, and can then perform their own
internal coercion. If the slot function is passed a type it can't handle, it
can indicate the failure by returning a reference to the
Py_NotImplementedsingleton value. The numeric functions of the other
type will then be tried,
and perhaps they can handle the operation; if the other type also returns
Py_NotImplemented, then a TypeError will be raised. Numeric methods written
in Python can also return Py_NotImplemented, causing the interpreter to act
as if the method did not exist (perhaps raising a TypeError, perhaps trying
another object's numeric methods).

 Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080621/577b6ec1/attachment.html>

From stefan at sun.ac.za  Sat Jun 21 16:09:21 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sat, 21 Jun 2008 22:09:21 +0200
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <2219.65.102.174.35.1214074554.squirrel@webmail.uio.no>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<1e2af89e0806210932u744eea87x15efd3e00da530f3@mail.gmail.com>
	<2219.65.102.174.35.1214074554.squirrel@webmail.uio.no>
Message-ID: <9457e7c80806211309l3ef097c9h5dea6a2b84a47534@mail.gmail.com>

Hi Dag

2008/6/21 Dag Sverre Seljebotn <dagss at student.matnat.uio.no>:
> However, I could make it so that if you left out the type, it would be
> auto-detected. I.e.:
>
> cdef np.ndarray[2] arr = ...
> cdef np.float64 x = arr[3,4]

Would it not be possible for Cython to make the necessary C-API calls
to query the dimensions and type of the ndarray?  I don't really know
the full background here, so sorry if my comment is completely
off-base.

Eventually, I'd love for the Cython code to look very similar to
Python code.  Even if that is not technically feasible at the moment,
it should be the eventual target to have as little "markup" as
possible.

Regards
St?fan


From stefan at sun.ac.za  Sat Jun 21 16:49:58 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sat, 21 Jun 2008 22:49:58 +0200
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <3d375d730806201223q7965e15bof26a9365376e9210@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<3d375d730806191706h441912ebmba3ae981e32d4eb@mail.gmail.com>
	<db6b5ecc0806191718x5dfea216v8d2c94d331f4227a@mail.gmail.com>
	<3d375d730806191729h1d3d5953m586a5fb3f4fcacdc@mail.gmail.com>
	<db6b5ecc0806192118w25a6b530j4135c4e6cda90b8a@mail.gmail.com>
	<db6b5ecc0806201051s2b87de69k925b4d998c2f974d@mail.gmail.com>
	<485BF2D4.4080600@astraw.com>
	<db6b5ecc0806201125l42614bf2ga7d47f79c67cf753@mail.gmail.com>
	<1e2af89e0806201211g134c2c73m58b6583efaa6056b@mail.gmail.com>
	<3d375d730806201223q7965e15bof26a9365376e9210@mail.gmail.com>
Message-ID: <9457e7c80806211349n36609e15l8eb42525c463f752@mail.gmail.com>

2008/6/20 Robert Kern <robert.kern at gmail.com>:
>> Are they initializing different copies of the same thing?
>
> No. "import numpy" is essentially the same as the pure Python
> equivalent; it loads the module and puts it into the namespace.
> cnp.import_array() loads the module (or reuses the already loaded
> one), does not put it into any namespace, but then, most importantly,
> grabs the pointer to the table of function pointers and assigns it to
> the local void** variable. All of the numpy API is made available to
> third-party extension modules by #define macros which look up into
> this table.

Since "import numpy" already sets up the pointer table, is it then
necessary to use import_array?

St?fan


From matthew.brett at gmail.com  Sat Jun 21 17:25:19 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 21 Jun 2008 21:25:19 +0000
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <2219.65.102.174.35.1214074554.squirrel@webmail.uio.no>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<1e2af89e0806210932u744eea87x15efd3e00da530f3@mail.gmail.com>
	<2219.65.102.174.35.1214074554.squirrel@webmail.uio.no>
Message-ID: <1e2af89e0806211425q390a69asc1afc1bee821ef0@mail.gmail.com>

Hi,

> Well... the Cython compiler definitely needs to know about which type it
> is -- the array simply has a byte buffer, and one needs to know how to
> interpret that (how many bytes per element etc.).

It may be not practical, and, as I say, I haven't thought deeply
enough about this to offer this as anything but a silly question, but
would it not be possible to output C code switch statements that would
deal with (some of) the possible types np.array could contain?

Best,

Matthew


From charlesr.harris at gmail.com  Sat Jun 21 17:30:06 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 21 Jun 2008 15:30:06 -0600
Subject: [Numpy-discussion] NotImplementedType should be an exception
In-Reply-To: <485D0483.3050309@enthought.com>
References: <e06186140806202221n5ab6d587re987232dfe9dedcd@mail.gmail.com>
	<485D0483.3050309@enthought.com>
Message-ID: <e06186140806211430s5724f2a8y9d747289f257313@mail.gmail.com>

On Sat, Jun 21, 2008 at 7:39 AM, Travis E. Oliphant <oliphant at enthought.com>
wrote:

> Charles R Harris wrote:
> > Shouldn't this raise an NotImplementedError exception?
> >
> > In [7]: type(remainder(complex192(1), complex192(1)))
> > Out[7]: <type 'NotImplementedType'>
> I'm not sure if it is relevant in this case but we do need to keep in
> mind that Python uses the NotImplementedType as a signal to hand off the
> calculation to the other object in a binary operation.
>
> If we change this here we may affect that behavior, unintentionally.
>

It doesn't change the behavior of any of the ufuncs with array/array
arguments. I need to write up comprehensive tests for the combinations of
array/scalars and scalars,scalars, but there don't seem to be problems there
either. Can I also suggest that we not promote arrays of numeric types to
arrays of objects? Is there some way of preventing this?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080621/32c4b19b/attachment.html>

From garg1 at ualberta.ca  Sat Jun 21 18:05:56 2008
From: garg1 at ualberta.ca (Rahul Garg)
Date: Sat, 21 Jun 2008 16:05:56 -0600
Subject: [Numpy-discussion] parallel loop annotations
In-Reply-To: <20080618174349.myuvne1tco8c48cc@webmail.ualberta.ca>
References: <20080618094747.93p5popyoskgggok@webmail.ualberta.ca>
	<9457e7c80806181554h701bf5e5xf34863bdb6be14b9@mail.gmail.com>
	<20080618174349.myuvne1tco8c48cc@webmail.ualberta.ca>
Message-ID: <20080621160556.9fwrf5qqo0skw4ko@webmail.ualberta.ca>

The discussion has moved to :
http://wiki.cython.org/enhancements/parallel

Quoting Rahul Garg <garg1 at ualberta.ca>:

> Thanks for pointing them out. I will look into both of them. I like
> the "with" statement. From an implementation perspective, the "with"
> statement looks simpler since the Python parser seems to discard
> comments and I use the Python parser as front end for compiler.
>
> For other annotations, I have been using either decorators or strings.
> Decorators are used for type annotating functions. Strings just before
> class declarations are currently necessary for telling the compiler
> about fields of a class.
>
> Strings can get a little ugly. Its better to use language features
> whenever possible. I will look into ipython's notation.
>
> More comments welcome :)
>
> thanks,
> rahul
>
>
> Quoting St?fan van der Walt <stefan at sun.ac.za>:
>
>> 2008/6/18 Rahul Garg <garg1 at ualberta.ca>:
>>> I want to add a similar annotation to Python
>>> example usage :
>>>
>>> "pragma parallel for"
>>> for i in xrange(m): a[i] = b[i]*c[i]
>>>
>>> The interpreter ignores such strings and the code will of course
>>> execute serially but such strings can potentially be used by compilers.
>>> Any thoughts on this? What form of annotations would you like to see?
>>> Is it pythonic?
>>
>> Is the idea to have the annotations completely separated from the
>> executed source?  Otherwise, I'd just as well make it implicit:
>>
>> with parallel(i):
>>     for i in range(m):
>>         a[i] = b[i] * c[i]
>>
>> That is also roughly the form that Fernando implemented for IPython1,
>> and is very intuitive.
>>
>> What have you been using for other annotations so far?  Python itself
>> supports at least one other annotation:
>>
>> # encoding: utf-8
>>
>> so maybe
>>
>> # pragma: parallel for
>>
>> could work.
>>
>> Regards
>> St?fan
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>




From peridot.faceted at gmail.com  Sat Jun 21 18:08:43 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 21 Jun 2008 18:08:43 -0400
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <1834.65.102.174.34.1214031565.squirrel@webmail.uio.no>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<1834.65.102.174.34.1214031565.squirrel@webmail.uio.no>
Message-ID: <ce557a360806211508g32da759fx5039aa6b8917a782@mail.gmail.com>

2008/6/21 Dag Sverre Seljebotn <dagss at student.matnat.uio.no>:
> Dag wrote:
>> General feedback is welcome; in particular, I need more opinions about
>> what syntax people would like. We seem unable to find something that we
>> really like; this is the current best candidate (cdef is the way you
>> declare types on variables in Cython):
>>
>> cdef int i = 4, j = 6
>> cdef np.ndarray[np.float64, 2] arr = np.zeros((10, 10), dtype=np.float64)
>> arr[i, j] = 1
>> ...
>>
>
> Some more important points:
> - There will likely be a compiler flag (and perhaps per-block or
> per-variable pragmas) on whether bounds-checking is done or not)
> - It doesn't look like negative indices will be supported in this mode, as
> it adds another branch in potentially very tight loops. Opinions on this
> are welcome though. In safe mode, bounds checking will catch it, while in
> unsafe mode it will at best segfault and at worst corrupt data.
>
> The negative indices thing is potentially confusing to new users. Pex uses
> a different syntax (arr{i, j} for efficient array lookups) partly to make
> this fact very explicit. Thoughts?

I am very worried about the negative numbers issue. It's the sort of
thing that will readily lead to errors, and that produces a
significant difference between cython and python. I understand the
performance issues that motivate it, but cython really needs to be
easy to use or we might as well just use C. As I understand it, the
ultimate goal should be for users to be able to compile arbitrary
python code for tiny speed improvements, then add type annotations for
key variables to obtain large speed improvements. Forbidding negative
indices now prevents that goal.

My suggestion is this: allow negative indices, accepting the cost in
tight loops. (If bounds checking is enabled, the cost will be
negligible anyway.) Provide a #pragma allowing the user to assert that
a certain piece of code uses no negative indices. Ultimately, the
cython compiler will often be able to deduce than negative indices
will not be used within a particular loop and supply the pragma
itself, but in the meantime this solution allows people to use the
extremely-convenient negative indices without causing mysterious
problems, and it sets things up for the future. (Other compiler-based
cleverness can deal with the most common case by converting explicit
negative indices.)



Anne


From dagss at student.matnat.uio.no  Sat Jun 21 18:32:23 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 22 Jun 2008 00:32:23 +0200 (CEST)
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <ce557a360806211508g32da759fx5039aa6b8917a782@mail.gmail.com>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<1834.65.102.174.34.1214031565.squirrel@webmail.uio.no>
	<ce557a360806211508g32da759fx5039aa6b8917a782@mail.gmail.com>
Message-ID: <2987.65.102.174.35.1214087543.squirrel@webmail.uio.no>

Anne Archibald wrote:
> 2008/6/21 Dag Sverre Seljebotn <dagss at student.matnat.uio.no>:
>> Dag wrote:
>>> General feedback is welcome; in particular, I need more opinions about
>>> what syntax people would like. We seem unable to find something that we
>>> really like; this is the current best candidate (cdef is the way you
>>> declare types on variables in Cython):
>>>
>>> cdef int i = 4, j = 6
>>> cdef np.ndarray[np.float64, 2] arr = np.zeros((10, 10),
>>> dtype=np.float64)
>>> arr[i, j] = 1
>>> ...
>>>
>>
>> Some more important points:
>> - There will likely be a compiler flag (and perhaps per-block or
>> per-variable pragmas) on whether bounds-checking is done or not)
>> - It doesn't look like negative indices will be supported in this mode,
>> as
>> it adds another branch in potentially very tight loops. Opinions on this
>> are welcome though. In safe mode, bounds checking will catch it, while
>> in
>> unsafe mode it will at best segfault and at worst corrupt data.
>>
>> The negative indices thing is potentially confusing to new users. Pex
>> uses
>> a different syntax (arr{i, j} for efficient array lookups) partly to
>> make
>> this fact very explicit. Thoughts?
>
> I am very worried about the negative numbers issue. It's the sort of
> thing that will readily lead to errors, and that produces a
> significant difference between cython and python. I understand the
> performance issues that motivate it, but cython really needs to be
> easy to use or we might as well just use C. As I understand it, the
> ultimate goal should be for users to be able to compile arbitrary
> python code for tiny speed improvements, then add type annotations for
> key variables to obtain large speed improvements. Forbidding negative
> indices now prevents that goal.
>
> My suggestion is this: allow negative indices, accepting the cost in
> tight loops. (If bounds checking is enabled, the cost will be
> negligible anyway.) Provide a #pragma allowing the user to assert that
> a certain piece of code uses no negative indices. Ultimately, the
> cython compiler will often be able to deduce than negative indices
> will not be used within a particular loop and supply the pragma
> itself, but in the meantime this solution allows people to use the
> extremely-convenient negative indices without causing mysterious
> problems, and it sets things up for the future. (Other compiler-based
> cleverness can deal with the most common case by converting explicit
> negative indices.)

Thank you for supplying such useful feedback! It is likely that you have
convinced me.

Dag Sverre



From garg1 at ualberta.ca  Sat Jun 21 18:35:01 2008
From: garg1 at ualberta.ca (Rahul Garg)
Date: Sat, 21 Jun 2008 16:35:01 -0600
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <ce557a360806211508g32da759fx5039aa6b8917a782@mail.gmail.com>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<1834.65.102.174.34.1214031565.squirrel@webmail.uio.no>
	<ce557a360806211508g32da759fx5039aa6b8917a782@mail.gmail.com>
Message-ID: <20080621163501.0in54otjr4c4occ4@webmail.ualberta.ca>

> I am very worried about the negative numbers issue. It's the sort of
> thing that will readily lead to errors, and that produces a
> significant difference between cython and python. I understand the
> performance issues that motivate it, but cython really needs to be
> easy to use or we might as well just use C. As I understand it, the
> ultimate goal should be for users to be able to compile arbitrary
> python code for tiny speed improvements, then add type annotations for
> key variables to obtain large speed improvements. Forbidding negative
> indices now prevents that goal.

Since this also applies to my compiler unPython, I am adding this note.
In unPython, currently negative indices are not supported but I will  
support them in the next release. There will be a global compiler  
command line switch that will turn-on or turn-off the negative  
indices. Compiler will also try to infer if the indices are positive  
in a given loop.

thanks,
rahul



From dagss at student.matnat.uio.no  Sat Jun 21 18:44:36 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 22 Jun 2008 00:44:36 +0200 (CEST)
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <1e2af89e0806211425q390a69asc1afc1bee821ef0@mail.gmail.com>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<1e2af89e0806210932u744eea87x15efd3e00da530f3@mail.gmail.com>
	<2219.65.102.174.35.1214074554.squirrel@webmail.uio.no>
	<1e2af89e0806211425q390a69asc1afc1bee821ef0@mail.gmail.com>
Message-ID: <3070.65.102.174.35.1214088276.squirrel@webmail.uio.no>

Matthew Brett wrote:
> Hi,
>
>> Well... the Cython compiler definitely needs to know about which type it
>> is -- the array simply has a byte buffer, and one needs to know how to
>> interpret that (how many bytes per element etc.).
>
> It may be not practical, and, as I say, I haven't thought deeply
> enough about this to offer this as anything but a silly question, but
> would it not be possible to output C code switch statements that would
> deal with (some of) the possible types np.array could contain?

In the way you imagine it I don't think it would be a good solution. It is
possible that Cython might grow some support for type-generalized
programming a la C++ templates (though in a more Pythonic fashion); and in
that case, you could use those for ndarray types as well.

The feature of compiling code for multiple types is somewhat orthogonal to
ndarray support; better treat them seperately and take one at the time.
There's lots of non-NumPy code that would benefit for such a feature as
well.

Dag Sverre



From matthew.brett at gmail.com  Sat Jun 21 18:55:51 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 21 Jun 2008 22:55:51 +0000
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <3070.65.102.174.35.1214088276.squirrel@webmail.uio.no>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<1e2af89e0806210932u744eea87x15efd3e00da530f3@mail.gmail.com>
	<2219.65.102.174.35.1214074554.squirrel@webmail.uio.no>
	<1e2af89e0806211425q390a69asc1afc1bee821ef0@mail.gmail.com>
	<3070.65.102.174.35.1214088276.squirrel@webmail.uio.no>
Message-ID: <1e2af89e0806211555p3630357h78bab2d096fd7da0@mail.gmail.com>

Hi,

> The feature of compiling code for multiple types is somewhat orthogonal to
> ndarray support; better treat them seperately and take one at the time.

Well, it's relevant to numpy because if you want to implement - for
example - a numpy sort, then you've got to deal with an unspecified
number of dimensions, and one of a range of specified types, otherwise
you will end up copying, casting, rejecting types and so on...

Best,

Matthew


From robert.kern at gmail.com  Sat Jun 21 19:06:13 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 21 Jun 2008 18:06:13 -0500
Subject: [Numpy-discussion] Updating cython directory to pxd usage:
	objections?
In-Reply-To: <9457e7c80806211349n36609e15l8eb42525c463f752@mail.gmail.com>
References: <db6b5ecc0806182148q604e4ab3q22eb099c917eab36@mail.gmail.com>
	<db6b5ecc0806191718x5dfea216v8d2c94d331f4227a@mail.gmail.com>
	<3d375d730806191729h1d3d5953m586a5fb3f4fcacdc@mail.gmail.com>
	<db6b5ecc0806192118w25a6b530j4135c4e6cda90b8a@mail.gmail.com>
	<db6b5ecc0806201051s2b87de69k925b4d998c2f974d@mail.gmail.com>
	<485BF2D4.4080600@astraw.com>
	<db6b5ecc0806201125l42614bf2ga7d47f79c67cf753@mail.gmail.com>
	<1e2af89e0806201211g134c2c73m58b6583efaa6056b@mail.gmail.com>
	<3d375d730806201223q7965e15bof26a9365376e9210@mail.gmail.com>
	<9457e7c80806211349n36609e15l8eb42525c463f752@mail.gmail.com>
Message-ID: <3d375d730806211606qdf371e8i5a98d3fc45fe9fcc@mail.gmail.com>

On Sat, Jun 21, 2008 at 15:49, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2008/6/20 Robert Kern <robert.kern at gmail.com>:
>>> Are they initializing different copies of the same thing?
>>
>> No. "import numpy" is essentially the same as the pure Python
>> equivalent; it loads the module and puts it into the namespace.
>> cnp.import_array() loads the module (or reuses the already loaded
>> one), does not put it into any namespace, but then, most importantly,
>> grabs the pointer to the table of function pointers and assigns it to
>> the local void** variable. All of the numpy API is made available to
>> third-party extension modules by #define macros which look up into
>> this table.
>
> Since "import numpy" already sets up the pointer table, is it then
> necessary to use import_array?

None of the third party extension modules have a reference to it.
import_array() grabs the pointer to the start of the pointer table and
assigns it to a void** variable *local* to the extension module. Each
extension module has its own and needs it to be set up via
import_array(). "import numpy" sets up the table, but does not expose
it. import_array() does not set up the table itself, but rather
exposes it to the extension module.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco

From robert.kern at gmail.com  Sat Jun 21 19:10:01 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 21 Jun 2008 18:10:01 -0500
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <ce557a360806211508g32da759fx5039aa6b8917a782@mail.gmail.com>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<1834.65.102.174.34.1214031565.squirrel@webmail.uio.no>
	<ce557a360806211508g32da759fx5039aa6b8917a782@mail.gmail.com>
Message-ID: <3d375d730806211610i14a662e0i6ca7668e75c484f7@mail.gmail.com>

On Sat, Jun 21, 2008 at 17:08, Anne Archibald <peridot.faceted at gmail.com> wrote:
> My suggestion is this: allow negative indices, accepting the cost in
> tight loops. (If bounds checking is enabled, the cost will be
> negligible anyway.) Provide a #pragma allowing the user to assert that
> a certain piece of code uses no negative indices.

Instead of a #pragma, you could rely on the type of the index. If it
is unsigned, you can do the fast path; if it is signed, you need to
check for and handle potential negatives.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From peridot.faceted at gmail.com  Sat Jun 21 19:45:37 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 21 Jun 2008 19:45:37 -0400
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <3d375d730806211610i14a662e0i6ca7668e75c484f7@mail.gmail.com>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<1834.65.102.174.34.1214031565.squirrel@webmail.uio.no>
	<ce557a360806211508g32da759fx5039aa6b8917a782@mail.gmail.com>
	<3d375d730806211610i14a662e0i6ca7668e75c484f7@mail.gmail.com>
Message-ID: <ce557a360806211645r19127881l39e427d4ed15f561@mail.gmail.com>

2008/6/21 Robert Kern <robert.kern at gmail.com>:
> On Sat, Jun 21, 2008 at 17:08, Anne Archibald <peridot.faceted at gmail.com> wrote:
>> My suggestion is this: allow negative indices, accepting the cost in
>> tight loops. (If bounds checking is enabled, the cost will be
>> negligible anyway.) Provide a #pragma allowing the user to assert that
>> a certain piece of code uses no negative indices.
>
> Instead of a #pragma, you could rely on the type of the index. If it
> is unsigned, you can do the fast path; if it is signed, you need to
> check for and handle potential negatives.

Cute! And then it's easy to make "for i in range(n)" produce unsigned
results with no effort on the part of the user.

Anne


From robert.kern at gmail.com  Sat Jun 21 19:51:02 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 21 Jun 2008 18:51:02 -0500
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <ce557a360806211645r19127881l39e427d4ed15f561@mail.gmail.com>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<1834.65.102.174.34.1214031565.squirrel@webmail.uio.no>
	<ce557a360806211508g32da759fx5039aa6b8917a782@mail.gmail.com>
	<3d375d730806211610i14a662e0i6ca7668e75c484f7@mail.gmail.com>
	<ce557a360806211645r19127881l39e427d4ed15f561@mail.gmail.com>
Message-ID: <3d375d730806211651l9ce9403n4772f3e022a4d54@mail.gmail.com>

On Sat, Jun 21, 2008 at 18:45, Anne Archibald <peridot.faceted at gmail.com> wrote:
> 2008/6/21 Robert Kern <robert.kern at gmail.com>:
>> On Sat, Jun 21, 2008 at 17:08, Anne Archibald <peridot.faceted at gmail.com> wrote:
>>> My suggestion is this: allow negative indices, accepting the cost in
>>> tight loops. (If bounds checking is enabled, the cost will be
>>> negligible anyway.) Provide a #pragma allowing the user to assert that
>>> a certain piece of code uses no negative indices.
>>
>> Instead of a #pragma, you could rely on the type of the index. If it
>> is unsigned, you can do the fast path; if it is signed, you need to
>> check for and handle potential negatives.
>
> Cute! And then it's easy to make "for i in range(n)" produce unsigned
> results with no effort on the part of the user.

Not entirely. You still need to declare "cdef unsigned int i". Cython
does not translate the for loop into fast C unless if the variable has
been declared.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From charlesr.harris at gmail.com  Sun Jun 22 00:25:22 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 21 Jun 2008 22:25:22 -0600
Subject: [Numpy-discussion] Test framework changes
In-Reply-To: <1d36917a0806201722p2b2ed4f4y613435efc42a6fba@mail.gmail.com>
References: <1d36917a0806201401y65187588s5bae0f6c79061269@mail.gmail.com>
	<3d375d730806201435x23d14c1fr696c82a417700bb@mail.gmail.com>
	<1d36917a0806201504x3b7dd98aqae05ffa2f5f4a82f@mail.gmail.com>
	<db6b5ecc0806201622q331e8e84m8838f609b3f913bb@mail.gmail.com>
	<1d36917a0806201722p2b2ed4f4y613435efc42a6fba@mail.gmail.com>
Message-ID: <e06186140806212125i7a1a2ba8t6a904221507b6189@mail.gmail.com>

On Fri, Jun 20, 2008 at 6:22 PM, Alan McIntyre <alan.mcintyre at gmail.com>
wrote:

> On Fri, Jun 20, 2008 at 7:22 PM, Fernando Perez <fperez.net at gmail.com>
> wrote:
> > It may be worth bringing it up wtih the nose guys here:
> >
> > http://lists.idyll.org/listinfo/testing-in-python
> >
> > The nose author seems very responsive, and Titus Brown is on the list
> > and cares a lot about numpy/scipy, and he may offer suggestions as
> > well.
>

Speaking of nose, I discovered why it couldn't find test_linalg.py; there
was a working copy in the tests directory named test_linalg.pure.py that got
copied into the numpy install and the name apparently confused nose. Moral:
don't leave working files in the test directories.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080621/dcc64533/attachment.html>

From alan.mcintyre at gmail.com  Sun Jun 22 01:03:35 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Sun, 22 Jun 2008 01:03:35 -0400
Subject: [Numpy-discussion] Test framework changes
In-Reply-To: <1d36917a0806201401y65187588s5bae0f6c79061269@mail.gmail.com>
References: <1d36917a0806201401y65187588s5bae0f6c79061269@mail.gmail.com>
Message-ID: <1d36917a0806212203w2f74748fpa3c2788964643621@mail.gmail.com>

These changes are now checked in, with the changes Robert suggested:
numpy.test() returns a TextTestResult object again, and coverage (if
enabled) is limited to the package from which test is called.

The tests still fail because the buildbots don't have nose installed, though.

On Fri, Jun 20, 2008 at 5:01 PM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> Hi all,
>
> Just wanted to get feedback about the following changes before I make
> them.  Please speak up if any of this seems objectionable to you.
>
> - The old test framework classes will be restored, but will not be
> used anywhere in NumPy's tests.  If your old tests still don't work
> with the restored classes, please let me know and I'll fix it.
>
> - Add a function to numpy.testing to execute a module's tests via the
> "if __name__ == '__main__'" hook.  It takes one optional argument, the
> name of the file to run (if not present, it uses the same method as
> NoseTester to look it up from the stack).  The intent is to make the
> boilerplate as simple as possible.
>
>  if __name__ == '__main__':
>      run_module_suite()
>
>  If somebody has a suggestion for a better name, I'd love to hear it.
> I didn't want "test" in the name, because then I have to explicitly
> tell nose to ignore it when it's looking for test functions.
>
> - Remove numpy/testing/pkgtester.py since it now just has one line of
> code that imports NoseTester (as Tester) from nosetester.py (it used
> to create a null tester if nose wasn't present, but this was removed
> by porting of r4424 from SciPy).  NoseTester will still be made
> available as numpy.testing.Tester.
>
> - numpy.test (and all the other test functions in subpackages) will
> take the following positional arguments: label, verbose, extra_argv,
> doctests, coverage (the same arguments as SciPy.test except for the
> new coverage option).  The old arguments can be passed in as keyword
> arguments (which seems to be how they were passed in in all the
> examples I could find), and they will be emulated as much as possible
> by the new test suite, but a deprecation warning will be issued.
>
> - numpy.test now returns an object with a wasSuccessful method; under
> the old test framework it returned a unittest._TextTestResult, but
> since nose.run only returns success/failure, I'm just wrapping this
> result in a dummy object to match the old behavior until I can figure
> out how to get a real _TextTestResult from nose.
>
> (The two changes to numpy.test should allow the buildbots to run the
> tests properly again)
>
> Thanks,
> Alan
>


From lists at informa.tiker.net  Sun Jun 22 11:02:24 2008
From: lists at informa.tiker.net (Andreas =?utf-8?q?Kl=C3=B6ckner?=)
Date: Sun, 22 Jun 2008 11:02:24 -0400
Subject: [Numpy-discussion] ANN: PyCuda
Message-ID: <200806221102.29819.lists@informa.tiker.net>

Hi all,

I am happy to announce the availability of PyCuda [1,8], which is a 
value-added Python wrapper around Nvidia's CUDA [2] GPU Computation 
framework. In the presence of other wrapping modules [3,4], why would you 
want to use PyCuda?

* It's designed to work and interact with numpy.

* RAII, [5] i.e. object cleanup is tied to lifetime of objects. This idiom 
makes it much easier to write correct, leak- and crash-free code. PyCuda 
knows about liftime dependencies, too, so (for example) it won?t detach from 
a context before all memory allocated in it is also freed.

* Convenience. Abstractions like pycuda.driver.SourceModule [6] and 
pycuda.gpuarray.GPUArray [7] make CUDA programming even more convenient than 
with Nvidia?s C-based runtime.

* Completeness. PyCuda puts the full power of CUDA?s driver API at your 
disposal, if you wish.

* Automatic Error Checking. All CUDA errors are automatically translated into 
Python exceptions.

* Speed. PyCuda?s base layer is written in C++, so all the niceties above are 
virtually free.

* Helpful documentation [8] with plenty of examples.

If you run into any issues using the code, don't hesitate to post here or get 
in touch.

Andreas

[1] http://mathema.tician.de/software/pycuda
[2] http://nvidia.com/cuda
[3] http://code.google.com/p/pystream/
[4] ftp://ftp.graviscom.com/pub/code/python-cuda
[5] http://en.wikipedia.org/wiki/Resource_Acquisition_Is_Initialization
[6] http://tiker.net/doc/pycuda/driver.html#pycuda.driver.SourceModule
[7] http://tiker.net/doc/pycuda/array.html#pycuda.gpuarray.GPUArray
[8] http://tiker.net/doc/pycuda <-- click here!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080622/6e94d75f/attachment.sig>

From bioinformed at gmail.com  Sun Jun 22 16:04:15 2008
From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>)
Date: Sun, 22 Jun 2008 16:04:15 -0400
Subject: [Numpy-discussion] ANN: PyCuda
In-Reply-To: <200806221558.59997.inform@tiker.net>
References: <200806221102.29819.lists@informa.tiker.net>
	<2e1434c10806221224k636a7390v78374638b93b315c@mail.gmail.com>
	<200806221558.59997.inform@tiker.net>
Message-ID: <2e1434c10806221304r43fc86c4ub926bff94ec8789@mail.gmail.com>

On Sun, Jun 22, 2008 at 3:58 PM, Andreas Kl?ckner <inform at tiker.net> wrote:

> PyCuda is based on the driver API. CUBLAS uses the high-level API. Once
> *can*
> violate this rule without crashing immediately. But sketchy stuff does
> happen. Instead, for BLAS-1 operations, PyCuda comes with a class called
> GPUArray that essentially reimplements that part of CUBLAS.
>


Thanks for the clarification.  That makes perfect sense.  Do you have any
feelings on the relative performance of GPUArray versus CUBLAS?


>
> PyUblas used to be a dependency of PyCuda, but isn't any more. Where did
> you
> see that reference? It should probably be removed.



The first  part of install.rst still says: "This tutorial will walk you
through the process of building PyUblas."

Thanks,
-Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080622/bce22174/attachment.html>

From lists at informa.tiker.net  Sun Jun 22 16:21:53 2008
From: lists at informa.tiker.net (Andreas =?utf-8?q?Kl=C3=B6ckner?=)
Date: Sun, 22 Jun 2008 16:21:53 -0400
Subject: [Numpy-discussion] ANN: PyCuda
In-Reply-To: <2e1434c10806221304r43fc86c4ub926bff94ec8789@mail.gmail.com>
References: <200806221102.29819.lists@informa.tiker.net>
	<200806221558.59997.inform@tiker.net>
	<2e1434c10806221304r43fc86c4ub926bff94ec8789@mail.gmail.com>
Message-ID: <200806221621.54451.lists@informa.tiker.net>

On Sonntag 22 Juni 2008, Kevin Jacobs <jacobs at bioinformed.com> wrote:
> Thanks for the clarification.  That makes perfect sense.  Do you have any
> feelings on the relative performance of GPUArray versus CUBLAS?

Same. If you check out the past version of PyCuda that still has CUBLAS, there 
are files test/test_{cublas,gpuarray}_speed.py. In fact, since CUBLAS does 
not implement three-operand "z = x + y", it requires an extra copy that 
GPUArray can avoid. If you're into "lies, damned lies and benchmarks", you 
could say that GPUArray is actually twice as fast.  :)

> The first  part of install.rst still says: "This tutorial will walk you
> through the process of building PyUblas."

Oops. Thanks. Fixed.

Andreas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080622/140b2e1b/attachment.sig>

From charlesr.harris at gmail.com  Sun Jun 22 18:46:35 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 22 Jun 2008 16:46:35 -0600
Subject: [Numpy-discussion] Travis: this looks like a bug in ufuncobject.c
Message-ID: <e06186140806221546k77be3867u78197c6de5e48d43@mail.gmail.com>

select_types(PyUFuncObject *self, int *arg_types,
             PyUFuncGenericFunction *function, void **data,
             PyArray_SCALARKIND *scalars,
             PyObject *typetup)
{
    int i, j;
    char start_type;
    int userdef=-1;
    int userdef_ind=-1;

    if (self->userloops) {
        for(i=0; i<self->nin; i++) {
            if (PyTypeNum_ISUSERDEF(arg_types[i])) {
                userdef = arg_types[i];
        userdef_ind = i;
                break;
            }
        }
    }

    if (typetup != NULL)
        return extract_specified_loop(self, arg_types, function, data,
                                      typetup, userdef);

    if (userdef > 0) {
       ^^^^^^^^^^^^
        PyObject *key, *obj;
        int ret=-1;
        obj = NULL;
    /* Look through all the registered loops for all the user-defined
       types to find a match.
     */
    while (ret == -1) {
        if (userdef_ind >= self->nin) break;
        userdef = arg_types[userdef_ind++];
        if (!(PyTypeNum_ISUSERDEF(userdef))) continue;
        key = PyInt_FromLong((long) userdef);
        if (key == NULL) return -1;
        obj = PyDict_GetItem(self->userloops, key);
        Py_DECREF(key);
        if (obj == NULL) continue;
        /* extract the correct function
           data and argtypes for this user-defined type.
        */
        ret = _find_matching_userloop(obj, arg_types, scalars,
                      function, data, self->nargs,
                      self->nin);
    }
    if (ret == 0) return ret;
    PyErr_SetString(PyExc_TypeError, _types_msg);
    return ret;
    }
...

It looks like the test should be userdef >= 0.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080622/3230f01d/attachment.html>

From alan.mcintyre at gmail.com  Sun Jun 22 19:01:02 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Sun, 22 Jun 2008 19:01:02 -0400
Subject: [Numpy-discussion] Using numpy.testing for SciPy
Message-ID: <1d36917a0806221601p3b4f5117nd35fd0b7ea79ebc1@mail.gmail.com>

For what it's worth, I tried deleting scipy/testing, changed all the
scipy.testing references to numpy.testing, and it would appear that
the NumPy test setup is able to run all the SciPy tests just fine
(which shouldn't be too surprising, since that's where I stole it from
in the first place).  So whenever it's appropriate, it should be easy
to switch SciPy over to use the NumPy test code.

It does seem that the coverage option doesn't work correctly, though;
it reports that no SciPy code is being executed.  I'll have a look
into that this week.

Alan


From dagss at student.matnat.uio.no  Sun Jun 22 19:14:51 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 23 Jun 2008 01:14:51 +0200 (CEST)
Subject: [Numpy-discussion] On my Cython/NumPy project
In-Reply-To: <1e2af89e0806211555p3630357h78bab2d096fd7da0@mail.gmail.com>
References: <1776.65.102.174.34.1214031027.squirrel@webmail.uio.no>
	<1e2af89e0806210932u744eea87x15efd3e00da530f3@mail.gmail.com>
	<2219.65.102.174.35.1214074554.squirrel@webmail.uio.no>
	<1e2af89e0806211425q390a69asc1afc1bee821ef0@mail.gmail.com>
	<3070.65.102.174.35.1214088276.squirrel@webmail.uio.no>
	<1e2af89e0806211555p3630357h78bab2d096fd7da0@mail.gmail.com>
Message-ID: <1758.67.170.117.81.1214176491.squirrel@webmail.uio.no>

Matthew Brett wrote:
> Hi,
>
>> The feature of compiling code for multiple types is somewhat orthogonal
>> to
>> ndarray support; better treat them seperately and take one at the time.
>
> Well, it's relevant to numpy because if you want to implement - for
> example - a numpy sort, then you've got to deal with an unspecified
> number of dimensions, and one of a range of specified types, otherwise
> you will end up copying, casting, rejecting types and so on...

Thanks for the feedback.

Sure; I didn't mean to imply that it was irrelevant, indeed I think it
would be very useful to NumPy users. I just mean to say that it is
orthogonal -- it is also important, but should be treated as a seperate
feature that is independent of NumPy support as such. To give you a flavor
of why this is a nontrivial issue, consider a function with four different
numpy arrays. Then you would need to create approx. 15**4 different
versions of it, one for each combination of types -- not feasible at all.
I.e., you need some way of specifying that "these two arrays have the same
datatype", and so on, i.e. some kind of generalized/template programming.
I'd rather focus on the easy case for now; not precluding more features
later.

When it comes to not knowing the number of dimensions, the right way to go
about that would be to support NumPy dimension-neutral array iterators in
a nice way. I have  a lot of thoughts about that too, but it's lower on
the priority list and I'll wait with discussing it until the more direct
case is solved (as that is the one new NumPy/Cython users will miss most).

On the negative indices side; I really like the unsigned int idea. The
problem with "range" is that it can, at least in theory, grow larger than
MAX_INT, and so we are cautious about automatically inferring that the
iterator variable should be unsigned int (Python ints can be arbitrarily
large). Of course, one could compile two versions of each loop and have an
if-statement...again, I'll probably specifically drop the negative test
for explicitly declared unsigned int for now, while having "range" imply
unsigned int will be dealt with together with more general type inference
which Cython developers are also thinking about (though whether we'll have
the developer resources for it is another issue).

Dag Svere






From gael.varoquaux at normalesup.org  Sun Jun 22 20:37:30 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 23 Jun 2008 02:37:30 +0200
Subject: [Numpy-discussion] Surprising performance tweak in Cython
Message-ID: <20080623003730.GF15908@phare.normalesup.org>

I tried tweak my Cython code for performance by manually inlining a small
function, and ended up with a less performant code. I must confess I
don't really understand what is going on here. If somebody has an
explaination, I'd be delighted. The code follows.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
from numpy import zeros

# Make sure numpy is initialized.
include "c_numpy.pxd"

##############################################################################
cdef int inner_loop(float c_x, float c_y):
    cdef float x, y, x_buffer
    x = 0; y = 0
    cdef int i
    for i in range(50):
        x_buffer = x*x - y*y + c_x
        y = 2*x*y + c_y
        x = x_buffer
        if (x*x + x*y > 100):
            return 50 - i

def do_Mandelbrot_cython():
    cdef ndarray threshold_time 
    threshold_time = zeros((500, 500))
    cdef double *tp
    cdef float c_x, c_y
    cdef int i, j
    c_x = -1.5
    tp = <double*>threshold_time.data
    for i in range(500):
        c_y = -1
        for j in range(500):
            tp += 1
            c_y += 0.004
            tp[0] = inner_loop(c_x, c_y)
        c_x += 0.004
    return threshold_time


def do_Mandelbrot_cython2():
    cdef ndarray threshold_time
    threshold_time = zeros((500, 500))
    cdef double *tp
    tp = <double*>threshold_time.data
    cdef float x, y, xbuffer, c_x, c_y
    cdef int i, j, n 
    c_x = -1.5
    for i in range(500):
        c_y = -1
        for j in range(500):
            tp += 1
            c_y += 0.004
            x = 0; y = 0
            for n in range(50):
                x_buffer = x*x - y*y + c_x
                y = 2*x*y + c_y
                x = x_buffer
                if (x*x + y*y > 100):
                    tp[0] = 50 -n
                    break
        c_x += 0.004
    return threshold_time
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

And the timing I get are:

In [2]: %timeit C.do_Mandelbrot_cython2()
10 loops, best of 3: 342 ms per loop

In [3]: %timeit C.do_Mandelbrot_cython()
10 loops, best of 3: 126 ms per loop

Cheers,

Ga?l


From efiring at hawaii.edu  Sun Jun 22 23:02:38 2008
From: efiring at hawaii.edu (Eric Firing)
Date: Sun, 22 Jun 2008 17:02:38 -1000
Subject: [Numpy-discussion] Surprising performance tweak in Cython
In-Reply-To: <20080623003730.GF15908@phare.normalesup.org>
References: <20080623003730.GF15908@phare.normalesup.org>
Message-ID: <485F124E.3000104@hawaii.edu>

Gael Varoquaux wrote:
> I tried tweak my Cython code for performance by manually inlining a small
> function, and ended up with a less performant code. I must confess I
> don't really understand what is going on here. If somebody has an
> explaination, I'd be delighted. The code follows.
> 
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> from numpy import zeros
> 
> # Make sure numpy is initialized.
> include "c_numpy.pxd"
> 
> ##############################################################################
> cdef int inner_loop(float c_x, float c_y):
>     cdef float x, y, x_buffer
>     x = 0; y = 0
>     cdef int i
>     for i in range(50):
>         x_buffer = x*x - y*y + c_x
>         y = 2*x*y + c_y
>         x = x_buffer
>         if (x*x + x*y > 100):

The line above looks like a typo, and does not match your inline 
version.  Could that be making the difference?

Eric


>             return 50 - i
> 
> def do_Mandelbrot_cython():
>     cdef ndarray threshold_time 
>     threshold_time = zeros((500, 500))
>     cdef double *tp
>     cdef float c_x, c_y
>     cdef int i, j
>     c_x = -1.5
>     tp = <double*>threshold_time.data
>     for i in range(500):
>         c_y = -1
>         for j in range(500):
>             tp += 1
>             c_y += 0.004
>             tp[0] = inner_loop(c_x, c_y)
>         c_x += 0.004
>     return threshold_time
> 
> 
> def do_Mandelbrot_cython2():
>     cdef ndarray threshold_time
>     threshold_time = zeros((500, 500))
>     cdef double *tp
>     tp = <double*>threshold_time.data
>     cdef float x, y, xbuffer, c_x, c_y
>     cdef int i, j, n 
>     c_x = -1.5
>     for i in range(500):
>         c_y = -1
>         for j in range(500):
>             tp += 1
>             c_y += 0.004
>             x = 0; y = 0
>             for n in range(50):
>                 x_buffer = x*x - y*y + c_x
>                 y = 2*x*y + c_y
>                 x = x_buffer
>                 if (x*x + y*y > 100):
>                     tp[0] = 50 -n
>                     break
>         c_x += 0.004
>     return threshold_time
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 
> And the timing I get are:
> 
> In [2]: %timeit C.do_Mandelbrot_cython2()
> 10 loops, best of 3: 342 ms per loop
> 
> In [3]: %timeit C.do_Mandelbrot_cython()
> 10 loops, best of 3: 126 ms per loop
> 
> Cheers,
> 
> Ga?l
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From gael.varoquaux at normalesup.org  Sun Jun 22 23:10:28 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 23 Jun 2008 05:10:28 +0200
Subject: [Numpy-discussion] Surprising performance tweak in Cython
In-Reply-To: <485F124E.3000104@hawaii.edu>
References: <20080623003730.GF15908@phare.normalesup.org>
	<485F124E.3000104@hawaii.edu>
Message-ID: <20080623031028.GI15908@phare.normalesup.org>

On Sun, Jun 22, 2008 at 05:02:38PM -1000, Eric Firing wrote:
> The line above looks like a typo, and does not match your inline 
> version.  Could that be making the difference?

Thank you Eric! It was indeed a typo (the plot is nicer with the typo,
surprising, but that also explains why it didn't look exactly like the
weave version).

That doesn't explain the performance, however, as with the typo
corrected, the non-inlined version goes a bit faster: 110ms compared to
126ms. weave.inline is 62.5ms, surprisingly. Trying to understand the
difference between weave.inline and cython lead me to inlining, and to
the surprising result.

Thanks for your input,

Ga?l


From efiring at hawaii.edu  Mon Jun 23 00:39:21 2008
From: efiring at hawaii.edu (Eric Firing)
Date: Sun, 22 Jun 2008 18:39:21 -1000
Subject: [Numpy-discussion] Surprising performance tweak in Cython
In-Reply-To: <20080623031028.GI15908@phare.normalesup.org>
References: <20080623003730.GF15908@phare.normalesup.org>
	<485F124E.3000104@hawaii.edu>
	<20080623031028.GI15908@phare.normalesup.org>
Message-ID: <485F28F9.3040902@hawaii.edu>

Gael,

Another typo is the culprit:

In [2]:timeit do_Mandelbrot_cython()
10 loops, best of 3: 53.8 ms per loop

In [3]:timeit do_Mandelbrot_cython2()
10 loops, best of 3: 54 ms per loop

This is after I put the underscore in the x_buffer declaration in 
do_Mandlebrot_cython2.  As it was, with no underscore, x_buffer was 
being treated as a python double.

With the typo fixed, the versions are virtually identical, with perhaps 
a slight edge for the non-inlined version.  This suggests to me that the 
compiler is smarter than we are.

Eric




Gael Varoquaux wrote:
> On Sun, Jun 22, 2008 at 05:02:38PM -1000, Eric Firing wrote:
>> The line above looks like a typo, and does not match your inline 
>> version.  Could that be making the difference?
> 
> Thank you Eric! It was indeed a typo (the plot is nicer with the typo,
> surprising, but that also explains why it didn't look exactly like the
> weave version).
> 
> That doesn't explain the performance, however, as with the typo
> corrected, the non-inlined version goes a bit faster: 110ms compared to
> 126ms. weave.inline is 62.5ms, surprisingly. Trying to understand the
> difference between weave.inline and cython lead me to inlining, and to
> the surprising result.
> 
> Thanks for your input,
> 
> Ga?l
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From gael.varoquaux at normalesup.org  Mon Jun 23 00:55:16 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 23 Jun 2008 06:55:16 +0200
Subject: [Numpy-discussion] Surprising performance tweak in Cython
In-Reply-To: <485F28F9.3040902@hawaii.edu>
References: <20080623003730.GF15908@phare.normalesup.org>
	<485F124E.3000104@hawaii.edu>
	<20080623031028.GI15908@phare.normalesup.org>
	<485F28F9.3040902@hawaii.edu>
Message-ID: <20080623045516.GK15908@phare.normalesup.org>

On Sun, Jun 22, 2008 at 06:39:21PM -1000, Eric Firing wrote:
> Another typo is the culprit:

> In [2]:timeit do_Mandelbrot_cython()
> 10 loops, best of 3: 53.8 ms per loop

> In [3]:timeit do_Mandelbrot_cython2()
> 10 loops, best of 3: 54 ms per loop

> This is after I put the underscore in the x_buffer declaration in 
> do_Mandlebrot_cython2.  As it was, with no underscore, x_buffer was 
> being treated as a python double.

Indeed, I was looking for a forgotten type definition, but I forgotten to
look for typos in the type definitions. As a results, the cython code is
now of comparable speed to the weave.inline: cython=59ms and weave=62.5ms
on my box.

Thanks a lot, next time I'll now where to look.

Ga?l


From peridot.faceted at gmail.com  Mon Jun 23 02:03:06 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 23 Jun 2008 02:03:06 -0400
Subject: [Numpy-discussion] Surprising performance tweak in Cython
In-Reply-To: <20080623045516.GK15908@phare.normalesup.org>
References: <20080623003730.GF15908@phare.normalesup.org>
	<485F124E.3000104@hawaii.edu>
	<20080623031028.GI15908@phare.normalesup.org>
	<485F28F9.3040902@hawaii.edu>
	<20080623045516.GK15908@phare.normalesup.org>
Message-ID: <ce557a360806222303h589762bald8a064e520d70da5@mail.gmail.com>

2008/6/23 Gael Varoquaux <gael.varoquaux at normalesup.org>:
> On Sun, Jun 22, 2008 at 06:39:21PM -1000, Eric Firing wrote:
>> Another typo is the culprit:
>
>> In [2]:timeit do_Mandelbrot_cython()
>> 10 loops, best of 3: 53.8 ms per loop
>
>> In [3]:timeit do_Mandelbrot_cython2()
>> 10 loops, best of 3: 54 ms per loop
>
>> This is after I put the underscore in the x_buffer declaration in
>> do_Mandlebrot_cython2.  As it was, with no underscore, x_buffer was
>> being treated as a python double.
>
> Indeed, I was looking for a forgotten type definition, but I forgotten to
> look for typos in the type definitions. As a results, the cython code is
> now of comparable speed to the weave.inline: cython=59ms and weave=62.5ms
> on my box.
>
> Thanks a lot, next time I'll now where to look.

One way to track down this kind of problem is to look at the C code
that's generated. Easier, I admit, with a little familiarity with C,
but in any case code working with python variables will be obtrusively
strewn with functions like "PyFoo"; efficient code will be mostly
devoid of underscores and the "Py" prefix.

Anne


From alan.mcintyre at gmail.com  Mon Jun 23 03:37:23 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 23 Jun 2008 03:37:23 -0400
Subject: [Numpy-discussion] Code samples in docstrings mistaken as doctests
Message-ID: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>

Hi all,

Some docstrings have examples of how to use the function that aren't
executable code (see numpy.core.defmatrix.bmat for an example) in
their current form.  Should these examples have the ">>>" removed from
them to avoid them being picked up as doctests?

Thanks,
Alan


From silva at lma.cnrs-mrs.fr  Mon Jun 23 03:38:06 2008
From: silva at lma.cnrs-mrs.fr (Fabrice Silva)
Date: Mon, 23 Jun 2008 09:38:06 +0200
Subject: [Numpy-discussion] error importing a f2py compiled module.
Message-ID: <1214206686.3133.14.camel@Portable-s2m.cnrs-mrs.fr>

?Dear all
I've tried to run f2py on a fortran file which used to be usable from
python some months ago.
Following command lines are applied with success (no errors raised) :
        f2py -m modulename -h tmpo.pyf --overwrite-signature  tmpo.f
        f2py -m modulename -c --f90exec=/usr/bin/f95 tmpo.f

The output of these commands is available here:
http://paste.debian.net/7307

When importing in Python with "import modulename", I have an
ImportError:
        Traceback (most recent call last):
          File "Solveur.py", line 44, in <module>
            import modulename as Modele
        ImportError: modulename.so: failed to map segment from shared
        object: Operation not permitted

How can that be fixed ? Any suggestion ?
Thanks
-- 
Fabrice Silva
LMA UPR CNRS 7051 - ?quipe S2M




From stefan at sun.ac.za  Mon Jun 23 04:03:28 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 23 Jun 2008 10:03:28 +0200
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
Message-ID: <9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>

2008/6/23 Alan McIntyre <alan.mcintyre at gmail.com>:
> Some docstrings have examples of how to use the function that aren't
> executable code (see numpy.core.defmatrix.bmat for an example) in
> their current form.  Should these examples have the ">>>" removed from
> them to avoid them being picked up as doctests?

The examples written for the random module warrants the same question.
 First and foremost, the docstrings are there to illustrate to users
how to use the code; second, to serve as tests.

Example codes should run, but I'm not sure whether they should always
be valid doctests.

In the `bmat` example, I would remove the '>>>' like you suggested.

Regards
St?fan


From rjd4 at cam.ac.uk  Mon Jun 23 04:31:24 2008
From: rjd4 at cam.ac.uk (Bob Dowling)
Date: Mon, 23 Jun 2008 09:31:24 +0100
Subject: [Numpy-discussion] ndarray methods vs numpy module functions
Message-ID: <485F5F5C.5040903@cam.ac.uk>

[ I'm new here and this has the feel of an FAQ but I couldn't find 
anything at http://www.scipy.org/FAQ .  If I should have looked 
somewhere else a URL would be gratefully received. ]


What's the reasoning behind functions like sum() and cumsum() being 
provided both as module functions (numpy.sum(data, axis=1)) and as 
object methods (data.sum(axis=1)) but other functions - and I stumbled 
over diff() - only being provided as module functions?


 >>> print numpy.__version__
1.1.0

 >>> data = numpy.array([[1,2,3],[4,5,6]])

 >>> numpy.sum(data,axis=1)
array([ 6, 15])

 >>> data.sum(axis=1)
array([ 6, 15])

 >>> numpy.diff(data,axis=1)
array([[1, 1],
        [1, 1]])

 >>> data.diff(axis=1)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
AttributeError: 'numpy.ndarray' object has no attribute 'diff'


From pearu at cens.ioc.ee  Mon Jun 23 07:00:10 2008
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Mon, 23 Jun 2008 14:00:10 +0300 (EEST)
Subject: [Numpy-discussion] error importing a f2py compiled module.
In-Reply-To: <1214206686.3133.14.camel@Portable-s2m.cnrs-mrs.fr>
References: <1214206686.3133.14.camel@Portable-s2m.cnrs-mrs.fr>
Message-ID: <63206.88.89.32.166.1214218810.squirrel@cens.ioc.ee>

On Mon, June 23, 2008 10:38 am, Fabrice Silva wrote:
> ???Dear all
> I've tried to run f2py on a fortran file which used to be usable from
> python some months ago.
> Following command lines are applied with success (no errors raised) :
>         f2py -m modulename -h tmpo.pyf --overwrite-signature  tmpo.f
>         f2py -m modulename -c --f90exec=/usr/bin/f95 tmpo.f

First, it is not clear what compiler is f95. If it is gfortran, then
use the command
  f2py -m modulename -c --fcompiler=gnu95 tmpo.f

If it is something else, check the output of

  f2py -c --help-fcompiler

and use appropiate --fcompiler switch.

Second, I hope you realize that the first command has no effect to
the second command. If you have edited the tmpo.pyf file, then use
the following second command:

  f2py tmpo.pyf  -c --fcompiler=gnu95 tmpo.f

> The output of these commands is available here:
> http://paste.debian.net/7307
>
> When importing in Python with "import modulename", I have an
> ImportError:
>         Traceback (most recent call last):
>           File "Solveur.py", line 44, in <module>
>             import modulename as Modele
>         ImportError: modulename.so: failed to map segment from shared
>         object: Operation not permitted
>
> How can that be fixed ? Any suggestion ?

I don't  have ideas what is causing this import error. Try
the instructions above, may be it is due to some compile object
conflicts.

HTH,
Pearu



From silva at lma.cnrs-mrs.fr  Mon Jun 23 08:10:58 2008
From: silva at lma.cnrs-mrs.fr (Fabrice Silva)
Date: Mon, 23 Jun 2008 14:10:58 +0200
Subject: [Numpy-discussion] error importing a f2py compiled module.
In-Reply-To: <63206.88.89.32.166.1214218810.squirrel@cens.ioc.ee>
References: <1214206686.3133.14.camel@Portable-s2m.cnrs-mrs.fr>
	<63206.88.89.32.166.1214218810.squirrel@cens.ioc.ee>
Message-ID: <1214223058.3133.25.camel@Portable-s2m.cnrs-mrs.fr>

Le lundi 23 juin 2008 ? 14:00 +0300, Pearu Peterson a ?crit :
> First, it is not clear what compiler is f95. If it is gfortran, then
> use the command
>   f2py -m modulename -c --fcompiler=gnu95 tmpo.f
> 
> If it is something else, check the output of
> 
>   f2py -c --help-fcompiler
> 
> and use appropiate --fcompiler switch.
> 
> Second, I hope you realize that the first command has no effect to
> the second command. If you have edited the tmpo.pyf file, then use
> the following second command:
> 
>   f2py tmpo.pyf  -c --fcompiler=gnu95 tmpo.f

Thanks for these comments. The compiler was, following all the symbolic
links, gfortran so that I corrected the switch to gnu95.
The first line was the residue of some tests I've done in the past.
With the command line 
	f2py -c --quiet -m ?modulename --fcompiler=gnu95  fortranfile.f
it compiles as before, but I still can not import the module in python.

> I don't  have ideas what is causing this import error. Try
> the instructions above, may be it is due to some compile object
> conflicts.
The only posts on mailing lists I've read mention security policies
(SElinux) and /tmp execution limitations...
-- 
Fabrice Silva <silva at lma.cnrs-mrs.fr>
LMA UPR CNRS 7051 - ?quipe S2M



From gael.varoquaux at normalesup.org  Mon Jun 23 08:16:00 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 23 Jun 2008 14:16:00 +0200
Subject: [Numpy-discussion] Surprising performance tweak in Cython
In-Reply-To: <ce557a360806222303h589762bald8a064e520d70da5@mail.gmail.com>
References: <20080623003730.GF15908@phare.normalesup.org>
	<485F124E.3000104@hawaii.edu>
	<20080623031028.GI15908@phare.normalesup.org>
	<485F28F9.3040902@hawaii.edu>
	<20080623045516.GK15908@phare.normalesup.org>
	<ce557a360806222303h589762bald8a064e520d70da5@mail.gmail.com>
Message-ID: <20080623121600.GA6589@phare.normalesup.org>

On Mon, Jun 23, 2008 at 02:03:06AM -0400, Anne Archibald wrote:
> One way to track down this kind of problem is to look at the C code
> that's generated. Easier, I admit, with a little familiarity with C,
> but in any case code working with python variables will be obtrusively
> strewn with functions like "PyFoo"; efficient code will be mostly
> devoid of underscores and the "Py" prefix.

Good point. And cython makes this easy to do with the -a switch that
generates anhtml page for that.

Thanks for the tip,

Ga?l



From oliphant at enthought.com  Mon Jun 23 10:35:58 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Mon, 23 Jun 2008 09:35:58 -0500
Subject: [Numpy-discussion] Travis: this looks like a bug in
	ufuncobject.c
In-Reply-To: <e06186140806221546k77be3867u78197c6de5e48d43@mail.gmail.com>
References: <e06186140806221546k77be3867u78197c6de5e48d43@mail.gmail.com>
Message-ID: <485FB4CE.1030603@enthought.com>

Charles R Harris wrote:
> select_types(PyUFuncObject *self, int *arg_types,
>              PyUFuncGenericFunction *function, void **data,
>              PyArray_SCALARKIND *scalars,
>              PyObject *typetup)
> {
>     int i, j;
>     char start_type;
>     int userdef=-1;
>     int userdef_ind=-1;
>
>     if (self->userloops) {
>         for(i=0; i<self->nin; i++) {
>             if (PyTypeNum_ISUSERDEF(arg_types[i])) {
>                 userdef = arg_types[i];
>         userdef_ind = i;
>                 break;
>             }
>         }
>     }
>
>     if (typetup != NULL)
>         return extract_specified_loop(self, arg_types, function, data,
>                                       typetup, userdef);
>
>     if (userdef > 0) {
>        ^^^^^^^^^^^^
>         PyObject *key, *obj;
>         int ret=-1;
>         obj = NULL;
>     /* Look through all the registered loops for all the user-defined
>        types to find a match.
>      */
>     while (ret == -1) {
>         if (userdef_ind >= self->nin) break;
>         userdef = arg_types[userdef_ind++];
>         if (!(PyTypeNum_ISUSERDEF(userdef))) continue;
>         key = PyInt_FromLong((long) userdef);
>         if (key == NULL) return -1;
>         obj = PyDict_GetItem(self->userloops, key);
>         Py_DECREF(key);
>         if (obj == NULL) continue;
>         /* extract the correct function
>            data and argtypes for this user-defined type.
>         */
>         ret = _find_matching_userloop(obj, arg_types, scalars,
>                       function, data, self->nargs,
>                       self->nin);
>     }
>     if (ret == 0) return ret;
>     PyErr_SetString(PyExc_TypeError, _types_msg);
>     return ret;
>     }
> ...
>
> It looks like the test should be userdef >= 0.
>
What leads you to say that? 

-Travis



From charlesr.harris at gmail.com  Mon Jun 23 13:23:51 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 23 Jun 2008 11:23:51 -0600
Subject: [Numpy-discussion] Py_NotImplementedType leak
Message-ID: <e06186140806231023m1912f18dqe63caa5f43e9997a@mail.gmail.com>

Leaks of Py_NotImplementedType to the user are regarded as errors by the
Python developers and google turns up several patches fixing such issues. In
numpy we have this problem because we issue the same call for, say,
right_shift as we do for >>. This leads to things like:

In [1]: int64(1) >> float32(1)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

/home/charris/<ipython console> in <module>()

TypeError: unsupported operand type(s) for >>: 'int' and 'numpy.float32'

In [2]: right_shift(int64(1),float32(1))
Out[2]: NotImplemented

There are several things going on here. First, all the numbers are promoted
to 0-D object arrays, so numpy is using the logic for numpy.object scalars.
Second, the Python interpreter knows how to deal with the >> operator for
Python ints, which has the left and right slots, but can't deal with the
explicit call to right_shift because it knows nothing about it. Since proper
behavior in this case depends on knowledge of how right_shift is called I
think the thing to do is raise a TypeError{NotImplemented} exception in the
explicit right_shift call and move the NotImplementedType logic up to the
__rrshift__ slot for the numpy array types. This also applies to the other
standard numeric methods.

Thoughts?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080623/5bb35edd/attachment.html>

From fperez.net at gmail.com  Mon Jun 23 14:02:44 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 23 Jun 2008 11:02:44 -0700
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
Message-ID: <db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>

On Mon, Jun 23, 2008 at 1:03 AM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2008/6/23 Alan McIntyre <alan.mcintyre at gmail.com>:
>> Some docstrings have examples of how to use the function that aren't
>> executable code (see numpy.core.defmatrix.bmat for an example) in
>> their current form.  Should these examples have the ">>>" removed from
>> them to avoid them being picked up as doctests?
>
> The examples written for the random module warrants the same question.
>  First and foremost, the docstrings are there to illustrate to users
> how to use the code; second, to serve as tests.
>
> Example codes should run, but I'm not sure whether they should always
> be valid doctests.
>
> In the `bmat` example, I would remove the '>>>' like you suggested.

There's also the option of marking them so doctest skips them via

#doctest: +SKIP

http://docs.python.org/lib/doctest-options.html

Cheers,

f


From cimrman3 at ntc.zcu.cz  Mon Jun 23 14:11:36 2008
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Mon, 23 Jun 2008 20:11:36 +0200
Subject: [Numpy-discussion] set_printoptions - floating point format option?
Message-ID: <485FE758.5070408@ntc.zcu.cz>

Hi,

I need to display some numpy arrays in mantissa+exponent format (e.g. 
'%.2e' using C syntax). In numpy.set_printoptions(), there is currently 
only 'precision' option, which does not allow this.

What about having an option related to 'precision', named possibly 
'float_format', with the following function:

     :Parameters:
         format : string
             Format string for floating point output, has precedence 
over 'precision' (default '%.8f').

r.


From alan.mcintyre at gmail.com  Mon Jun 23 14:17:09 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 23 Jun 2008 14:17:09 -0400
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
Message-ID: <1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>

On Mon, Jun 23, 2008 at 2:02 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> There's also the option of marking them so doctest skips them via
>
> #doctest: +SKIP
>
> http://docs.python.org/lib/doctest-options.html

For short examples, that seems like a good option, but it seems like
you have to have that comment on every line that you want skipped.
There are some long examples (like the one in
lib/function_base.py:bartlett) that (to me) would look pretty ugly
having that comment tacked on to every line.

Either way is fine with me in the end, though, so long as it doesn't
produce test failures. :)


From pav at iki.fi  Mon Jun 23 14:37:39 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 23 Jun 2008 18:37:39 +0000 (UTC)
Subject: [Numpy-discussion] Code samples in docstrings mistaken
	as	doctests
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
Message-ID: <g3oqhj$q9h$2@ger.gmane.org>

Mon, 23 Jun 2008 14:17:09 -0400, Alan McIntyre wrote:

> On Mon, Jun 23, 2008 at 2:02 PM, Fernando Perez <fperez.net at gmail.com>
> wrote:
>> There's also the option of marking them so doctest skips them via
>>
>> #doctest: +SKIP
>>
>> http://docs.python.org/lib/doctest-options.html
> 
> For short examples, that seems like a good option, but it seems like you
> have to have that comment on every line that you want skipped. There are
> some long examples (like the one in lib/function_base.py:bartlett) that
> (to me) would look pretty ugly having that comment tacked on to every
> line.
> 
> Either way is fine with me in the end, though, so long as it doesn't
> produce test failures. :)

Can you make the convention chosen for the examples (currently only in 
the doc wiki, not yet in SVN) to work: assuming "import numpy as np" in 
examples?

This would remove the need for those "from numpy import *" lines in the 
examples that I see were added in r5311.

-- 
Pauli Virtanen



From fperez.net at gmail.com  Mon Jun 23 14:46:11 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 23 Jun 2008 11:46:11 -0700
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
Message-ID: <db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>

On Mon, Jun 23, 2008 at 11:17 AM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> On Mon, Jun 23, 2008 at 2:02 PM, Fernando Perez <fperez.net at gmail.com> wrote:
>> There's also the option of marking them so doctest skips them via
>>
>> #doctest: +SKIP
>>
>> http://docs.python.org/lib/doctest-options.html
>
> For short examples, that seems like a good option, but it seems like
> you have to have that comment on every line that you want skipped.
> There are some long examples (like the one in
> lib/function_base.py:bartlett) that (to me) would look pretty ugly
> having that comment tacked on to every line.

Ugh.  Definitely too ugly if it has to go in every line.  From reading
the docs I interpreted it as affecting the whole example, which would
be far more sensible...

> Either way is fine with me in the end, though, so long as it doesn't
> produce test failures. :)

Yes, but we also want to make these really easy for users to cleanly
paste in with minimal effort.  I wonder if a decorator could be
applied to those functions so that nose would then skip the doctests:

@skip_doctest
def foo()....

but the nose doctest plugin isn't exactly a model of modularity, so
I'm not sure you want to go down that particular road...

The no-prompts option seems to be the cleanest right now.  You may
want then to instead just use a reST block for those, so that sphinx
eventually renders them nicely at least as blocks:

    Plot the window and its frequency response::    ### <<< Added
double colon here

      from numpy import clip, log10, array, bartlett   ### <<< Extra
indent so this is a reST block
      from scipy.fftpack import fft

etc...

Just some ideas...


Cheers,

f


From alan.mcintyre at gmail.com  Mon Jun 23 15:03:24 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 23 Jun 2008 15:03:24 -0400
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <g3oqhj$q9h$2@ger.gmane.org>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
	<g3oqhj$q9h$2@ger.gmane.org>
Message-ID: <1d36917a0806231203g512382fp1e8eb015eeb4f11e@mail.gmail.com>

On Mon, Jun 23, 2008 at 2:37 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Can you make the convention chosen for the examples (currently only in
> the doc wiki, not yet in SVN) to work: assuming "import numpy as np" in
> examples?
>
> This would remove the need for those "from numpy import *" lines in the
> examples that I see were added in r5311.

Sure, I'll look at that.  It seems like every possible option for
importing stuff from numpy is used in doctests (sometimes even in the
same module), so having them standardized with that implicit import is
much better.


From stefan at sun.ac.za  Mon Jun 23 15:21:51 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 23 Jun 2008 21:21:51 +0200
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
	<db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>
Message-ID: <9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>

2008/6/23 Fernando Perez <fperez.net at gmail.com>:
> On Mon, Jun 23, 2008 at 11:17 AM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
>> On Mon, Jun 23, 2008 at 2:02 PM, Fernando Perez <fperez.net at gmail.com> wrote:
>>> There's also the option of marking them so doctest skips them via
>>>
>>> #doctest: +SKIP
>>>
>>> http://docs.python.org/lib/doctest-options.html
>>
>> For short examples, that seems like a good option, but it seems like
>> you have to have that comment on every line that you want skipped.
>> There are some long examples (like the one in
>> lib/function_base.py:bartlett) that (to me) would look pretty ugly
>> having that comment tacked on to every line.
>
> Ugh.  Definitely too ugly if it has to go in every line.  From reading
> the docs I interpreted it as affecting the whole example, which would
> be far more sensible...
>
>> Either way is fine with me in the end, though, so long as it doesn't
>> produce test failures. :)
>
> Yes, but we also want to make these really easy for users to cleanly
> paste in with minimal effort.  I wonder if a decorator could be
> applied to those functions so that nose would then skip the doctests:
>
> @skip_doctest
> def foo()....

Another alternative is to replace +SKIP with something like +IGNORE.
That way, the statement is still executed, we just don't care about
its outcome.  If we skip the line entirely, it often affects the rest
of the tests later on.

See

http://aroberge.blogspot.com/2008/06/monkeypatching-doctest.html

for an example implementation.

Cheers
St?fan


From charlesr.harris at gmail.com  Mon Jun 23 15:28:51 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 23 Jun 2008 13:28:51 -0600
Subject: [Numpy-discussion] Py_NotImplementedType leak
In-Reply-To: <e06186140806231023m1912f18dqe63caa5f43e9997a@mail.gmail.com>
References: <e06186140806231023m1912f18dqe63caa5f43e9997a@mail.gmail.com>
Message-ID: <e06186140806231228j2f80b357wb134e029dcae8b8e@mail.gmail.com>

On Mon, Jun 23, 2008 at 11:23 AM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

> Leaks of Py_NotImplementedType to the user are regarded as errors by the
> Python developers and google turns up several patches fixing such issues. In
> numpy we have this problem because we issue the same call for, say,
> right_shift as we do for >>. This leads to things like:
>
> In [1]: int64(1) >> float32(1)
> ---------------------------------------------------------------------------
> TypeError                                 Traceback (most recent call last)
>
> /home/charris/<ipython console> in <module>()
>
> TypeError: unsupported operand type(s) for >>: 'int' and 'numpy.float32'
>
> In [2]: right_shift(int64(1),float32(1))
> Out[2]: NotImplemented
>
> There are several things going on here. First, all the numbers are promoted
> to 0-D object arrays, so numpy is using the logic for numpy.object scalars.
> Second, the Python interpreter knows how to deal with the >> operator for
> Python ints, which has the left and right slots, but can't deal with the
> explicit call to right_shift because it knows nothing about it. Since proper
> behavior in this case depends on knowledge of how right_shift is called I
> think the thing to do is raise a TypeError{NotImplemented} exception in the
> explicit right_shift call and move the NotImplementedType logic up to the
> __rrshift__ slot for the numpy array types. This also applies to the other
> standard numeric methods.
>
> Thoughts?
>

In particular:

Extension types can now set the type flag Py_TPFLAGS_CHECKTYPES in their
> PyTypeObject structure to indicate that they support the new coercion
> model. In such extension types, the numeric slot functions can no longer
> assume that they'll be passed two arguments of the same type; instead they
> may be passed two arguments of differing types, and can then perform their
> own internal coercion.
>

So the NotImplemented return is restricted to numeric slot functions. Given
this, and the fact that leaking NotImplemented to the user is a bug, I'm
going to make the fix unless someone objects.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080623/a60845fc/attachment.html>

From alan.mcintyre at gmail.com  Mon Jun 23 15:37:14 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 23 Jun 2008 15:37:14 -0400
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
	<db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>
	<9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>
Message-ID: <1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>

On Mon, Jun 23, 2008 at 3:21 PM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> Another alternative is to replace +SKIP with something like +IGNORE.
> That way, the statement is still executed, we just don't care about
> its outcome.  If we skip the line entirely, it often affects the rest
> of the tests later on.

Ugh.  That just seems like a lot of unreadable ugliness to me.  If
this comment magic is the only way to make that stuff execute properly
under doctest, I think I'd rather just skip it in favor of clean,
uncluttered, non-doctestable code samples in the docstrings.  If the
code that's currently in docstrings needs to be retained as test code,
I'll gladly take the time to put it into a test_ module where it
doesn't get in the way of documentation.  I'll defer to the consensus,
though.


From peridot.faceted at gmail.com  Mon Jun 23 15:53:55 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 23 Jun 2008 15:53:55 -0400
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
	<db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>
	<9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>
	<1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
Message-ID: <ce557a360806231253h46062a66g1f4071ed4d77829a@mail.gmail.com>

2008/6/23 Alan McIntyre <alan.mcintyre at gmail.com>:
> On Mon, Jun 23, 2008 at 3:21 PM, St?fan van der Walt <stefan at sun.ac.za> wrote:
>> Another alternative is to replace +SKIP with something like +IGNORE.
>> That way, the statement is still executed, we just don't care about
>> its outcome.  If we skip the line entirely, it often affects the rest
>> of the tests later on.
>
> Ugh.  That just seems like a lot of unreadable ugliness to me.  If
> this comment magic is the only way to make that stuff execute properly
> under doctest, I think I'd rather just skip it in favor of clean,
> uncluttered, non-doctestable code samples in the docstrings.  If the
> code that's currently in docstrings needs to be retained as test code,
> I'll gladly take the time to put it into a test_ module where it
> doesn't get in the way of documentation.  I'll defer to the consensus,
> though.

I think doctests are valuable: it's very hard for the documentation to
get out of sync with the code, and it makes it very easy to write
tests, particularly in light of the wiki documentation framework. But
I think encrusting examples with weird comments will be a pain for
documentors and off-putting to users. Perhaps doctests can be
positively marked, in some relatively unobtrusive way?

Anne

From pav at iki.fi  Mon Jun 23 15:57:08 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 23 Jun 2008 19:57:08 +0000 (UTC)
Subject: [Numpy-discussion] Code samples in docstrings mistaken
	as	doctests
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
Message-ID: <g3ov6j$q9h$3@ger.gmane.org>

Mon, 23 Jun 2008 10:03:28 +0200, St?fan van der Walt wrote:

> 2008/6/23 Alan McIntyre <alan.mcintyre at gmail.com>:
>> Some docstrings have examples of how to use the function that aren't
>> executable code (see numpy.core.defmatrix.bmat for an example) in their
>> current form.  Should these examples have the ">>>" removed from them
>> to avoid them being picked up as doctests?
> 
> The examples written for the random module warrants the same question.
> First and foremost, the docstrings are there to illustrate to users
> how to use the code; second, to serve as tests.
> 
> Example codes should run, but I'm not sure whether they should always be
> valid doctests.
> 
> In the `bmat` example, I would remove the '>>>' like you suggested.

"Schematic" code (such as that currently in numpy.bmat) that doesn't run 
probably shouldn't be written with >>>, and for it the ReST block quote 
syntax is also looks OK.

But I'm personally not in favor of a distinction between a "doctest" and 
a "code sample", as the difference is not of interest to the main target 
audience who reads the docstrings (or the reference documentation 
generated based on them). As I see it, Numpy has a test architecture that 
is separate from doctests, so that most of the bonus doctests gives us is 
ensuring that all of our examples run without errors and produce expected 
results.

It's a bit unfortunate though that the doctest directives are as 
obtrusive as they are and only apply to a single line. One problem that I 
see now is quite annoying in the sample codes using matplotlib: 
matplotlib functions tend to return some objects whose <repr> contains a 
memory address, which causes the code to fit badly in a doctest. This can 
be worked around (ELLIPSIS, assigning to a variable), but I don't see a 
clean way. (I'm not so worried here about plot windows popping up as they 
can be worked around by monkey-patching matplotlib.show and choosing a 
non-graphical backend.)

Another point related to numpy are blank lines often appearing in array 
printouts (the text <BLANKLINE> is not a pretty sight in documentation). 
Also, NORMALIZE_WHITESPACE is useful for reducing the whitespace in array 
printout.

-- 
Pauli Virtanen



From mforbes at physics.ubc.ca  Mon Jun 23 16:04:45 2008
From: mforbes at physics.ubc.ca (Michael McNeil Forbes)
Date: Mon, 23 Jun 2008 13:04:45 -0700
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
	<db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>
	<9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>
	<1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
Message-ID: <01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>

> On 23 Jun 2008, at 12:37 PM, Alan McIntyre wrote:
>> Ugh.  That just seems like a lot of unreadable ugliness to me.  If
>> this comment magic is the only way to make that stuff execute  
>> properly
>> under doctest, I think I'd rather just skip it in favor of clean,
>> uncluttered, non-doctestable code samples in the docstrings.

Another perspective: doctests ensure that documentation stays up to  
date (if the behaviour or interface changes, then tests will fail  
indicating that the documentation also needs to be updated.)

Thus, one can argue that all examples should also be doctests.  This  
generally makes things a little more ugly, but much less ambiguous.

...
Examples:
---------
If A, B, C, and D are appropriately shaped 2-d arrays, then one can  
produce

     [ A  B ]
     [ C  D ]

using any of these methods:
 >>> A, B, C, D = [[1,1]], [[2,2]], [[3,3]], [[4,4]]
 >>> np.bmat('A, B; C, D')                  # From a string
matrix([[ 1,  1,  2,  2],
         [ 3,  3,  4,  4]])
 >>> np.bmat([[A,B],[C,D]])                 # From a nested sequence
matrix([[ 1,  1,  2,  2],
         [ 3,  3,  4,  4]])
 >>> np.bmat(np.r_[np.c_[A,B],np.c_[C,D]])  # From an array
matrix([[ 1,  1,  2,  2],
         [ 3,  3,  4,  4]])

Michael.


From pav at iki.fi  Mon Jun 23 16:07:13 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 23 Jun 2008 20:07:13 +0000 (UTC)
Subject: [Numpy-discussion] Code samples in docstrings mistaken
	as	doctests
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
	<db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>
	<9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>
	<1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
	<ce557a360806231253h46062a66g1f4071ed4d77829a@mail.gmail.com>
Message-ID: <g3ovph$q9h$4@ger.gmane.org>

Mon, 23 Jun 2008 15:53:55 -0400, Anne Archibald wrote:

> 2008/6/23 Alan McIntyre <alan.mcintyre at gmail.com>:
>> On Mon, Jun 23, 2008 at 3:21 PM, St?fan van der Walt <stefan at sun.ac.za>
>> wrote:
>>> Another alternative is to replace +SKIP with something like +IGNORE.
>>> That way, the statement is still executed, we just don't care about
>>> its outcome.  If we skip the line entirely, it often affects the rest
>>> of the tests later on.
>>
>> Ugh.  That just seems like a lot of unreadable ugliness to me.  If this
>> comment magic is the only way to make that stuff execute properly under
>> doctest, I think I'd rather just skip it in favor of clean,
>> uncluttered, non-doctestable code samples in the docstrings.  If the
>> code that's currently in docstrings needs to be retained as test code,
>> I'll gladly take the time to put it into a test_ module where it
>> doesn't get in the way of documentation.  I'll defer to the consensus,
>> though.
> 
> I think doctests are valuable: it's very hard for the documentation to
> get out of sync with the code, and it makes it very easy to write tests,
> particularly in light of the wiki documentation framework. But I think
> encrusting examples with weird comments will be a pain for documentors
> and off-putting to users. Perhaps doctests can be positively marked, in
> some relatively unobtrusive way?

I also think being able to test that the examples in docstrings run 
correctly could be valuable, but I'm not sure if it makes sense to have 
this enabled in the default test set.

Another idea (in addition to whitelisting): how easy would it be to 
subclass doctest.DocTestParser so that it would eg. automatically +IGNORE 
any doctest lines containing ">>> plt."?

-- 
Pauli Virtanen



From alan.mcintyre at gmail.com  Mon Jun 23 16:12:59 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 23 Jun 2008 16:12:59 -0400
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <g3ov6j$q9h$3@ger.gmane.org>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<g3ov6j$q9h$3@ger.gmane.org>
Message-ID: <1d36917a0806231312p2d0beb67pd2118823176e1ecc@mail.gmail.com>

On Mon, Jun 23, 2008 at 3:57 PM, Pauli Virtanen <pav at iki.fi> wrote:
> "Schematic" code (such as that currently in numpy.bmat) that doesn't run
> probably shouldn't be written with >>>, and for it the ReST block quote
> syntax is also looks OK.
>
> But I'm personally not in favor of a distinction between a "doctest" and
> a "code sample", as the difference is not of interest to the main target
> audience who reads the docstrings (or the reference documentation
> generated based on them). As I see it, Numpy has a test architecture that
> is separate from doctests, so that most of the bonus doctests gives us is
> ensuring that all of our examples run without errors and produce expected
> results.

I agree with you, Anne and Michael that ensuring that the
documentation examples run is important.   The more I think about it,
the more I'd rather have examples that are a bit verbose.  In the
particular example of bmat, as a new user, I'd really honestly rather
see those three cases fully coded:

>>> A=nd.arange(1,5).reshape(2,2)
>>> B= etc.
>>> F=bmat('A,B;C,D')
>>> F
matrix([[1,2,5,6],
etc.


From alan.mcintyre at gmail.com  Mon Jun 23 16:14:47 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 23 Jun 2008 16:14:47 -0400
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <g3ovph$q9h$4@ger.gmane.org>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
	<db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>
	<9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>
	<1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
	<ce557a360806231253h46062a66g1f4071ed4d77829a@mail.gmail.com>
	<g3ovph$q9h$4@ger.gmane.org>
Message-ID: <1d36917a0806231314h6b6dd631u22d66d5ba2c28e2a@mail.gmail.com>

On Mon, Jun 23, 2008 at 4:07 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Another idea (in addition to whitelisting): how easy would it be to
> subclass doctest.DocTestParser so that it would eg. automatically +IGNORE
> any doctest lines containing ">>> plt."?

I'll play around with that and see how hard it is to just ignore the
ugly bits that currently require all that per-line directive stuff.


From peridot.faceted at gmail.com  Mon Jun 23 16:28:37 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 23 Jun 2008 16:28:37 -0400
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
	<db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>
	<9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>
	<1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
	<01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>
Message-ID: <ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>

2008/6/23 Michael McNeil Forbes <mforbes at physics.ubc.ca>:
>> On 23 Jun 2008, at 12:37 PM, Alan McIntyre wrote:
>>> Ugh.  That just seems like a lot of unreadable ugliness to me.  If
>>> this comment magic is the only way to make that stuff execute
>>> properly
>>> under doctest, I think I'd rather just skip it in favor of clean,
>>> uncluttered, non-doctestable code samples in the docstrings.
>
> Another perspective: doctests ensure that documentation stays up to
> date (if the behaviour or interface changes, then tests will fail
> indicating that the documentation also needs to be updated.)
>
> Thus, one can argue that all examples should also be doctests.  This
> generally makes things a little more ugly, but much less ambiguous.

This is a bit awkward. How do you give an example for a random-number
generator? Even if you are willing to include a seed in each
statement, misleading users into thinking it's necessary, the value
returned for a given seed is not necessarily part of the interface a
random-number generator agrees to support.

I do agree that as many examples as possible should be doctests, but I
don't think we should restrict the examples we are allowed to give to
only those that can be made to serve as doctests.

Anne


From dyamins at gmail.com  Mon Jun 23 16:36:30 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Mon, 23 Jun 2008 16:36:30 -0400
Subject: [Numpy-discussion] Sparse Matrices in Numpy -- (with eigenvalue
	algorithms if possible)
Message-ID: <15e4667e0806231336s1c21f4b5xc604d3335fd09d87@mail.gmail.com>

I'm wondering if there is a currently-available Sparse-Matrix package for
numpy?  If so, how do I get it?    And, if there is a good sparse-matrix
package, does it include an eigenvalue-computation algorithm?  How would a
sparse-matrix package interact with something like numpy.linalg.eig, or for
that matter any of the other numpy modules?

Thanks,
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080623/bdf76e1f/attachment.html>

From mforbes at physics.ubc.ca  Mon Jun 23 16:44:58 2008
From: mforbes at physics.ubc.ca (Michael McNeil Forbes)
Date: Mon, 23 Jun 2008 13:44:58 -0700
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
	<db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>
	<9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>
	<1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
	<01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>
	<ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>
Message-ID: <B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>

On 23 Jun 2008, at 1:28 PM, Anne Archibald wrote:

> 2008/6/23 Michael McNeil Forbes <mforbes at physics.ubc.ca>:
>> Thus, one can argue that all examples should also be doctests.  This
>> generally makes things a little more ugly, but much less ambiguous.
>
> This is a bit awkward. How do you give an example for a random-number
> generator? Even if you are willing to include a seed in each
> statement, misleading users into thinking it's necessary, the value
> returned for a given seed is not necessarily part of the interface a
> random-number generator agrees to support.

I agree that this can be awkward sometimes, and should certainly not  
be policy, but one can usually get around this.  Instead of printing  
the result, you can use it, or demonstrate porperties:

 >>> random_array = np.random.rand(3,4)
 >>> random_array.shape
(3,4)
 >>> random_array.max() < 1
True
 >>> random_array.min() > 0
True

etc.

Michael.



From robert.kern at gmail.com  Mon Jun 23 16:51:20 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 23 Jun 2008 15:51:20 -0500
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
	<db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>
	<9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>
	<1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
	<01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>
	<ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>
	<B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>
Message-ID: <3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>

On Mon, Jun 23, 2008 at 15:44, Michael McNeil Forbes
<mforbes at physics.ubc.ca> wrote:
> On 23 Jun 2008, at 1:28 PM, Anne Archibald wrote:
>
>> 2008/6/23 Michael McNeil Forbes <mforbes at physics.ubc.ca>:
>>> Thus, one can argue that all examples should also be doctests.  This
>>> generally makes things a little more ugly, but much less ambiguous.
>>
>> This is a bit awkward. How do you give an example for a random-number
>> generator? Even if you are willing to include a seed in each
>> statement, misleading users into thinking it's necessary, the value
>> returned for a given seed is not necessarily part of the interface a
>> random-number generator agrees to support.
>
> I agree that this can be awkward sometimes, and should certainly not
> be policy, but one can usually get around this.  Instead of printing
> the result, you can use it, or demonstrate porperties:
>
>  >>> random_array = np.random.rand(3,4)
>  >>> random_array.shape
> (3,4)
>  >>> random_array.max() < 1
> True
>  >>> random_array.min() > 0
> True

Yes, this makes it doctestable, but you've destroyed the exampleness.
It should be policy *not* to do this.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From alan.mcintyre at gmail.com  Mon Jun 23 17:31:19 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 23 Jun 2008 17:31:19 -0400
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
	<db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>
	<9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>
	<1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
	<01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>
	<ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>
	<B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>
	<3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>
Message-ID: <1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>

On Mon, Jun 23, 2008 at 4:51 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> I agree that this can be awkward sometimes, and should certainly not
>> be policy, but one can usually get around this.  Instead of printing
>> the result, you can use it, or demonstrate porperties:
>>
>>  >>> random_array = np.random.rand(3,4)
>>  >>> random_array.shape
>> (3,4)
>>  >>> random_array.max() < 1
>> True
>>  >>> random_array.min() > 0
>> True
>
> Yes, this makes it doctestable, but you've destroyed the exampleness.
> It should be policy *not* to do this.

So it seems we have:
 1. Example code that is doctestable
 2. Example code that probably can't ever be doctestable (random
number stuff, etc.), but is still executable
 3. Schematic examples that aren't executable

Personally, I'm in favor of filling out examples of type #3 to make
them at least #2, but maybe that's not always practical.  I don't
think #3 should ever have ">>>" prompts, so it shouldn't ever be
picked up by doctest.

I suppose I could go for a decorator option to flag #2. If we execute
them, but not look at the results, then at least we find out about
examples that are broken enough to raise exceptions.


From mforbes at physics.ubc.ca  Mon Jun 23 17:51:21 2008
From: mforbes at physics.ubc.ca (Michael McNeil Forbes)
Date: Mon, 23 Jun 2008 14:51:21 -0700
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
	<db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>
	<9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>
	<1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
	<01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>
	<ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>
	<B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>
	<3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>
	<1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>
Message-ID: <C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>

> On Mon, Jun 23, 2008 at 4:51 PM, Robert Kern  
> <robert.kern at gmail.com> wrote:
>>>>>> random_array = np.random.rand(3,4)
>>>>>> random_array.shape
>>> (3,4)
>>>>>> random_array.max() < 1
>>> True
>>>>>> random_array.min() > 0
>>> True
>>
>> Yes, this makes it doctestable, but you've destroyed the exampleness.
>> It should be policy *not* to do this.

Well perhaps... but do you think that

rand(d0, d1, ..., dn) -> random values

is more exampley than

 >>> r = np.random.rand(3,2,4)
 >>> r.shape
(3,2,4)

?

On 23 Jun 2008, at 2:31 PM, Alan McIntyre wrote:
> So it seems we have:
>  1. Example code that is doctestable
>  2. Example code that probably can't ever be doctestable (random
> number stuff, etc.), but is still executable
>  3. Schematic examples that aren't executable
>
> Personally, I'm in favor of filling out examples of type #3 to make
> them at least #2, but maybe that's not always practical.  I don't
> think #3 should ever have ">>>" prompts, so it shouldn't ever be
> picked up by doctest.
>
> I suppose I could go for a decorator option to flag #2. If we execute
> them, but not look at the results, then at least we find out about
> examples that are broken enough to raise exceptions.

One can usually do #3 -> #1 or #2 by just leave bare assignments  
without printing a result (the user can always execute them and look  
at the result if they want).

 >>> r = np.random.rand(3,2,4)

which is cleaner than adding any flags...

Michael.




From robert.kern at gmail.com  Mon Jun 23 17:58:36 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 23 Jun 2008 16:58:36 -0500
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>
	<9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>
	<1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
	<01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>
	<ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>
	<B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>
	<3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>
	<1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>
	<C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>
Message-ID: <3d375d730806231458r580cd94eg59d39242dc18d8e6@mail.gmail.com>

On Mon, Jun 23, 2008 at 16:51, Michael McNeil Forbes
<mforbes at physics.ubc.ca> wrote:
>> On Mon, Jun 23, 2008 at 4:51 PM, Robert Kern
>> <robert.kern at gmail.com> wrote:
>>>>>>> random_array = np.random.rand(3,4)
>>>>>>> random_array.shape
>>>> (3,4)
>>>>>>> random_array.max() < 1
>>>> True
>>>>>>> random_array.min() > 0
>>>> True
>>>
>>> Yes, this makes it doctestable, but you've destroyed the exampleness.
>>> It should be policy *not* to do this.
>
> Well perhaps... but do you think that
>
> rand(d0, d1, ..., dn) -> random values
>
> is more exampley than
>
>  >>> r = np.random.rand(3,2,4)
>  >>> r.shape
> (3,2,4)
>
> ?

No. It wasn't an example. It was a specification of the call signature
because it is in an extension module, so the call signature is not
available like it is for pure Python functions. Thus, it needs to be
given in the docstring.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From haase at msg.ucsf.edu  Mon Jun 23 19:10:26 2008
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue, 24 Jun 2008 01:10:26 +0200
Subject: [Numpy-discussion] ndarray methods vs numpy module functions
In-Reply-To: <485F5F5C.5040903@cam.ac.uk>
References: <485F5F5C.5040903@cam.ac.uk>
Message-ID: <bc657ead0806231610t6a3a9a4bw2c046b89c782b6f4@mail.gmail.com>

On Mon, Jun 23, 2008 at 10:31 AM, Bob Dowling <rjd4 at cam.ac.uk> wrote:
> [ I'm new here and this has the feel of an FAQ but I couldn't find
> anything at http://www.scipy.org/FAQ .  If I should have looked
> somewhere else a URL would be gratefully received. ]
>
>
> What's the reasoning behind functions like sum() and cumsum() being
> provided both as module functions (numpy.sum(data, axis=1)) and as
> object methods (data.sum(axis=1)) but other functions - and I stumbled
> over diff() - only being provided as module functions?
>
>
Hi Bob,
this is a very good question.
I think the answers are
a) historical reasons AND, more importantly, differing personal preferences
b) I would file  the missing data.diff() as a bug.  There are many
inconsistencies left in such a big project like numpy.  And filing
bugs might be the best way of keeping track of them and getting them
fixes eventually...
(( a much more dangerous example is numpy.resize and data.resize,
which do (slightly) different things !!))


Others, please correct my .....

Welcome on the list.

-
Sebastian Haase


From robert.kern at gmail.com  Mon Jun 23 19:14:01 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 23 Jun 2008 18:14:01 -0500
Subject: [Numpy-discussion] ndarray methods vs numpy module functions
In-Reply-To: <bc657ead0806231610t6a3a9a4bw2c046b89c782b6f4@mail.gmail.com>
References: <485F5F5C.5040903@cam.ac.uk>
	<bc657ead0806231610t6a3a9a4bw2c046b89c782b6f4@mail.gmail.com>
Message-ID: <3d375d730806231614u3f7f2a43u225a60e679f1a5c0@mail.gmail.com>

On Mon, Jun 23, 2008 at 18:10, Sebastian Haase <haase at msg.ucsf.edu> wrote:
> On Mon, Jun 23, 2008 at 10:31 AM, Bob Dowling <rjd4 at cam.ac.uk> wrote:
>> [ I'm new here and this has the feel of an FAQ but I couldn't find
>> anything at http://www.scipy.org/FAQ .  If I should have looked
>> somewhere else a URL would be gratefully received. ]
>>
>>
>> What's the reasoning behind functions like sum() and cumsum() being
>> provided both as module functions (numpy.sum(data, axis=1)) and as
>> object methods (data.sum(axis=1)) but other functions - and I stumbled
>> over diff() - only being provided as module functions?
>>
>>
> Hi Bob,
> this is a very good question.
> I think the answers are
> a) historical reasons AND, more importantly, differing personal preferences
> b) I would file  the missing data.diff() as a bug.

It's not.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From stefan at sun.ac.za  Mon Jun 23 19:29:19 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 24 Jun 2008 01:29:19 +0200
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<db6b5ecc0806231146i7fec90d7l675d6c811a2c9b94@mail.gmail.com>
	<9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>
	<1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
	<01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>
	<ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>
	<B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>
	<3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>
	<1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>
	<C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>
Message-ID: <9457e7c80806231629h6693259bke09484ef16d5398f@mail.gmail.com>

2008/6/23 Michael McNeil Forbes <mforbes at physics.ubc.ca>:
> One can usually do #3 -> #1 or #2 by just leave bare assignments
> without printing a result (the user can always execute them and look
> at the result if they want).
>
>  >>> r = np.random.rand(3,2,4)
>
> which is cleaner than adding any flags...

Purposefully reducing the clarity of an example to satisfy some tool
is not an option.  We might be able to work around this specific case,
but there will be others.

It should be fairly easy to execute the example code, just to make
sure it runs.  We can always work out a scheme to test its validity
later.

One route is to use the same docstring scraper we use for the
reference guide, to extract all tests.  We can then choose a markup
which identifies tests with unpredictable results, and refrain from
executing them.  In some instances, we can even infer which tests to
ignore, e.g. the '>>> plt.' example Pauli mentioned.

Regards
St?fan


From stefan at sun.ac.za  Mon Jun 23 19:43:36 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 24 Jun 2008 01:43:36 +0200
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <9457e7c80806231629h6693259bke09484ef16d5398f@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>
	<1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
	<01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>
	<ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>
	<B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>
	<3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>
	<1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>
	<C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>
	<9457e7c80806231629h6693259bke09484ef16d5398f@mail.gmail.com>
Message-ID: <9457e7c80806231643m7ac12096uc134445b3c8db209@mail.gmail.com>

2008/6/24 St?fan van der Walt <stefan at sun.ac.za>:
> It should be fairly easy to execute the example code, just to make
> sure it runs.  We can always work out a scheme to test its validity
> later.

Mike Hansen just explained to me that the Sage doctest system sets the
random seed before executing each test.  If we address

a) Random variables
b) Plotting representations and
c) Endianness

we're probably halfway there.

Regards
St?fan


From stefan at sun.ac.za  Mon Jun 23 19:50:48 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 24 Jun 2008 01:50:48 +0200
Subject: [Numpy-discussion] Sparse Matrices in Numpy -- (with eigenvalue
	algorithms if possible)
In-Reply-To: <15e4667e0806231336s1c21f4b5xc604d3335fd09d87@mail.gmail.com>
References: <15e4667e0806231336s1c21f4b5xc604d3335fd09d87@mail.gmail.com>
Message-ID: <9457e7c80806231650l54060278qccd9242608b06bc8@mail.gmail.com>

Hi Dan

2008/6/23 Dan Yamins <dyamins at gmail.com>:
> I'm wondering if there is a currently-available Sparse-Matrix package for
> numpy?  If so, how do I get it?    And, if there is a good sparse-matrix
> package, does it include an eigenvalue-computation algorithm?  How would a
> sparse-matrix package interact with something like numpy.linalg.eig, or for
> that matter any of the other numpy modules?

SciPy contains all this functionality as `scipy.sparse`.  The NumPy
algorithms only work on dense matrices, but SciPy provides special
linear algebra routines.

Regards
St?fan


From michael.abshoff at googlemail.com  Mon Jun 23 19:58:19 2008
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Mon, 23 Jun 2008 16:58:19 -0700
Subject: [Numpy-discussion] Code samples in docstrings mistaken
	as	doctests
In-Reply-To: <9457e7c80806231643m7ac12096uc134445b3c8db209@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>	<9457e7c80806231221m551fdf5bsd26f19e33d443715@mail.gmail.com>	<1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>	<01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>	<ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>	<B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>	<3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>	<1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>	<C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>	<9457e7c80806231629h6693259bke09484ef16d5398f@mail.gmail.com>
	<9457e7c80806231643m7ac12096uc134445b3c8db209@mail.gmail.com>
Message-ID: <4860389B.2030606@gmail.com>

St?fan van der Walt wrote:
> 2008/6/24 St?fan van der Walt <stefan at sun.ac.za>:
>> It should be fairly easy to execute the example code, just to make
>> sure it runs.  We can always work out a scheme to test its validity
>> later.

Hi,

> Mike Hansen just explained to me that the Sage doctest system sets the
> random seed before executing each test.  If we address
> 
> a) Random variables

we have some small extensions to the doctesting framework that allow us 
to mark doctests as "#random" so that the result it not checked. Carl 
Witty wrote some code that makes the random number generator in a lot of 
the Sage components behave consistently on all supported platforms.

> b) Plotting representations and
> c) Endianness

Yeah, the Sage test suite seems to catch at least one of those in every 
release cycle.

Another thing we just implemented is a "jar of pickles" that lets us 
verify that there is no cross platform issues (32 vs. 64 bits and big 
vs. little endian) as well as no problems with loading pickles from 
previous releases.

> we're probably halfway there.
> 
> Regards
> St?fan

Cheers,

Michael

> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From fperez.net at gmail.com  Mon Jun 23 20:19:58 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 23 Jun 2008 17:19:58 -0700
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <4860389B.2030606@gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>
	<ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>
	<B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>
	<3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>
	<1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>
	<C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>
	<9457e7c80806231629h6693259bke09484ef16d5398f@mail.gmail.com>
	<9457e7c80806231643m7ac12096uc134445b3c8db209@mail.gmail.com>
	<4860389B.2030606@gmail.com>
Message-ID: <db6b5ecc0806231719o3240a910uf0ef74aa7f1f413a@mail.gmail.com>

On Mon, Jun 23, 2008 at 4:58 PM, Michael Abshoff
<michael.abshoff at googlemail.com> wrote:

>> a) Random variables
>
> we have some small extensions to the doctesting framework that allow us
> to mark doctests as "#random" so that the result it not checked. Carl
> Witty wrote some code that makes the random number generator in a lot of
> the Sage components behave consistently on all supported platforms.

Care to share? (BSD, we can't even look at the Sage code).

Cheers,

f


From michael.abshoff at googlemail.com  Mon Jun 23 20:26:26 2008
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Mon, 23 Jun 2008 17:26:26 -0700
Subject: [Numpy-discussion] Code samples in docstrings mistaken
	as	doctests
In-Reply-To: <db6b5ecc0806231719o3240a910uf0ef74aa7f1f413a@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>	<01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>	<ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>	<B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>	<3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>	<1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>	<C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>	<9457e7c80806231629h6693259bke09484ef16d5398f@mail.gmail.com>	<9457e7c80806231643m7ac12096uc134445b3c8db209@mail.gmail.com>	<4860389B.2030606@gmail.com>
	<db6b5ecc0806231719o3240a910uf0ef74aa7f1f413a@mail.gmail.com>
Message-ID: <48603F32.2080207@gmail.com>

Fernando Perez wrote:
> On Mon, Jun 23, 2008 at 4:58 PM, Michael Abshoff
> <michael.abshoff at googlemail.com> wrote:

Hi Fernando,

>>> a) Random variables
>> we have some small extensions to the doctesting framework that allow us
>> to mark doctests as "#random" so that the result it not checked. Carl
>> Witty wrote some code that makes the random number generator in a lot of
>> the Sage components behave consistently on all supported platforms.
> 
> Care to share? (BSD, we can't even look at the Sage code).

I am not the author, so I need to find out who wrote the code, but I am 
sure it can be made BSD. We are also working on "doctest+timeit" to hunt 
for performance regressions, but that one is not ready for prime time yet.

> Cheers,
> 
> f

Cheers,

Michael


> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From rmay31 at gmail.com  Mon Jun 23 20:35:35 2008
From: rmay31 at gmail.com (Ryan May)
Date: Mon, 23 Jun 2008 20:35:35 -0400
Subject: [Numpy-discussion] ndarray methods vs numpy module functions
In-Reply-To: <3d375d730806231614u3f7f2a43u225a60e679f1a5c0@mail.gmail.com>
References: <485F5F5C.5040903@cam.ac.uk>	<bc657ead0806231610t6a3a9a4bw2c046b89c782b6f4@mail.gmail.com>
	<3d375d730806231614u3f7f2a43u225a60e679f1a5c0@mail.gmail.com>
Message-ID: <48604157.4080207@gmail.com>

Robert Kern wrote:
> On Mon, Jun 23, 2008 at 18:10, Sebastian Haase <haase at msg.ucsf.edu> wrote:
>> On Mon, Jun 23, 2008 at 10:31 AM, Bob Dowling <rjd4 at cam.ac.uk> wrote:
>>> [ I'm new here and this has the feel of an FAQ but I couldn't find
>>> anything at http://www.scipy.org/FAQ .  If I should have looked
>>> somewhere else a URL would be gratefully received. ]
>>>
>>>
>>> What's the reasoning behind functions like sum() and cumsum() being
>>> provided both as module functions (numpy.sum(data, axis=1)) and as
>>> object methods (data.sum(axis=1)) but other functions - and I stumbled
>>> over diff() - only being provided as module functions?
>>>
>>>
>> Hi Bob,
>> this is a very good question.
>> I think the answers are
>> a) historical reasons AND, more importantly, differing personal preferences
>> b) I would file  the missing data.diff() as a bug.
> 
> It's not.
> 
Care to elaborate?

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From fperez.net at gmail.com  Mon Jun 23 20:37:33 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 23 Jun 2008 17:37:33 -0700
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <48603F32.2080207@gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>
	<3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>
	<1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>
	<C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>
	<9457e7c80806231629h6693259bke09484ef16d5398f@mail.gmail.com>
	<9457e7c80806231643m7ac12096uc134445b3c8db209@mail.gmail.com>
	<4860389B.2030606@gmail.com>
	<db6b5ecc0806231719o3240a910uf0ef74aa7f1f413a@mail.gmail.com>
	<48603F32.2080207@gmail.com>
Message-ID: <db6b5ecc0806231737h419c672fuba3fbabc26877fec@mail.gmail.com>

On Mon, Jun 23, 2008 at 5:26 PM, Michael Abshoff
<michael.abshoff at googlemail.com> wrote:

> I am not the author, so I need to find out who wrote the code, but I am
> sure it can be made BSD. We are also working on "doctest+timeit" to hunt
> for performance regressions, but that one is not ready for prime time yet.

Great, thanks.

Cheers,

f


From wnbell at gmail.com  Mon Jun 23 21:08:01 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Mon, 23 Jun 2008 20:08:01 -0500
Subject: [Numpy-discussion] Sparse Matrices in Numpy -- (with eigenvalue
	algorithms if possible)
In-Reply-To: <15e4667e0806231336s1c21f4b5xc604d3335fd09d87@mail.gmail.com>
References: <15e4667e0806231336s1c21f4b5xc604d3335fd09d87@mail.gmail.com>
Message-ID: <d05265cb0806231808y7a5644dfs969a8c4e8861539@mail.gmail.com>

On Mon, Jun 23, 2008 at 3:36 PM, Dan Yamins <dyamins at gmail.com> wrote:
> I'm wondering if there is a currently-available Sparse-Matrix package for
> numpy?  If so, how do I get it?    And, if there is a good sparse-matrix
> package, does it include an eigenvalue-computation algorithm?  How would a
> sparse-matrix package interact with something like numpy.linalg.eig, or for
> that matter any of the other numpy modules?
>

The next version of SciPy will include two sparse eigensolvers: ARPACK
and LOBPCG

http://scipy.org/scipy/scipy/browser/trunk/scipy/sparse/linalg/eigen

If you're familiar with MATLAB's eigs(), then you'll find ARPACK easy to use.


-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From robert.kern at gmail.com  Mon Jun 23 21:11:03 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 23 Jun 2008 20:11:03 -0500
Subject: [Numpy-discussion] ndarray methods vs numpy module functions
In-Reply-To: <48604157.4080207@gmail.com>
References: <485F5F5C.5040903@cam.ac.uk>
	<bc657ead0806231610t6a3a9a4bw2c046b89c782b6f4@mail.gmail.com>
	<3d375d730806231614u3f7f2a43u225a60e679f1a5c0@mail.gmail.com>
	<48604157.4080207@gmail.com>
Message-ID: <3d375d730806231811y55a47ab5m2d31caf2af2e017f@mail.gmail.com>

On Mon, Jun 23, 2008 at 19:35, Ryan May <rmay31 at gmail.com> wrote:
> Robert Kern wrote:
>> On Mon, Jun 23, 2008 at 18:10, Sebastian Haase <haase at msg.ucsf.edu> wrote:
>>> On Mon, Jun 23, 2008 at 10:31 AM, Bob Dowling <rjd4 at cam.ac.uk> wrote:
>>>> [ I'm new here and this has the feel of an FAQ but I couldn't find
>>>> anything at http://www.scipy.org/FAQ .  If I should have looked
>>>> somewhere else a URL would be gratefully received. ]
>>>>
>>>>
>>>> What's the reasoning behind functions like sum() and cumsum() being
>>>> provided both as module functions (numpy.sum(data, axis=1)) and as
>>>> object methods (data.sum(axis=1)) but other functions - and I stumbled
>>>> over diff() - only being provided as module functions?
>>>>
>>>>
>>> Hi Bob,
>>> this is a very good question.
>>> I think the answers are
>>> a) historical reasons AND, more importantly, differing personal preferences
>>> b) I would file  the missing data.diff() as a bug.
>>
>> It's not.
>>
> Care to elaborate?

There is not supposed to be a one-to-one correspondence between the
functions in numpy and the methods on an ndarray. There is some
duplication between the two, but that is not a reason to make more
duplication.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From charlesr.harris at gmail.com  Mon Jun 23 22:09:54 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 23 Jun 2008 20:09:54 -0600
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <4860389B.2030606@gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>
	<ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>
	<B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>
	<3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>
	<1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>
	<C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>
	<9457e7c80806231629h6693259bke09484ef16d5398f@mail.gmail.com>
	<9457e7c80806231643m7ac12096uc134445b3c8db209@mail.gmail.com>
	<4860389B.2030606@gmail.com>
Message-ID: <e06186140806231909i122d390dv2d81d6852ff73dba@mail.gmail.com>

On Mon, Jun 23, 2008 at 5:58 PM, Michael Abshoff <
michael.abshoff at googlemail.com> wrote:

> St?fan van der Walt wrote:
> > 2008/6/24 St?fan van der Walt <stefan at sun.ac.za>:
> >> It should be fairly easy to execute the example code, just to make
> >> sure it runs.  We can always work out a scheme to test its validity
> >> later.
>
> Hi,
>
> > Mike Hansen just explained to me that the Sage doctest system sets the
> > random seed before executing each test.  If we address
> >
> > a) Random variables
>
> we have some small extensions to the doctesting framework that allow us
> to mark doctests as "#random" so that the result it not checked. Carl
> Witty wrote some code that makes the random number generator in a lot of
> the Sage components behave consistently on all supported platforms.
>

But there is more than one possible random number generator. If you do that
you are tied into one kind of generator and one kind of initialization
implementation.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080623/48eee13c/attachment.html>

From michael.abshoff at googlemail.com  Mon Jun 23 22:27:05 2008
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Mon, 23 Jun 2008 19:27:05 -0700
Subject: [Numpy-discussion] Code samples in docstrings mistaken
	as	doctests
In-Reply-To: <e06186140806231909i122d390dv2d81d6852ff73dba@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>	<01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>	<ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>	<B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>	<3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>	<1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>	<C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>	<9457e7c80806231629h6693259bke09484ef16d5398f@mail.gmail.com>	<9457e7c80806231643m7ac12096uc134445b3c8db209@mail.gmail.com>	<4860389B.2030606@gmail.com>
	<e06186140806231909i122d390dv2d81d6852ff73dba@mail.gmail.com>
Message-ID: <48605B79.2060000@gmail.com>

Charles R Harris wrote:
> 
> 
> On Mon, Jun 23, 2008 at 5:58 PM, Michael Abshoff 
> <michael.abshoff at googlemail.com <mailto:michael.abshoff at googlemail.com>> 
> wrote:
> 
>     St?fan van der Walt wrote:
>      > 2008/6/24 St?fan van der Walt <stefan at sun.ac.za
>     <mailto:stefan at sun.ac.za>>:
>      >> It should be fairly easy to execute the example code, just to make
>      >> sure it runs.  We can always work out a scheme to test its validity
>      >> later.
> 
>     Hi,
> 
>      > Mike Hansen just explained to me that the Sage doctest system
>     sets the
>      > random seed before executing each test.  If we address
>      >
>      > a) Random variables
> 
>     we have some small extensions to the doctesting framework that allow us
>     to mark doctests as "#random" so that the result it not checked. Carl
>     Witty wrote some code that makes the random number generator in a lot of
>     the Sage components behave consistently on all supported platforms.

Hi,

> 
> But there is more than one possible random number generator. If you do 
> that you are tied into one kind of generator and one kind of 
> initialization implementation.
> 
> Chuck
> 

Correct, but so far Carl has hooked into six out of the many random 
number generators in the various components of Sage. This way we can set 
a global seed and also more easily reproduce issues with algorithms 
where randomness plays a role without being forced to be on the same 
platform. There are still doctests in Sage where the randomness comes 
from sources not in randgen (Carl's code), but sooner or later we will 
get around to all of them.

Cheers,

Michael

> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From dyamins at gmail.com  Mon Jun 23 23:23:06 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Mon, 23 Jun 2008 23:23:06 -0400
Subject: [Numpy-discussion] Sparse Matrices in Numpy -- (with eigenvalue
	algorithms if possible)
In-Reply-To: <d05265cb0806231808y7a5644dfs969a8c4e8861539@mail.gmail.com>
References: <15e4667e0806231336s1c21f4b5xc604d3335fd09d87@mail.gmail.com>
	<d05265cb0806231808y7a5644dfs969a8c4e8861539@mail.gmail.com>
Message-ID: <15e4667e0806232023m78eac31fhf185978223f092f1@mail.gmail.com>

>
> >
>
> The next version of SciPy will include two sparse eigensolvers: ARPACK
> and LOBPCG
>


>
> http://scipy.org/scipy/scipy/browser/trunk/scipy/sparse/linalg/eigen
>


OK, so I then I have three questions:

1) Does your commend imply that the current version of scipy does _not_ have
an eigensolver? (I installed it but can't find an eigensolver in it ... )

2) How would I go about installing the ARPACK and/or LOBPCG routines that do
exist now?  Or is it too early and they're not quite ready yet?

3) Finally: (and this is slightly off topic) -- I've just installed SciPy
for the first time.  In the readme.txt for the download it mentioned
something about having LAPACK and ATLAS installed.  However, on the scipy
install page (for OSX, http://www.scipy.org/Installing_SciPy/Mac_OS_X), it
doesn't mention anything about LAPACK and ATLAS -- and the instructions
there that I followed worked without any need for those packages.    Do I
need them?  Or are they only necessary for making certain routines faster?

Thanks!
Dan



Also:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080623/f5a4075d/attachment.html>

From wnbell at gmail.com  Mon Jun 23 23:41:55 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Mon, 23 Jun 2008 22:41:55 -0500
Subject: [Numpy-discussion] Sparse Matrices in Numpy -- (with eigenvalue
	algorithms if possible)
In-Reply-To: <15e4667e0806232023m78eac31fhf185978223f092f1@mail.gmail.com>
References: <15e4667e0806231336s1c21f4b5xc604d3335fd09d87@mail.gmail.com>
	<d05265cb0806231808y7a5644dfs969a8c4e8861539@mail.gmail.com>
	<15e4667e0806232023m78eac31fhf185978223f092f1@mail.gmail.com>
Message-ID: <d05265cb0806232041w2ee1c1a4ida80cbf7c68635cf@mail.gmail.com>

On Mon, Jun 23, 2008 at 10:23 PM, Dan Yamins <dyamins at gmail.com> wrote:

>
> 1) Does your commend imply that the current version of scipy does _not_ have
> an eigensolver? (I installed it but can't find an eigensolver in it ... )

That's correct, SciPy 0.6 does not have a sparse eigensolver.

>
> 2) How would I go about installing the ARPACK and/or LOBPCG routines that do
> exist now?  Or is it too early and they're not quite ready yet?
>

You will need to install the development version of SciPy from SVN:
http://www.scipy.org/Download#head-d0ec9f4cfbf6ed3029a2c409b6d9899586eee0e3
http://www.scipy.org/Installing_SciPy

I can confirm that both solvers work.

> 3) Finally: (and this is slightly off topic) -- I've just installed SciPy
> for the first time.  In the readme.txt for the download it mentioned
> something about having LAPACK and ATLAS installed.  However, on the scipy
> install page (for OSX, http://www.scipy.org/Installing_SciPy/Mac_OS_X), it
> doesn't mention anything about LAPACK and ATLAS -- and the instructions
> there that I followed worked without any need for those packages.    Do I
> need them?  Or are they only necessary for making certain routines faster?
>

I don't know about OSX specifically, but my understanding is that you
can build NumPy and SciPy without those libraries.  Performance in
certain dense linear algebra operations will be slower, but
scipy.sparse will be unchanged.

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From peridot.faceted at gmail.com  Mon Jun 23 23:51:19 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 23 Jun 2008 23:51:19 -0400
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <9457e7c80806231643m7ac12096uc134445b3c8db209@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<1d36917a0806231237p1727bb0ct7c8632600f6e22e3@mail.gmail.com>
	<01DD9DF2-6978-46A7-BB35-689B3A776D32@physics.ubc.ca>
	<ce557a360806231328p148f45b8yd65cf7b0a4165569@mail.gmail.com>
	<B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>
	<3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>
	<1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>
	<C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>
	<9457e7c80806231629h6693259bke09484ef16d5398f@mail.gmail.com>
	<9457e7c80806231643m7ac12096uc134445b3c8db209@mail.gmail.com>
Message-ID: <ce557a360806232051k4e885c63h4f372050adaa5b39@mail.gmail.com>

2008/6/23 St?fan van der Walt <stefan at sun.ac.za>:
> 2008/6/24 St?fan van der Walt <stefan at sun.ac.za>:
>> It should be fairly easy to execute the example code, just to make
>> sure it runs.  We can always work out a scheme to test its validity
>> later.
>
> Mike Hansen just explained to me that the Sage doctest system sets the
> random seed before executing each test.  If we address
>
> a) Random variables
> b) Plotting representations and
> c) Endianness
>
> we're probably halfway there.

I agree (though I have reservations about how they are to be
addressed). But in the current setting, "halfway there" is still a
problem - it seems to me we need, now and later, a way to deal with
generic examples that are not doctests. There may not be many of them,
and most may be dealt with by falling into categories a, b, and c
above, but it is important that we not make it difficult to write new
examples even if they can't readily be made into doctests. In
particular, we don't want some documentor saying "well, I'd like to
write an example, but I don't remember the arcane syntax to prevent
this failing a doctest, so I'm not going to bother."

Anne

From peridot.faceted at gmail.com  Mon Jun 23 23:53:09 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 23 Jun 2008 23:53:09 -0400
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <48605B79.2060000@gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<B1716BE9-BAAA-47D7-B5D3-CD74BD6B36F1@physics.ubc.ca>
	<3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>
	<1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>
	<C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>
	<9457e7c80806231629h6693259bke09484ef16d5398f@mail.gmail.com>
	<9457e7c80806231643m7ac12096uc134445b3c8db209@mail.gmail.com>
	<4860389B.2030606@gmail.com>
	<e06186140806231909i122d390dv2d81d6852ff73dba@mail.gmail.com>
	<48605B79.2060000@gmail.com>
Message-ID: <ce557a360806232053k2ba35143hfcdd38c74d6e5f0b@mail.gmail.com>

2008/6/23 Michael Abshoff <michael.abshoff at googlemail.com>:
> Charles R Harris wrote:
>>
>>
>> On Mon, Jun 23, 2008 at 5:58 PM, Michael Abshoff
>> <michael.abshoff at googlemail.com <mailto:michael.abshoff at googlemail.com>>
>> wrote:
>>
>>     St?fan van der Walt wrote:
>>      > 2008/6/24 St?fan van der Walt <stefan at sun.ac.za
>>     <mailto:stefan at sun.ac.za>>:
>>      >> It should be fairly easy to execute the example code, just to make
>>      >> sure it runs.  We can always work out a scheme to test its validity
>>      >> later.
>>
>>     Hi,
>>
>>      > Mike Hansen just explained to me that the Sage doctest system
>>     sets the
>>      > random seed before executing each test.  If we address
>>      >
>>      > a) Random variables
>>
>>     we have some small extensions to the doctesting framework that allow us
>>     to mark doctests as "#random" so that the result it not checked. Carl
>>     Witty wrote some code that makes the random number generator in a lot of
>>     the Sage components behave consistently on all supported platforms.
>
> Hi,
>
>>
>> But there is more than one possible random number generator. If you do
>> that you are tied into one kind of generator and one kind of
>> initialization implementation.
>>
>> Chuck
>>
>
> Correct, but so far Carl has hooked into six out of the many random
> number generators in the various components of Sage. This way we can set
> a global seed and also more easily reproduce issues with algorithms
> where randomness plays a role without being forced to be on the same
> platform. There are still doctests in Sage where the randomness comes
> from sources not in randgen (Carl's code), but sooner or later we will
> get around to all of them.

Doesn't this mean you can't change your implementation of random
number generators (for example choosing a different implementation of
generation of normally-distributed random numbers, or replacing the
Mersenne Twister) without causing countless doctests to fail
meaninglessly?

Anne

From robert.kern at gmail.com  Mon Jun 23 23:57:25 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 23 Jun 2008 22:57:25 -0500
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <ce557a360806232053k2ba35143hfcdd38c74d6e5f0b@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<3d375d730806231351u6daaba28kf04470c5a1dd4443@mail.gmail.com>
	<1d36917a0806231431g71e82a04g12efe2e3d9044c41@mail.gmail.com>
	<C9FC4F91-0808-4980-B5E9-4DA741782AB7@physics.ubc.ca>
	<9457e7c80806231629h6693259bke09484ef16d5398f@mail.gmail.com>
	<9457e7c80806231643m7ac12096uc134445b3c8db209@mail.gmail.com>
	<4860389B.2030606@gmail.com>
	<e06186140806231909i122d390dv2d81d6852ff73dba@mail.gmail.com>
	<48605B79.2060000@gmail.com>
	<ce557a360806232053k2ba35143hfcdd38c74d6e5f0b@mail.gmail.com>
Message-ID: <3d375d730806232057g42ed6db0wec317ecd01a04754@mail.gmail.com>

On Mon, Jun 23, 2008 at 22:53, Anne Archibald <peridot.faceted at gmail.com> wrote:
> 2008/6/23 Michael Abshoff <michael.abshoff at googlemail.com>:

>> Correct, but so far Carl has hooked into six out of the many random
>> number generators in the various components of Sage. This way we can set
>> a global seed and also more easily reproduce issues with algorithms
>> where randomness plays a role without being forced to be on the same
>> platform. There are still doctests in Sage where the randomness comes
>> from sources not in randgen (Carl's code), but sooner or later we will
>> get around to all of them.
>
> Doesn't this mean you can't change your implementation of random
> number generators (for example choosing a different implementation of
> generation of normally-distributed random numbers, or replacing the
> Mersenne Twister) without causing countless doctests to fail
> meaninglessly?

It's not that bad. After you've verified that your new code works, you
regenerate the examples. You check in both at the same time.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From robince at gmail.com  Tue Jun 24 00:10:14 2008
From: robince at gmail.com (Robin)
Date: Tue, 24 Jun 2008 05:10:14 +0100
Subject: [Numpy-discussion] Sparse Matrices in Numpy -- (with eigenvalue
	algorithms if possible)
In-Reply-To: <15e4667e0806232023m78eac31fhf185978223f092f1@mail.gmail.com>
References: <15e4667e0806231336s1c21f4b5xc604d3335fd09d87@mail.gmail.com>
	<d05265cb0806231808y7a5644dfs969a8c4e8861539@mail.gmail.com>
	<15e4667e0806232023m78eac31fhf185978223f092f1@mail.gmail.com>
Message-ID: <b14b627d0806232110y5c8ce408gfa3edaa173c7c54c@mail.gmail.com>

On Tue, Jun 24, 2008 at 4:23 AM, Dan Yamins <dyamins at gmail.com> wrote:
> 3) Finally: (and this is slightly off topic) -- I've just installed SciPy
> for the first time.  In the readme.txt for the download it mentioned
> something about having LAPACK and ATLAS installed.  However, on the scipy
> install page (for OSX, http://www.scipy.org/Installing_SciPy/Mac_OS_X), it
> doesn't mention anything about LAPACK and ATLAS -- and the instructions
> there that I followed worked without any need for those packages.    Do I
> need them?  Or are they only necessary for making certain routines faster?

On OS X it is usual to use the optimised BLAS provided by the
Accelerate framework (at least that's what it used to be called I
think). In any case - you don't need them because Apple provides an
optimised blas (which is used as an alternative to lapack and
atlas)... You could use lapack/atlas if you wanted but installation is
probably simpler following the instructions on the wiki to use the
apple one...

Robin


From charlesr.harris at gmail.com  Tue Jun 24 02:12:37 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 24 Jun 2008 00:12:37 -0600
Subject: [Numpy-discussion] Sparse Matrices in Numpy -- (with eigenvalue
	algorithms if possible)
In-Reply-To: <15e4667e0806232023m78eac31fhf185978223f092f1@mail.gmail.com>
References: <15e4667e0806231336s1c21f4b5xc604d3335fd09d87@mail.gmail.com>
	<d05265cb0806231808y7a5644dfs969a8c4e8861539@mail.gmail.com>
	<15e4667e0806232023m78eac31fhf185978223f092f1@mail.gmail.com>
Message-ID: <e06186140806232312w3da432c5l3d9ecfd1f3e1f96d@mail.gmail.com>

Hi Dan,

Did you finally get numpy installed as 64 bits?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080624/2f9a67fa/attachment.html>

From rjd4 at cam.ac.uk  Tue Jun 24 03:33:37 2008
From: rjd4 at cam.ac.uk (Bob Dowling)
Date: Tue, 24 Jun 2008 08:33:37 +0100
Subject: [Numpy-discussion] ndarray methods vs numpy module functions
In-Reply-To: <3d375d730806231811y55a47ab5m2d31caf2af2e017f@mail.gmail.com>
References: <485F5F5C.5040903@cam.ac.uk>	<bc657ead0806231610t6a3a9a4bw2c046b89c782b6f4@mail.gmail.com>	<3d375d730806231614u3f7f2a43u225a60e679f1a5c0@mail.gmail.com>	<48604157.4080207@gmail.com>
	<3d375d730806231811y55a47ab5m2d31caf2af2e017f@mail.gmail.com>
Message-ID: <4860A351.2020305@cam.ac.uk>

> There is not supposed to be a one-to-one correspondence between the
> functions in numpy and the methods on an ndarray. There is some
> duplication between the two, but that is not a reason to make more
> duplication.

I would make a plea for consistency, to start with.

Those of us who write in an OO style are required to switch backwards 
and forwards between OO and not-OO, or to abandon OO altogether in our 
NumPy code.  Neither is an attractive option.

The reason I tripped over this is that I am currently writing a course 
which introduces students to NumPy.  I am going to be asked this 
question from the audience.  As yet I don't have any answer except 
"history".


From haase at msg.ucsf.edu  Tue Jun 24 03:43:52 2008
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue, 24 Jun 2008 09:43:52 +0200
Subject: [Numpy-discussion] ndarray methods vs numpy module functions
In-Reply-To: <3d375d730806231811y55a47ab5m2d31caf2af2e017f@mail.gmail.com>
References: <485F5F5C.5040903@cam.ac.uk>
	<bc657ead0806231610t6a3a9a4bw2c046b89c782b6f4@mail.gmail.com>
	<3d375d730806231614u3f7f2a43u225a60e679f1a5c0@mail.gmail.com>
	<48604157.4080207@gmail.com>
	<3d375d730806231811y55a47ab5m2d31caf2af2e017f@mail.gmail.com>
Message-ID: <bc657ead0806240043k54fe31dxcc7b27751e523616@mail.gmail.com>

On Tue, Jun 24, 2008 at 3:11 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Mon, Jun 23, 2008 at 19:35, Ryan May <rmay31 at gmail.com> wrote:
>> Robert Kern wrote:
>>> On Mon, Jun 23, 2008 at 18:10, Sebastian Haase <haase at msg.ucsf.edu> wrote:
>>>> On Mon, Jun 23, 2008 at 10:31 AM, Bob Dowling <rjd4 at cam.ac.uk> wrote:
>>>>> [ I'm new here and this has the feel of an FAQ but I couldn't find
>>>>> anything at http://www.scipy.org/FAQ .  If I should have looked
>>>>> somewhere else a URL would be gratefully received. ]
>>>>>
>>>>>
>>>>> What's the reasoning behind functions like sum() and cumsum() being
>>>>> provided both as module functions (numpy.sum(data, axis=1)) and as
>>>>> object methods (data.sum(axis=1)) but other functions - and I stumbled
>>>>> over diff() - only being provided as module functions?
>>>>>
>>>>>
>>>> Hi Bob,
>>>> this is a very good question.
>>>> I think the answers are
>>>> a) historical reasons AND, more importantly, differing personal preferences
>>>> b) I would file  the missing data.diff() as a bug.
>>>
>>> It's not.
>>>
>> Care to elaborate?
>
> There is not supposed to be a one-to-one correspondence between the
> functions in numpy and the methods on an ndarray. There is some
> duplication between the two, but that is not a reason to make more
> duplication.
>
Are you saying the duplication is "just random" ?  It would be better
-- as in: principle of minimum surprise -- if there would be some sort
"reasonable set" of duplicates ....
If there are only a handful functions missing, why not try to make it complete.

( Again, a list of functions vs. methods on the wiki would clarify
what we are talking about ....)

Just thinking loudly of course. Don't take this as an offense .....
-Sebastian


From peridot.faceted at gmail.com  Tue Jun 24 04:36:32 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 24 Jun 2008 04:36:32 -0400
Subject: [Numpy-discussion] ndarray methods vs numpy module functions
In-Reply-To: <4860A351.2020305@cam.ac.uk>
References: <485F5F5C.5040903@cam.ac.uk>
	<bc657ead0806231610t6a3a9a4bw2c046b89c782b6f4@mail.gmail.com>
	<3d375d730806231614u3f7f2a43u225a60e679f1a5c0@mail.gmail.com>
	<48604157.4080207@gmail.com>
	<3d375d730806231811y55a47ab5m2d31caf2af2e017f@mail.gmail.com>
	<4860A351.2020305@cam.ac.uk>
Message-ID: <ce557a360806240136m745ca089o6026393d4bdf522a@mail.gmail.com>

2008/6/24 Bob Dowling <rjd4 at cam.ac.uk>:
>> There is not supposed to be a one-to-one correspondence between the
>> functions in numpy and the methods on an ndarray. There is some
>> duplication between the two, but that is not a reason to make more
>> duplication.
>
> I would make a plea for consistency, to start with.
>
> Those of us who write in an OO style are required to switch backwards
> and forwards between OO and not-OO, or to abandon OO altogether in our
> NumPy code.  Neither is an attractive option.
>
> The reason I tripped over this is that I am currently writing a course
> which introduces students to NumPy.  I am going to be asked this
> question from the audience.  As yet I don't have any answer except
> "history".

As a rule, I personally avoid methods whenever reasonable. The primary
reason is that the function versions generally work fine on lists,
giving my functions some extra genericity for free. I generally make
an exception only for attributes - X.shape, for example, rather than
np.shape(X).

It's true, it's a mess, but I'd just set down some simple rules that
work, and mention that some functions exist in other forms. There's
something to be said for rationalizing numpy's rules - or at least
writing them down! - but there's no need to use every version of every
function. And I believe they're all accessible as module-level
functions.

Anne


From david at ar.media.kyoto-u.ac.jp  Tue Jun 24 04:23:24 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 24 Jun 2008 17:23:24 +0900
Subject: [Numpy-discussion] ndarray methods vs numpy module functions
In-Reply-To: <bc657ead0806240043k54fe31dxcc7b27751e523616@mail.gmail.com>
References: <485F5F5C.5040903@cam.ac.uk>	<bc657ead0806231610t6a3a9a4bw2c046b89c782b6f4@mail.gmail.com>	<3d375d730806231614u3f7f2a43u225a60e679f1a5c0@mail.gmail.com>	<48604157.4080207@gmail.com>	<3d375d730806231811y55a47ab5m2d31caf2af2e017f@mail.gmail.com>
	<bc657ead0806240043k54fe31dxcc7b27751e523616@mail.gmail.com>
Message-ID: <4860AEFC.4090708@ar.media.kyoto-u.ac.jp>

Sebastian Haase wrote:
> Are you saying the duplication is "just random" ?  It would be better
> -- as in: principle of minimum surprise -- if there would be some sort
> "reasonable set" of duplicates ....

Yes it would be better. But how do you do it without breaking other
people code and avoiding duplication ? That's a trade-off: if we remove
say numpy.sum, people will complain that numpy.sum does not exist.
Adding all the functions in numpy arrays to be able to say a.foo instead
of foo(a) is not that great either. I think we can spend our time on
more interesting/important problems.

> If there are only a handful functions missing, why not try to make it complete.
Duplication is bad, and should be avoided as much as possible. a.foo vs
foo(a) does not sound like a good case to introduce more duplication ot me.

cheers,

David


From stefan at sun.ac.za  Tue Jun 24 04:39:17 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 24 Jun 2008 10:39:17 +0200
Subject: [Numpy-discussion] ndarray methods vs numpy module functions
In-Reply-To: <4860A351.2020305@cam.ac.uk>
References: <485F5F5C.5040903@cam.ac.uk>
	<bc657ead0806231610t6a3a9a4bw2c046b89c782b6f4@mail.gmail.com>
	<3d375d730806231614u3f7f2a43u225a60e679f1a5c0@mail.gmail.com>
	<48604157.4080207@gmail.com>
	<3d375d730806231811y55a47ab5m2d31caf2af2e017f@mail.gmail.com>
	<4860A351.2020305@cam.ac.uk>
Message-ID: <9457e7c80806240139x47bcac6as1c03fbae9ce015c5@mail.gmail.com>

Hi Bob

2008/6/24 Bob Dowling <rjd4 at cam.ac.uk>:
> I would make a plea for consistency, to start with.
>
> Those of us who write in an OO style are required to switch backwards
> and forwards between OO and not-OO, or to abandon OO altogether in our
> NumPy code.  Neither is an attractive option.

There are an infinite number of operations to be performed on arrays,
so the question becomes: which of those belong as members of the
class?  In my opinion, none do; we have them simply for backward
compatibility.

In general, my feeling (not the status quo) is to:

a) Use array methods for in-place operations and operations pertaining
specifically to ndarrays.  This would include `sort`, but not `sum` or
`dump`.
b) Use numpy functions for operations that copy the object, or do
calculations that yield new objects.

Even if you subscribe to such a rule, having x.sum() at hand is
convenient, so many people use it.  There's bound to be a big outcry
if we try to remove them now.  I'm not even sure most people would
agree with these guidelines.

Regards
St?fan


From dyamins at gmail.com  Tue Jun 24 11:45:08 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Tue, 24 Jun 2008 11:45:08 -0400
Subject: [Numpy-discussion] Sparse Matrices in Numpy -- (with eigenvalue
	algorithms if possible)
In-Reply-To: <e06186140806232312w3da432c5l3d9ecfd1f3e1f96d@mail.gmail.com>
References: <15e4667e0806231336s1c21f4b5xc604d3335fd09d87@mail.gmail.com>
	<d05265cb0806231808y7a5644dfs969a8c4e8861539@mail.gmail.com>
	<15e4667e0806232023m78eac31fhf185978223f092f1@mail.gmail.com>
	<e06186140806232312w3da432c5l3d9ecfd1f3e1f96d@mail.gmail.com>
Message-ID: <15e4667e0806240845l6998ebd2o2c5b7ec6cbca16f9@mail.gmail.com>

On Tue, Jun 24, 2008 at 2:12 AM, Charles R Harris <charlesr.harris at gmail.com>
wrote:

> Hi Dan,
>
> Did you finally get numpy installed as 64 bits?
>
> Chuck
>
> Hey, thanks for asking.   I did in fact get it installed -- at least, I
think so.

First, I had to built python in 64bit.  I did this using exactly the
instructions that mabshoff gave (they should be accessible in an email in
the archives of this mailinglist).  Then, when it came to installing numpy I
ran into a problem that at first I couldn't get numpy to import properly --
the multiarray.so module was build for 32-bit, not 64.  But then I followed
mabshoff's instructions again and deleted the build folder from the numpy
download; this time when I ran setup.py, multiarray.so appeared to have been
built properly.  I _think_ at least it's working -- when I run
   numpy.dtype(numpy.uintp).itemsize
the result is '8'.  Moreover, I can run numpy array functions efficienctly
on objects of size >> 3 GB, so it looks like it's working.

Btw, I had to do the same to get scipy to install properly as 64-bit -- at
least, I think it's been installed properly.    (Again,
scipy.dtype(scipy.uintp).itemsize returns 8 --  I assume that's diagnostic
here as well?)

It might be useful to have a repository of 64-bit builds and/or build
instructions.

Thanks,
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080624/f099252e/attachment.html>

From doutriaux1 at llnl.gov  Tue Jun 24 12:07:31 2008
From: doutriaux1 at llnl.gov (Charles Doutriaux)
Date: Tue, 24 Jun 2008 09:07:31 -0700
Subject: [Numpy-discussion] _md5 module?
Message-ID: <48611BC3.1090307@llnl.gov>

Hello on redhat enterprise 5 i get this with numpy 1.1 and python 2.5

Any idea if we can get around tihs? short of rebuilding python with md5 support i guess.
Shouldn't numpy catch that before building?

C.


"/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/__init__.py", line 42, in <module>
    import add_newdocs
  File
"/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/add_newdocs.py", line 5, in <module>
    from lib import add_newdoc
  File
"/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/lib/__init__.py", line 18, in <module>
    from io import *
  File
"/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/lib/io.py", line 16, in <module>
    from _datasource import DataSource
  File
"/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/lib/_datasource.py", line 42, in <module>
    from urllib2 import urlopen, URLError
  File
"/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/urllib2.py",
line 91, in <module>
    import hashlib
  File
"/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/hashlib.py",
line 133, in <module>
    md5 = __get_builtin_constructor('md5')
  File
"/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/hashlib.py",
line 60, in __get_builtin_constructor
    import _md5
ImportError: No module named _md5



From charlesr.harris at gmail.com  Tue Jun 24 12:15:47 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 24 Jun 2008 10:15:47 -0600
Subject: [Numpy-discussion] Sparse Matrices in Numpy -- (with eigenvalue
	algorithms if possible)
In-Reply-To: <15e4667e0806240845l6998ebd2o2c5b7ec6cbca16f9@mail.gmail.com>
References: <15e4667e0806231336s1c21f4b5xc604d3335fd09d87@mail.gmail.com>
	<d05265cb0806231808y7a5644dfs969a8c4e8861539@mail.gmail.com>
	<15e4667e0806232023m78eac31fhf185978223f092f1@mail.gmail.com>
	<e06186140806232312w3da432c5l3d9ecfd1f3e1f96d@mail.gmail.com>
	<15e4667e0806240845l6998ebd2o2c5b7ec6cbca16f9@mail.gmail.com>
Message-ID: <e06186140806240915r2c57b756qda7d5505e3cf3ef4@mail.gmail.com>

On Tue, Jun 24, 2008 at 9:45 AM, Dan Yamins <dyamins at gmail.com> wrote:

>
>
> On Tue, Jun 24, 2008 at 2:12 AM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>> Hi Dan,
>>
>> Did you finally get numpy installed as 64 bits?
>>
>> Chuck
>>
>> Hey, thanks for asking.   I did in fact get it installed -- at least, I
> think so.
>
> First, I had to built python in 64bit.  I did this using exactly the
> instructions that mabshoff gave (they should be accessible in an email in
> the archives of this mailinglist).  Then, when it came to installing numpy I
> ran into a problem that at first I couldn't get numpy to import properly --
> the multiarray.so module was build for 32-bit, not 64.  But then I followed
> mabshoff's instructions again and deleted the build folder from the numpy
> download; this time when I ran setup.py, multiarray.so appeared to have been
> built properly.  I _think_ at least it's working -- when I run
>    numpy.dtype(numpy.uintp).itemsize
> the result is '8'.  Moreover, I can run numpy array functions efficienctly
> on objects of size >> 3 GB, so it looks like it's working.
>
> Btw, I had to do the same to get scipy to install properly as 64-bit -- at
> least, I think it's been installed properly.    (Again,
> scipy.dtype(scipy.uintp).itemsize returns 8 --  I assume that's diagnostic
> here as well?)
>

Looks like you got it right. The reason you needed to delete the build
folder was that setup didn't know that python had changed and, since nothing
else had changed either, it just copied over the old build.


> It might be useful to have a repository of 64-bit builds and/or build
> instructions.
>

We should probably put mabshoff's instructions for building 64-bit python on
the MAC somewhere in the wiki.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080624/f329c2b2/attachment.html>

From charlesr.harris at gmail.com  Tue Jun 24 12:30:44 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 24 Jun 2008 10:30:44 -0600
Subject: [Numpy-discussion] _md5 module?
In-Reply-To: <48611BC3.1090307@llnl.gov>
References: <48611BC3.1090307@llnl.gov>
Message-ID: <e06186140806240930k7afcc7cfn17f593f3ec2bd2df@mail.gmail.com>

On Tue, Jun 24, 2008 at 10:07 AM, Charles Doutriaux <doutriaux1 at llnl.gov>
wrote:

> Hello on redhat enterprise 5 i get this with numpy 1.1 and python 2.5
>
> Any idea if we can get around tihs? short of rebuilding python with md5
> support i guess.
> Shouldn't numpy catch that before building?
>
> C.
>
>
> "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/__init__.py",
> line 42, in <module>
>    import add_newdocs
>  File
> "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/add_newdocs.py",
> line 5, in <module>
>    from lib import add_newdoc
>  File
> "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/lib/__init__.py",
> line 18, in <module>
>    from io import *
>  File
> "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/lib/io.py",
> line 16, in <module>
>    from _datasource import DataSource
>  File
> "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/lib/_datasource.py",
> line 42, in <module>
>    from urllib2 import urlopen, URLError
>  File
> "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/urllib2.py",
> line 91, in <module>
>    import hashlib
>  File
> "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/hashlib.py",
> line 133, in <module>
>    md5 = __get_builtin_constructor('md5')
>  File
> "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/hashlib.py",
> line 60, in __get_builtin_constructor
>    import _md5
> ImportError: No module named _md5
>
>
Python2.5 should come with md5, at least it does on fedora 8:
/usr/lib/python2.5/lib-dynload/_md5module.so. It is also deprecated as of
2.5, so we might need some modifications anyway.

Are you using the python built by CDAT? I found the easiest way to get CDAT
4.3 working was to copy over the relevant site-packages from the CDAT built
python to the system python (of the same version). You could maybe go the
other way also.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080624/0e22d7d7/attachment.html>

From doutriaux1 at llnl.gov  Tue Jun 24 13:18:41 2008
From: doutriaux1 at llnl.gov (Charles Doutriaux)
Date: Tue, 24 Jun 2008 10:18:41 -0700
Subject: [Numpy-discussion] _md5 module?
In-Reply-To: <e06186140806240930k7afcc7cfn17f593f3ec2bd2df@mail.gmail.com>
References: <48611BC3.1090307@llnl.gov>
	<e06186140806240930k7afcc7cfn17f593f3ec2bd2df@mail.gmail.com>
Message-ID: <48612C71.5050704@llnl.gov>

Hi Charles,

Yes.. unfortunately we have to be able to build out of the box, we can't 
rely on much anything (Except compiler and X) to be there already...

It would be much easier for CDAT to simply say getpython 2.5 and numpy 
1.1 and all the externals, and if you ever get it together come back see 
us :)

Unfortunately i don't think we'd have much success that way :)

I'll keep looking... I don't understand why python didn't build with 
md5. Never happened before...

I guess it's more of a python issue than numpy... But a syou said it's 
deprecated so you might as well want to replace it.

C.

Charles R Harris wrote:
>
>
> On Tue, Jun 24, 2008 at 10:07 AM, Charles Doutriaux 
> <doutriaux1 at llnl.gov <mailto:doutriaux1 at llnl.gov>> wrote:
>
>     Hello on redhat enterprise 5 i get this with numpy 1.1 and python 2.5
>
>     Any idea if we can get around tihs? short of rebuilding python
>     with md5 support i guess.
>     Shouldn't numpy catch that before building?
>
>     C.
>
>
>     "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/__init__.py",
>     line 42, in <module>
>        import add_newdocs
>      File
>     "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/add_newdocs.py",
>     line 5, in <module>
>        from lib import add_newdoc
>      File
>     "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/lib/__init__.py",
>     line 18, in <module>
>        from io import *
>      File
>     "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/lib/io.py",
>     line 16, in <module>
>        from _datasource import DataSource
>      File
>     "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/site-packages/numpy/lib/_datasource.py",
>     line 42, in <module>
>        from urllib2 import urlopen, URLError
>      File
>     "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/urllib2.py",
>     line 91, in <module>
>        import hashlib
>      File
>     "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/hashlib.py",
>     line 133, in <module>
>        md5 = __get_builtin_constructor('md5')
>      File
>     "/usr/local/cdat/release/5.0e/5.0.0.alpha7/lib/python2.5/hashlib.py",
>     line 60, in __get_builtin_constructor
>        import _md5
>     ImportError: No module named _md5
>
>
> Python2.5 should come with md5, at least it does on fedora 8: 
> /usr/lib/python2.5/lib-dynload/_md5module.so. It is also deprecated as 
> of 2.5, so we might need some modifications anyway.
>
> Are you using the python built by CDAT? I found the easiest way to get 
> CDAT 4.3 working was to copy over the relevant site-packages from the 
> CDAT built python to the system python (of the same version). You 
> could maybe go the other way also.
>
> Chuck
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From robert.kern at gmail.com  Tue Jun 24 13:30:06 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 24 Jun 2008 12:30:06 -0500
Subject: [Numpy-discussion] ndarray methods vs numpy module functions
In-Reply-To: <4860A351.2020305@cam.ac.uk>
References: <485F5F5C.5040903@cam.ac.uk>
	<bc657ead0806231610t6a3a9a4bw2c046b89c782b6f4@mail.gmail.com>
	<3d375d730806231614u3f7f2a43u225a60e679f1a5c0@mail.gmail.com>
	<48604157.4080207@gmail.com>
	<3d375d730806231811y55a47ab5m2d31caf2af2e017f@mail.gmail.com>
	<4860A351.2020305@cam.ac.uk>
Message-ID: <3d375d730806241030t4639384bu55716e2b8306420d@mail.gmail.com>

On Tue, Jun 24, 2008 at 02:33, Bob Dowling <rjd4 at cam.ac.uk> wrote:
>> There is not supposed to be a one-to-one correspondence between the
>> functions in numpy and the methods on an ndarray. There is some
>> duplication between the two, but that is not a reason to make more
>> duplication.
>
> I would make a plea for consistency, to start with.

It's way too late to make changes like this.

> Those of us who write in an OO style are required to switch backwards
> and forwards between OO and not-OO, or to abandon OO altogether in our
> NumPy code.  Neither is an attractive option.

OO does not mean "always use methods."

> The reason I tripped over this is that I am currently writing a course
> which introduces students to NumPy.  I am going to be asked this
> question from the audience.  As yet I don't have any answer except
> "history".

Well, "history," usually along with "it seemed like a good idea at the
time," are valid reasons for things to continue to exist in any
nontrivial software project with a userbase. Your students will need
to learn this if they use software.

If you want a slightly better answer, the implementation of many of
the C functions were somewhat easier to do as methods on ndarray than
separate functions particularly since numpy.ndarray has subclasses.
The functions could then be implemented similar to the following:

  def myfunc(a):
      return asanyarray(a).myfunc()

One thing you will notice about numpy.diff() is that it is a pure
Python function rather than a C function, so it's certainly not going
to be a method on ndarray.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From robert.kern at gmail.com  Tue Jun 24 13:40:45 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 24 Jun 2008 12:40:45 -0500
Subject: [Numpy-discussion] ndarray methods vs numpy module functions
In-Reply-To: <bc657ead0806240043k54fe31dxcc7b27751e523616@mail.gmail.com>
References: <485F5F5C.5040903@cam.ac.uk>
	<bc657ead0806231610t6a3a9a4bw2c046b89c782b6f4@mail.gmail.com>
	<3d375d730806231614u3f7f2a43u225a60e679f1a5c0@mail.gmail.com>
	<48604157.4080207@gmail.com>
	<3d375d730806231811y55a47ab5m2d31caf2af2e017f@mail.gmail.com>
	<bc657ead0806240043k54fe31dxcc7b27751e523616@mail.gmail.com>
Message-ID: <3d375d730806241040h45d66286hd075b7a4f0064018@mail.gmail.com>

On Tue, Jun 24, 2008 at 02:43, Sebastian Haase <haase at msg.ucsf.edu> wrote:
> On Tue, Jun 24, 2008 at 3:11 AM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Mon, Jun 23, 2008 at 19:35, Ryan May <rmay31 at gmail.com> wrote:
>>> Robert Kern wrote:
>>>> On Mon, Jun 23, 2008 at 18:10, Sebastian Haase <haase at msg.ucsf.edu> wrote:
>>>>> On Mon, Jun 23, 2008 at 10:31 AM, Bob Dowling <rjd4 at cam.ac.uk> wrote:
>>>>>> [ I'm new here and this has the feel of an FAQ but I couldn't find
>>>>>> anything at http://www.scipy.org/FAQ .  If I should have looked
>>>>>> somewhere else a URL would be gratefully received. ]
>>>>>>
>>>>>>
>>>>>> What's the reasoning behind functions like sum() and cumsum() being
>>>>>> provided both as module functions (numpy.sum(data, axis=1)) and as
>>>>>> object methods (data.sum(axis=1)) but other functions - and I stumbled
>>>>>> over diff() - only being provided as module functions?
>>>>>>
>>>>>>
>>>>> Hi Bob,
>>>>> this is a very good question.
>>>>> I think the answers are
>>>>> a) historical reasons AND, more importantly, differing personal preferences
>>>>> b) I would file  the missing data.diff() as a bug.
>>>>
>>>> It's not.
>>>>
>>> Care to elaborate?
>>
>> There is not supposed to be a one-to-one correspondence between the
>> functions in numpy and the methods on an ndarray. There is some
>> duplication between the two, but that is not a reason to make more
>> duplication.
>>
> Are you saying the duplication is "just random" ?

No. If you want a clearer (but still imperfect) dividing line is that
all of the methods are implemented in C. numpy.diff(), for example, is
not. A lot of the C functions in Numeric's core (but not FFT, SVD,
etc.) got moved into methods, partly for implementation reasons
(ndarray is subclassable now, so methods are easier to make generic),
and partly for "it seemed like a good idea at the time." We couldn't
remove the functions, then, if we wanted any kind of continuity with
Numeric, and we certainly can't now.

> It would be better
> -- as in: principle of minimum surprise -- if there would be some sort
> "reasonable set" of duplicates ....
> If there are only a handful functions missing, why not try to make it complete.

There aren't.

> ( Again, a list of functions vs. methods on the wiki would clarify
> what we are talking about ....)

Go ahead.

> Just thinking loudly of course. Don't take this as an offense .....

It's not that you're being offensive, but you are kicking up dust on
an old argument that was settled before 1.0, when it actually
mattered.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From pgmdevlist at gmail.com  Tue Jun 24 14:03:19 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 24 Jun 2008 14:03:19 -0400
Subject: [Numpy-discussion] Keywords in wrapped functions
Message-ID: <200806241403.19747.pgmdevlist@gmail.com>

All,
Sorry to bumpt the post, accept my apologies for my rudeness, but I'm 
curious...
So, let me rephrase my question:

Many numpy functions (min, max, sum...) based on ndarray methods have a 
construction of the style
#-----------------------------------------------
def amin(a, axis=None, out=None):
    try:
        amin = a.min
    except AttributeError:
        return _wrapit(a, 'min', axis, out)
    return amin(axis, out)
#-----------------------------------------------

I'm wondering why the initial keywords arguments are transformed into 
positional arguments in the return. In the previous example, I would expect a 
line like
return amin(axis=axis, out=out)

[Actually, I would even prefer a 
return amin(axis=axis, out=out, **other_optional_parameters), which would 
permit to add some extra parameters such as a "fill_value", when dealing with 
masked arrays ?]

Is this for efficiency purposes ?






From charlesr.harris at gmail.com  Tue Jun 24 14:06:47 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 24 Jun 2008 12:06:47 -0600
Subject: [Numpy-discussion] Request for clarification from Travis
Message-ID: <e06186140806241106h1a66f957l178420fa06256aee@mail.gmail.com>

Hi Travis,

Could you expand on your thinking  concerning NotImplemented? The relevant
code is:

    /*
     * FAIL with NotImplemented if the other object has
     * the __r<op>__ method and has __array_priority__ as
     * an attribute (signalling it can handle ndarray's)
     * and is not already an ndarray
     */
    if ((arg_types[1] == PyArray_OBJECT) &&                         \
        (loop->ufunc->nin==2) && (loop->ufunc->nout == 1)) {
        PyObject *_obj = PyTuple_GET_ITEM(args, 1);
        if (!PyArray_CheckExact(_obj) &&                        \
            PyObject_HasAttrString(_obj, "__array_priority__") && \
            _has_reflected_op(_obj, loop->ufunc->name)) {
            loop->notimplemented = 1;
            return nargs;
        }
    }

And the check is made after the needed promotions have been determined and
stored in arg_types. Consequently if a type is promoted to an object array,
as happens for the shift operations with floating scalars, the rest of the
conditions will be checked and pass because numpy scalar types aren't
arrays. I wonder if that is what you intended here? The reason I ask is that
this determination should be associated with the standard slot functions
when the call is made and I don't want to worry about type promotions.
Further, I wonder why you want this return before the attempt to run the
function with the current arguments is made. So it would be very helpful if
you could clarify what you aim to achieve here.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080624/f676f4e3/attachment.html>

From robert.kern at gmail.com  Tue Jun 24 14:11:12 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 24 Jun 2008 13:11:12 -0500
Subject: [Numpy-discussion] Keywords in wrapped functions
In-Reply-To: <200806241403.19747.pgmdevlist@gmail.com>
References: <200806241403.19747.pgmdevlist@gmail.com>
Message-ID: <3d375d730806241111r9a0593nfd0adb2aee422d77@mail.gmail.com>

On Tue, Jun 24, 2008 at 13:03, Pierre GM <pgmdevlist at gmail.com> wrote:
> All,
> Sorry to bumpt the post, accept my apologies for my rudeness, but I'm
> curious...
> So, let me rephrase my question:
>
> Many numpy functions (min, max, sum...) based on ndarray methods have a
> construction of the style
> #-----------------------------------------------
> def amin(a, axis=None, out=None):
>    try:
>        amin = a.min
>    except AttributeError:
>        return _wrapit(a, 'min', axis, out)
>    return amin(axis, out)
> #-----------------------------------------------
>
> I'm wondering why the initial keywords arguments are transformed into
> positional arguments in the return. In the previous example, I would expect a
> line like
> return amin(axis=axis, out=out)
>
> [Actually, I would even prefer a
> return amin(axis=axis, out=out, **other_optional_parameters), which would
> permit to add some extra parameters such as a "fill_value", when dealing with
> masked arrays ?]
>
> Is this for efficiency purposes ?

Most likely, the author just didn't realize that it might matter. Go
ahead with your changes.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From pgmdevlist at gmail.com  Tue Jun 24 14:20:02 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 24 Jun 2008 14:20:02 -0400
Subject: [Numpy-discussion] Keywords in wrapped functions
In-Reply-To: <3d375d730806241111r9a0593nfd0adb2aee422d77@mail.gmail.com>
References: <200806241403.19747.pgmdevlist@gmail.com>
	<3d375d730806241111r9a0593nfd0adb2aee422d77@mail.gmail.com>
Message-ID: <200806241420.02994.pgmdevlist@gmail.com>

On Tuesday 24 June 2008 14:11:12 Robert Kern wrote:
> On Tue, Jun 24, 2008 at 13:03, Pierre GM <pgmdevlist at gmail.com> wrote:

> > Is this for efficiency purposes ?
>
> Most likely, the author just didn't realize that it might matter. Go
> ahead with your changes.

OK, great. Thanks a lot!


From oliphant at enthought.com  Tue Jun 24 14:46:34 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Tue, 24 Jun 2008 13:46:34 -0500
Subject: [Numpy-discussion] Request for clarification from Travis
In-Reply-To: <e06186140806241106h1a66f957l178420fa06256aee@mail.gmail.com>
References: <e06186140806241106h1a66f957l178420fa06256aee@mail.gmail.com>
Message-ID: <4861410A.5020501@enthought.com>

Charles R Harris wrote:
> Hi Travis,
>
> Could you expand on your thinking  concerning NotImplemented? The 
> relevant code is:
>
>     /*
>      * FAIL with NotImplemented if the other object has
>      * the __r<op>__ method and has __array_priority__ as
>      * an attribute (signalling it can handle ndarray's)
>      * and is not already an ndarray
>      */
>     if ((arg_types[1] == PyArray_OBJECT) &&                         \
>         (loop->ufunc->nin==2) && (loop->ufunc->nout == 1)) {
>         PyObject *_obj = PyTuple_GET_ITEM(args, 1);
>         if (!PyArray_CheckExact(_obj) &&                        \
>             PyObject_HasAttrString(_obj, "__array_priority__") && \
>             _has_reflected_op(_obj, loop->ufunc->name)) {
>             loop->notimplemented = 1;
>             return nargs;
>         }
>     }
>
> And the check is made after the needed promotions have been determined 
> and stored in arg_types. Consequently if a type is promoted to an 
> object array, as happens for the shift operations with floating 
> scalars, the rest of the conditions will be checked and pass because 
> numpy scalar types aren't arrays. I wonder if that is what you 
> intended here? 
I'm not sure, I don't understand what problem you are trying to solve.

> The reason I ask is that this determination should be associated with 
> the standard slot functions when the call is made and I don't want to 
> worry about type promotions. Further, I wonder why you want this 
> return before the attempt to run the function with the current 
> arguments is made. So it would be very helpful if you could clarify 
> what you aim to achieve here.
This is to ensure that object arrays get a chance to run their own code 
if they have the required right-hand special name (e.g. __radd__).

The problem is that NumPy is very aggressive and tries to handle 
everything so if I let the code get past this point, it would run 
successfully as an object array which is not what is necessarily wanted

-Travis



From oliphant at enthought.com  Tue Jun 24 14:48:04 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Tue, 24 Jun 2008 13:48:04 -0500
Subject: [Numpy-discussion] Keywords in wrapped functions
In-Reply-To: <200806241403.19747.pgmdevlist@gmail.com>
References: <200806241403.19747.pgmdevlist@gmail.com>
Message-ID: <48614164.20501@enthought.com>

Pierre GM wrote:
> All,
> Sorry to bumpt the post, accept my apologies for my rudeness, but I'm 
> curious...
> So, let me rephrase my question:
>
> Many numpy functions (min, max, sum...) based on ndarray methods have a 
> construction of the style
> #-----------------------------------------------
> def amin(a, axis=None, out=None):
>     try:
>         amin = a.min
>     except AttributeError:
>         return _wrapit(a, 'min', axis, out)
>     return amin(axis, out)
> #-----------------------------------------------
>
> I'm wondering why the initial keywords arguments are transformed into 
> positional arguments in the return. In the previous example, I would expect a 
> line like
> return amin(axis=axis, out=out)
>
> [Actually, I would even prefer a 
> return amin(axis=axis, out=out, **other_optional_parameters), which would 
> permit to add some extra parameters such as a "fill_value", when dealing with 
> masked arrays ?]
>
> Is this for efficiency purposes ?
>
>
>   
The keyword arguments to the methods were added later and so positional 
argument were initially necessary.  I don't think they are now, but I 
have not verified that.

-Travis



From thrabe at burnham.org  Tue Jun 24 14:59:40 2008
From: thrabe at burnham.org (Thomas Hrabe)
Date: Tue, 24 Jun 2008 11:59:40 -0700
Subject: [Numpy-discussion] embedded arrays- still an issue
References: <A7E1AE993C63414B8B2F762E13BF4D898E4E9F@MAIL.burnham.org>
	<3d375d730806061527w42fccb9dt8e5cc857934d4196@mail.gmail.com>
Message-ID: <A7E1AE993C63414B8B2F762E13BF4D898E4EA9@MAIL.burnham.org>

Hi all, 


I finally managed to come back to my memory troubles which I have described recently to this group (see below). Here is the requested backtrace of my program:

#0  0x94659b88 in strlen ()
#1  0x004437b8 in PyString_FromFormatV (format=0x4f2360 "'%.50s' object has no attribute '%.400s'", vargs=0xbfffee00 "") at /Users/ronald/r252/Objects/stringobject.c:202
#2  0x004a57e8 in PyErr_Format (exception=0x505fb4, format=0xbfffee00 "") at /Users/ronald/r252/Python/errors.c:522
#3  0x00432430 in PyObject_GetAttr (v=0x80ef40, name=0xc5278) at /Users/ronald/r252/Objects/object.c:1130
#4  0x004324b8 in PyObject_GetAttrString (v=0x80ef40, name=0x0) at /Users/ronald/r252/Objects/object.c:1069
#5  0x00b47660 in PyArray_FromStructInterface ()
#6  0x00b47938 in PyArray_FromAny ()
#7  0x00b72fb4 in array_richcompare ()
#8  0x004341f0 in PyObject_RichCompare (v=0x80ef40, w=0x8b9440, op=2) at /Users/ronald/r252/Objects/object.c:874
#9  0x00490d20 in PyEval_EvalFrameEx (f=0x8b9090, throwflag=9146828) at /Users/ronald/r252/Python/ceval.c:3989
#10 0x00493e50 in PyEval_EvalCodeEx (co=0xf4d578, globals=0xbfffee00, locals=0x4437b8, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at /Users/ronald/r252/Python/ceval.c:2836
#11 0x00493ff0 in PyEval_EvalCode (co=0x0, globals=0xbfffee00, locals=0x0) at /Users/ronald/r252/Python/ceval.c:494
#12 0x004b9b24 in PyRun_InteractiveOneFlags (fp=0x0, filename=0x4f88f0 "<stdin>", flags=0xbffff448) at /Users/ronald/r252/Python/pythonrun.c:1273
#13 0x004b9d30 in PyRun_InteractiveLoopFlags (fp=0xa08bf374, filename=0x4f88f0 "<stdin>", flags=0xbffff448) at /Users/ronald/r252/Python/pythonrun.c:723
#14 0x004ba3f0 in PyRun_AnyFileExFlags (fp=0xa08bf374, filename=0x4f88f0 "<stdin>", closeit=0, flags=0xbffff448) at /Users/ronald/r252/Python/pythonrun.c:692
#15 0x004c9a9c in Py_Main (argc=1, argv=0xbffff5d8) at /Users/ronald/r252/Modules/main.c:523
#16 0x000019bc in ?? ()
#17 0x000016c0 in ?? ()
(gdb) 
in /Users/ronald/r252/Objects/stringobject.c

Please let me know if there are any ways how to go on from this point. I am new into this kind of debugging...

Thank you in advance for your help,
Thomas

-----Urspr?ngliche Nachricht-----
Von: numpy-discussion-bounces at scipy.org im Auftrag von Robert Kern
Gesendet: Fr 06.06.2008 15:27
An: Discussion of Numerical Python
Betreff: Re: [Numpy-discussion] embedded arrays
 
On Fri, Jun 6, 2008 at 17:10, Thomas Hrabe <thrabe at burnham.org> wrote:
> Hi all,
>
> while writing a extension module in C++ for python & numpy, I find a strange
> error.
>
> I can send and retrieve numpy arrays to and from my module.
> But python stops if I do the following:
>
> a = numpy.array([[1.1,2,3],[4,5,6]])
> PM.put(a,'a')   //send a to the module
> b = PM.get('a') //get a identical copy from the module
> print b
> array([[ 1.1,  2. ,  3. ],
>        [ 4. ,  5. ,  6. ]])
>
> b.__class__
> Out[36]: <type 'numpy.ndarray'>
>
>
> perfect, until
> a == b
>
> the interpreter does not continue from here...
> I can add values to to b, everything, but a == b simply crashes ...?
>
> Does anybody have a clue for this problem?

Not really. It probably depends on some details with your interfacing.
Since we don't have access to your code, we don't have much to go on.
You might have buggy reference counting or perhaps you gave the numpy
ndarray ownership of the array's memory when it actually shouldn't.
Memory issues can be a bit finicky where everything will work for a
while, then crash.

Try running your program under a C debugger like gdb so we can get a
backtrace. That might give us some more ideas about exactly where
problems are occurring.




-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 4731 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080624/9497ce5d/attachment.bin>

From haase at msg.ucsf.edu  Tue Jun 24 15:28:49 2008
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue, 24 Jun 2008 21:28:49 +0200
Subject: [Numpy-discussion] ndarray methods vs numpy module functions
In-Reply-To: <3d375d730806241040h45d66286hd075b7a4f0064018@mail.gmail.com>
References: <485F5F5C.5040903@cam.ac.uk>
	<bc657ead0806231610t6a3a9a4bw2c046b89c782b6f4@mail.gmail.com>
	<3d375d730806231614u3f7f2a43u225a60e679f1a5c0@mail.gmail.com>
	<48604157.4080207@gmail.com>
	<3d375d730806231811y55a47ab5m2d31caf2af2e017f@mail.gmail.com>
	<bc657ead0806240043k54fe31dxcc7b27751e523616@mail.gmail.com>
	<3d375d730806241040h45d66286hd075b7a4f0064018@mail.gmail.com>
Message-ID: <bc657ead0806241228u1dee32d2ia6d38c7ffc3843b6@mail.gmail.com>

On Tue, Jun 24, 2008 at 7:40 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Tue, Jun 24, 2008 at 02:43, Sebastian Haase <haase at msg.ucsf.edu> wrote:
>> On Tue, Jun 24, 2008 at 3:11 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>> On Mon, Jun 23, 2008 at 19:35, Ryan May <rmay31 at gmail.com> wrote:
>>>> Robert Kern wrote:
>>>>> On Mon, Jun 23, 2008 at 18:10, Sebastian Haase <haase at msg.ucsf.edu> wrote:
>>>>>> On Mon, Jun 23, 2008 at 10:31 AM, Bob Dowling <rjd4 at cam.ac.uk> wrote:
>>>>>>> [ I'm new here and this has the feel of an FAQ but I couldn't find
>>>>>>> anything at http://www.scipy.org/FAQ .  If I should have looked
>>>>>>> somewhere else a URL would be gratefully received. ]
>>>>>>>
>>>>>>>
>>>>>>> What's the reasoning behind functions like sum() and cumsum() being
>>>>>>> provided both as module functions (numpy.sum(data, axis=1)) and as
>>>>>>> object methods (data.sum(axis=1)) but other functions - and I stumbled
>>>>>>> over diff() - only being provided as module functions?
>>>>>>>
>>>>>>>
>>>>>> Hi Bob,
>>>>>> this is a very good question.
>>>>>> I think the answers are
>>>>>> a) historical reasons AND, more importantly, differing personal preferences
>>>>>> b) I would file  the missing data.diff() as a bug.
>>>>>
>>>>> It's not.
>>>>>
>>>> Care to elaborate?
>>>
>>> There is not supposed to be a one-to-one correspondence between the
>>> functions in numpy and the methods on an ndarray. There is some
>>> duplication between the two, but that is not a reason to make more
>>> duplication.
>>>
>> Are you saying the duplication is "just random" ?
>
> No. If you want a clearer (but still imperfect) dividing line is that
> all of the methods are implemented in C. numpy.diff(), for example, is
> not. A lot of the C functions in Numeric's core (but not FFT, SVD,
> etc.) got moved into methods, partly for implementation reasons
> (ndarray is subclassable now, so methods are easier to make generic),
> and partly for "it seemed like a good idea at the time." We couldn't
> remove the functions, then, if we wanted any kind of continuity with
> Numeric, and we certainly can't now.
>
>> It would be better
>> -- as in: principle of minimum surprise -- if there would be some sort
>> "reasonable set" of duplicates ....
>> If there are only a handful functions missing, why not try to make it complete.
>
> There aren't.
>
>> ( Again, a list of functions vs. methods on the wiki would clarify
>> what we are talking about ....)
>
> Go ahead.
>
>> Just thinking loudly of course. Don't take this as an offense .....
>
> It's not that you're being offensive, but you are kicking up dust on
> an old argument that was settled before 1.0, when it actually
> mattered.

Just for the record: I like it the way it is. I did follow the
discussion at the time, and while there was a real (historical) need
to keep functions, there were arguments for supporting a two paradigms
and thus also having methods. "Numpy serves both people coming from
the Matlab/IDL community and people coming from (strict /
method-based) OO programming".

Regarding the OP, if he stumbled over this, like probably many other
"newcomers", it should go into the FAQ section. (at
http://www.scipy.org/FAQ )
("For implementation reasons some operations are only available as
functions or as methods respectively")

Cheers,
- Sebastian Haase


From charlesr.harris at gmail.com  Tue Jun 24 15:31:50 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 24 Jun 2008 13:31:50 -0600
Subject: [Numpy-discussion] Request for clarification from Travis
In-Reply-To: <4861410A.5020501@enthought.com>
References: <e06186140806241106h1a66f957l178420fa06256aee@mail.gmail.com>
	<4861410A.5020501@enthought.com>
Message-ID: <e06186140806241231i15a03f31ra39ae40b2b1293e2@mail.gmail.com>

On Tue, Jun 24, 2008 at 12:46 PM, Travis E. Oliphant <oliphant at enthought.com>
wrote:

> Charles R Harris wrote:
> > Hi Travis,
> >
> > Could you expand on your thinking  concerning NotImplemented? The
> > relevant code is:
> >
> >     /*
> >      * FAIL with NotImplemented if the other object has
> >      * the __r<op>__ method and has __array_priority__ as
> >      * an attribute (signalling it can handle ndarray's)
> >      * and is not already an ndarray
> >      */
> >     if ((arg_types[1] == PyArray_OBJECT) &&                         \
> >         (loop->ufunc->nin==2) && (loop->ufunc->nout == 1)) {
> >         PyObject *_obj = PyTuple_GET_ITEM(args, 1);
> >         if (!PyArray_CheckExact(_obj) &&                        \
> >             PyObject_HasAttrString(_obj, "__array_priority__") && \
> >             _has_reflected_op(_obj, loop->ufunc->name)) {
> >             loop->notimplemented = 1;
> >             return nargs;
> >         }
> >     }
> >
> > And the check is made after the needed promotions have been determined
> > and stored in arg_types. Consequently if a type is promoted to an
> > object array, as happens for the shift operations with floating
> > scalars, the rest of the conditions will be checked and pass because
> > numpy scalar types aren't arrays. I wonder if that is what you
> > intended here?
> I'm not sure, I don't understand what problem you are trying to solve.
>

1) NotImplemented is associated to specific types, hence the CHECKABLE flag
on the type.
2) It is a message to the python interpreter, not to the user.
3) It is associated with the slot functions of numeric types, not with
general functions.

When python encounters something like a + b, it looks at the CHECKABLE flag
on a. If set, it calls a.__add__ with b as is, not attempting to promote the
type. If that fails, a.__add__ returns NotImplemented, telling the
interpreter to do something, probably try the b.__radd__. If that fails it
raises an error.

If the *user* ever sees NotImplemented, it is a bug. Some of the numpy
ufuncs exhibit this bug and we should fix it.


> The problem is that NumPy is very aggressive and tries to handle
> everything so if I let the code get past this point, it would run
> successfully as an object array which is not what is necessarily wanted
>

The problem is that numpy has a *bug*. I am trying to fix it and I want your
help figuring out how to do so while preserving the behavior you had in
mind. So if you could point out specific cases you were thinking of it would
help me with this.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080624/d274d26c/attachment.html>

From oliphant at enthought.com  Tue Jun 24 15:41:39 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Tue, 24 Jun 2008 14:41:39 -0500
Subject: [Numpy-discussion] Request for clarification from Travis
In-Reply-To: <e06186140806241231i15a03f31ra39ae40b2b1293e2@mail.gmail.com>
References: <e06186140806241106h1a66f957l178420fa06256aee@mail.gmail.com>	<4861410A.5020501@enthought.com>
	<e06186140806241231i15a03f31ra39ae40b2b1293e2@mail.gmail.com>
Message-ID: <48614DF3.4020308@enthought.com>


> The problem is that numpy has a *bug*. I am trying to fix it and I 
> want your help figuring out how to do so while preserving the behavior 
> you had in mind. So if you could point out specific cases you were 
> thinking of it would help me with this.
This is a hack to support matrices and other subclasses of ndarray like 
masked arrays.

-Travis



From charlesr.harris at gmail.com  Tue Jun 24 17:39:58 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 24 Jun 2008 15:39:58 -0600
Subject: [Numpy-discussion] Request for clarification from Travis
In-Reply-To: <48614DF3.4020308@enthought.com>
References: <e06186140806241106h1a66f957l178420fa06256aee@mail.gmail.com>
	<4861410A.5020501@enthought.com>
	<e06186140806241231i15a03f31ra39ae40b2b1293e2@mail.gmail.com>
	<48614DF3.4020308@enthought.com>
Message-ID: <e06186140806241439v1ec439bm9562f639caf0dfde@mail.gmail.com>

On Tue, Jun 24, 2008 at 1:41 PM, Travis E. Oliphant <oliphant at enthought.com>
wrote:

>
> > The problem is that numpy has a *bug*. I am trying to fix it and I
> > want your help figuring out how to do so while preserving the behavior
> > you had in mind. So if you could point out specific cases you were
> > thinking of it would help me with this.
> This is a hack to support matrices and other subclasses of ndarray like
> masked arrays.
>

That's what I was thinking. Do you have a test case? One that failed without
the NotImplemented return would be very helpful.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080624/49db7230/attachment.html>

From oliphant at enthought.com  Tue Jun 24 22:40:32 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Tue, 24 Jun 2008 21:40:32 -0500
Subject: [Numpy-discussion] Request for clarification from Travis
In-Reply-To: <e06186140806241439v1ec439bm9562f639caf0dfde@mail.gmail.com>
References: <e06186140806241106h1a66f957l178420fa06256aee@mail.gmail.com>	<4861410A.5020501@enthought.com>	<e06186140806241231i15a03f31ra39ae40b2b1293e2@mail.gmail.com>	<48614DF3.4020308@enthought.com>
	<e06186140806241439v1ec439bm9562f639caf0dfde@mail.gmail.com>
Message-ID: <4861B020.8080101@enthought.com>

Charles R Harris wrote:
>
>
> On Tue, Jun 24, 2008 at 1:41 PM, Travis E. Oliphant 
> <oliphant at enthought.com <mailto:oliphant at enthought.com>> wrote:
>
>
>     > The problem is that numpy has a *bug*. I am trying to fix it and I
>     > want your help figuring out how to do so while preserving the
>     behavior
>     > you had in mind. So if you could point out specific cases you were
>     > thinking of it would help me with this.
>     This is a hack to support matrices and other subclasses of ndarray
>     like
>     masked arrays.
>
>
> That's what I was thinking. Do you have a test case? One that failed 
> without the NotImplemented return would be very helpful.
There should be some in the test suite, but I'm not sure.  Perhaps the 
masked array test suite also has test cases.

The NotImplemented is being returned so that the other operation will 
try and use it if it can. 

-Travis



From charlesr.harris at gmail.com  Tue Jun 24 23:12:35 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 24 Jun 2008 21:12:35 -0600
Subject: [Numpy-discussion] Request for clarification from Travis
In-Reply-To: <4861B020.8080101@enthought.com>
References: <e06186140806241106h1a66f957l178420fa06256aee@mail.gmail.com>
	<4861410A.5020501@enthought.com>
	<e06186140806241231i15a03f31ra39ae40b2b1293e2@mail.gmail.com>
	<48614DF3.4020308@enthought.com>
	<e06186140806241439v1ec439bm9562f639caf0dfde@mail.gmail.com>
	<4861B020.8080101@enthought.com>
Message-ID: <e06186140806242012w2410f0f9j38463d532e08f6c0@mail.gmail.com>

On Tue, Jun 24, 2008 at 8:40 PM, Travis E. Oliphant <oliphant at enthought.com>
wrote:

> Charles R Harris wrote:
> >
> >
> > On Tue, Jun 24, 2008 at 1:41 PM, Travis E. Oliphant
> > <oliphant at enthought.com <mailto:oliphant at enthought.com>> wrote:
> >
> >
> >     > The problem is that numpy has a *bug*. I am trying to fix it and I
> >     > want your help figuring out how to do so while preserving the
> >     behavior
> >     > you had in mind. So if you could point out specific cases you were
> >     > thinking of it would help me with this.
> >     This is a hack to support matrices and other subclasses of ndarray
> >     like
> >     masked arrays.
> >
> >
> > That's what I was thinking. Do you have a test case? One that failed
> > without the NotImplemented return would be very helpful.
> There should be some in the test suite, but I'm not sure.  Perhaps the
> masked array test suite also has test cases.
>
> The NotImplemented is being returned so that the other operation will
> try and use it if it can.
>

Yes, but it is directed at the interpreter, which will raise a TypeError if
needed. But the interpreter doesn't know that some generic function might
return NotImplemented and wouldn't know what to do if it did. What the user
should see when they call something like right_shift(a,b) and it fails is an
exception, they shouldn't have to check the return value.

I think what you want is that subtypes of ndarray can override some ufuncs
so that, for instance, right_shift() should interpret itself as a call to
arg2.__rrshift__ if it exists, even if it isn't called through the
interpreter friendly slot function >>. Is that correct? If so, I would
rather have an explicit overload flag in the subtype so that the ufunc can
easily do the right thing.

I can't find __array_priority__ in your book, what exactly does it imply?

Chuck







> -Travis
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080624/06917c99/attachment.html>

From drnlmuller+scipy at gmail.com  Wed Jun 25 02:44:20 2008
From: drnlmuller+scipy at gmail.com (Neil Muller)
Date: Wed, 25 Jun 2008 08:44:20 +0200
Subject: [Numpy-discussion] Review of issue 825
Message-ID: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>

Could someone review the patch at
http://scipy.org/scipy/numpy/ticket/825, please?

The issue prevents the test suite running successfully on Sparc, so
I'd like to see it fixed.

-- 
Neil Muller
drnlmuller at gmail.com


From stefan at sun.ac.za  Wed Jun 25 08:42:08 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 25 Jun 2008 14:42:08 +0200
Subject: [Numpy-discussion] Infinity as nan as real part
Message-ID: <9457e7c80806250542k54e2eac1w87d819cc69664b56@mail.gmail.com>

Hi all,

Why can't a person construct a complex number with an infinite complex
part and a zero real part?

1 + 1j*np.inf == (nan + infj) # because np.inf * 1j == (nan + infj)

There is a workaround:

z = np.array([0], dtype=complex)
z.imag = np.inf

but that's not very pleasant.

Is this a bug, or intended behaviour?

Regards
St?fan


From charlesr.harris at gmail.com  Wed Jun 25 10:12:13 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 25 Jun 2008 08:12:13 -0600
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
Message-ID: <e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>

On Wed, Jun 25, 2008 at 12:44 AM, Neil Muller
<drnlmuller+scipy at gmail.com<drnlmuller%2Bscipy at gmail.com>>
wrote:

> Could someone review the patch at
> http://scipy.org/scipy/numpy/ticket/825, please?
>
> The issue prevents the test suite running successfully on Sparc, so
> I'd like to see it fixed.
>

Looks like the copybuffer should use the npy_ucs4 type, although I need to
look at rest of the code there to be sure. You can see examples in the
_sortmodule.c.src file.

Numpy has settled on a 4 byte encoding for unicode, what does python on
sparc use?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080625/3a8874f5/attachment.html>

From charlesr.harris at gmail.com  Wed Jun 25 10:17:51 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 25 Jun 2008 08:17:51 -0600
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
	<e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>
Message-ID: <e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>

On Wed, Jun 25, 2008 at 8:12 AM, Charles R Harris <charlesr.harris at gmail.com>
wrote:

>
>
> On Wed, Jun 25, 2008 at 12:44 AM, Neil Muller <drnlmuller+scipy at gmail.com<drnlmuller%2Bscipy at gmail.com>>
> wrote:
>
>> Could someone review the patch at
>> http://scipy.org/scipy/numpy/ticket/825, please?
>>
>> The issue prevents the test suite running successfully on Sparc, so
>> I'd like to see it fixed.
>>
>
> Looks like the copybuffer should use the npy_ucs4 type, although I need to
> look at rest of the code there to be sure. You can see examples in the
> _sortmodule.c.src file.
>
> Numpy has settled on a 4 byte encoding for unicode, what does python on
> sparc use?
>

Also, if you are just allocating temporary space in the function you can use
the normal malloc/free combination if that helps.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080625/823f7e41/attachment.html>

From charlesr.harris at gmail.com  Wed Jun 25 11:14:23 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 25 Jun 2008 09:14:23 -0600
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
	<e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>
	<e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>
Message-ID: <e06186140806250814q24e3e649le2551f31133a72f@mail.gmail.com>

On Wed, Jun 25, 2008 at 8:17 AM, Charles R Harris <charlesr.harris at gmail.com>
wrote:

>
>
> On Wed, Jun 25, 2008 at 8:12 AM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>>
>>
>> On Wed, Jun 25, 2008 at 12:44 AM, Neil Muller <drnlmuller+scipy at gmail.com<drnlmuller%2Bscipy at gmail.com>>
>> wrote:
>>
>>> Could someone review the patch at
>>> http://scipy.org/scipy/numpy/ticket/825, please?
>>>
>>> The issue prevents the test suite running successfully on Sparc, so
>>> I'd like to see it fixed.
>>>
>>
>> Looks like the copybuffer should use the npy_ucs4 type, although I need to
>> look at rest of the code there to be sure. You can see examples in the
>> _sortmodule.c.src file.
>>
>> Numpy has settled on a 4 byte encoding for unicode, what does python on
>> sparc use?
>>
>
> Also, if you are just allocating temporary space in the function you can
> use the normal malloc/free combination if that helps.
>

OK, the problem in the UNICODE_{get,set}item routines is converting between
ucs4 and the encoding python is using, which may be ucs2.  But there is
something strange if sparc is using ucs4 (Py_UNICODE_WIDE) and the pointer
ip is aligned on two bytes instead of 4, that would seem to indicate a
problem further up the call chain. Could you check that that is actually
happening, i.e., ip is not 4 byte aligned and Py_UNICODE_WIDE is defined?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080625/02d59706/attachment.html>

From drnlmuller+scipy at gmail.com  Wed Jun 25 12:49:33 2008
From: drnlmuller+scipy at gmail.com (Neil Muller)
Date: Wed, 25 Jun 2008 18:49:33 +0200
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <e06186140806250814q24e3e649le2551f31133a72f@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
	<e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>
	<e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>
	<e06186140806250814q24e3e649le2551f31133a72f@mail.gmail.com>
Message-ID: <a4cf72520806250949m71931131qcecbecd10efc6464@mail.gmail.com>

On Wed, Jun 25, 2008 at 5:14 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> OK, the problem in the UNICODE_{get,set}item routines is converting between
> ucs4 and the encoding python is using, which may be ucs2.  But there is
> something strange if sparc is using ucs4 (Py_UNICODE_WIDE) and the pointer
> ip is aligned on two bytes instead of 4, that would seem to indicate a
> problem further up the call chain. Could you check that that is actually
> happening, i.e., ip is not 4 byte aligned and Py_UNICODE_WIDE is defined?

You need to keep the test case in the 1st comment of the issue in mind
here - the problem is extracting the unicode string for a dtype
specified as (unsigned char, unicode string). This is allocated as 5
bytes, and the string is not correctly aligned within these 5 bytes
for access via a long pointer, as is needed for the current check in
UNICODE_getitem to work.

-- 
Neil Muller
drnlmuller at gmail.com


From alan.mcintyre at gmail.com  Wed Jun 25 13:19:05 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Wed, 25 Jun 2008 13:19:05 -0400
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <1d36917a0806231203g512382fp1e8eb015eeb4f11e@mail.gmail.com>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
	<g3oqhj$q9h$2@ger.gmane.org>
	<1d36917a0806231203g512382fp1e8eb015eeb4f11e@mail.gmail.com>
Message-ID: <1d36917a0806251019k1db95d8fucb495bcda19b95cf@mail.gmail.com>

On Mon, Jun 23, 2008 at 3:03 PM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> On Mon, Jun 23, 2008 at 2:37 PM, Pauli Virtanen <pav at iki.fi> wrote:
>> Can you make the convention chosen for the examples (currently only in
>> the doc wiki, not yet in SVN) to work: assuming "import numpy as np" in
>> examples?
>>
>> This would remove the need for those "from numpy import *" lines in the
>> examples that I see were added in r5311.
>
> Sure, I'll look at that.  It seems like every possible option for
> importing stuff from numpy is used in doctests (sometimes even in the
> same module), so having them standardized with that implicit import is
> much better.

It turns out it's possible to give all the doctests an implicit
"import numpy as np" (and probably any other arbitrary tweaks to their
execution context, if need be).  Once I can include some of the other
doctest tweaks discussed in this thread, and it's checked in, I'll go
back and remove all those "import numpy" statements I inserted into
the docstrings.

Alan


From charlesr.harris at gmail.com  Wed Jun 25 13:53:12 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 25 Jun 2008 11:53:12 -0600
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <a4cf72520806250949m71931131qcecbecd10efc6464@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
	<e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>
	<e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>
	<e06186140806250814q24e3e649le2551f31133a72f@mail.gmail.com>
	<a4cf72520806250949m71931131qcecbecd10efc6464@mail.gmail.com>
Message-ID: <e06186140806251053i7a8dc63bo7db73c635dd49091@mail.gmail.com>

On Wed, Jun 25, 2008 at 10:49 AM, Neil Muller
<drnlmuller+scipy at gmail.com<drnlmuller%2Bscipy at gmail.com>>
wrote:

> On Wed, Jun 25, 2008 at 5:14 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> > OK, the problem in the UNICODE_{get,set}item routines is converting
> between
> > ucs4 and the encoding python is using, which may be ucs2.  But there is
> > something strange if sparc is using ucs4 (Py_UNICODE_WIDE) and the
> pointer
> > ip is aligned on two bytes instead of 4, that would seem to indicate a
> > problem further up the call chain. Could you check that that is actually
> > happening, i.e., ip is not 4 byte aligned and Py_UNICODE_WIDE is defined?
>
> You need to keep the test case in the 1st comment of the issue in mind
> here - the problem is extracting the unicode string for a dtype
> specified as (unsigned char, unicode string). This is allocated as 5
> bytes, and the string is not correctly aligned within these 5 bytes
> for access via a long pointer, as is needed for the current check in
> UNICODE_getitem to work.
>

Umm, OK. So what we have is a packed structure from which to extract the
unicode and if we want to have portability it probably needs to remain
packed. I think the fix should be made generic rather than depending on
sparc.  But I wonder if this case isn't supposed to be caught by this
fragment:

    if (!PyArray_ISBEHAVED(ap)) {
        buffer = _pya_malloc(mysize << 2);
        if (buffer == NULL)
            return PyErr_NoMemory();
        alloc = 1;
        memcpy(buffer, ip, mysize << 2);
        if (!PyArray_ISNOTSWAPPED(ap)) {
            byte_swap_vector(buffer, mysize, 4);
        }
    }

Maybe we should be using PyArray_ISBEHAVED_RO(ap) instead.

$[charris at f8 numpy]$ grep "define PyArray_ISBEHAVED" *
ndarrayobject.h:#define PyArray_ISBEHAVED(m) PyArray_FLAGSWAP(m,
NPY_BEHAVED)
ndarrayobject.h:#define PyArray_ISBEHAVED_RO(m) PyArray_FLAGSWAP(m,
NPY_ALIGNED)
$[charris at f8 numpy]$ grep "define PyArray_FLAGSWAP" *
ndarrayobject.h:#define PyArray_FLAGSWAP(m, flags) (PyArray_CHKFLAGS(m,
flags) &&       \


Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080625/58e6c0c8/attachment.html>

From charlesr.harris at gmail.com  Wed Jun 25 14:46:39 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 25 Jun 2008 12:46:39 -0600
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <e06186140806251053i7a8dc63bo7db73c635dd49091@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
	<e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>
	<e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>
	<e06186140806250814q24e3e649le2551f31133a72f@mail.gmail.com>
	<a4cf72520806250949m71931131qcecbecd10efc6464@mail.gmail.com>
	<e06186140806251053i7a8dc63bo7db73c635dd49091@mail.gmail.com>
Message-ID: <e06186140806251146i643eeb7bxf8642bde97a887d5@mail.gmail.com>

On Wed, Jun 25, 2008 at 11:53 AM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

>
>
> On Wed, Jun 25, 2008 at 10:49 AM, Neil Muller <drnlmuller+scipy at gmail.com<drnlmuller%2Bscipy at gmail.com>>
> wrote:
>
>> On Wed, Jun 25, 2008 at 5:14 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> > OK, the problem in the UNICODE_{get,set}item routines is converting
>> between
>> > ucs4 and the encoding python is using, which may be ucs2.  But there is
>> > something strange if sparc is using ucs4 (Py_UNICODE_WIDE) and the
>> pointer
>> > ip is aligned on two bytes instead of 4, that would seem to indicate a
>> > problem further up the call chain. Could you check that that is actually
>> > happening, i.e., ip is not 4 byte aligned and Py_UNICODE_WIDE is
>> defined?
>>
>> You need to keep the test case in the 1st comment of the issue in mind
>> here - the problem is extracting the unicode string for a dtype
>> specified as (unsigned char, unicode string). This is allocated as 5
>> bytes, and the string is not correctly aligned within these 5 bytes
>> for access via a long pointer, as is needed for the current check in
>> UNICODE_getitem to work.
>>
>
> Umm, OK. So what we have is a packed structure from which to extract the
> unicode and if we want to have portability it probably needs to remain
> packed. I think the fix should be made generic rather than depending on
> sparc.  But I wonder if this case isn't supposed to be caught by this
> fragment:
>
>     if (!PyArray_ISBEHAVED(ap)) {
>         buffer = _pya_malloc(mysize << 2);
>         if (buffer == NULL)
>             return PyErr_NoMemory();
>         alloc = 1;
>         memcpy(buffer, ip, mysize << 2);
>         if (!PyArray_ISNOTSWAPPED(ap)) {
>             byte_swap_vector(buffer, mysize, 4);
>         }
>     }
>

#define NPY_BEHAVED (NPY_ALIGNED | NPY_WRITEABLE)

Looks to me like the NPY_ALIGNED flag is incorrectly set. Can you check
this  by printing the results of

PyArray_CHKFLAGS(ap, NPY_ALIGNED)

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080625/3c383b37/attachment.html>

From robert.kern at gmail.com  Wed Jun 25 15:05:31 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 25 Jun 2008 14:05:31 -0500
Subject: [Numpy-discussion] Infinity as nan as real part
In-Reply-To: <9457e7c80806250542k54e2eac1w87d819cc69664b56@mail.gmail.com>
References: <9457e7c80806250542k54e2eac1w87d819cc69664b56@mail.gmail.com>
Message-ID: <3d375d730806251205i35670762q9d6798ea86102fd4@mail.gmail.com>

On Wed, Jun 25, 2008 at 07:42, St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi all,
>
> Why can't a person construct a complex number with an infinite complex
> part and a zero real part?
>
> 1 + 1j*np.inf == (nan + infj) # because np.inf * 1j == (nan + infj)

In [1]: from numpy import *

In [3]: 1j == (0+1j)
Out[3]: True

In [4]: 0*inf
Out[4]: nan

> There is a workaround:
>
> z = np.array([0], dtype=complex)
> z.imag = np.inf
>
> but that's not very pleasant.

In [5]: complex(0, inf)
Out[5]: infj

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco

From charlesr.harris at gmail.com  Wed Jun 25 15:13:35 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 25 Jun 2008 13:13:35 -0600
Subject: [Numpy-discussion] Infinity as nan as real part
In-Reply-To: <9457e7c80806250542k54e2eac1w87d819cc69664b56@mail.gmail.com>
References: <9457e7c80806250542k54e2eac1w87d819cc69664b56@mail.gmail.com>
Message-ID: <e06186140806251213j33b966f8od459442e66000497@mail.gmail.com>

On Wed, Jun 25, 2008 at 6:42 AM, St?fan van der Walt <stefan at sun.ac.za>
wrote:

> Hi all,
>
> Why can't a person construct a complex number with an infinite complex
> part and a zero real part?
>
> 1 + 1j*np.inf == (nan + infj) # because np.inf * 1j == (nan + infj)
>
> There is a workaround:
>
> z = np.array([0], dtype=complex)
> z.imag = np.inf
>
> but that's not very pleasant.
>
> Is this a bug, or intended behaviour?
>

Probably a bug.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080625/bb279829/attachment.html>

From drnlmuller+scipy at gmail.com  Wed Jun 25 16:04:18 2008
From: drnlmuller+scipy at gmail.com (Neil Muller)
Date: Wed, 25 Jun 2008 22:04:18 +0200
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <e06186140806251053i7a8dc63bo7db73c635dd49091@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
	<e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>
	<e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>
	<e06186140806250814q24e3e649le2551f31133a72f@mail.gmail.com>
	<a4cf72520806250949m71931131qcecbecd10efc6464@mail.gmail.com>
	<e06186140806251053i7a8dc63bo7db73c635dd49091@mail.gmail.com>
Message-ID: <a4cf72520806251304h13c51319q17e8d7bf2c5d088a@mail.gmail.com>

On Wed, Jun 25, 2008 at 7:53 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> But I wonder if this case isn't supposed to be caught by this
> fragment:
>
>     if (!PyArray_ISBEHAVED(ap)) {
>         buffer = _pya_malloc(mysize << 2);
>         if (buffer == NULL)
>             return PyErr_NoMemory();
>         alloc = 1;
>         memcpy(buffer, ip, mysize << 2);
>         if (!PyArray_ISNOTSWAPPED(ap)) {
>             byte_swap_vector(buffer, mysize, 4);
>         }
>     }

This actually works fine - the problem is immediately before this,
with the loop to find the end of the unicode string.

An alternative solution is to avoid the long pointer dptr entirely and
use a different approach to find the string length, but I worry that
that would obscure the intent of the code with very little positive
benefit.

-- 
Neil Muller
drnlmuller at gmail.com

I've got a gmail account. Why haven't I become cool?


From stefan at sun.ac.za  Wed Jun 25 16:16:49 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 25 Jun 2008 22:16:49 +0200
Subject: [Numpy-discussion] Infinity as nan as real part
In-Reply-To: <3d375d730806251205i35670762q9d6798ea86102fd4@mail.gmail.com>
References: <9457e7c80806250542k54e2eac1w87d819cc69664b56@mail.gmail.com>
	<3d375d730806251205i35670762q9d6798ea86102fd4@mail.gmail.com>
Message-ID: <9457e7c80806251316r1f220c0dx6bbe9b0b2b68eeaf@mail.gmail.com>

2008/6/25 Robert Kern <robert.kern at gmail.com>:
> In [1]: from numpy import *
>
> In [3]: 1j == (0+1j)
> Out[3]: True
>
> In [4]: 0*inf
> Out[4]: nan

Fair enough.

How about

z = np.complex(0, np.inf)
z**2 == (nannanj)

Shouldn't that be -inf?

Regards
St?fan


From drnlmuller+scipy at gmail.com  Wed Jun 25 16:26:25 2008
From: drnlmuller+scipy at gmail.com (Neil Muller)
Date: Wed, 25 Jun 2008 22:26:25 +0200
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <e06186140806251146i643eeb7bxf8642bde97a887d5@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
	<e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>
	<e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>
	<e06186140806250814q24e3e649le2551f31133a72f@mail.gmail.com>
	<a4cf72520806250949m71931131qcecbecd10efc6464@mail.gmail.com>
	<e06186140806251053i7a8dc63bo7db73c635dd49091@mail.gmail.com>
	<e06186140806251146i643eeb7bxf8642bde97a887d5@mail.gmail.com>
Message-ID: <a4cf72520806251326w66a9fe57p1ed2e8bbee578719@mail.gmail.com>

On Wed, Jun 25, 2008 at 8:46 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> Looks to me like the NPY_ALIGNED flag is incorrectly set. Can you check
> this  by printing the results of
>
> PyArray_CHKFLAGS(ap, NPY_ALIGNED)

For the test listed in the ticket, the array isn't flagged as aligned
- ap->flags =  NPY_CONTIGUOUS | NPY_FORTRAN | NPY_WRITEABLE

I'm under the impression that NPY_ALIGNED refers to the alignment of
the entire entry, though, in which case it's irrelevant here, since
the problem is with the internal alignment of items within a single
array entry.

-- 
Neil Muller
drnlmuller at gmail.com

I've got a gmail account. Why haven't I become cool?


From robert.kern at gmail.com  Wed Jun 25 16:30:39 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 25 Jun 2008 15:30:39 -0500
Subject: [Numpy-discussion] Infinity as nan as real part
In-Reply-To: <9457e7c80806251316r1f220c0dx6bbe9b0b2b68eeaf@mail.gmail.com>
References: <9457e7c80806250542k54e2eac1w87d819cc69664b56@mail.gmail.com>
	<3d375d730806251205i35670762q9d6798ea86102fd4@mail.gmail.com>
	<9457e7c80806251316r1f220c0dx6bbe9b0b2b68eeaf@mail.gmail.com>
Message-ID: <3d375d730806251330m419141f9y7fcbc6c40b55c6d2@mail.gmail.com>

On Wed, Jun 25, 2008 at 15:16, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2008/6/25 Robert Kern <robert.kern at gmail.com>:
>> In [1]: from numpy import *
>>
>> In [3]: 1j == (0+1j)
>> Out[3]: True
>>
>> In [4]: 0*inf
>> Out[4]: nan
>
> Fair enough.
>
> How about
>
> z = np.complex(0, np.inf)
> z**2 == (nannanj)
>
> Shouldn't that be -inf?

Well, complex(-inf, nan).

It's a bit difficult with standard type conversion rules, though. You
have to special-case integer powers (probably worthwhile, but it is
another special case). In this case, this is Python's problem, not
numpy's. numpy.complex is just __builtin__.complex. With numpy arrays,
we get the expected result.

In [9]: from numpy import *

In [10]: za = array([complex(0,inf)])

In [11]: za**2
Out[11]: array([-Inf NaNj])

In [12]: za*za
Out[12]: array([-Inf NaNj])

Unfortunately, not so with numpy scalars:

In [17]: z = complex(0, inf)

In [18]: z128 = complex128(z)

In [19]: z128**2
Out[19]: (nan+nanj)

In [20]: z64 = complex64(z)

In [21]: z64**2
Out[21]: (nan+nanj)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco

From charlesr.harris at gmail.com  Wed Jun 25 17:09:41 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 25 Jun 2008 15:09:41 -0600
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <a4cf72520806250949m71931131qcecbecd10efc6464@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
	<e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>
	<e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>
	<e06186140806250814q24e3e649le2551f31133a72f@mail.gmail.com>
	<a4cf72520806250949m71931131qcecbecd10efc6464@mail.gmail.com>
Message-ID: <e06186140806251409k513f8932m6fb3173d7fa1d208@mail.gmail.com>

On Wed, Jun 25, 2008 at 10:49 AM, Neil Muller
<drnlmuller+scipy at gmail.com<drnlmuller%2Bscipy at gmail.com>>
wrote:

> On Wed, Jun 25, 2008 at 5:14 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> > OK, the problem in the UNICODE_{get,set}item routines is converting
> between
> > ucs4 and the encoding python is using, which may be ucs2.  But there is
> > something strange if sparc is using ucs4 (Py_UNICODE_WIDE) and the
> pointer
> > ip is aligned on two bytes instead of 4, that would seem to indicate a
> > problem further up the call chain. Could you check that that is actually
> > happening, i.e., ip is not 4 byte aligned and Py_UNICODE_WIDE is defined?
>
> You need to keep the test case in the 1st comment of the issue in mind
> here - the problem is extracting the unicode string for a dtype
> specified as (unsigned char, unicode string). This is allocated as 5
> bytes, and the string is not correctly aligned within these 5 bytes
> for access via a long pointer, as is needed for the current check in
> UNICODE_getitem to work.
>

UNICODE_getitem can be called from several places. Here is looks like print
is the caller. Can you check if you can  extract the string with an explicit
call? Something like

In [1]: desc = [ ('x', 'u1'), ('s', 'U2'), ]

In [2]: buffer = [ (5, 'cc'), (6, 'dd') ]

In [3]: ta = array(buffer, dtype(desc))

In [4]: ta[0]['s']
Out[4]: u'cc'

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080625/848c3f5d/attachment.html>

From charlesr.harris at gmail.com  Wed Jun 25 17:32:05 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 25 Jun 2008 15:32:05 -0600
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <a4cf72520806251304h13c51319q17e8d7bf2c5d088a@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
	<e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>
	<e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>
	<e06186140806250814q24e3e649le2551f31133a72f@mail.gmail.com>
	<a4cf72520806250949m71931131qcecbecd10efc6464@mail.gmail.com>
	<e06186140806251053i7a8dc63bo7db73c635dd49091@mail.gmail.com>
	<a4cf72520806251304h13c51319q17e8d7bf2c5d088a@mail.gmail.com>
Message-ID: <e06186140806251432h7e4ad5di7ba872333fa5c20d@mail.gmail.com>

On Wed, Jun 25, 2008 at 2:04 PM, Neil Muller
<drnlmuller+scipy at gmail.com<drnlmuller%2Bscipy at gmail.com>>
wrote:

> On Wed, Jun 25, 2008 at 7:53 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> > But I wonder if this case isn't supposed to be caught by this
> > fragment:
> >
> >     if (!PyArray_ISBEHAVED(ap)) {
> >         buffer = _pya_malloc(mysize << 2);
> >         if (buffer == NULL)
> >             return PyErr_NoMemory();
> >         alloc = 1;
> >         memcpy(buffer, ip, mysize << 2);
> >         if (!PyArray_ISNOTSWAPPED(ap)) {
> >             byte_swap_vector(buffer, mysize, 4);
> >         }
> >     }
>
> This actually works fine - the problem is immediately before this,
> with the loop to find the end of the unicode string.
>

Ah, then we should move that to after the checks. Looking at some of the
other code that calls through UNICODE_getitem, I see that there are a ton of
alignment checks and buffer allocations. That is, a ton of code duplication.
The usual check is something like

                    ((((intp)(ip+offset)) % ap->alignment) != 0))

Which may be used to set the flag. I think we should should put this upfront
in UNICODE_getitem, make the copy if needed, check byte order, and then trim
the zeros and proceed. Duplicated code can then be removed at leisure some
point.

Chuck


> An alternative solution is to avoid the long pointer dptr entirely and
> use a different approach to find the string length, but I worry that
> that would obscure the intent of the code with very little positive
> benefit.
>
> --
> Neil Muller
> drnlmuller at gmail.com
>
> I've got a gmail account. Why haven't I become cool?
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080625/91e3e5fe/attachment.html>

From alan.mcintyre at gmail.com  Wed Jun 25 19:33:54 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Wed, 25 Jun 2008 19:33:54 -0400
Subject: [Numpy-discussion] Code samples in docstrings mistaken as
	doctests
In-Reply-To: <g3oqhj$q9h$2@ger.gmane.org>
References: <1d36917a0806230037r4417af1ek46bf6fd5240d22b4@mail.gmail.com>
	<9457e7c80806230103g1d46e0a7w5bcf48f927829d01@mail.gmail.com>
	<db6b5ecc0806231102q1755a5ecya47e9c70e4a311c1@mail.gmail.com>
	<1d36917a0806231117k58dec303jf49eb5e55631cf5d@mail.gmail.com>
	<g3oqhj$q9h$2@ger.gmane.org>
Message-ID: <1d36917a0806251633u3db014f9x87a68e7ae9d30eac@mail.gmail.com>

On Mon, Jun 23, 2008 at 2:37 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Can you make the convention chosen for the examples (currently only in
> the doc wiki, not yet in SVN) to work: assuming "import numpy as np" in
> examples?

Are there any other implicit imports that we will need?  How about for
SciPy, since the NumPy test framework will also be handling SciPy
tests in 0.7?


From charlesr.harris at gmail.com  Wed Jun 25 22:42:51 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 25 Jun 2008 20:42:51 -0600
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <a4cf72520806251326w66a9fe57p1ed2e8bbee578719@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
	<e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>
	<e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>
	<e06186140806250814q24e3e649le2551f31133a72f@mail.gmail.com>
	<a4cf72520806250949m71931131qcecbecd10efc6464@mail.gmail.com>
	<e06186140806251053i7a8dc63bo7db73c635dd49091@mail.gmail.com>
	<e06186140806251146i643eeb7bxf8642bde97a887d5@mail.gmail.com>
	<a4cf72520806251326w66a9fe57p1ed2e8bbee578719@mail.gmail.com>
Message-ID: <e06186140806251942q43a64124r518c79a03f34456a@mail.gmail.com>

Hi Neil,

On Wed, Jun 25, 2008 at 2:26 PM, Neil Muller
<drnlmuller+scipy at gmail.com<drnlmuller%2Bscipy at gmail.com>>
wrote:

> On Wed, Jun 25, 2008 at 8:46 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> >
> > Looks to me like the NPY_ALIGNED flag is incorrectly set. Can you check
> > this  by printing the results of
> >
> > PyArray_CHKFLAGS(ap, NPY_ALIGNED)
>
> For the test listed in the ticket, the array isn't flagged as aligned
> - ap->flags =  NPY_CONTIGUOUS | NPY_FORTRAN | NPY_WRITEABLE
>
> I'm under the impression that NPY_ALIGNED refers to the alignment of
> the entire entry, though, in which case it's irrelevant here, since
> the problem is with the internal alignment of items within a single
> array entry.
>

It looks like it refers to the item. Look  at VOID_getitem to see how it is
set item by item.

Can you try the attached patch?

TIA,
Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080625/70fb1b9e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sparc.patch
Type: text/x-patch
Size: 2587 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080625/70fb1b9e/attachment.bin>

From charlesr.harris at gmail.com  Wed Jun 25 23:13:16 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 25 Jun 2008 21:13:16 -0600
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <e06186140806251942q43a64124r518c79a03f34456a@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
	<e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>
	<e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>
	<e06186140806250814q24e3e649le2551f31133a72f@mail.gmail.com>
	<a4cf72520806250949m71931131qcecbecd10efc6464@mail.gmail.com>
	<e06186140806251053i7a8dc63bo7db73c635dd49091@mail.gmail.com>
	<e06186140806251146i643eeb7bxf8642bde97a887d5@mail.gmail.com>
	<a4cf72520806251326w66a9fe57p1ed2e8bbee578719@mail.gmail.com>
	<e06186140806251942q43a64124r518c79a03f34456a@mail.gmail.com>
Message-ID: <e06186140806252013k1f208b8el78e3049a48d6c578@mail.gmail.com>

On Wed, Jun 25, 2008 at 8:42 PM, Charles R Harris <charlesr.harris at gmail.com>
wrote:

> Hi Neil,
>
> On Wed, Jun 25, 2008 at 2:26 PM, Neil Muller <drnlmuller+scipy at gmail.com<drnlmuller%2Bscipy at gmail.com>>
> wrote:
>
>> On Wed, Jun 25, 2008 at 8:46 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> >
>> > Looks to me like the NPY_ALIGNED flag is incorrectly set. Can you check
>> > this  by printing the results of
>> >
>> > PyArray_CHKFLAGS(ap, NPY_ALIGNED)
>>
>> For the test listed in the ticket, the array isn't flagged as aligned
>> - ap->flags =  NPY_CONTIGUOUS | NPY_FORTRAN | NPY_WRITEABLE
>>
>> I'm under the impression that NPY_ALIGNED refers to the alignment of
>> the entire entry, though, in which case it's irrelevant here, since
>> the problem is with the internal alignment of items within a single
>> array entry.
>>
>
> It looks like it refers to the item. Look  at VOID_getitem to see how it is
> set item by item.
>
> Can you try the attached patch?
>

Oops, try this one instead. I'm not sure why the last one worked.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080625/535285b0/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sparc.patch
Type: text/x-patch
Size: 2613 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080625/535285b0/attachment.bin>

From charlesr.harris at gmail.com  Thu Jun 26 01:25:03 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 25 Jun 2008 23:25:03 -0600
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <e06186140806252013k1f208b8el78e3049a48d6c578@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
	<e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>
	<e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>
	<e06186140806250814q24e3e649le2551f31133a72f@mail.gmail.com>
	<a4cf72520806250949m71931131qcecbecd10efc6464@mail.gmail.com>
	<e06186140806251053i7a8dc63bo7db73c635dd49091@mail.gmail.com>
	<e06186140806251146i643eeb7bxf8642bde97a887d5@mail.gmail.com>
	<a4cf72520806251326w66a9fe57p1ed2e8bbee578719@mail.gmail.com>
	<e06186140806251942q43a64124r518c79a03f34456a@mail.gmail.com>
	<e06186140806252013k1f208b8el78e3049a48d6c578@mail.gmail.com>
Message-ID: <e06186140806252225y53ee08c8i3384dcb6726ffac7@mail.gmail.com>

On Wed, Jun 25, 2008 at 9:13 PM, Charles R Harris <charlesr.harris at gmail.com>
wrote:

>
>
> On Wed, Jun 25, 2008 at 8:42 PM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>> Hi Neil,
>>
>> On Wed, Jun 25, 2008 at 2:26 PM, Neil Muller <drnlmuller+scipy at gmail.com<drnlmuller%2Bscipy at gmail.com>>
>> wrote:
>>
>>> On Wed, Jun 25, 2008 at 8:46 PM, Charles R Harris
>>> <charlesr.harris at gmail.com> wrote:
>>> >
>>> >
>>> > Looks to me like the NPY_ALIGNED flag is incorrectly set. Can you check
>>> > this  by printing the results of
>>> >
>>> > PyArray_CHKFLAGS(ap, NPY_ALIGNED)
>>>
>>> For the test listed in the ticket, the array isn't flagged as aligned
>>> - ap->flags =  NPY_CONTIGUOUS | NPY_FORTRAN | NPY_WRITEABLE
>>>
>>> I'm under the impression that NPY_ALIGNED refers to the alignment of
>>> the entire entry, though, in which case it's irrelevant here, since
>>> the problem is with the internal alignment of items within a single
>>> array entry.
>>>
>>
>> It looks like it refers to the item. Look  at VOID_getitem to see how it
>> is set item by item.
>>
>> Can you try the attached patch?
>>
>
> Oops, try this one instead. I'm not sure why the last one worked.
>

Seems to work, no bus errors.  Etch has a problem with memmap.roundtrip
while Lenny does fine, but that looks to be some other problem.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080625/e745c98d/attachment.html>

From oliphant at enthought.com  Thu Jun 26 03:42:36 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 26 Jun 2008 02:42:36 -0500
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <e06186140806251432h7e4ad5di7ba872333fa5c20d@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>	<e06186140806250712w2209071aq62cf2bb79ac3ec8a@mail.gmail.com>	<e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>	<e06186140806250814q24e3e649le2551f31133a72f@mail.gmail.com>	<a4cf72520806250949m71931131qcecbecd10efc6464@mail.gmail.com>	<e06186140806251053i7a8dc63bo7db73c635dd49091@mail.gmail.com>	<a4cf72520806251304h13c51319q17e8d7bf2c5d088a@mail.gmail.com>
	<e06186140806251432h7e4ad5di7ba872333fa5c20d@mail.gmail.com>
Message-ID: <4863486C.7070106@enthought.com>

Charles R Harris wrote:
>
>
> On Wed, Jun 25, 2008 at 2:04 PM, Neil Muller 
> <drnlmuller+scipy at gmail.com <mailto:drnlmuller%2Bscipy at gmail.com>> wrote:
>
>     On Wed, Jun 25, 2008 at 7:53 PM, Charles R Harris
>     <charlesr.harris at gmail.com <mailto:charlesr.harris at gmail.com>> wrote:
>     > But I wonder if this case isn't supposed to be caught by this
>     > fragment:
>     >
>     >     if (!PyArray_ISBEHAVED(ap)) {
>     >         buffer = _pya_malloc(mysize << 2);
>     >         if (buffer == NULL)
>     >             return PyErr_NoMemory();
>     >         alloc = 1;
>     >         memcpy(buffer, ip, mysize << 2);
>     >         if (!PyArray_ISNOTSWAPPED(ap)) {
>     >             byte_swap_vector(buffer, mysize, 4);
>     >         }
>     >     }
>
>     This actually works fine - the problem is immediately before this,
>     with the loop to find the end of the unicode string.
>
This is indeed the problem.   No de-referencing of anything but a char 
pointer should be done unless you are sure you have an aligned array.    
This length-of-string adjustment code should definitely be placed after 
the copy is done so that an aligned array is obtained.  

The other approach is to check the length of string differently (i.e. 
using only byte dereferencing) so that when the string is misaligned 
un-necessary copies are not done.  But,  Chuck's patch looks sufficient.

Best regards,

-Travis



From oliphant at enthought.com  Thu Jun 26 03:50:03 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 26 Jun 2008 02:50:03 -0500
Subject: [Numpy-discussion] Request for clarification from Travis
In-Reply-To: <e06186140806242012w2410f0f9j38463d532e08f6c0@mail.gmail.com>
References: <e06186140806241106h1a66f957l178420fa06256aee@mail.gmail.com>	<4861410A.5020501@enthought.com>	<e06186140806241231i15a03f31ra39ae40b2b1293e2@mail.gmail.com>	<48614DF3.4020308@enthought.com>	<e06186140806241439v1ec439bm9562f639caf0dfde@mail.gmail.com>	<4861B020.8080101@enthought.com>
	<e06186140806242012w2410f0f9j38463d532e08f6c0@mail.gmail.com>
Message-ID: <48634A2B.2060304@enthought.com>


> Yes, but it is directed at the interpreter, which will raise a 
> TypeError if needed. But the interpreter doesn't know that some 
> generic function might return NotImplemented and wouldn't know what to 
> do if it did. What the user should see when they call something like 
> right_shift(a,b) and it fails is an exception, they shouldn't have to 
> check the return value
Yes, you are right.   So, the questions is how to handle this correctly.
>
> I think what you want is that subtypes of ndarray can override some 
> ufuncs so that, for instance, right_shift() should interpret itself as 
> a call to arg2.__rrshift__ if it exists, even if it isn't called 
> through the interpreter friendly slot function >>. Is that correct? If 
> so, I would rather have an explicit overload flag in the subtype so 
> that the ufunc can easily do the right thing.
So, how would this overload flag work and how would a subtype set it?   
Would this be a per-instance flag in the FLAGS field? 
>  
> I can't find __array_priority__ in your book, what exactly does it imply?
The subtype with the largest priority wins in any decision about which 
subtype to create during operations involving more than one subtype 
(like adding a matrix to an array --- what should be returned?).

-Travis



From drnlmuller+scipy at gmail.com  Thu Jun 26 06:35:10 2008
From: drnlmuller+scipy at gmail.com (Neil Muller)
Date: Thu, 26 Jun 2008 12:35:10 +0200
Subject: [Numpy-discussion] Review of issue 825
In-Reply-To: <e06186140806252225y53ee08c8i3384dcb6726ffac7@mail.gmail.com>
References: <a4cf72520806242344o46244de9sf4e222cb21912f24@mail.gmail.com>
	<e06186140806250717h162a3543u9bd069aec62b894d@mail.gmail.com>
	<e06186140806250814q24e3e649le2551f31133a72f@mail.gmail.com>
	<a4cf72520806250949m71931131qcecbecd10efc6464@mail.gmail.com>
	<e06186140806251053i7a8dc63bo7db73c635dd49091@mail.gmail.com>
	<e06186140806251146i643eeb7bxf8642bde97a887d5@mail.gmail.com>
	<a4cf72520806251326w66a9fe57p1ed2e8bbee578719@mail.gmail.com>
	<e06186140806251942q43a64124r518c79a03f34456a@mail.gmail.com>
	<e06186140806252013k1f208b8el78e3049a48d6c578@mail.gmail.com>
	<e06186140806252225y53ee08c8i3384dcb6726ffac7@mail.gmail.com>
Message-ID: <a4cf72520806260335h5dfbf11ei2e42f2946c93fb7d@mail.gmail.com>

On Thu, Jun 26, 2008 at 7:25 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>   Etch has a problem with memmap.roundtrip
> while Lenny does fine, but that looks to be some other problem.

Indeed - I see this is already filed as #828. I'll look at this and
add any comments to that ticket.

-- 
Neil Muller
drnlmuller at gmail.com

I've got a gmail account. Why haven't I become cool?


From stefan at sun.ac.za  Thu Jun 26 07:35:00 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 26 Jun 2008 13:35:00 +0200
Subject: [Numpy-discussion] Topical documentation
Message-ID: <9457e7c80806260435t3185668ci9fc37af6fb7b4c6a@mail.gmail.com>

Hi all,

We are busy writing several documents that address general NumPy
topics, such as indexing, broadcasting, testing, etc.  I would like
for those documents to be available inside of NumPy, so that they
could be accessed as docstrings:

>>> help(np.doc.indexing)

or

In [15]: np.doc.indexing?

If I had to implement it right now, I'd probably

- Have a directory full of topic.txt files.
- Upon importing the doc sub-module, setup doc.topic and add docstring
accordingly.

Does anyone have comments or suggestions regarding such a framework?

Thanks for your time,
St?fan


From peridot.faceted at gmail.com  Thu Jun 26 11:08:02 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 26 Jun 2008 11:08:02 -0400
Subject: [Numpy-discussion] Topical documentation
In-Reply-To: <9457e7c80806260435t3185668ci9fc37af6fb7b4c6a@mail.gmail.com>
References: <9457e7c80806260435t3185668ci9fc37af6fb7b4c6a@mail.gmail.com>
Message-ID: <ce557a360806260808u49269eactf3bedcc06aee0236@mail.gmail.com>

2008/6/26 St?fan van der Walt <stefan at sun.ac.za>:
> Hi all,
>
> We are busy writing several documents that address general NumPy
> topics, such as indexing, broadcasting, testing, etc.  I would like
> for those documents to be available inside of NumPy, so that they
> could be accessed as docstrings:
>
>>>> help(np.doc.indexing)
>
> or
>
> In [15]: np.doc.indexing?
>
> If I had to implement it right now, I'd probably
>
> - Have a directory full of topic.txt files.
> - Upon importing the doc sub-module, setup doc.topic and add docstring
> accordingly.
>
> Does anyone have comments or suggestions regarding such a framework?

I think making them available inside np.doc is a good idea. Should
np.doc.indexing contain anything but the single docstring?

It's also worth making some of the examples work as doctests, both to
get better testing - some of the indexing operations don't work, or at
least, don't do anything I can understand - and to keep the examples
in syn with the code. To make this happen, might it not be better to
create doc/indexing.py, containing
__doc__="""..."""?

The topical documentation would presumably also be ReST. Any further
conventions it should follow?

Anne

From stefan at sun.ac.za  Thu Jun 26 12:13:59 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 26 Jun 2008 18:13:59 +0200
Subject: [Numpy-discussion] Record arrays
Message-ID: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>

Hi all,

I am documenting `recarray`, and have a question:

Is its use still recommended, or has it been superseded by fancy data-types?

Regards
St?fan


From jh at physics.ucf.edu  Thu Jun 26 12:26:51 2008
From: jh at physics.ucf.edu (Joe Harrington)
Date: Thu, 26 Jun 2008 12:26:51 -0400
Subject: [Numpy-discussion] Topical documentation
In-Reply-To: <mailman.67126.1214492891.27196.numpy-discussion@scipy.org>
	(numpy-discussion-request@scipy.org)
References: <mailman.67126.1214492891.27196.numpy-discussion@scipy.org>
Message-ID: <wl87icctb5g.fsf@glup.physics.ucf.edu>

Just to clarify, the documentation Stefan refers to is topical
*reference* documentation for the numpy package infrastructure,
conventions, etc.  The contemplated .doc module will be a few kB of
distilled reference text.  Its contents will be included in the PDF
and HTML reference guides.

It may include such topics as:

summary of numpy package capabilities and conventions
	includes a list of these topics and related
	docstrings elsewhere, like np.ndarray.__doc__
indexing and slicing
broadcasting
list of functions by topic
essential performance hints (i.e., "don't loop", not "how to write
	parallel C extensions")
etc.

Brief words on how to use the documentation will become part of
numpy.__doc__.

We're not contemplating turning tutorial documentation (cookbook
recipes, user guide, discipline-specific stuff) into numpy objects.

ReST formatting conventions will be identical to that for docstrings,
since these will be docstrings.  Doctests need to work, of course.

Given that there will be a small number of these topics (fewer than
20, perhaps fewer than 10), should they be imported into the top level
on numpy load?  That would allow the more intuitive

help(np.indexing)

We should only say "yes" to this question if they will load on every
import of numpy.  I would not want them to land in the main level only
after importing the doc module.

--jh--


From chanley at stsci.edu  Thu Jun 26 12:31:40 2008
From: chanley at stsci.edu (Christopher Hanley)
Date: Thu, 26 Jun 2008 12:31:40 -0400
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
Message-ID: <4863C46C.40501@stsci.edu>

St?fan van der Walt wrote:
> Hi all,
> 
> I am documenting `recarray`, and have a question:
> 
> Is its use still recommended, or has it been superseded by fancy data-types?
> 
> Regards
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

I would say that it has been superseded by fancy data-types.  It is 
necessary that recarray remain for backward compatibility with large, 
legacy numarray projects.  However, I would encourage all new code be 
written with the native object arrays.

Chris



-- 
Christopher Hanley
Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338


From oliphant at enthought.com  Thu Jun 26 12:38:29 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 26 Jun 2008 11:38:29 -0500
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
Message-ID: <4863C605.1080103@enthought.com>

St?fan van der Walt wrote:
> Hi all,
>
> I am documenting `recarray`, and have a question:
>
> Is its use still recommended, or has it been superseded by fancy data-types?
>   
I rarely recommend it's use (but some people do like attribute access to 
the fields).    It is wrong, however, to say that recarray has been 
superseded by fancy data types because fancy data types have existed for 
as long as recarrays. 

I believe pyfits uses them quite a bit, and so they deserve to be 
documented.

-Travis



From charlesr.harris at gmail.com  Thu Jun 26 12:43:34 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 26 Jun 2008 10:43:34 -0600
Subject: [Numpy-discussion] Request for clarification from Travis
In-Reply-To: <48634A2B.2060304@enthought.com>
References: <e06186140806241106h1a66f957l178420fa06256aee@mail.gmail.com>
	<4861410A.5020501@enthought.com>
	<e06186140806241231i15a03f31ra39ae40b2b1293e2@mail.gmail.com>
	<48614DF3.4020308@enthought.com>
	<e06186140806241439v1ec439bm9562f639caf0dfde@mail.gmail.com>
	<4861B020.8080101@enthought.com>
	<e06186140806242012w2410f0f9j38463d532e08f6c0@mail.gmail.com>
	<48634A2B.2060304@enthought.com>
Message-ID: <e06186140806260943m30eb000fp2a362014046db443@mail.gmail.com>

Hi Travis,

On Thu, Jun 26, 2008 at 1:50 AM, Travis E. Oliphant <oliphant at enthought.com>
wrote:

>
> > Yes, but it is directed at the interpreter, which will raise a
> > TypeError if needed. But the interpreter doesn't know that some
> > generic function might return NotImplemented and wouldn't know what to
> > do if it did. What the user should see when they call something like
> > right_shift(a,b) and it fails is an exception, they shouldn't have to
> > check the return value
> Yes, you are right.   So, the questions is how to handle this correctly.
> >
> > I think what you want is that subtypes of ndarray can override some
> > ufuncs so that, for instance, right_shift() should interpret itself as
> > a call to arg2.__rrshift__ if it exists, even if it isn't called
> > through the interpreter friendly slot function >>. Is that correct? If
> > so, I would rather have an explicit overload flag in the subtype so
> > that the ufunc can easily do the right thing.
> So, how would this overload flag work and how would a subtype set it?
> Would this be a per-instance flag in the FLAGS field?


It's been a few days, but my last thoughts were that  ufunc_generic _call,
or one of the other gateways, should play the role that the interpreter does
for NotImplemented. How exactly it determines that some function is
overridden I'm not sure. Subtypes could, for instance, have a method with
the ufunc name, but we need to be sure we don't end up in an endless loop.


> >
> > I can't find __array_priority__ in your book, what exactly does it imply?
> The subtype with the largest priority wins in any decision about which
> subtype to create during operations involving more than one subtype
> (like adding a matrix to an array --- what should be returned?).
>

So it has a numeric value?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080626/dcf2d179/attachment.html>

From chanley at stsci.edu  Thu Jun 26 12:45:12 2008
From: chanley at stsci.edu (Christopher Hanley)
Date: Thu, 26 Jun 2008 12:45:12 -0400
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <4863C605.1080103@enthought.com>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
	<4863C605.1080103@enthought.com>
Message-ID: <4863C798.5020700@stsci.edu>

Travis E. Oliphant wrote:
> St?fan van der Walt wrote:
>> Hi all,
>>
>> I am documenting `recarray`, and have a question:
>>
>> Is its use still recommended, or has it been superseded by fancy data-types?
>>   
> I rarely recommend it's use (but some people do like attribute access to 
> the fields).    It is wrong, however, to say that recarray has been 
> superseded by fancy data types because fancy data types have existed for 
> as long as recarrays. 
> 
> I believe pyfits uses them quite a bit, and so they deserve to be 
> documented.
> 
> -Travis
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

Travis is correct.  PyFITS uses recarrays quite extensively.  It was the 
large, legacy numarray project I was referring too. ;-)

I had forgotten about the attribute access.  I know a number of people 
who use that feature in conjunction with matplotlib for plotting data in 
tables, especially during interactive use.

Chris


-- 
Christopher Hanley
Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338


From jdh2358 at gmail.com  Thu Jun 26 12:48:06 2008
From: jdh2358 at gmail.com (John Hunter)
Date: Thu, 26 Jun 2008 11:48:06 -0500
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <4863C605.1080103@enthought.com>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
	<4863C605.1080103@enthought.com>
Message-ID: <88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>

On Thu, Jun 26, 2008 at 11:38 AM, Travis E. Oliphant
<oliphant at enthought.com> wrote:
> St?fan van der Walt wrote:
>> Hi all,
>>
>> I am documenting `recarray`, and have a question:
>>
>> Is its use still recommended, or has it been superseded by fancy data-types?
>>
> I rarely recommend it's use (but some people do like attribute access to
> the fields).    It is wrong, however, to say that recarray has been
> superseded by fancy data types because fancy data types have existed for
> as long as recarrays.

I personally think they are the best thing since sliced bread, and
everyone here who uses them becomes immediately addicted to them.  I
would like to see better support for them, especially making the attrs
exposed to dir so tab completion would work.

People in the financial/business world work with spreadsheet data a
lot, and record arrays are the natural data structure to represent
tabular, heterogeneous data.    If you work with this data all day,
you save a lot of ugly keystrokes doing r.date rather than r['date'],
and the code is prettier in my opinion.

JDH


From oliphant at enthought.com  Thu Jun 26 12:52:49 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 26 Jun 2008 11:52:49 -0500
Subject: [Numpy-discussion] Request for clarification from Travis
In-Reply-To: <e06186140806260943m30eb000fp2a362014046db443@mail.gmail.com>
References: <e06186140806241106h1a66f957l178420fa06256aee@mail.gmail.com>	<4861410A.5020501@enthought.com>	<e06186140806241231i15a03f31ra39ae40b2b1293e2@mail.gmail.com>	<48614DF3.4020308@enthought.com>	<e06186140806241439v1ec439bm9562f639caf0dfde@mail.gmail.com>	<4861B020.8080101@enthought.com>	<e06186140806242012w2410f0f9j38463d532e08f6c0@mail.gmail.com>	<48634A2B.2060304@enthought.com>
	<e06186140806260943m30eb000fp2a362014046db443@mail.gmail.com>
Message-ID: <4863C961.5050205@enthought.com>


>
> So it has a numeric value?
Yes, it's just a floating point number.  It's not a very elegant thing, 
but it does allow some ability to specify an ordering.

-Travis




From vfulco1 at gmail.com  Thu Jun 26 13:22:37 2008
From: vfulco1 at gmail.com (Vince Fulco)
Date: Thu, 26 Jun 2008 13:22:37 -0400
Subject: [Numpy-discussion] Record arrays
Message-ID: <34f2770f0806261022h13ff3d64pd6deb33f9c498066@mail.gmail.com>

Would someone kindly provide links to references for the fancy data types
superseding the recarrays?  Having come from an R-Project background,
manipulating the latter takes time to become acclimated to when building
data.frame like objects.

TIA,

-- 
Vince Fulco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080626/e257fdd5/attachment.html>

From charlesr.harris at gmail.com  Thu Jun 26 13:37:19 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 26 Jun 2008 11:37:19 -0600
Subject: [Numpy-discussion] nosetester.py can't use decorators and work with
	python2.3
Message-ID: <e06186140806261037p6e3f8745gf7424e560a811b91@mail.gmail.com>

Hi Alan,

Decorators weren't introduced until Python2.4. I'm not sure what version we
require now, that information should probably be added to a DEPENDENCY file,
or maybe the README. Anyway, if we still support Python2.3 we can't use
decorators.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080626/a0c5b59f/attachment.html>

From millman at berkeley.edu  Thu Jun 26 13:51:27 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Thu, 26 Jun 2008 10:51:27 -0700
Subject: [Numpy-discussion] nosetester.py can't use decorators and work
	with python2.3
In-Reply-To: <e06186140806261037p6e3f8745gf7424e560a811b91@mail.gmail.com>
References: <e06186140806261037p6e3f8745gf7424e560a811b91@mail.gmail.com>
Message-ID: <c7009a550806261051r24a19cbdicde29166ef071469@mail.gmail.com>

On Thu, Jun 26, 2008 at 10:37 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> Decorators weren't introduced until Python2.4. I'm not sure what version we
> require now, that information should probably be added to a DEPENDENCY file,
> or maybe the README. Anyway, if we still support Python2.3 we can't use
> decorators.

Starting with NumPy 1.2 and SciPy 0.7 we will require Python 2.4 or
greater.  I will make sure to update the documentation later today, if
no one gets to it before me.

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From pgmdevlist at gmail.com  Thu Jun 26 13:47:06 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 26 Jun 2008 13:47:06 -0400
Subject: [Numpy-discussion] nosetester.py can't use decorators and work
	with python2.3
In-Reply-To: <e06186140806261037p6e3f8745gf7424e560a811b91@mail.gmail.com>
References: <e06186140806261037p6e3f8745gf7424e560a811b91@mail.gmail.com>
Message-ID: <200806261347.06421.pgmdevlist@gmail.com>

On Thursday 26 June 2008 13:37:19 Charles R Harris wrote:
> Hi Alan,
>
> Decorators weren't introduced until Python2.4. I'm not sure what version we
> require now, that information should probably be added to a DEPENDENCY
> file, or maybe the README. Anyway, if we still support Python2.3 we can't
> use decorators.

I thought Python2.3 was supported for the 1.1.x branches only, not the 1.2 
one. If the nosetester approach works for 1.2.x, then we should be OK, no?



From alan.mcintyre at gmail.com  Thu Jun 26 14:01:53 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Thu, 26 Jun 2008 14:01:53 -0400
Subject: [Numpy-discussion] nosetester.py can't use decorators and work
	with python2.3
In-Reply-To: <e06186140806261037p6e3f8745gf7424e560a811b91@mail.gmail.com>
References: <e06186140806261037p6e3f8745gf7424e560a811b91@mail.gmail.com>
Message-ID: <1d36917a0806261101n22bf6fdfk44025cc15f726ba6@mail.gmail.com>

On Thu, Jun 26, 2008 at 1:37 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> Decorators weren't introduced until Python2.4. I'm not sure what version we
> require now, that information should probably be added to a DEPENDENCY file,
> or maybe the README. Anyway, if we still support Python2.3 we can't use
> decorators.

I can remove those decorators; there's no point in requiring 2.4 just
for updating some docstrings.


From alan.mcintyre at gmail.com  Thu Jun 26 14:11:04 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Thu, 26 Jun 2008 14:11:04 -0400
Subject: [Numpy-discussion] nosetester.py can't use decorators and work
	with python2.3
In-Reply-To: <c7009a550806261051r24a19cbdicde29166ef071469@mail.gmail.com>
References: <e06186140806261037p6e3f8745gf7424e560a811b91@mail.gmail.com>
	<c7009a550806261051r24a19cbdicde29166ef071469@mail.gmail.com>
Message-ID: <1d36917a0806261111x5e279a13x244d4913b64d60df@mail.gmail.com>

On Thu, Jun 26, 2008 at 1:51 PM, Jarrod Millman <millman at berkeley.edu> wrote:
> Starting with NumPy 1.2 and SciPy 0.7 we will require Python 2.4 or
> greater.  I will make sure to update the documentation later today, if
> no one gets to it before me.

In that case I will leave the decorators in nosetester.py.

By the way, what about 2.6 and 3.0?  I know they're not close to being
out yet, but are we going to worry about them at all for 1.2?


From charlesr.harris at gmail.com  Thu Jun 26 14:32:50 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 26 Jun 2008 12:32:50 -0600
Subject: [Numpy-discussion] nosetester.py can't use decorators and work
	with python2.3
In-Reply-To: <c7009a550806261051r24a19cbdicde29166ef071469@mail.gmail.com>
References: <e06186140806261037p6e3f8745gf7424e560a811b91@mail.gmail.com>
	<c7009a550806261051r24a19cbdicde29166ef071469@mail.gmail.com>
Message-ID: <e06186140806261132o3d9edbe8mb206e9c711d3ac3f@mail.gmail.com>

On Thu, Jun 26, 2008 at 11:51 AM, Jarrod Millman <millman at berkeley.edu>
wrote:

> On Thu, Jun 26, 2008 at 10:37 AM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> > Decorators weren't introduced until Python2.4. I'm not sure what version
> we
> > require now, that information should probably be added to a DEPENDENCY
> file,
> > or maybe the README. Anyway, if we still support Python2.3 we can't use
> > decorators.
>
> Starting with NumPy 1.2 and SciPy 0.7 we will require Python 2.4 or
> greater.  I will make sure to update the documentation later today, if
> no one gets to it before me.
>

Maybe we should also add a version check that runs when numpy is imported so
as to avoid spurious error reports. I'm thinking of the latest reply to
ticket #821.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080626/7794355e/attachment.html>

From gael.varoquaux at normalesup.org  Thu Jun 26 15:34:25 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 26 Jun 2008 21:34:25 +0200
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
	<4863C605.1080103@enthought.com>
	<88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>
Message-ID: <20080626193425.GB3848@phare.normalesup.org>

On Thu, Jun 26, 2008 at 11:48:06AM -0500, John Hunter wrote:
> I personally think they are the best thing since sliced bread, and
> everyone here who uses them becomes immediately addicted to them.  I
> would like to see better support for them, especially making the attrs
> exposed to dir so tab completion would work.

> People in the financial/business world work with spreadsheet data a
> lot, and record arrays are the natural data structure to represent
> tabular, heterogeneous data.    If you work with this data all day,
> you save a lot of ugly keystrokes doing r.date rather than r['date'],
> and the code is prettier in my opinion.

I am +1 on all that.

Gael



From dyamins at gmail.com  Thu Jun 26 16:13:27 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Thu, 26 Jun 2008 16:13:27 -0400
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <20080626193425.GB3848@phare.normalesup.org>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
	<4863C605.1080103@enthought.com>
	<88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>
	<20080626193425.GB3848@phare.normalesup.org>
Message-ID: <15e4667e0806261313g6f3140aaj4be0e4ac306d0295@mail.gmail.com>

On Thu, Jun 26, 2008 at 3:34 PM, Gael Varoquaux <
gael.varoquaux at normalesup.org> wrote:

> On Thu, Jun 26, 2008 at 11:48:06AM -0500, John Hunter wrote:
> > I personally think they are the best thing since sliced bread, and
> > everyone here who uses them becomes immediately addicted to them.  I
> > would like to see better support for them, especially making the attrs
> > exposed to dir so tab completion would work.
>
> > People in the financial/business world work with spreadsheet data a
> > lot, and record arrays are the natural data structure to represent
> > tabular, heterogeneous data.    If you work with this data all day,
> > you save a lot of ugly keystrokes doing r.date rather than r['date'],
> > and the code is prettier in my opinion.
>
> I am +1 on all that.
>
>
I also completely second this.  I use them all the time -- for finance data
as well as biological/genomics data.  It is essential for these applications
to have spread-sheet like objects that can have mixed types and from which
good numpy numerical arrays can be extracted when necessary.   I hope to
continue having access to them or something like them.  I also hope that
they will be better documented, since not only do I use them all the time,
I'm hoping to teach their use to many more people whom I am training and in
spread-sheet like data analysis.

(If they have some flaw I don't understand, it would be great if someone
could explain it to me.   And if there's something out there that fixes that
flaw, I'd love to hear about it.  But it seems to me at least that recarrays
are very useful.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080626/3a734f39/attachment.html>

From gael.varoquaux at normalesup.org  Thu Jun 26 15:34:25 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 26 Jun 2008 21:34:25 +0200
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
	<4863C605.1080103@enthought.com>
	<88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>
Message-ID: <20080626193425.GB3848@phare.normalesup.org>

On Thu, Jun 26, 2008 at 11:48:06AM -0500, John Hunter wrote:
> I personally think they are the best thing since sliced bread, and
> everyone here who uses them becomes immediately addicted to them.  I
> would like to see better support for them, especially making the attrs
> exposed to dir so tab completion would work.

> People in the financial/business world work with spreadsheet data a
> lot, and record arrays are the natural data structure to represent
> tabular, heterogeneous data.    If you work with this data all day,
> you save a lot of ugly keystrokes doing r.date rather than r['date'],
> and the code is prettier in my opinion.

I am +1 on all that.

Gael



From robert.kern at gmail.com  Thu Jun 26 16:25:11 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 26 Jun 2008 15:25:11 -0500
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <15e4667e0806261313g6f3140aaj4be0e4ac306d0295@mail.gmail.com>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
	<4863C605.1080103@enthought.com>
	<88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>
	<20080626193425.GB3848@phare.normalesup.org>
	<15e4667e0806261313g6f3140aaj4be0e4ac306d0295@mail.gmail.com>
Message-ID: <3d375d730806261325g194c0488lcdc0bcc4bf13f31f@mail.gmail.com>

On Thu, Jun 26, 2008 at 15:13, Dan Yamins <dyamins at gmail.com> wrote:
>
> On Thu, Jun 26, 2008 at 3:34 PM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
>>
>> On Thu, Jun 26, 2008 at 11:48:06AM -0500, John Hunter wrote:
>> > I personally think they are the best thing since sliced bread, and
>> > everyone here who uses them becomes immediately addicted to them.  I
>> > would like to see better support for them, especially making the attrs
>> > exposed to dir so tab completion would work.
>>
>> > People in the financial/business world work with spreadsheet data a
>> > lot, and record arrays are the natural data structure to represent
>> > tabular, heterogeneous data.    If you work with this data all day,
>> > you save a lot of ugly keystrokes doing r.date rather than r['date'],
>> > and the code is prettier in my opinion.
>>
>> I am +1 on all that.
>>
>
> I also completely second this.  I use them all the time -- for finance data
> as well as biological/genomics data.  It is essential for these applications
> to have spread-sheet like objects that can have mixed types and from which
> good numpy numerical arrays can be extracted when necessary.   I hope to
> continue having access to them or something like them.  I also hope that
> they will be better documented, since not only do I use them all the time,
> I'm hoping to teach their use to many more people whom I am training and in
> spread-sheet like data analysis.
>
> (If they have some flaw I don't understand, it would be great if someone
> could explain it to me.   And if there's something out there that fixes that
> flaw, I'd love to hear about it.  But it seems to me at least that recarrays
> are very useful.)

Let's be clear, there are two very closely related things: recarrays
and record arrays. Record arrays are just ndarrays with a complicated
dtype. E.g.

In [1]: from numpy import *

In [2]: ones(3, dtype=dtype([('foo', int), ('bar', float)]))
Out[2]:
array([(1, 1.0), (1, 1.0), (1, 1.0)],
      dtype=[('foo', '<i4'), ('bar', '<f8')])

In [3]: r = _

In [4]: r['foo']
Out[4]: array([1, 1, 1])


recarray is a subclass of ndarray that just adds attribute access to
record arrays.

In [10]: r2 = r.view(recarray)

In [11]: r2
Out[11]:
recarray([(1, 1.0), (1, 1.0), (1, 1.0)],
      dtype=[('foo', '<i4'), ('bar', '<f8')])

In [12]: r2.foo
Out[12]: array([1, 1, 1])


One downside of this is that the attribute access feature slows down
all field accesses, even the r['foo'] form, because it sticks a bunch
of pure Python code in the middle. Much code won't notice this, but if
you end up having to iterate over an array of records (as I have),
this will be a hotspot for you.

Record arrays are fundamentally a part of numpy, and no one is even
suggesting that they would go away. No one is seriously suggesting
that we should remove recarray, but some of us hesitate to recommend
its use over plain record arrays.

Does that clarify the discussion for you?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From ggellner at uoguelph.ca  Thu Jun 26 16:38:36 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Thu, 26 Jun 2008 16:38:36 -0400
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <3d375d730806261325g194c0488lcdc0bcc4bf13f31f@mail.gmail.com>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
	<4863C605.1080103@enthought.com>
	<88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>
	<20080626193425.GB3848@phare.normalesup.org>
	<15e4667e0806261313g6f3140aaj4be0e4ac306d0295@mail.gmail.com>
	<3d375d730806261325g194c0488lcdc0bcc4bf13f31f@mail.gmail.com>
Message-ID: <20080626203836.GA9083@basestar>

> Let's be clear, there are two very closely related things: recarrays
> and record arrays. Record arrays are just ndarrays with a complicated
> dtype. E.g.
> 
> In [1]: from numpy import *
> 
> In [2]: ones(3, dtype=dtype([('foo', int), ('bar', float)]))
> Out[2]:
> array([(1, 1.0), (1, 1.0), (1, 1.0)],
>       dtype=[('foo', '<i4'), ('bar', '<f8')])
> 
> In [3]: r = _
> 
> In [4]: r['foo']
> Out[4]: array([1, 1, 1])
> 
> 
> recarray is a subclass of ndarray that just adds attribute access to
> record arrays.
> 
> In [10]: r2 = r.view(recarray)
> 
> In [11]: r2
> Out[11]:
> recarray([(1, 1.0), (1, 1.0), (1, 1.0)],
>       dtype=[('foo', '<i4'), ('bar', '<f8')])
> 
> In [12]: r2.foo
> Out[12]: array([1, 1, 1])
> 
> 
> One downside of this is that the attribute access feature slows down
> all field accesses, even the r['foo'] form, because it sticks a bunch
> of pure Python code in the middle. Much code won't notice this, but if
> you end up having to iterate over an array of records (as I have),
> this will be a hotspot for you.
> 
> Record arrays are fundamentally a part of numpy, and no one is even
> suggesting that they would go away. No one is seriously suggesting
> that we should remove recarray, but some of us hesitate to recommend
> its use over plain record arrays.
> 
> Does that clarify the discussion for you?
> 
Thanks! This has always been something that has confused me . . . This is
awesome, I guess I build by DataFrame object for nothing :-)

Gabriel


From dyamins at gmail.com  Thu Jun 26 16:39:25 2008
From: dyamins at gmail.com (Dan Yamins)
Date: Thu, 26 Jun 2008 16:39:25 -0400
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <3d375d730806261325g194c0488lcdc0bcc4bf13f31f@mail.gmail.com>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
	<4863C605.1080103@enthought.com>
	<88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>
	<20080626193425.GB3848@phare.normalesup.org>
	<15e4667e0806261313g6f3140aaj4be0e4ac306d0295@mail.gmail.com>
	<3d375d730806261325g194c0488lcdc0bcc4bf13f31f@mail.gmail.com>
Message-ID: <15e4667e0806261339n7ec46b67xe9a8892b0849c6ad@mail.gmail.com>

>
> In [12]: r2.foo
> Out[12]: array([1, 1, 1])
>
>
> One downside of this is that the attribute access feature slows down
> all field accesses, even the r['foo'] form, because it sticks a bunch
> of pure Python code in the middle. Much code won't notice this, but if
> you end up having to iterate over an array of records (as I have),
> this will be a hotspot for you.
>
> Record arrays are fundamentally a part of numpy, and no one is even
> suggesting that they would go away. No one is seriously suggesting
> that we should remove recarray, but some of us hesitate to recommend
> its use over plain record arrays.
>
> Does that clarify the discussion for you?




Yes, thanks very much, this is very helpful.  (I think I was confused by the
fact that, AFAICT, the Guide to Numpy only mentions recarray -- as distinct
from Record arrays -- in one somewhat cryptic line.)   But I guess that the
numpy documentation work going on now will provide good documentation for
using Record Arrays proper?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080626/7b8a2114/attachment.html>

From oliphant at enthought.com  Thu Jun 26 18:07:14 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 26 Jun 2008 17:07:14 -0500
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <15e4667e0806261339n7ec46b67xe9a8892b0849c6ad@mail.gmail.com>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>	<4863C605.1080103@enthought.com>	<88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>	<20080626193425.GB3848@phare.normalesup.org>	<15e4667e0806261313g6f3140aaj4be0e4ac306d0295@mail.gmail.com>	<3d375d730806261325g194c0488lcdc0bcc4bf13f31f@mail.gmail.com>
	<15e4667e0806261339n7ec46b67xe9a8892b0849c6ad@mail.gmail.com>
Message-ID: <48641312.9020003@enthought.com>


>     Does that clarify the discussion for you?
>
>
>
>
> Yes, thanks very much, this is very helpful.  (I think I was confused 
> by the fact that, AFAICT, the Guide to Numpy only mentions recarray -- 
> as distinct from Record arrays -- in one somewhat cryptic line.)   But 
> I guess that the numpy documentation work going on now will provide 
> good documentation for using Record Arrays proper?

Incidentally.   Eric and I use the term "structured arrays" to refer to 
NumPy arrays with a complicated dtype, precisely because of the 
confusion with the recarray subclass that record arrays sometimes 
engenders.

-Travis



From gael.varoquaux at normalesup.org  Thu Jun 26 22:24:21 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 27 Jun 2008 04:24:21 +0200
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <3d375d730806261325g194c0488lcdc0bcc4bf13f31f@mail.gmail.com>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
	<4863C605.1080103@enthought.com>
	<88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>
	<20080626193425.GB3848@phare.normalesup.org>
	<15e4667e0806261313g6f3140aaj4be0e4ac306d0295@mail.gmail.com>
	<3d375d730806261325g194c0488lcdc0bcc4bf13f31f@mail.gmail.com>
Message-ID: <20080627022421.GB11323@phare.normalesup.org>

I understand all your comments and thank you for making this distinction
explicit. I can see why recarray can slow code down, but I find attribute
lookup make code much more readable, and interactive work fantastic (tab
completion). For many of my applications I do have a strong use case for
these recarrays, and I am willing to take the speek cost (many of the
things I do are very for from being numerically intensiv).

On a side note, a pattern I use a lot (and incidently that Fernando and
Brian also came up with in ipython1) is a mixed object that acts like a
dictionary (and thus comes with all the goodies like the keys, iterkeys,
... methods, and the "in"), but exposes its keys as attributes:

class Bunch(dict):

    def __init__(self, **kwargs):
        dict.__init__(self, **kwargs)
        self.__dict__ = self

a = Bunch(a=1, b=2)

This is not directly related to the discussion, as the recarrays add more
to this (eg operations uniform over all the fields), but it does show
that this pattern is liked by many people.

My 2 cents,

Ga?l

On Thu, Jun 26, 2008 at 03:25:11PM -0500, Robert Kern wrote:
> Let's be clear, there are two very closely related things: recarrays
> and record arrays. Record arrays are just ndarrays with a complicated
> dtype. E.g.

> In [1]: from numpy import *

> In [2]: ones(3, dtype=dtype([('foo', int), ('bar', float)]))
> Out[2]:
> array([(1, 1.0), (1, 1.0), (1, 1.0)],
>       dtype=[('foo', '<i4'), ('bar', '<f8')])

> In [3]: r = _

> In [4]: r['foo']
> Out[4]: array([1, 1, 1])


> recarray is a subclass of ndarray that just adds attribute access to
> record arrays.

> In [10]: r2 = r.view(recarray)

> In [11]: r2
> Out[11]:
> recarray([(1, 1.0), (1, 1.0), (1, 1.0)],
>       dtype=[('foo', '<i4'), ('bar', '<f8')])

> In [12]: r2.foo
> Out[12]: array([1, 1, 1])


> One downside of this is that the attribute access feature slows down
> all field accesses, even the r['foo'] form, because it sticks a bunch
> of pure Python code in the middle. Much code won't notice this, but if
> you end up having to iterate over an array of records (as I have),
> this will be a hotspot for you.

> Record arrays are fundamentally a part of numpy, and no one is even
> suggesting that they would go away. No one is seriously suggesting
> that we should remove recarray, but some of us hesitate to recommend
> its use over plain record arrays.

> Does that clarify the discussion for you?


From robert.kern at gmail.com  Thu Jun 26 22:36:38 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 26 Jun 2008 21:36:38 -0500
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <20080627022421.GB11323@phare.normalesup.org>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
	<4863C605.1080103@enthought.com>
	<88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>
	<20080626193425.GB3848@phare.normalesup.org>
	<15e4667e0806261313g6f3140aaj4be0e4ac306d0295@mail.gmail.com>
	<3d375d730806261325g194c0488lcdc0bcc4bf13f31f@mail.gmail.com>
	<20080627022421.GB11323@phare.normalesup.org>
Message-ID: <3d375d730806261936m34b2006fg492834b4e461e657@mail.gmail.com>

On Thu, Jun 26, 2008 at 21:24, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> I understand all your comments and thank you for making this distinction
> explicit. I can see why recarray can slow code down, but I find attribute
> lookup make code much more readable, and interactive work fantastic (tab
> completion).

I'm confused. recarray fields do not show up in any standard
tab-completion schemes.

> For many of my applications I do have a strong use case for
> these recarrays, and I am willing to take the speek cost (many of the
> things I do are very for from being numerically intensiv).
>
> On a side note, a pattern I use a lot (and incidently that Fernando and
> Brian also came up with in ipython1) is a mixed object that acts like a
> dictionary (and thus comes with all the goodies like the keys, iterkeys,
> ... methods, and the "in"), but exposes its keys as attributes:
>
> class Bunch(dict):
>
>    def __init__(self, **kwargs):
>        dict.__init__(self, **kwargs)
>        self.__dict__ = self
>
> a = Bunch(a=1, b=2)

Actually, I wrote that particular snippet in the IPython codebase, but
the idea comes from a Cookbook entry:

  http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52308

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From gael.varoquaux at normalesup.org  Thu Jun 26 22:46:18 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 27 Jun 2008 04:46:18 +0200
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <3d375d730806261936m34b2006fg492834b4e461e657@mail.gmail.com>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
	<4863C605.1080103@enthought.com>
	<88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>
	<20080626193425.GB3848@phare.normalesup.org>
	<15e4667e0806261313g6f3140aaj4be0e4ac306d0295@mail.gmail.com>
	<3d375d730806261325g194c0488lcdc0bcc4bf13f31f@mail.gmail.com>
	<20080627022421.GB11323@phare.normalesup.org>
	<3d375d730806261936m34b2006fg492834b4e461e657@mail.gmail.com>
Message-ID: <20080627024618.GF11323@phare.normalesup.org>

On Thu, Jun 26, 2008 at 09:36:38PM -0500, Robert Kern wrote:
> On Thu, Jun 26, 2008 at 21:24, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
> > I understand all your comments and thank you for making this distinction
> > explicit. I can see why recarray can slow code down, but I find attribute
> > lookup make code much more readable, and interactive work fantastic (tab
> > completion).

> I'm confused. recarray fields do not show up in any standard
> tab-completion schemes.

Damn you are right. So I what just making noise I guess. But my point
about code readability remains, and it is shorter to type.

> Actually, I wrote that particular snippet in the IPython codebase, but
> the idea comes from a Cookbook entry:

:)

Ga?l


From fperez.net at gmail.com  Fri Jun 27 01:10:18 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 26 Jun 2008 22:10:18 -0700
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <3d375d730806261325g194c0488lcdc0bcc4bf13f31f@mail.gmail.com>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
	<4863C605.1080103@enthought.com>
	<88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>
	<20080626193425.GB3848@phare.normalesup.org>
	<15e4667e0806261313g6f3140aaj4be0e4ac306d0295@mail.gmail.com>
	<3d375d730806261325g194c0488lcdc0bcc4bf13f31f@mail.gmail.com>
Message-ID: <db6b5ecc0806262210k2abef9a8k14ebd57f2372a536@mail.gmail.com>

On Thu, Jun 26, 2008 at 1:25 PM, Robert Kern <robert.kern at gmail.com> wrote:

> One downside of this is that the attribute access feature slows down
> all field accesses, even the r['foo'] form, because it sticks a bunch
> of pure Python code in the middle. Much code won't notice this, but if
> you end up having to iterate over an array of records (as I have),
> this will be a hotspot for you.

I wonder if it wouldn't be useful for *all* numpy arrays to have a .f
attribute that would provide attribute access to fields for complex
dtypes:

In [13]: r['foo']
Out[13]: array([1, 1, 1])

In [14]: r.f.foo
-> Hypothetically, same as [13] above

This object would be in general an empty namespace, thus avoiding the
potential for collisions that recarrays have, could normalize field
names to be valid python identifiers (spaces to _, etc) and could
provide name TAB completion. Since the .f object would be a *separate*
object, the main array wouldn't need to have complex python code in
the fast path and there would be no speed penalty for other uses of
the top level object.

I've never quite liked recarrays because of the fact that they blend
the named fields with the main namespace, and because they don't tab
complete.  I'd happily pay the price of accessing a sub-object for a
cleaner and more useful access to fields (I could always do xf=x.f if
I am really going to use the field object a lot).

Just an idea, perhaps it's already been shut down in the past.

Cheers,

f


From gael.varoquaux at normalesup.org  Fri Jun 27 01:18:47 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 27 Jun 2008 07:18:47 +0200
Subject: [Numpy-discussion] Student sponsorship for the SciPy08 conference
Message-ID: <20080627051847.GM11323@phare.normalesup.org>

We are delighted to announce that the Python Software Foundation has
answered our call and is providing sponsoring to the SciPy08 conference.

We will use this money to sponsor the registration fees and travel for up
to 10 college or graduate students to attend the conference. The PSF did
not provide all the founds required for all 10 students and once again
Enthought Inc. (http://www.enthought.com) is stepping up to fill in.

To apply, please send a short description of what you are studying and
why you?d like to attend to info at enthought.com. Please include telephone
contact information.

Thanks a lot to Travis Vaught from Enthought for bringing this project to
a success.

Please don't hesitate to forward this announcement to anybody who might
be interested.

Ga?l, on behalf of the Scipy08 organisation committee

SciPy coneference site:   http://conference.scipy.org


From alfreale74 at gmail.com  Fri Jun 27 04:58:33 2008
From: alfreale74 at gmail.com (Alfredo Alessandrini)
Date: Fri, 27 Jun 2008 10:58:33 +0200
Subject: [Numpy-discussion] working with matrix....
Message-ID: <d2c05c5a0806270158q78a769d1xc0767c1ff17c3ed6@mail.gmail.com>

I must analyse many data (about 2000 historical series of two variables).

This data are organized in matrix.

I must calculate some simple statistical analysis, and I must write a
script for to automate the processing.

I've a simple question:

What's the better method or working (I'm a beginner of python and R):

- Numpy
- Bash shell + R
- Rpy + R
- Other possibilities??

Thanks in advance,

Alfredo


From gael.varoquaux at normalesup.org  Fri Jun 27 09:17:41 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 27 Jun 2008 15:17:41 +0200
Subject: [Numpy-discussion] working with matrix....
In-Reply-To: <d2c05c5a0806270158q78a769d1xc0767c1ff17c3ed6@mail.gmail.com>
References: <d2c05c5a0806270158q78a769d1xc0767c1ff17c3ed6@mail.gmail.com>
Message-ID: <20080627131741.GE24133@phare.normalesup.org>

On Fri, Jun 27, 2008 at 10:58:33AM +0200, Alfredo Alessandrini wrote:
> What's the better method or working (I'm a beginner of python and R):

> - Numpy
> - Bash shell + R
> - Rpy + R
> - Other possibilities??

numpy + scipy.stats and when you need more Rpy, but I like to keep it to
a minimum.

Ga?l


From theller at ctypes.org  Fri Jun 27 09:29:53 2008
From: theller at ctypes.org (Thomas Heller)
Date: Fri, 27 Jun 2008 15:29:53 +0200
Subject: [Numpy-discussion] Windows_XP_x86_64_MSVC buildbot
Message-ID: <g42pvq$v43$1@ger.gmane.org>

I'm going to have a 2 or 3-weeks vacation, and will shut down
the Windows_XP_x86_64_MSVC during this time.  I will restart it
when I come back, sorry for the inconvenience.

Thanks,
Thomas



From charlesr.harris at gmail.com  Fri Jun 27 11:27:32 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 27 Jun 2008 09:27:32 -0600
Subject: [Numpy-discussion] Should the fields in void scalars be available?
Message-ID: <e06186140806270827s799d1165vcca7f4b6cfd8f5a2@mail.gmail.com>

Question,

In [24]: x = array(('1', u'2'), dtype=[('a', '|S4'), ('b', '<U2')])

In [25]: x.shape
Out[25]: ()

In [26]: x['a']
Out[26]:
array('1',
      dtype='|S4')

Shouldn't the last be a string?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080627/fced8d17/attachment.html>

From gael.varoquaux at normalesup.org  Fri Jun 27 14:24:29 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 27 Jun 2008 20:24:29 +0200
Subject: [Numpy-discussion] Scipy08 Paper submission deadline extention to
	Monday 30th
Message-ID: <20080627182429.GH5103@phare.normalesup.org>

The deadline for submitting abstracts to the Scipy conference was tonight.
In order to give you more time to submit excellent abstracts, the review
committee is extending the deadline to Monday (June 30th), and will work
hastily to get all of them reviewed in time for the program announcement,
on Thursday July 3rd.

----

The SciPy 2008 Conference will be held 21-22 August 2008 at the
California Institute of Technology, Pasadena, California. SciPy is a
scientific computing package, written in the Python language. It is
widely used in research, the industry and academia.

The program features tutorials, contributed papers, lightning talks, and
bird-of-a-feather sessions. We are soliciting talks and accompanying
papers (either formal academic or magazine-style articles) that discuss
topics which center around scientific computing using Python. These
include applications, teaching, future development directions and
research. A collection of peer-reviewed articles will be published as
part of the proceedings.

Proposals for talks are submitted as extended abstracts. There are two
categories of talks:
Lightning talks

These talks are 10 minutes in duration. An abstract of between 300 and
700 words should describe the topic and motivate its relevance to
scientific computing. Lightning talks do not require an accompanying
article (although, if submitted, these will still be published).
Paper presentations

These talks are 35 minutes in duration (including questions). A one page
abstract of no less than 500 words (excluding figures and references)
should give an outline of the final paper. Papers are due two weeks
before the conference, and may be in a formal academic style, or in a
more relaxed magazine-style format.

If you wish to present a talk at the conference, please create an account
on the website http://conference.scipy.org. You may then submit an
abstract by logging in, clicking on your profile and following the "
Submit an abstract " link.

Ga?l, on behalf on the SciPy08 organizing committee.


From charlesr.harris at gmail.com  Fri Jun 27 14:41:25 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 27 Jun 2008 12:41:25 -0600
Subject: [Numpy-discussion] Scipy08 Paper submission deadline extention
	to Monday 30th
In-Reply-To: <20080627182429.GH5103@phare.normalesup.org>
References: <20080627182429.GH5103@phare.normalesup.org>
Message-ID: <e06186140806271141n103c0059l47f1e9df60694e5c@mail.gmail.com>

On Fri, Jun 27, 2008 at 12:24 PM, Gael Varoquaux <
gael.varoquaux at normalesup.org> wrote:

> The deadline for submitting abstracts to the Scipy conference was tonight.
> In order to give you more time to submit excellent abstracts, the review
> committee is extending the deadline to Monday (June 30th), and will work
> hastily to get all of them reviewed in time for the program announcement,
> on Thursday July 3rd.
>
> ----
>
> The SciPy 2008 Conference will be held 21-22 August 2008 at the
> California Institute of Technology, Pasadena, California. SciPy is a
> scientific computing package, written in the Python language. It is
> widely used in research, the industry and academia.
>
> The program features tutorials, contributed papers, lightning talks, and
> bird-of-a-feather sessions. We are soliciting talks and accompanying
> papers (either formal academic or magazine-style articles) that discuss
> topics which center around scientific computing using Python. These
> include applications, teaching, future development directions and
> research. A collection of peer-reviewed articles will be published as
> part of the proceedings.
>
> Proposals for talks are submitted as extended abstracts. There are two
> categories of talks:
> Lightning talks
>
> These talks are 10 minutes in duration. An abstract of between 300 and
> 700 words should describe the topic and motivate its relevance to
> scientific computing. Lightning talks do not require an accompanying
> article (although, if submitted, these will still be published).
> Paper presentations
>
> These talks are 35 minutes in duration (including questions). A one page
> abstract of no less than 500 words (excluding figures and references)
> should give an outline of the final paper. Papers are due two weeks
> before the conference, and may be in a formal academic style, or in a
> more relaxed magazine-style format.
>
> If you wish to present a talk at the conference, please create an account
> on the website http://conference.scipy.org. You may then submit an
> abstract by logging in, clicking on your profile and following the "
> Submit an abstract " link.
>
> Ga?l, on behalf on the SciPy08 organizing committee.
> ______


So how do we find the abstracts/papers for review?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080627/b0adb946/attachment.html>

From gael.varoquaux at normalesup.org  Fri Jun 27 14:53:07 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 27 Jun 2008 20:53:07 +0200
Subject: [Numpy-discussion] Scipy08 Paper submission deadline	extention
	to Monday 30th
In-Reply-To: <e06186140806271141n103c0059l47f1e9df60694e5c@mail.gmail.com>
References: <20080627182429.GH5103@phare.normalesup.org>
	<e06186140806271141n103c0059l47f1e9df60694e5c@mail.gmail.com>
Message-ID: <20080627185307.GJ5103@phare.normalesup.org>

On Fri, Jun 27, 2008 at 12:41:25PM -0600, Charles R Harris wrote:
>    So how do we find the abstracts/papers for review?

Hey Chuck,

If you question is: how do you, as a member of the review committee, get
to read the abstracts, the answer is that I will send a pdf with all of
them.

I am really sorry, I sent an e-mail to the other reviewers giving this
information, but somehow I forgot to put you on my list. Thank you for
reminding me.

Ga?l


From robert.kern at gmail.com  Fri Jun 27 15:34:40 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 27 Jun 2008 14:34:40 -0500
Subject: [Numpy-discussion] Should the fields in void scalars be
	available?
In-Reply-To: <e06186140806270827s799d1165vcca7f4b6cfd8f5a2@mail.gmail.com>
References: <e06186140806270827s799d1165vcca7f4b6cfd8f5a2@mail.gmail.com>
Message-ID: <3d375d730806271234n78b41a2bj3db561bb49bfe86f@mail.gmail.com>

On Fri, Jun 27, 2008 at 10:27, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> Question,
>
> In [24]: x = array(('1', u'2'), dtype=[('a', '|S4'), ('b', '<U2')])
>
> In [25]: x.shape
> Out[25]: ()
>
> In [26]: x['a']
> Out[26]:
> array('1',
>       dtype='|S4')
>
> Shouldn't the last be a string?

No.

In [6]: x = array(('1', u'2', 3), dtype=[('a', '|S4'), ('b', '<U2'),
('c', 'i4')])

In [7]: x
Out[7]:
array(('1', u'2', 3),
      dtype=[('a', '|S4'), ('b', '<U2'), ('c', '<i4')])

In [8]: x['c']
Out[8]: array(3)


x is not a scalar. It is a rank-0 array. Picking out a field from it
will also be a rank-0 array. If x were a rank-1 array, picking out a
field from it will give a rank-1 array.

In [12]: y = array([('1', u'2', 3)], dtype=[('a', '|S4'), ('b',
'<U2'), ('c', 'i4')])

In [13]: y['a']
Out[13]:
array(['1'],
      dtype='|S4')

The only inconsistency I can see here is that x.item() gives a tuple
rather than a scalar record. A scalar record does have field access,
but the tuple doesn't.

In [28]: x.item()['a']
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

/Users/rkern/bzr/ipython/<ipython console> in <module>()

TypeError: tuple indices must be integers

In [29]: y[0]['a']
Out[29]: '1'


-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From jdh2358 at gmail.com  Fri Jun 27 16:06:24 2008
From: jdh2358 at gmail.com (John Hunter)
Date: Fri, 27 Jun 2008 15:06:24 -0500
Subject: [Numpy-discussion] sampling based on running sums
Message-ID: <88e473830806271306vda5d118l225759d8f08a4b30@mail.gmail.com>

I would like to find the sample points where the running sum of some
vector exceeds some threshold -- at those points I want to collect all
the data in the vector since the last time the criteria was reached
and compute some stats on it.  For example, in python

    tot = 0.
    xs = []
    ys = []

    samples1 = []
    for thisx, thisy in zip(x, y):
        tot += thisx
        xs.append(thisx)
        ys.append(thisy)
        if tot>=threshold:
            samples1.append(func(xs,ys))
            tot = 0.
            xs = []
            ys = []


The following is close in numpy

    sx = np.cumsum(x)
    n = (sx/threshold).astype(int)
    ind = np.nonzero(np.diff(n)>0)[0]+1

    lasti = 0
    samples2 = []
    for i in ind:
        xs = x[lasti:i+1]
        ys = y[lasti:i+1]
        samples2.append(func(xs, ys))
        lasti = i

But the sample points in ind do no guarantee that at least threshold
points are between the sample points due to truncation error.

What is a good numpy way to do this?

Thanks,
JDH


From oliphant at enthought.com  Fri Jun 27 18:13:44 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Fri, 27 Jun 2008 17:13:44 -0500
Subject: [Numpy-discussion] Should the fields in void scalars
	be	available?
In-Reply-To: <3d375d730806271234n78b41a2bj3db561bb49bfe86f@mail.gmail.com>
References: <e06186140806270827s799d1165vcca7f4b6cfd8f5a2@mail.gmail.com>
	<3d375d730806271234n78b41a2bj3db561bb49bfe86f@mail.gmail.com>
Message-ID: <48656618.1010304@enthought.com>

Robert Kern wrote:
>
> The only inconsistency I can see here is that x.item() gives a tuple
> rather than a scalar record. A scalar record does have field access,
> but the tuple doesn't.
>
> In [28]: x.item()['a']
>   
.item() always returns a standard Python type.  So, this is expected 
behavior.

If you want to index a 0-d array to get the array scalar do

x[()] or x[...]

-Travis



From robert.kern at gmail.com  Fri Jun 27 18:37:26 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 27 Jun 2008 17:37:26 -0500
Subject: [Numpy-discussion] Should the fields in void scalars be
	available?
In-Reply-To: <48656618.1010304@enthought.com>
References: <e06186140806270827s799d1165vcca7f4b6cfd8f5a2@mail.gmail.com>
	<3d375d730806271234n78b41a2bj3db561bb49bfe86f@mail.gmail.com>
	<48656618.1010304@enthought.com>
Message-ID: <3d375d730806271537m241e1a35ydd07b22c08744678@mail.gmail.com>

On Fri, Jun 27, 2008 at 17:13, Travis E. Oliphant
<oliphant at enthought.com> wrote:
> Robert Kern wrote:
>>
>> The only inconsistency I can see here is that x.item() gives a tuple
>> rather than a scalar record. A scalar record does have field access,
>> but the tuple doesn't.
>>
>> In [28]: x.item()['a']
>>
> .item() always returns a standard Python type.  So, this is expected
> behavior.

Okey-dokey.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From travis at enthought.com  Fri Jun 27 19:04:47 2008
From: travis at enthought.com (Travis Vaught)
Date: Fri, 27 Jun 2008 18:04:47 -0500
Subject: [Numpy-discussion] SciPy Conference Updates
Message-ID: <E8FB9C93-6FDA-4801-8595-C6DB0B5259A6@enthought.com>

Greetings,

The SciPy Conference is not too far away.  I thought I'd summarize  
some recent news about the conference in case some of you missed it:


- Accommodations (news!): We've negotiated a group rate with a nearby  
Marriott hotel, for those that would like to take advantage of it.   
The hotel has set up a web site for our event here:

http://cwp.marriott.com/laxot/scipyworkshop/

- Student Sponsorship: As you may have seen, the Python Software  
Foundation has agreed to partner with Enthought to sponsor 10  
students' travel, registration, and accommodation for the tutorials,  
conference and (most importantly) sprints.  If you're in college or a  
graduate program, please check out the details here:

http://conference.scipy.org/sponsoring

- Abstracts Submission Deadline Extended:  The review committee is  
extending the deadline to Monday, June 30th.  Please see the Call for  
Papers for instructions on abstract submission here:

http://conference.scipy.org/call_for_papers


Please drop me an email if you have any questions or comments.

Best,

Travis


From haase at msg.ucsf.edu  Fri Jun 27 20:10:57 2008
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Sat, 28 Jun 2008 02:10:57 +0200
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <db6b5ecc0806262210k2abef9a8k14ebd57f2372a536@mail.gmail.com>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
	<4863C605.1080103@enthought.com>
	<88e473830806260948t7774bb8ene889a3f808acd5e9@mail.gmail.com>
	<20080626193425.GB3848@phare.normalesup.org>
	<15e4667e0806261313g6f3140aaj4be0e4ac306d0295@mail.gmail.com>
	<3d375d730806261325g194c0488lcdc0bcc4bf13f31f@mail.gmail.com>
	<db6b5ecc0806262210k2abef9a8k14ebd57f2372a536@mail.gmail.com>
Message-ID: <bc657ead0806271710t328fef1an8f3f0824b4412a69@mail.gmail.com>

On Fri, Jun 27, 2008 at 7:10 AM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Thu, Jun 26, 2008 at 1:25 PM, Robert Kern <robert.kern at gmail.com> wrote:
>
>> One downside of this is that the attribute access feature slows down
>> all field accesses, even the r['foo'] form, because it sticks a bunch
>> of pure Python code in the middle. Much code won't notice this, but if
>> you end up having to iterate over an array of records (as I have),
>> this will be a hotspot for you.
>
> I wonder if it wouldn't be useful for *all* numpy arrays to have a .f
> attribute that would provide attribute access to fields for complex
> dtypes:
>
> In [13]: r['foo']
> Out[13]: array([1, 1, 1])
>
> In [14]: r.f.foo
> -> Hypothetically, same as [13] above
>
> This object would be in general an empty namespace, thus avoiding the
> potential for collisions that recarrays have, could normalize field
> names to be valid python identifiers (spaces to _, etc) and could
> provide name TAB completion. Since the .f object would be a *separate*
> object, the main array wouldn't need to have complex python code in
> the fast path and there would be no speed penalty for other uses of
> the top level object.
>
> I've never quite liked recarrays because of the fact that they blend
> the named fields with the main namespace, and because they don't tab
> complete.  I'd happily pay the price of accessing a sub-object for a
> cleaner and more useful access to fields (I could always do xf=x.f if
> I am really going to use the field object a lot).
>
> Just an idea, perhaps it's already been shut down in the past.
>
+ 1

-- Sebastian Haase


From robert.kern at gmail.com  Sat Jun 28 01:40:38 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 28 Jun 2008 00:40:38 -0500
Subject: [Numpy-discussion] sampling based on running sums
In-Reply-To: <88e473830806271306vda5d118l225759d8f08a4b30@mail.gmail.com>
References: <88e473830806271306vda5d118l225759d8f08a4b30@mail.gmail.com>
Message-ID: <3d375d730806272240q19505c1cj5834386976cf7bdf@mail.gmail.com>

On Fri, Jun 27, 2008 at 15:06, John Hunter <jdh2358 at gmail.com> wrote:
> I would like to find the sample points where the running sum of some
> vector

All non-negative, right?

> exceeds some threshold -- at those points I want to collect all
> the data in the vector since the last time the criteria was reached
> and compute some stats on it.  For example, in python
>
>    tot = 0.
>    xs = []
>    ys = []
>
>    samples1 = []
>    for thisx, thisy in zip(x, y):
>        tot += thisx
>        xs.append(thisx)
>        ys.append(thisy)
>        if tot>=threshold:
>            samples1.append(func(xs,ys))
>            tot = 0.
>            xs = []
>            ys = []
>
>
> The following is close in numpy
>
>    sx = np.cumsum(x)
>    n = (sx/threshold).astype(int)
>    ind = np.nonzero(np.diff(n)>0)[0]+1
>
>    lasti = 0
>    samples2 = []
>    for i in ind:
>        xs = x[lasti:i+1]
>        ys = y[lasti:i+1]
>        samples2.append(func(xs, ys))
>        lasti = i
>
> But the sample points in ind do no guarantee that at least threshold
> points are between the sample points due to truncation error.

Truncation error?

One reason you get different results between the two is that you are
finding the locations where the sum exceeds an integer multiple of the
threshold *starting from 0*. In the pure-Python version, you reset the
count to 0 every time you hit the threshold.

Assuming that the data are all non-negative, the cumsum is sorted. So
use searchsorted() to find the next index the threshold is exceeded.
Separate the data into two parts by the index. Use the below part to
compute your function. Shift the above part down by the last value in
the below part to reset the sum. Rinse and repeat with the shifted
above part.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From charlesr.harris at gmail.com  Sat Jun 28 18:31:22 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 28 Jun 2008 16:31:22 -0600
Subject: [Numpy-discussion] Time to fix ticket #390?
Message-ID: <e06186140806281531p58384780ye4a08dd5adbdec6e@mail.gmail.com>

Questions about ticket #390<http://projects.scipy.org/scipy/numpy/ticket/390>:

1. Should we fix this for 1.2?
2. Is the attached patch the correct fix for this?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080628/b817797f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: records.patch
Type: text/x-patch
Size: 476 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080628/b817797f/attachment.bin>

From oliphant at enthought.com  Sat Jun 28 21:16:09 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Sat, 28 Jun 2008 20:16:09 -0500
Subject: [Numpy-discussion] Time to fix ticket #390?
In-Reply-To: <e06186140806281531p58384780ye4a08dd5adbdec6e@mail.gmail.com>
References: <e06186140806281531p58384780ye4a08dd5adbdec6e@mail.gmail.com>
Message-ID: <4866E259.2040301@enthought.com>

Charles R Harris wrote:
> Questions about ticket #390 
> <http://projects.scipy.org/scipy/numpy/ticket/390> :
>
> 1. Should we fix this for 1.2?
+1
> 2. Is the attached patch the correct fix for this?
It looks good, except for the rare case where the array contains the 
word "rec"  Perhaps the fix should be changed to
only replace the rec only at the first of the string to "rec."

-Travis



From charlesr.harris at gmail.com  Sat Jun 28 22:23:31 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 28 Jun 2008 20:23:31 -0600
Subject: [Numpy-discussion] Time to fix ticket #390?
In-Reply-To: <4866E259.2040301@enthought.com>
References: <e06186140806281531p58384780ye4a08dd5adbdec6e@mail.gmail.com>
	<4866E259.2040301@enthought.com>
Message-ID: <e06186140806281923m2e509504h1ff22293651a6138@mail.gmail.com>

On Sat, Jun 28, 2008 at 7:16 PM, Travis E. Oliphant <oliphant at enthought.com>
wrote:

> Charles R Harris wrote:
> > Questions about ticket #390
> > <http://projects.scipy.org/scipy/numpy/ticket/390> :
> >
> > 1. Should we fix this for 1.2?
> +1
> > 2. Is the attached patch the correct fix for this?
> It looks good, except for the rare case where the array contains the
> word "rec"  Perhaps the fix should be changed to
> only replace the rec only at the first of the string to "rec."
>

Good point.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080628/5663f3e6/attachment.html>

From zachary.pincus at yale.edu  Sun Jun 29 01:28:52 2008
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Sun, 29 Jun 2008 01:28:52 -0400
Subject: [Numpy-discussion] numpy.distutils problem with py2app/py2exe?
Message-ID: <76C48C8C-2AF9-4474-86FF-7F345367ED4D@yale.edu>

Hello all,

I've noticed that py2app and py2exe work strangely on my project,  
which (having fortran code) is driven by the numpy distutils. Now,  
these two distutils commands need to peek into the build/lib. 
[whatever] directories to grab the files to package up. Indeed, the  
docs for py2exe and py2app say that the build directories are  
automatically added to the path.

However, I found that for my project, I need to explicitly add lines  
to the setup.py file like:

if have_py2exe:
   sys.path.insert(0, './build/lib.win32-%d.%d'%(sys.version_info[0],  
sys.version_info[1]))


I got around to digging a little deeper, and at least in the py2app  
case, py2app distutils Command subclass gets the build dir information  
as follows:

     def run(self):
         build = self.reinitialize_command('build')
         build.run()
         [... snip ...]
         extra_paths.extend([build.build_platlib, build.build_lib])

Now, when I run py2app, the build.build_platlib and build.build_lib  
values are None. So py2app never finds the library directory, and thus  
packages the files incorrectly. Is this likely to be (a) a problem in  
numpy.distutils, (b) a problem with my local setup, or (c) a problem  
with py2app?

Any thoughts?

Zach


From charlesr.harris at gmail.com  Sun Jun 29 02:03:26 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 29 Jun 2008 00:03:26 -0600
Subject: [Numpy-discussion] Ticket #164
Message-ID: <e06186140806282303p2a82a80cu5a1418190729913@mail.gmail.com>

Is the patch ticket in
#164<http://projects.scipy.org/scipy/numpy/ticket/164>still needed?
There was some discussion along similar lines here not so long
ago...

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080629/1376fd06/attachment.html>

From charlesr.harris at gmail.com  Sun Jun 29 11:57:52 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 29 Jun 2008 09:57:52 -0600
Subject: [Numpy-discussion] Should we fix Ticket #709?
Message-ID: <e06186140806290857x49d27bcfgf37d7a452f6be4ea@mail.gmail.com>

That's Ticket #709 <http://projects.scipy.org/scipy/numpy/ticket/709>:

> I'm faily sure that:
> numpy.isnan(datetime.datetime.now()
> ...should just return False and not raise an exception.
> rationale: anything that is not nan should just return False, not raise
> TypeErrors? <http://projects.scipy.org/scipy/numpy/wiki/TypeErrors>
>
<http://projects.scipy.org/scipy/numpy/ticket/709>Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080629/c790945b/attachment.html>

From perry at stsci.edu  Sun Jun 29 19:51:16 2008
From: perry at stsci.edu (Perry Greenfield)
Date: Sun, 29 Jun 2008 19:51:16 -0400
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <4863C798.5020700@stsci.edu>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>
	<4863C605.1080103@enthought.com> <4863C798.5020700@stsci.edu>
Message-ID: <859B44EE-F5BD-4547-BD6C-E2FDA5E4A8C7@stsci.edu>

Hi Chris,

Didn't we remove all dependence on recarray? I could have sworn we  
did that.

Perry

On Jun 26, 2008, at 12:45 PM, Christopher Hanley wrote:

> Travis E. Oliphant wrote:
>> St?fan van der Walt wrote:
>>> Hi all,
>>>
>>> I am documenting `recarray`, and have a question:
>>>
>>> Is its use still recommended, or has it been superseded by fancy  
>>> data-types?
>>>
>> I rarely recommend it's use (but some people do like attribute  
>> access to
>> the fields).    It is wrong, however, to say that recarray has been
>> superseded by fancy data types because fancy data types have  
>> existed for
>> as long as recarrays.
>>
>> I believe pyfits uses them quite a bit, and so they deserve to be
>> documented.
>>
>> -Travis
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> Travis is correct.  PyFITS uses recarrays quite extensively.  It  
> was the
> large, legacy numarray project I was referring too. ;-)
>
> I had forgotten about the attribute access.  I know a number of people
> who use that feature in conjunction with matplotlib for plotting  
> data in
> tables, especially during interactive use.
>
> Chris
>
>
> -- 
> Christopher Hanley
> Systems Software Engineer
> Space Telescope Science Institute
> 3700 San Martin Drive
> Baltimore MD, 21218
> (410) 338-4338
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From eads at soe.ucsc.edu  Sun Jun 29 20:24:43 2008
From: eads at soe.ucsc.edu (Damian Eads)
Date: Sun, 29 Jun 2008 18:24:43 -0600
Subject: [Numpy-discussion] copying in nanXXX
Message-ID: <486827CB.5010401@soe.ucsc.edu>

Hi there,

I'm using nansum for some code and noticed it does a bit of copying. 
Specifically, the nanxxx functions copy the input array, create an isnan 
boolean mask, set the nan values to make them insignificant (nansum: 0, 
nanmin: inf, or nanmax: -inf), and then call xxx to compute the statistic.

def nansum(a, axis=None):
     """Sum the array over the given axis, treating NaNs as 0.
     """
     y = array(a,subok=True)
     if not issubclass(y.dtype.type, _nx.integer):
         y[isnan(a)] = 0
     return y.sum(axis)

Unless someone has already done this, I will offer to write C versions 
of these functions. Does anyone else think this is useful?

Cheers,

Damian


From chanley at stsci.edu  Sun Jun 29 20:41:40 2008
From: chanley at stsci.edu (Christopher Hanley)
Date: Sun, 29 Jun 2008 20:41:40 -0400
Subject: [Numpy-discussion] Record arrays
In-Reply-To: <859B44EE-F5BD-4547-BD6C-E2FDA5E4A8C7@stsci.edu>
References: <9457e7c80806260913m15badd84sdf895fea5cdf7556@mail.gmail.com>	<4863C605.1080103@enthought.com>
	<4863C798.5020700@stsci.edu>
	<859B44EE-F5BD-4547-BD6C-E2FDA5E4A8C7@stsci.edu>
Message-ID: <48682BC4.2060201@stsci.edu>

Perry Greenfield wrote:
> Hi Chris,
> 
> Didn't we remove all dependence on recarray? I could have sworn we  
> did that.
> 
> Perry
> 


Perry,

You are right.  We no longer import the recarray module from numpy.

Chris


-- 
Christopher Hanley
Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338


From saketn at gmail.com  Sun Jun 29 20:47:46 2008
From: saketn at gmail.com (Saket)
Date: Sun, 29 Jun 2008 20:47:46 -0400
Subject: [Numpy-discussion] Eigenvectors in Matlab vs. Numpy
Message-ID: <120930130806291747x4e4fe9b9y3f45a6108455d087@mail.gmail.com>

Hi,

I'm having this weird problem when computing eigenvalues/vectors with
Numpy. I have the following symmetric matrix, B:

   -0.3462    0.6538    0.5385   -0.4615    0.6538   -0.3462   -0.3462   -0.3462
    0.6538   -0.3462    0.5385   -0.4615    0.6538   -0.3462   -0.3462   -0.3462
    0.5385    0.5385   -0.6154    0.3846    0.5385   -0.4615   -0.4615   -0.4615
   -0.4615   -0.4615    0.3846   -0.6154   -0.4615    0.5385    0.5385    0.5385
    0.6538    0.6538    0.5385   -0.4615   -0.3462   -0.3462   -0.3462   -0.3462
   -0.3462   -0.3462   -0.4615    0.5385   -0.3462   -0.3462    0.6538    0.6538
   -0.3462   -0.3462   -0.4615    0.5385   -0.3462    0.6538   -0.3462    0.6538
   -0.3462   -0.3462   -0.4615    0.5385   -0.3462    0.6538    0.6538   -0.3462

I compute the eigenvalues and eigenvectors of B using
numpy.linalg.eig(B). I get the following eigenvalues:

[  2.79128785e+00  -1.79128785e+00   1.64060486e-16  -3.07692308e-01
  -1.00000000e+00  -1.00000000e+00  -1.00000000e+00  -1.00000000e+00]

I do the same thing in Matlab and get the SAME eigenvalues. However,
my eigenVECTORS in Matlab versus numpy are different. It makes no
sense to me. In general, the following relationship should hold: Bx =
Lx, where B is my matrix, x is an eigenvector, and L is the
corresponding eigenvalue. For the eigenvectors that Matlab returns, I
have confirmed that the relationship does hold. But for the Numpy
eigenvectors, it doesn't!

Any idea why this might be happening? I did some computations myself
and it looks like the Matlab output is correct. Just seems like the
eigenvectors that Numpy is returning are wrong...

Thanks for any suggestions.

Saket


From charlesr.harris at gmail.com  Sun Jun 29 20:54:44 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 29 Jun 2008 18:54:44 -0600
Subject: [Numpy-discussion] copying in nanXXX
In-Reply-To: <486827CB.5010401@soe.ucsc.edu>
References: <486827CB.5010401@soe.ucsc.edu>
Message-ID: <e06186140806291754p3d529a45k213b150cab0f2cc9@mail.gmail.com>

On Sun, Jun 29, 2008 at 6:24 PM, Damian Eads <eads at soe.ucsc.edu> wrote:

> Hi there,
>
> I'm using nansum for some code and noticed it does a bit of copying.
> Specifically, the nanxxx functions copy the input array, create an isnan
> boolean mask, set the nan values to make them insignificant (nansum: 0,
> nanmin: inf, or nanmax: -inf), and then call xxx to compute the statistic.
>
> def nansum(a, axis=None):
>     """Sum the array over the given axis, treating NaNs as 0.
>     """
>     y = array(a,subok=True)
>     if not issubclass(y.dtype.type, _nx.integer):
>         y[isnan(a)] = 0
>     return y.sum(axis)
>
> Unless someone has already done this, I will offer to write C versions
> of these functions. Does anyone else think this is useful?
>

There are several places where numpy code can be made more efficient, even
in the c code. How to go about improving things needs to be discussed at
some point, but I don't think it's the first priority right now. That said,
every bit of code helps. So I think you should open an ticket for
enhancement and attach any code you come up with.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080629/440e8f10/attachment.html>

From charlesr.harris at gmail.com  Sun Jun 29 21:10:24 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 29 Jun 2008 19:10:24 -0600
Subject: [Numpy-discussion] Eigenvectors in Matlab vs. Numpy
In-Reply-To: <120930130806291747x4e4fe9b9y3f45a6108455d087@mail.gmail.com>
References: <120930130806291747x4e4fe9b9y3f45a6108455d087@mail.gmail.com>
Message-ID: <e06186140806291810g3b52807arcf2c9e229fe86de1@mail.gmail.com>

On Sun, Jun 29, 2008 at 6:47 PM, Saket <saketn at gmail.com> wrote:

> Hi,
>
> I'm having this weird problem when computing eigenvalues/vectors with
> Numpy. I have the following symmetric matrix, B:
>
>   -0.3462    0.6538    0.5385   -0.4615    0.6538   -0.3462   -0.3462
> -0.3462
>    0.6538   -0.3462    0.5385   -0.4615    0.6538   -0.3462   -0.3462
> -0.3462
>    0.5385    0.5385   -0.6154    0.3846    0.5385   -0.4615   -0.4615
> -0.4615
>   -0.4615   -0.4615    0.3846   -0.6154   -0.4615    0.5385    0.5385
>  0.5385
>    0.6538    0.6538    0.5385   -0.4615   -0.3462   -0.3462   -0.3462
> -0.3462
>   -0.3462   -0.3462   -0.4615    0.5385   -0.3462   -0.3462    0.6538
>  0.6538
>   -0.3462   -0.3462   -0.4615    0.5385   -0.3462    0.6538   -0.3462
>  0.6538
>   -0.3462   -0.3462   -0.4615    0.5385   -0.3462    0.6538    0.6538
> -0.3462
>
> I compute the eigenvalues and eigenvectors of B using
> numpy.linalg.eig(B). I get the following eigenvalues:
>
> [  2.79128785e+00  -1.79128785e+00   1.64060486e-16  -3.07692308e-01
>  -1.00000000e+00  -1.00000000e+00  -1.00000000e+00  -1.00000000e+00]
>
> I do the same thing in Matlab and get the SAME eigenvalues. However,
> my eigenVECTORS in Matlab versus numpy are different. It makes no
> sense to me. In general, the following relationship should hold: Bx =
> Lx, where B is my matrix, x is an eigenvector, and L is the
> corresponding eigenvalue. For the eigenvectors that Matlab returns, I
> have confirmed that the relationship does hold. But for the Numpy
> eigenvectors, it doesn't!
>
> Any idea why this might be happening? I did some computations myself
> and it looks like the Matlab output is correct. Just seems like the
> eigenvectors that Numpy is returning are wrong...
>
> Thanks for any suggestions.
>

Works for me:

In [16]: d,v = linalg.eig(A)

In [17]: abs(dot(A,v) - dot(v,diag(d))).max()
Out[17]: 1.1102230246251565e-15

Perhaps you are not applying the results correctly. You should also use eigh
for symmetric matrices. Note that Matlab, IIRC, returns the eigenvalues as a
diagonal matrix when you ask for the eigenvectors, while numpy returns a 1D
array that needs to be made into a diagonal array or simply multiplied
pointwise from the right, i.e.,

In [21]: abs(dot(A,v) - v*d).max()
Out[21]: 1.1102230246251565e-15

This is with arrays, matrices will be slightly different. If your problem
persists, please attach your configuration info.

In [28]: numpy.__config__.show()
atlas_threads_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/local/atlas/lib']
    language = f77
    include_dirs = ['/usr/local/atlas/include']

blas_opt_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/local/atlas/lib']
    define_macros = [('ATLAS_INFO', '"\\"3.7.35\\""')]
    language = c
    include_dirs = ['/usr/local/atlas/include']

atlas_blas_threads_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/local/atlas/lib']
    language = c
    include_dirs = ['/usr/local/atlas/include']

lapack_opt_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/local/atlas/lib']
    define_macros = [('ATLAS_INFO', '"\\"3.7.35\\""')]
    language = f77
    include_dirs = ['/usr/local/atlas/include']

lapack_mkl_info:
  NOT AVAILABLE

blas_mkl_info:
  NOT AVAILABLE

mkl_info:
  NOT AVAILABLE


Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080629/0d824394/attachment.html>

From charlesr.harris at gmail.com  Sun Jun 29 21:15:58 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 29 Jun 2008 19:15:58 -0600
Subject: [Numpy-discussion] Eigenvectors in Matlab vs. Numpy
In-Reply-To: <120930130806291747x4e4fe9b9y3f45a6108455d087@mail.gmail.com>
References: <120930130806291747x4e4fe9b9y3f45a6108455d087@mail.gmail.com>
Message-ID: <e06186140806291815h6b3a07d7wb42ec730434f24b1@mail.gmail.com>

On Sun, Jun 29, 2008 at 6:47 PM, Saket <saketn at gmail.com> wrote:

> Hi,
>
> I'm having this weird problem when computing eigenvalues/vectors with
> Numpy. I have the following symmetric matrix, B:
>
>   -0.3462    0.6538    0.5385   -0.4615    0.6538   -0.3462   -0.3462
> -0.3462
>    0.6538   -0.3462    0.5385   -0.4615    0.6538   -0.3462   -0.3462
> -0.3462
>    0.5385    0.5385   -0.6154    0.3846    0.5385   -0.4615   -0.4615
> -0.4615
>   -0.4615   -0.4615    0.3846   -0.6154   -0.4615    0.5385    0.5385
>  0.5385
>    0.6538    0.6538    0.5385   -0.4615   -0.3462   -0.3462   -0.3462
> -0.3462
>   -0.3462   -0.3462   -0.4615    0.5385   -0.3462   -0.3462    0.6538
>  0.6538
>   -0.3462   -0.3462   -0.4615    0.5385   -0.3462    0.6538   -0.3462
>  0.6538
>   -0.3462   -0.3462   -0.4615    0.5385   -0.3462    0.6538    0.6538
> -0.3462
>
> I compute the eigenvalues and eigenvectors of B using
> numpy.linalg.eig(B). I get the following eigenvalues:
>
> [  2.79128785e+00  -1.79128785e+00   1.64060486e-16  -3.07692308e-01
>  -1.00000000e+00  -1.00000000e+00  -1.00000000e+00  -1.00000000e+00]
>
> I do the same thing in Matlab and get the SAME eigenvalues. However,
> my eigenVECTORS in Matlab versus numpy are different. It makes no
> sense to me. In general, the following relationship should hold: Bx =
> Lx, where B is my matrix, x is an eigenvector, and L is the
> corresponding eigenvalue. For the eigenvectors that Matlab returns, I
> have confirmed that the relationship does hold. But for the Numpy
> eigenvectors, it doesn't!
>
> Any idea why this might be happening? I did some computations myself
> and it looks like the Matlab output is correct. Just seems like the
> eigenvectors that Numpy is returning are wrong...
>
> Thanks for any suggestions.
>

Also note that the -1 eigenvalue has multiplicity 4. This means that any set
of orthogonal vectors spanning the same eigenspace will do for eigenvectors,
i.e., they aren't unique and roundoff error is likely to have a large effect
on what you end up with.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080629/9f8fcac2/attachment.html>

From saketn at gmail.com  Sun Jun 29 22:00:22 2008
From: saketn at gmail.com (Saket)
Date: Sun, 29 Jun 2008 22:00:22 -0400
Subject: [Numpy-discussion] Eigenvectors in Matlab vs. Numpy
In-Reply-To: <e06186140806291815h6b3a07d7wb42ec730434f24b1@mail.gmail.com>
References: <120930130806291747x4e4fe9b9y3f45a6108455d087@mail.gmail.com>
	<e06186140806291815h6b3a07d7wb42ec730434f24b1@mail.gmail.com>
Message-ID: <120930130806291900g2f28f941q770c9314c50495e5@mail.gmail.com>

Hmm... so the relationship Ax = Lx should hold for every eigenvalue
and corresponding eigenvector of A, right? But, consider the first
eigenvalue,eigenvector pair:

    for i,eval in enumerate(d):
        print abs(numpy.dot(A,v[i]) - numpy.dot(eval,v[i])).max()
        return

Outputs: 1.928

I thought maybe the ith eigenvector corresponds to a different (not
the ith) eigenvalue, but there doesn't seem to be any eigenvalue which
corresponds to the ith eigenvector such that the relationship holds...

Thanks again.

Saket

atlas_threads_info:
  NOT AVAILABLE

blas_opt_info:
    libraries = ['f77blas', 'cblas', 'atlas']
    library_dirs = ['C:\\local\\lib\\yop\\sse2']
    define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')]
    language = c

atlas_blas_threads_info:
  NOT AVAILABLE

lapack_opt_info:
    libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
    library_dirs = ['C:\\local\\lib\\yop\\sse2']
    define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')]
    language = f77

atlas_info:
    libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
    library_dirs = ['C:\\local\\lib\\yop\\sse2']
    language = f77

lapack_mkl_info:
  NOT AVAILABLE

blas_mkl_info:
  NOT AVAILABLE

atlas_blas_info:
    libraries = ['f77blas', 'cblas', 'atlas']
    library_dirs = ['C:\\local\\lib\\yop\\sse2']
    language = c

mkl_info:
  NOT AVAILABLE



On Sun, Jun 29, 2008 at 9:15 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Sun, Jun 29, 2008 at 6:47 PM, Saket <saketn at gmail.com> wrote:
>>
>> Hi,
>>
>> I'm having this weird problem when computing eigenvalues/vectors with
>> Numpy. I have the following symmetric matrix, B:
>>
>>   -0.3462    0.6538    0.5385   -0.4615    0.6538   -0.3462   -0.3462
>> -0.3462
>>    0.6538   -0.3462    0.5385   -0.4615    0.6538   -0.3462   -0.3462
>> -0.3462
>>    0.5385    0.5385   -0.6154    0.3846    0.5385   -0.4615   -0.4615
>> -0.4615
>>   -0.4615   -0.4615    0.3846   -0.6154   -0.4615    0.5385    0.5385
>>  0.5385
>>    0.6538    0.6538    0.5385   -0.4615   -0.3462   -0.3462   -0.3462
>> -0.3462
>>   -0.3462   -0.3462   -0.4615    0.5385   -0.3462   -0.3462    0.6538
>>  0.6538
>>   -0.3462   -0.3462   -0.4615    0.5385   -0.3462    0.6538   -0.3462
>>  0.6538
>>   -0.3462   -0.3462   -0.4615    0.5385   -0.3462    0.6538    0.6538
>> -0.3462
>>
>> I compute the eigenvalues and eigenvectors of B using
>> numpy.linalg.eig(B). I get the following eigenvalues:
>>
>> [  2.79128785e+00  -1.79128785e+00   1.64060486e-16  -3.07692308e-01
>>  -1.00000000e+00  -1.00000000e+00  -1.00000000e+00  -1.00000000e+00]
>>
>> I do the same thing in Matlab and get the SAME eigenvalues. However,
>> my eigenVECTORS in Matlab versus numpy are different. It makes no
>> sense to me. In general, the following relationship should hold: Bx =
>> Lx, where B is my matrix, x is an eigenvector, and L is the
>> corresponding eigenvalue. For the eigenvectors that Matlab returns, I
>> have confirmed that the relationship does hold. But for the Numpy
>> eigenvectors, it doesn't!
>>
>> Any idea why this might be happening? I did some computations myself
>> and it looks like the Matlab output is correct. Just seems like the
>> eigenvectors that Numpy is returning are wrong...
>>
>> Thanks for any suggestions.
>
> Also note that the -1 eigenvalue has multiplicity 4. This means that any set
> of orthogonal vectors spanning the same eigenspace will do for eigenvectors,
> i.e., they aren't unique and roundoff error is likely to have a large effect
> on what you end up with.
>
> Chuck
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From saketn at gmail.com  Sun Jun 29 22:14:56 2008
From: saketn at gmail.com (Saket)
Date: Sun, 29 Jun 2008 22:14:56 -0400
Subject: [Numpy-discussion] Eigenvectors in Matlab vs. Numpy
In-Reply-To: <120930130806291900g2f28f941q770c9314c50495e5@mail.gmail.com>
References: <120930130806291747x4e4fe9b9y3f45a6108455d087@mail.gmail.com>
	<e06186140806291815h6b3a07d7wb42ec730434f24b1@mail.gmail.com>
	<120930130806291900g2f28f941q770c9314c50495e5@mail.gmail.com>
Message-ID: <120930130806291914g160ccacj7b3ccd3898154a34@mail.gmail.com>

Also, I'm getting different numbers when I do the above using eig(A)
and eigh(A). I thought eigh would be faster, but would still give the
same results...


On Sun, Jun 29, 2008 at 10:00 PM, Saket <saketn at gmail.com> wrote:
> Hmm... so the relationship Ax = Lx should hold for every eigenvalue
> and corresponding eigenvector of A, right? But, consider the first
> eigenvalue,eigenvector pair:
>
>    for i,eval in enumerate(d):
>        print abs(numpy.dot(A,v[i]) - numpy.dot(eval,v[i])).max()
>        return
>
> Outputs: 1.928
>
> I thought maybe the ith eigenvector corresponds to a different (not
> the ith) eigenvalue, but there doesn't seem to be any eigenvalue which
> corresponds to the ith eigenvector such that the relationship holds...
>
> Thanks again.
>
> Saket
>
> atlas_threads_info:
>  NOT AVAILABLE
>
> blas_opt_info:
>    libraries = ['f77blas', 'cblas', 'atlas']
>    library_dirs = ['C:\\local\\lib\\yop\\sse2']
>    define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')]
>    language = c
>
> atlas_blas_threads_info:
>  NOT AVAILABLE
>
> lapack_opt_info:
>    libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
>    library_dirs = ['C:\\local\\lib\\yop\\sse2']
>    define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')]
>    language = f77
>
> atlas_info:
>    libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
>    library_dirs = ['C:\\local\\lib\\yop\\sse2']
>    language = f77
>
> lapack_mkl_info:
>  NOT AVAILABLE
>
> blas_mkl_info:
>  NOT AVAILABLE
>
> atlas_blas_info:
>    libraries = ['f77blas', 'cblas', 'atlas']
>    library_dirs = ['C:\\local\\lib\\yop\\sse2']
>    language = c
>
> mkl_info:
>  NOT AVAILABLE
>
>
>
> On Sun, Jun 29, 2008 at 9:15 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>>
>>
>> On Sun, Jun 29, 2008 at 6:47 PM, Saket <saketn at gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I'm having this weird problem when computing eigenvalues/vectors with
>>> Numpy. I have the following symmetric matrix, B:
>>>
>>>   -0.3462    0.6538    0.5385   -0.4615    0.6538   -0.3462   -0.3462
>>> -0.3462
>>>    0.6538   -0.3462    0.5385   -0.4615    0.6538   -0.3462   -0.3462
>>> -0.3462
>>>    0.5385    0.5385   -0.6154    0.3846    0.5385   -0.4615   -0.4615
>>> -0.4615
>>>   -0.4615   -0.4615    0.3846   -0.6154   -0.4615    0.5385    0.5385
>>>  0.5385
>>>    0.6538    0.6538    0.5385   -0.4615   -0.3462   -0.3462   -0.3462
>>> -0.3462
>>>   -0.3462   -0.3462   -0.4615    0.5385   -0.3462   -0.3462    0.6538
>>>  0.6538
>>>   -0.3462   -0.3462   -0.4615    0.5385   -0.3462    0.6538   -0.3462
>>>  0.6538
>>>   -0.3462   -0.3462   -0.4615    0.5385   -0.3462    0.6538    0.6538
>>> -0.3462
>>>
>>> I compute the eigenvalues and eigenvectors of B using
>>> numpy.linalg.eig(B). I get the following eigenvalues:
>>>
>>> [  2.79128785e+00  -1.79128785e+00   1.64060486e-16  -3.07692308e-01
>>>  -1.00000000e+00  -1.00000000e+00  -1.00000000e+00  -1.00000000e+00]
>>>
>>> I do the same thing in Matlab and get the SAME eigenvalues. However,
>>> my eigenVECTORS in Matlab versus numpy are different. It makes no
>>> sense to me. In general, the following relationship should hold: Bx =
>>> Lx, where B is my matrix, x is an eigenvector, and L is the
>>> corresponding eigenvalue. For the eigenvectors that Matlab returns, I
>>> have confirmed that the relationship does hold. But for the Numpy
>>> eigenvectors, it doesn't!
>>>
>>> Any idea why this might be happening? I did some computations myself
>>> and it looks like the Matlab output is correct. Just seems like the
>>> eigenvectors that Numpy is returning are wrong...
>>>
>>> Thanks for any suggestions.
>>
>> Also note that the -1 eigenvalue has multiplicity 4. This means that any set
>> of orthogonal vectors spanning the same eigenspace will do for eigenvectors,
>> i.e., they aren't unique and roundoff error is likely to have a large effect
>> on what you end up with.
>>
>> Chuck
>>
>>
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>


From mattknox_ca at hotmail.com  Sun Jun 29 22:17:01 2008
From: mattknox_ca at hotmail.com (Matt Knox)
Date: Sun, 29 Jun 2008 19:17:01 -0700 (PDT)
Subject: [Numpy-discussion] SphinxDocString class in new numpy doc framework
Message-ID: <18187697.post@talk.nabble.com>


I guess this question is mostly for Stefan... but I am trying to port the
scikits.timeseries module wiki documentation into a sphinx framework, and
also
trying to follow the numpy doc string standards (which can't be parsed
directly
by sphinx), so I'm trying to use the SphinxDocString class in
docscrape_sphinx.py to pre-process the doc strings. However, the doc strings
do
not seem to be getting processed correctly. I didn't dig deep into the
issue,
but I thought I'd ask first before I go any further in case I am approaching
this incorrectly all together.

If I try to process the numpy.swapaxes docstring for example, I get the
following output (note the formatting of the parameters):

>>> print SphinxDocString(np.swapaxes.__doc__)

Return a view of array a with axis1 and axis2 interchanged.


**Parameters**
``````````````

    **a** : array_like

        Input array.
        axis1 : int
        First axis.
        axis2 : int
        Second axis.

**Examples**
````````````

        >>> x = np.array([[1,2,3]])
        >>> np.swapaxes(x,0,1)
        array([[1],
               [2],
               [3]])

        >>> x = np.array([[[0,1],[2,3]],[[4,5],[6,7]]])
        >>> x
        array([[[0, 1],
                [2, 3]],

               [[4, 5],
                [6, 7]]])
        >>> np.swapaxes(x,0,2)
        array([[[0, 4],
                [2, 6]],

               [[1, 5],
                [3, 7]]])
================================================================

I don't know if the SphinxDocString class is even ready for use yet or not,
but
this didn't look right to me.

The other thing I was wondering is if anyone knows how to pre-process doc
strings on the fly such that you can mix them in with your .rst files
similar to
how the autodoc sphinx extension works (rather than just pre-generating a
big
dump of all the doc strings into a single .rst file). This is probably a
question more for the sphinx mailing list, but I thought I'd ask while I'm
on
the topic in case anyone has any quick tricks they can share.

Thanks,

- Matt

-- 
View this message in context: http://www.nabble.com/SphinxDocString-class-in-new-numpy-doc-framework-tp18187697p18187697.html
Sent from the Numpy-discussion mailing list archive at Nabble.com.



From robert.kern at gmail.com  Sun Jun 29 22:34:33 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 29 Jun 2008 21:34:33 -0500
Subject: [Numpy-discussion] Eigenvectors in Matlab vs. Numpy
In-Reply-To: <120930130806291900g2f28f941q770c9314c50495e5@mail.gmail.com>
References: <120930130806291747x4e4fe9b9y3f45a6108455d087@mail.gmail.com>
	<e06186140806291815h6b3a07d7wb42ec730434f24b1@mail.gmail.com>
	<120930130806291900g2f28f941q770c9314c50495e5@mail.gmail.com>
Message-ID: <3d375d730806291934o780e4f77u48f71dc84ab071e8@mail.gmail.com>

On Sun, Jun 29, 2008 at 21:00, Saket <saketn at gmail.com> wrote:
> Hmm... so the relationship Ax = Lx should hold for every eigenvalue
> and corresponding eigenvector of A, right? But, consider the first
> eigenvalue,eigenvector pair:
>
>    for i,eval in enumerate(d):
>        print abs(numpy.dot(A,v[i]) - numpy.dot(eval,v[i])).max()
>        return
>
> Outputs: 1.928
>
> I thought maybe the ith eigenvector corresponds to a different (not
> the ith) eigenvalue, but there doesn't seem to be any eigenvalue which
> corresponds to the ith eigenvector such that the relationship holds...

The eigenvectors are the columns, not the rows.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From robert.kern at gmail.com  Sun Jun 29 22:35:47 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 29 Jun 2008 21:35:47 -0500
Subject: [Numpy-discussion] Eigenvectors in Matlab vs. Numpy
In-Reply-To: <120930130806291914g160ccacj7b3ccd3898154a34@mail.gmail.com>
References: <120930130806291747x4e4fe9b9y3f45a6108455d087@mail.gmail.com>
	<e06186140806291815h6b3a07d7wb42ec730434f24b1@mail.gmail.com>
	<120930130806291900g2f28f941q770c9314c50495e5@mail.gmail.com>
	<120930130806291914g160ccacj7b3ccd3898154a34@mail.gmail.com>
Message-ID: <3d375d730806291935p77bb77ecw3c9d2a9f6c7830a8@mail.gmail.com>

On Sun, Jun 29, 2008 at 21:14, Saket <saketn at gmail.com> wrote:
> Also, I'm getting different numbers when I do the above using eig(A)
> and eigh(A). I thought eigh would be faster, but would still give the
> same results...

What different numbers? Remember, as Chuck pointed out, you have a
multiplicity of -1 eigenvalues, so the actual eigenvectors
corresponding to those eigenvalues are not uniquely determined. The
details of the algorithm will determine exactly which ones you get.
eigh() has a different algorithm than eig(); that's the reason that it
is faster.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From saketn at gmail.com  Sun Jun 29 22:41:12 2008
From: saketn at gmail.com (Saket)
Date: Sun, 29 Jun 2008 22:41:12 -0400
Subject: [Numpy-discussion] Eigenvectors in Matlab vs. Numpy
In-Reply-To: <3d375d730806291935p77bb77ecw3c9d2a9f6c7830a8@mail.gmail.com>
References: <120930130806291747x4e4fe9b9y3f45a6108455d087@mail.gmail.com>
	<e06186140806291815h6b3a07d7wb42ec730434f24b1@mail.gmail.com>
	<120930130806291900g2f28f941q770c9314c50495e5@mail.gmail.com>
	<120930130806291914g160ccacj7b3ccd3898154a34@mail.gmail.com>
	<3d375d730806291935p77bb77ecw3c9d2a9f6c7830a8@mail.gmail.com>
Message-ID: <120930130806291941y3a011f54pef9253ad4197127c@mail.gmail.com>

Everything's working now. I just messed up the rows and columns.
Thanks a lot, guys.



On Sun, Jun 29, 2008 at 10:35 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Sun, Jun 29, 2008 at 21:14, Saket <saketn at gmail.com> wrote:
>> Also, I'm getting different numbers when I do the above using eig(A)
>> and eigh(A). I thought eigh would be faster, but would still give the
>> same results...
>
> What different numbers? Remember, as Chuck pointed out, you have a
> multiplicity of -1 eigenvalues, so the actual eigenvectors
> corresponding to those eigenvalues are not uniquely determined. The
> details of the algorithm will determine exactly which ones you get.
> eigh() has a different algorithm than eig(); that's the reason that it
> is faster.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From aisaac at american.edu  Mon Jun 30 01:23:55 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Mon, 30 Jun 2008 01:23:55 -0400
Subject: [Numpy-discussion] trace not in linalg namespace
Message-ID: <Mahogany-0.67.0-1032-20080630-012355.00@american.edu>

I just reached for trace in the numpy.linalg namespace 
instead of the numpy namespace, once again, and it made me 
wonder: is the current location most natural?
Related: is it surprising not to find it where I expected?  

Cheers,
Alan Isaac





From robert.kern at gmail.com  Mon Jun 30 01:23:08 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 30 Jun 2008 00:23:08 -0500
Subject: [Numpy-discussion] trace not in linalg namespace
In-Reply-To: <Mahogany-0.67.0-1032-20080630-012355.00@american.edu>
References: <Mahogany-0.67.0-1032-20080630-012355.00@american.edu>
Message-ID: <3d375d730806292223u6835590bh75721a0f901ea513@mail.gmail.com>

On Mon, Jun 30, 2008 at 00:23, Alan G Isaac <aisaac at american.edu> wrote:
> I just reached for trace in the numpy.linalg namespace
> instead of the numpy namespace, once again, and it made me
> wonder: is the current location most natural?

I don't think that changing it improves matters significantly,
certainly not enough to justify the cost of the change.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From pgmdevlist at gmail.com  Mon Jun 30 01:32:33 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 30 Jun 2008 01:32:33 -0400
Subject: [Numpy-discussion] SphinxDocString class in new numpy doc
	framework
In-Reply-To: <18187697.post@talk.nabble.com>
References: <18187697.post@talk.nabble.com>
Message-ID: <200806300132.33551.pgmdevlist@gmail.com>

Matt,
David Huard and myself have also been working on that. So far, we have a set 
of directives (numpyxxx, where xxx is function, attribute, method...) that 
satisfy the numpy standard and can be processed by sphinx. We also have the 
equivalent autodirectives.

https://code.launchpad.net/~david-huard/scipy/numpydoc
https://code.launchpad.net/~pierregm/scipy/numpydocs

So far, we focused on making sure that the numpy standard was satisfied. I 
haven't tried to mix numpy standard and additional rst directives, I'll try 
to check that some time this week.

Let me know how it goes.


From stefan at sun.ac.za  Mon Jun 30 02:14:41 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 30 Jun 2008 08:14:41 +0200
Subject: [Numpy-discussion] SphinxDocString class in new numpy doc
	framework
In-Reply-To: <18187697.post@talk.nabble.com>
References: <18187697.post@talk.nabble.com>
Message-ID: <9457e7c80806292314j26f20e73u78a67f2309f14a90@mail.gmail.com>

Hi Matt

The docstring for ``np.swapaxes`` is broken -- notice the indent after
the first line.  If you remove that, things should be fine:

In [31]: d =  np.swapaxes.__doc__

In [32]: d = d.split('\n')

In [33]: d[0] = '    Return a view of an array a with axis1 and axis2
interchanged.'

In [34]: print SphinxDocString('\n'.join(d))

Return a view of an array a with axis1 and axis2 interchanged.


**Parameters**
``````````````

    **a** : array_like

        Input array.


etc.



Cheers
St?fan

2008/6/30 Matt Knox <mattknox_ca at hotmail.com>:
>
> I guess this question is mostly for Stefan... but I am trying to port the
> scikits.timeseries module wiki documentation into a sphinx framework, and
> also
> trying to follow the numpy doc string standards (which can't be parsed
> directly
> by sphinx), so I'm trying to use the SphinxDocString class in
> docscrape_sphinx.py to pre-process the doc strings. However, the doc strings
> do
> not seem to be getting processed correctly. I didn't dig deep into the
> issue,
> but I thought I'd ask first before I go any further in case I am approaching
> this incorrectly all together.
>
> If I try to process the numpy.swapaxes docstring for example, I get the
> following output (note the formatting of the parameters):
>
>>>> print SphinxDocString(np.swapaxes.__doc__)


From alan.mcintyre at gmail.com  Mon Jun 30 13:17:30 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 30 Jun 2008 13:17:30 -0400
Subject: [Numpy-discussion] Undefined "wrap" in linalg.py:tensorsolve.py
Message-ID: <1d36917a0806301017t7e527cfdg8a6ed1fc3141fade@mail.gmail.com>

While I was working on the doctest tweaks, I ran across a doctest that
failed because tensorsolve uses a function "wrap" that's not defined.
It seems that I should make the following change to fix it (at the
start of tensorsolve's code) :

-    a = asarray(a)
+    a,wrap = _makearray(a)

Doing so seems to make the example run, but I just wanted to see if
this was the appropriate thing to do.

Thanks,
Alan


From robert.kern at gmail.com  Mon Jun 30 13:20:16 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 30 Jun 2008 12:20:16 -0500
Subject: [Numpy-discussion] Undefined "wrap" in linalg.py:tensorsolve.py
In-Reply-To: <1d36917a0806301017t7e527cfdg8a6ed1fc3141fade@mail.gmail.com>
References: <1d36917a0806301017t7e527cfdg8a6ed1fc3141fade@mail.gmail.com>
Message-ID: <3d375d730806301020i52a05122m55cc392ef77637c2@mail.gmail.com>

On Mon, Jun 30, 2008 at 12:17, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> While I was working on the doctest tweaks, I ran across a doctest that
> failed because tensorsolve uses a function "wrap" that's not defined.
> It seems that I should make the following change to fix it (at the
> start of tensorsolve's code) :
>
> -    a = asarray(a)
> +    a,wrap = _makearray(a)
>
> Doing so seems to make the example run, but I just wanted to see if
> this was the appropriate thing to do.

It is.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From alan.mcintyre at gmail.com  Mon Jun 30 13:54:59 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 30 Jun 2008 13:54:59 -0400
Subject: [Numpy-discussion] More pending test framework changes (please give
	feedback)
Message-ID: <1d36917a0806301054t578dfc94n64ffc213cb3c79df@mail.gmail.com>

Hi all,

I can commit any/all of the following changes as long as they are
agreeable to everybody:

1. All doctests in NumPy will have the numpy module available in their
execution context as "np".

2. Turn on the normalized whitespace option for all doctests.  Having
a doctest fail just because there's a space after your result seems
like an unnecessary hassle for documenters.

3. Output will be ignored for each doctest expected output line that
contains "#random". I figured this can serve both as an ignore flag
and indication to the reader that the listed output may differ from
what they see if they execute the associated command. So you would be
able to do:
>>> random.random()
0.1234567890                           #random: output may differ on your system

And have the example executed but not cause a failure.  You could also
use this to ignore the <SomeObject at 0x1234ABCD> output from plot
methods as well.

If these get committed, I'll also revert doctests I changed earlier so
that they use the provided "np" instead of importing it, and fix any
currently failing doctests that can be addressed with the new
behavior.  I've already got a batch of these changes made locally just
to try out the new features, but I'll wait to see if adding the
features causes any troubles on the buildbots first.

There are a couple of options that I haven't got patches for yet:
4. Doctest commands starting with "plt." or "plot." will automatically
have their output ignored.
5. Add a decorator that causes all doctest output for that
function/method's docstring to be ignored. (This is for cases with
many examples, all of which should be executed but not compared, so
you don't have to clutter up each output line with #random or
ellipsis)

If there is still a big need for these, I'll work on them.  As an
alternative to 3,4 and 5, you could use the ELLIPSIS directive to
ignore the varying bits of the output, but you'd still have to put the
directive on each command line where you needed it.  Unless I turn on
the ellipsis feature for all doctests, that is; it's easy to do, and
it doesn't cause me any obvious problems locally.  I don't know
whether that's something that's generally a good idea, though.

Thanks,
Alan


From alan.mcintyre at gmail.com  Mon Jun 30 14:38:20 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 30 Jun 2008 14:38:20 -0400
Subject: [Numpy-discussion] Running doctests on buildbots
Message-ID: <1d36917a0806301138n3e2b799fwe73decb137f55bd5@mail.gmail.com>

Hi all,

It seems to me that the buildbots should be running the doctests in
addition to the unit tests, but right now the buildbot test command
"numpy.test(level=9999,verbosity=9999)" doesn't do that.  We could
either add "doctests=True" to the buildbot test command, or make
"level" above some threshhold run them (currently the level argument
is just ignored anyway).  Since the level and verbosity parameters are
going away in the release after 1.2, it seems more appropriate to just
add the doctests argument.  Any thoughts?

Thanks,
Alan

P.S. Thanks to whoever installed nose on the buildbots!


From robert.kern at gmail.com  Mon Jun 30 14:55:37 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 30 Jun 2008 13:55:37 -0500
Subject: [Numpy-discussion] Running doctests on buildbots
In-Reply-To: <1d36917a0806301138n3e2b799fwe73decb137f55bd5@mail.gmail.com>
References: <1d36917a0806301138n3e2b799fwe73decb137f55bd5@mail.gmail.com>
Message-ID: <3d375d730806301155n4e53414er56a28b8db339a1b4@mail.gmail.com>

On Mon, Jun 30, 2008 at 13:38, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> Hi all,
>
> It seems to me that the buildbots should be running the doctests in
> addition to the unit tests, but right now the buildbot test command
> "numpy.test(level=9999,verbosity=9999)" doesn't do that.  We could
> either add "doctests=True" to the buildbot test command, or make
> "level" above some threshhold run them (currently the level argument
> is just ignored anyway).  Since the level and verbosity parameters are
> going away in the release after 1.2, it seems more appropriate to just
> add the doctests argument.  Any thoughts?

Add the doctests argument.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From alan.mcintyre at gmail.com  Mon Jun 30 15:06:05 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 30 Jun 2008 15:06:05 -0400
Subject: [Numpy-discussion] Running doctests on buildbots
In-Reply-To: <3d375d730806301155n4e53414er56a28b8db339a1b4@mail.gmail.com>
References: <1d36917a0806301138n3e2b799fwe73decb137f55bd5@mail.gmail.com>
	<3d375d730806301155n4e53414er56a28b8db339a1b4@mail.gmail.com>
Message-ID: <1d36917a0806301206g574dbb96m7a3a485d0e04cfbc@mail.gmail.com>

On Mon, Jun 30, 2008 at 2:55 PM, Robert Kern <robert.kern at gmail.com> wrote:
> Add the doctests argument.

Where is the buildbot configuration kept?


From robert.kern at gmail.com  Mon Jun 30 15:08:21 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 30 Jun 2008 14:08:21 -0500
Subject: [Numpy-discussion] Running doctests on buildbots
In-Reply-To: <1d36917a0806301206g574dbb96m7a3a485d0e04cfbc@mail.gmail.com>
References: <1d36917a0806301138n3e2b799fwe73decb137f55bd5@mail.gmail.com>
	<3d375d730806301155n4e53414er56a28b8db339a1b4@mail.gmail.com>
	<1d36917a0806301206g574dbb96m7a3a485d0e04cfbc@mail.gmail.com>
Message-ID: <3d375d730806301208u35d1abe9y86a46d9866f35d8e@mail.gmail.com>

On Mon, Jun 30, 2008 at 14:06, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> On Mon, Jun 30, 2008 at 2:55 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> Add the doctests argument.
>
> Where is the buildbot configuration kept?

I have not the slightest idea. St?fan?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco

From fperez.net at gmail.com  Mon Jun 30 18:12:40 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 30 Jun 2008 15:12:40 -0700
Subject: [Numpy-discussion] Python/Sage minisymposium at SIAM Annual meeting
	July 9/10 2008
Message-ID: <db6b5ecc0806301512j4fff0558u45f635fc7ed524e2@mail.gmail.com>

Hi all,

this is just a reminder, in case you'll be attending the SIAM 2008
annual meeting next week in San Diego, that there will be a 3-part
minisymposium focusing on the uses of Python and Sage in scientific
computing:

http://meetings.siam.org/sess/dsp_programsess.cfm?SESSIONCODE=7369
http://meetings.siam.org/sess/dsp_programsess.cfm?SESSIONCODE=7370
http://meetings.siam.org/sess/dsp_programsess.cfm?SESSIONCODE=7447

We hope to see many of you there!

Regards,

Fernando and Randy


From dalke at dalkescientific.com  Mon Jun 30 19:32:02 2008
From: dalke at dalkescientific.com (Andrew Dalke)
Date: Tue, 1 Jul 2008 01:32:02 +0200
Subject: [Numpy-discussion] "import numpy" is slow
In-Reply-To: <F86FB1CD-C0FB-4D16-AB50-809E74B5920A@dalkescientific.com>
References: <F86FB1CD-C0FB-4D16-AB50-809E74B5920A@dalkescientific.com>
Message-ID: <92AFDF8C-418F-4B6D-895C-8E7F0FD47D90@dalkescientific.com>

(Trying again now that I'm subscribed.  BTW, there's no link to the  
subscription page from numpy.scipy.org .)


The initial 'import numpy' loads a huge number of modules, even when  
I don't need them.

Python 2.5 (r25:51918, Sep 19 2006, 08:49:13)
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> import sys
 >>> len(sys.modules)
28
 >>> import numpy
 >>> len(sys.modules)
256
 >>> len([s for s in sorted(sys.modules) if 'numpy' in s])
127
 >>> numpy.__version__
'1.1.0'
 >>>

As a result, I assume that's the reason my program's startup cost is  
quite high.

[josiah:~/src/fp] dalke% time python -c 'a=4'
0.014u 0.038s 0:00.05 80.0%     0+0k 0+1io 0pf+0w
[josiah:~/src/fp] dalke% time python -c 'import numpy'
0.161u 0.279s 0:00.44 97.7%     0+0k 0+9io 0pf+0w

My total runtime is something like 1.4 seconds, and the only thing  
I'm using NumPy for is to make an array of doubles that I can pass to  
a C extension.  (I could use the array module or ctypes, but figured  
numpy is more useful for downstream code.)

Why does numpy/__init__.py need to import all of these other modules  
and submodules?  Any chance of cutting down on the number, in order  
to improve startup costs?

				Andrew
				dalke at dalkescientific.com



From robert.kern at gmail.com  Mon Jun 30 20:22:22 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 30 Jun 2008 19:22:22 -0500
Subject: [Numpy-discussion] "import numpy" is slow
In-Reply-To: <92AFDF8C-418F-4B6D-895C-8E7F0FD47D90@dalkescientific.com>
References: <F86FB1CD-C0FB-4D16-AB50-809E74B5920A@dalkescientific.com>
	<92AFDF8C-418F-4B6D-895C-8E7F0FD47D90@dalkescientific.com>
Message-ID: <3d375d730806301722h335cd094n5cc28ff4c37002b1@mail.gmail.com>

On Mon, Jun 30, 2008 at 18:32, Andrew Dalke <dalke at dalkescientific.com> wrote:
> (Trying again now that I'm subscribed.  BTW, there's no link to the
> subscription page from numpy.scipy.org .)
>
>
> The initial 'import numpy' loads a huge number of modules, even when
> I don't need them.
>
> Python 2.5 (r25:51918, Sep 19 2006, 08:49:13)
> [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import sys
>  >>> len(sys.modules)
> 28
>  >>> import numpy
>  >>> len(sys.modules)
> 256
>  >>> len([s for s in sorted(sys.modules) if 'numpy' in s])
> 127
>  >>> numpy.__version__
> '1.1.0'
>  >>>
>
> As a result, I assume that's the reason my program's startup cost is
> quite high.
>
> [josiah:~/src/fp] dalke% time python -c 'a=4'
> 0.014u 0.038s 0:00.05 80.0%     0+0k 0+1io 0pf+0w
> [josiah:~/src/fp] dalke% time python -c 'import numpy'
> 0.161u 0.279s 0:00.44 97.7%     0+0k 0+9io 0pf+0w
>
> My total runtime is something like 1.4 seconds, and the only thing
> I'm using NumPy for is to make an array of doubles that I can pass to
> a C extension.  (I could use the array module or ctypes, but figured
> numpy is more useful for downstream code.)
>
> Why does numpy/__init__.py need to import all of these other modules
> and submodules?

Strictly speaking, there is no *need* for any of it. It was a judgment
call trading off import time for the convenience in fairly typical use
cases which do use functions across the breadth of the library. Your
use case isn't so typical and so suffers on the import time end of the
balance.

> Any chance of cutting down on the number, in order
> to improve startup costs?

Not at this point in time, no. That would break too much code.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


From dalke at dalkescientific.com  Mon Jun 30 21:00:59 2008
From: dalke at dalkescientific.com (Andrew Dalke)
Date: Tue, 1 Jul 2008 03:00:59 +0200
Subject: [Numpy-discussion] "import numpy" is slow
In-Reply-To: <3d375d730806301722h335cd094n5cc28ff4c37002b1@mail.gmail.com>
References: <F86FB1CD-C0FB-4D16-AB50-809E74B5920A@dalkescientific.com>
	<92AFDF8C-418F-4B6D-895C-8E7F0FD47D90@dalkescientific.com>
	<3d375d730806301722h335cd094n5cc28ff4c37002b1@mail.gmail.com>
Message-ID: <F91CA90F-073B-43D3-A2E8-3E5AEA296755@dalkescientific.com>

On Jul 1, 2008, at 2:22 AM, Robert Kern wrote:
> Your use case isn't so typical and so suffers on the import time  
> end of the
> balance.

I'm working on my presentation for EuroSciPy.  "Isn't so typical"  
seems to be a good summary of my first slide.  :)

>> Any chance of cutting down on the number, in order
>> to improve startup costs?
>
> Not at this point in time, no. That would break too much code.

Understood.

Thanks for the response,

				Andrew
				dalke at dalkescientific.com