Mailman 3 August 2006 - NumPy-Discussion

Patch against Image.py in the PIL
by Travis Oliphant Aug. 20, 2006

Aug. 20, 2006

Attached is a patch that makes PIL Image objects both export and consume the array interface. -Travis Index: PIL/Image.py =================================================================== --- PIL/Image.py (revision 358) +++ PIL/Image.py (working copy) @@ -187,6 +187,42 @@ } +if sys.byteorder == 'little': + _ENDIAN = '<' +else: + _ENDIAN = '>' + +_MODE_CONV = { + + # official modes + "1": ('|b1', None), + "L": ('|u1', None), + "I": ('%si4' % _ENDIAN, None), + "F": ('%sf4' % _ENDIAN, None), + "P": ('|u1', None), + "RGB": ('|u1', 3), + "RGBX": ('|u1', 4), + "RGBA": ('|u1', 4), + "CMYK": ('|u1', 4), + "YCbCr": ('|u1', 4), + + # Experimental modes include I;16, I;16B, RGBa, BGR;15, + # and BGR;24. Use these modes only if you know exactly + # what you're doing... + +} + +def _conv_type_shape(im): + shape = im.size[::-1] + typ, extra = _MODE_CONV[im.mode] + if extra is None: + return shape, typ + else: + return shape+(extra,), typ + + + + MODES = _MODEINFO.keys() MODES.sort() @@ -491,6 +527,22 @@ return string.join(data, "") ## + # Returns the array_interface dictionary + # + # @return A dictionary with keys 'shape', 'typestr', 'data' + + def __get_array_interface__(self): + new = {} + shape, typestr = _conv_type_shape(self) + new['shape'] = shape + new['typestr'] = typestr + new['data'] = self.tostring() + return new + + __array_interface__ = property(__get_array_interface__, None, doc="array interface") + + + ## # Returns the image converted to an X11 bitmap. This method # only works for mode "1" images. # @@ -1749,7 +1801,61 @@ return apply(fromstring, (mode, size, data, decoder_name, args)) + ## +# (New in 1.1.6) Create an image memory from an object exporting +# the array interface (using the buffer protocol). +# +# If obj is not contiguous, then the tostring method is called +# and frombuffer is used +# +# @param obj Object with array interface +# @param mode Mode to use (will be determined from type if None) + +def fromarray(obj, mode=None): + arr = obj.__array_interface__ + shape = arr['shape'] + ndim = len(shape) + try: + strides = arr['strides'] + except KeyError: + strides = None + if mode is None: + typestr = arr['typestr'] + if not (typestr[0] == '|' or typestr[0] == _ENDIAN or + typestr[1:] not in ['u1', 'b1', 'i4', 'f4']): + raise TypeError, "cannot handle data-type" + typestr = typestr[:2] + if typestr == 'i4': + mode = 'I' + elif typestr == 'f4': + mode = 'F' + elif typestr == 'b1': + mode = '1' + elif ndim == 2: + mode = 'L' + elif ndim == 3: + mode = 'RGB' + elif ndim == 4: + mode = 'RGBA' + else: + raise TypeError, "Do not understand data." + ndmax = 4 + bad_dims=0 + if mode in ['1','L','I','P','F']: + ndmax = 2 + elif mode == 'RGB': + ndmax = 3 + if ndim > ndmax: + raise ValueError, "Too many dimensions." + + size = shape[:2][::-1] + if strides is not None: + obj = obj.tostring() + + return frombuffer(mode, size, obj) + +## # Opens and identifies the given image file. # <p> # This is a lazy operation; this function identifies the file, but the

2 1

re:生产一线主管技能提升
by "8月26-27日/苏州" Aug. 20, 2006

Aug. 20, 2006

1 0

bugfix-patch for numpy-1.0b2 setup
by Norbert Nemec Aug. 20, 2006

Aug. 20, 2006

Hi there, in numpy-1.0b2 the logic in setup.py is slightly off. The attached patch fixes the issue. Greetings, Norbert PS: I would have preferred to submit this patch via the sourceforge bug-tracker, but that seems rather confusing: there are tabs "Numarray Patches" and "Numarray Bugs" but no "NumPy bugs" and the tab "Patches" seems to be used for Numeric. Why isn't NumPy handled via the Sourceforge page? Index: numpy/numpy/core/setup.py =================================================================== --- numpy.orig/numpy/core/setup.py 2006-08-13 02:06:58.000000000 +0200 +++ numpy/numpy/core/setup.py 2006-08-18 15:22:10.000000000 +0200 @@ -102,16 +102,15 @@ if check_func('strtod'): moredefs.append(('PyOS_ascii_strtod', 'strtod')) - if moredefs: - target_f = open(target,'a') - for d in moredefs: - if isinstance(d,str): - target_f.write('#define %s\n' % (d)) - else: - target_f.write('#define %s %s\n' % (d[0],d[1])) - if not nosmp: # default is to use WITH_THREAD - target_f.write('#ifdef WITH_THREAD\n#define NPY_ALLOW_THREADS 1\n#else\n#define NPY_ALLOW_THREADS 0\n#endif\n') - target_f.close() + target_f = open(target,'a') + for d in moredefs: + if isinstance(d,str): + target_f.write('#define %s\n' % (d)) + else: + target_f.write('#define %s %s\n' % (d[0],d[1])) + if not nosmp: # default is to use WITH_THREAD + target_f.write('#ifdef WITH_THREAD\n#define NPY_ALLOW_THREADS 1\n#else\n#define NPY_ALLOW_THREADS 0\n#endif\n') + target_f.close() else: mathlibs = [] target_f = open(target)

5 6

1.02b
by kortmann＠ideaworks.com Aug. 20, 2006

Aug. 20, 2006

I realize it was just released, but is there going to be a windows release for 1.02b?

3 2

install on Python 2.5
by Daniel Fish Aug. 20, 2006

Aug. 20, 2006

Any advice on installing numpy in Python 2.5 on WindowsXP?

1 0

Whitespace
by Charles R Harris Aug. 19, 2006

Aug. 19, 2006

Hi All, I've noticed a lot of trailing whitespace while browsing through the numpy subversion repository. So here is a perl script I pinched from the linux-kernel mailing list that does a good job of removing it. Chuck

1 0

Interesting memory leak
by David Grant Aug. 18, 2006

Aug. 18, 2006

Hello all, I had a massive memory leak in some of my code. It would basically end up using up all 1GB of my RAM or more if I don't kill the application. I managed to finally figure out which portion of the code was causing the leak (with great difficulty) and have a little example which exposes the leak. I am using numpy-0.9.8 and I'm wondering if perhaps this is already fixed in 1.0b2. Run this through valgrind with appropriate options (I used the recommended valgrind_py.sh that I found on scipy's site somewhere) and this will leak 100kB. Increase the xrange on the big loop and you can watch the memory increase over time in top. The interesting thing is that the only difference between the leaky and non-leaky code is: if not adjacencyMatrix[anInt2,anInt1] == 0: (leaky) vs. if not adjacencyMatrix[anInt2][anInt1] == 0: (non-leaky) however another way to make the leaky code non-leaky is to change anArrayOfInts to just be [1] Here's the code: from numpy import array def leakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix): ys = set() for aList in aListOfArrays: for anInt1 in anArrayOfInts: for anInt2 in aList: if not adjacencyMatrix[anInt2,anInt1] == 0: ys.add(anInt1) return ys def nonLeakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix): ys = set() for aList in aListOfArrays: for anInt1 in anArrayOfInts: for anInt2 in aList: if not adjacencyMatrix[anInt2][anInt1] == 0: ys.add(anInt1) return ys if __name__ == "__main__": for i in xrange(10000): aListOfArrays = [[0, 1]] anArrayOfInts = array([1]) adjacencyMatrix = array([[0,1],[1,0]]) #COMMENT OUT ONE OF THE 2 LINES BELOW #bar = nonLeakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix) bar = leakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix) -- David Grant http://www.davidgrant.ca

4 4

[Fwd: Re: Signal handling]
by Fernando Perez Aug. 18, 2006

Aug. 18, 2006

Hi all, here is the SAGE signal handling code, graciously donated by William Stein. I'd suggest putting (with any modifications to adapt it to numpy conventions) this into the actual numpy headers, so that not only all of our auto-generation tools (f2py, weave) can use it, but so that it also becomes trivial for end-users to user the same macros in their own code without doing anything additional. Regards, f -------- Original Message -------- Subject: Re: Signal handling Date: Fri, 18 Aug 2006 21:15:38 +0000 From: William Stein <wstein(a)gmail.com> To: Fernando Perez <Fernando.Perez(a)colorado.edu> References: <44E586D3.7010209(a)colorado.edu> Here you are (see attached). Let me know if you have any trouble with gmail mangling the attachment. On 8/18/06, Fernando Perez <Fernando.Perez(a)colorado.edu> wrote: > Hi William, > > could you please send me 'officially' an email with the interrupt.{c,h} files > and a notice of them being BSD licensed ? With that, I can then forward them > to the numpy list and work on their inclusion tomorrow. -- William Stein Associate Professor of Mathematics University of Washington /* Interrupt handling for extension code (header file) Copyright (c) 2006, William Stein, under "The BSD License" (see below). All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the SAGE Foundation nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include "interrupt.h" void sig_handle(int n) { signal(SIGINT, last_handler); /* restore python signal handler (or will crash when user presses ctrl-c in the interpreter). */ siglongjmp(env, n); } /* Interrupt handling for extension code (header file) Copyright (c) 2006, William Stein, under "The BSD License" (see below). All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the SAGE Foundation nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ /**************************************************************** WARNING: Only modify the version of this file in the sage/ext/ directory. It overwrites all other copies of this file during the SAGE build. ****************************************************************/ #ifndef FOO_H #define FOO_H #include <setjmp.h> #include <Python.h> #include <signal.h> sigjmp_buf env; /* In your Pyrex code, put _sig_on [pure c code block] _sig_off to get signal handling capabilities. VERY VERY IMPORTANT: 1. These *must* always come in pairs. E.g., if you have just a _sig_off without a corresponding _sig_on, then ctrl-c later in the interpreter will sigfault! 2. Do *not* put these in the __init__ method of a Pyrex extension class, or you'll get crashes. 3. Do not put these around any non-pure C code!!!! 4. If you need to do a check of control-c inside a loop, e.g., when constructing a matrix, put _sig_check instead of using _sig_on then _sig_off. */ /* Unfortunately signal handler function declarations vary HUGELY from platform to platform: */ #if defined(__CYGWIN32__) /* Windows XP */ _sig_func_ptr last, last_handler; #elif defined(__FreeBSD__) /* FreeBSD */ sig_t last, last_handler; #elif defined(__APPLE__) /* OSX */ sig_t last, last_handler; #elif defined (__sun__) || defined (__sun) /* Solaris */ typedef void (*__sighandler_t )(); __sighandler_t last, last_handler; #else /* Other, e.g., Linux */ __sighandler_t last, last_handler; #endif static PyObject *_m; void sig_handle(int n); static int __sig__n, __sig_str__n; #define _sig_on last = signal(SIGINT, sig_handle); \ if (last != sig_handle) \ last_handler = last; \ signal(SIGALRM, sig_handle); \ signal(SIGSEGV, sig_handle); \ signal(SIGABRT, sig_handle); \ signal(SIGFPE, sig_handle); \ if (__sig__n = sigsetjmp(env,1)) { \ if (__sig__n == SIGINT) \ PyErr_SetString(PyExc_KeyboardInterrupt, ""); \ else \ if (__sig__n == SIGALRM) \ PyErr_SetString(PyExc_KeyboardInterrupt, "Alarm received"); \ else \ PyErr_SetString(PyExc_RuntimeError, ""); \ return(0); \ } #define _sig_str(s) last = signal(SIGINT, sig_handle); \ if (last != sig_handle) \ last_handler = last; \ signal(SIGSEGV, sig_handle); \ signal(SIGABRT, sig_handle); \ signal(SIGFPE, sig_handle); \ signal(SIGALRM, sig_handle); \ if (__sig_str__n = sigsetjmp(env,1)) { \ if(__sig_str__n == SIGINT ) \ PyErr_SetString(PyExc_KeyboardInterrupt, s); \ else \ if (__sig__n == SIGALRM) \ PyErr_SetString(PyExc_KeyboardInterrupt, s); \ else \ PyErr_SetString(PyExc_RuntimeError, s); \ return(0); \ } #define _sig_off signal(SIGINT, last_handler); signal(SIGALRM, last_handler); signal(SIGFPE, last_handler); signal(SIGABRT, last_handler); #define _sig_check _sig_on _sig_off #endif /* FOO_H */ /* I thought maybe the following would work nicely, instead of just returning, but no (it crashes sometimes!). Also the line number is not set correctly. The only way I can imagine setting this correctly is if the following bit of code is generated within pyrex, since pyrex knows the line numbers. PyErr_SetString(PyExc_RuntimeError, "(complete C-level traceback not available)"); \ __pyx_filename = __pyx_f[1]; __pyx_lineno = 0; goto __pyx_L1; \ } */

2 1

numpy 0.9.8->1.0b2
by David Grant Aug. 18, 2006

Aug. 18, 2006

I'm contemplating upgrading to 1.0b2. The main reason is that I am experiencing a major memory leak and before I report a bug I think the developers would appeciate if I was using the most recent version. Am I correct in that the only major change that might actually break my code is that the following functions: take, repeat, sum, product, sometrue, cumsum, cumproduct, ptp, amax, amin, prod, cumprod, mean, std, var now have axis=None as argument? BTW, how come alter_code2.py ( http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/oldnumeric/alter_…) says in the docstring that it "converts functions that don't give axis= keyword that have changed" but I don't see it actually doing that anywhere in the code? Thanks, David -- David Grant http://www.davidgrant.ca

2 2

convertcode.py
by Chris Kees Aug. 18, 2006

Aug. 18, 2006

Hi, I just ran convertcode.py on my code (from the latest svn source of numpy) and it looks like it just changed the import statements to import numpy.oldnumeric as Numeric So it doesn't look like it's really helping me move over to the new usage. Is there a script that will converts code to use the new numpy as it's intended to be used? Thanks, Chris

3 2