Mailman 3 November 2007 - NumPy-Discussion

saving and loading as PNG
by Martin Teichmann 08 Nov '07

08 Nov '07

Hi all, i am working with numpy and mostly 16 bit matrices, and so I was looking for a standard way of saving them. I found that PNG actually supports that, but I could not find a way to use this feature from within python. I thought that it actually is a cool way of storing matrices and thus I wrote a little pyrex program that does the job. I figured this might be interesting to a broader audience, so I posted it here, maybe for inclusion in numpy? Some more thoughts: * Other implementations: There is other people who have done such a thing. the PIL knows how to read and write PNG, but only 8 bit. The same holds for matplotlib. * Problems: until now, numpy does not link agains libpng. This should not be a problem on linux-like systems, as those normally all have libpng installed. Windows however, I don't know. Same thing with pyrex. Numpy doesn't use pyrex until now. * other places to put: numpy could be considered to basic to have such a functionality, one could put it in scipy. Well, but png is not really a _scientific_ application (even though I personally use it to do science). Matplotlib would be another nice place to put this feature, big advantage is that matplotlib already links against libpng. So, now I shared my thoughts with you, comments? Greetings, Martin And here the code (188 lines are hopefully ok for a mailinglist?) cimport c_numpy import numpy cdef extern from "stdlib.h": cdef void *malloc(int) cdef void free(void *) cdef extern from "stdio.h": cdef struct file: pass ctypedef file FILE cdef FILE *fopen(char *, char *) cdef int fclose(FILE *) cdef extern from "errno.h": cdef int errno cdef extern from "png.h": cdef struct jmp_buf: pass cdef int setjmp(jmp_buf) cdef struct png_struct: jmp_buf jmpbuf ctypedef png_struct *png_structp cdef struct png_info: int width int height int bit_depth int color_type int channels ctypedef png_info *png_infop cdef struct png_text: int compression char *key char *text int text_length ctypedef png_text *png_textp cdef png_structp png_create_read_struct(char *, void *, void *, void*) cdef void png_init_io(png_structp, FILE *) cdef void png_set_sig_bytes(png_structp, int) cdef png_infop png_create_info_struct(png_structp) cdef void png_read_info(png_structp, png_infop) cdef int png_set_interlace_handling(png_structp) cdef void png_read_update_info(png_structp, png_infop) cdef void png_read_image(png_structp, unsigned char **) cdef void png_read_png(png_structp, png_infop, int, void *) cdef void png_read_end(png_structp, png_infop) cdef void png_destroy_read_struct(png_structp *, png_infop *, void *) cdef png_structp png_create_write_struct(char *, void *, void *, void*) cdef void png_set_IHDR(png_structp, png_infop, unsigned int, unsigned int, int, int, int, int, int) cdef void png_write_info(png_structp, png_infop) cdef void png_write_image(png_structp, unsigned char **) cdef void png_write_end(png_structp, png_infop) cdef void png_destroy_write_struct(png_structp *,png_infop *) cdef void png_set_swap(png_structp) cdef void png_set_rows(png_structp, png_infop, unsigned char **) cdef void png_write_png(png_structp, png_infop, int, void *) cdef void png_set_text(png_structp, png_infop, png_textp, int) cdef int PNG_COLOR_TYPE_GRAY cdef int PNG_COLOR_TYPE_PALETTE cdef int PNG_COLOR_TYPE_RGB cdef int PNG_COLOR_TYPE_RGB_ALPHA cdef int PNG_COLOR_TYPE_GRAY_ALPHA cdef int PNG_INTERLACE_NONE cdef int PNG_COMPRESSION_TYPE_BASE cdef int PNG_FILTER_TYPE_BASE cdef int PNG_TRANSFORM_IDENTITY cdef int PNG_TRANSFORM_SWAP_ENDIAN cdef char *PNG_LIBPNG_VER_STRING def load(fn): """ load a PNG file into a NumPy array. This function can read all kinds of PNG images in a senseful way: Depending on the number of bits per pixel, we chose numpy.bool, numpy.unit8 or numpy.uint16 as base type. For gray scale images or pallettes, this returns a 2-D array, for everything else a 3-D array, where the third dimension has a size of 2 for gray+alpha images, 3 for RGB and 4 for RGBA images. """ cdef FILE *f f = fopen(fn, "rb") if f == NULL: raise IOError(errno) cdef png_structp p p = png_create_read_struct(PNG_LIBPNG_VER_STRING, NULL, NULL, NULL) if setjmp(p.jmpbuf): fclose(f) raise RuntimeError("libpng reported error") cdef png_infop i i = png_create_info_struct(p) png_init_io(p, f) png_read_info(p, i) if i.color_type == PNG_COLOR_TYPE_GRAY: shape = (i.height, i.width) elif i.color_type == PNG_COLOR_TYPE_GRAY_ALPHA: shape = (i.height, i.width, 2) elif i.color_type == PNG_COLOR_TYPE_PALETTE: shape = (i.height, i.width) elif i.color_type == PNG_COLOR_TYPE_RGB: shape = (i.height, i.width, 3) elif i.color_type == PNG_COLOR_TYPE_RGB_ALPHA: shape = (i.height, i.width, 4) if i.bit_depth == 1: dtype = numpy.bool elif i.bit_depth == 8: dtype = numpy.uint8 elif i.bit_depth == 16: dtype = numpy.uint16 png_set_interlace_handling(p) png_read_update_info(p, i) r = numpy.zeros(shape, dtype) cdef c_numpy.ndarray s s = r cdef unsigned char **rp rp = <unsigned char **> malloc(sizeof(char *) * i.height) cdef int j for j from 0 <= j < i.height: rp[j] = <unsigned char *> &s.data[(j * i.channels * i.width * i.bit_depth) / 8] png_set_swap(p) png_read_image(p, rp) free(rp) png_destroy_read_struct(&p, &i, NULL) fclose(f) return r def save(fn, data): """ Save a NumPy array to PNG This function saves the NumPy array in data to the file fn. It saves the data the best way it can: 2-D arrays are saved as gray scale images, 3-D arrays are saved as gray+alpha, rgb or rgba depending on the size of the 3rd dimension. """ cdef c_numpy.ndarray d d = numpy.ascontiguousarray(data) cdef int j cdef int width if data.itemsize > 2: raise ValueError("PNG cannot handle BPP > 16") if data.ndim == 2: width = data.shape[1] * data.itemsize elif data.ndim == 3: width = data.shape[2] * data.shape[1] * data.itemsize else: raise ValueError("Can only save 2-D or 3-D arrays to PNG") cdef FILE *f f = fopen(fn, "wb") cdef png_structp p p = png_create_write_struct(PNG_LIBPNG_VER_STRING, NULL, NULL, NULL) cdef unsigned char **rp rp = NULL if setjmp(p.jmpbuf): free(rp) fclose(f) raise RuntimeError("libpng reported error") cdef png_infop i i = png_create_info_struct(p) png_init_io(p, f) if data.ndim == 2: color = PNG_COLOR_TYPE_GRAY elif data.shape[2] == 2: color = PNG_COLOR_TYPE_GRAY_ALPHA elif data.shape[2] == 3: color = PNG_COLOR_TYPE_RGB elif data.shape[2] == 4: color = PNG_COLOR_TYPE_RGB_ALPHA else: raise ValueError("Third dimension must be <= 4 for saving to PNG") png_set_IHDR(p, i, data.shape[1], data.shape[0], data.itemsize*8, color, PNG_INTERLACE_NONE, PNG_COMPRESSION_TYPE_BASE, PNG_FILTER_TYPE_BASE) rp = <unsigned char **> malloc(sizeof(char *) * data.shape[0]) for j from 0 <= j < data.shape[0]: rp[j] = <unsigned char *> &d.data[j * width] png_set_rows(p, i, rp) png_write_png(p, i, PNG_TRANSFORM_SWAP_ENDIAN, NULL) free(rp) png_destroy_write_struct(&p, &i) fclose(f)

2 1

SWIGed function does not like my boolean array
by Sebastian Haase 08 Nov '07

08 Nov '07

Hi, I have a templated function written in C++. My SWIG typemap instantiates this for many argument array types (such as unit8, int16, uint16, int32, float32,float64,...) All works well, except when feeding in a boolean array as in seb.mmms(a>6) I get this error: NotImplementedError: No matching function for overloaded 'mmms' What should I do ? Preferably I would avoid having to add another type-instantiation into the library (it looks already quite bloated having 6+ versions of every function). Isn't bool just a synonym for int32 ? Thanks Sebastian Haase

3 9

Why is g++ used for link with mingw ?
by David Cournapeau 08 Nov '07

08 Nov '07

Hi, I was wondering why the linking was done with g++ with mingw in numpy.distutils ? (I got some trouble because some function in math library are only defined for gcc and not g++, for example atanhf). This also means that g++ is required to build numpy, which is a bit strange since there is not a single C++ line; is there an issue I am missing, or is this legacy (the mingw tools in numpy.distutils seems to be 6 years old from the comments). cheers, David

1 0

ANN: NumPy 1.0.4
by Jarrod Millman 08 Nov '07

08 Nov '07

I'm pleased to announce the release of NumPy 1.0.4. NumPy is the fundamental package needed for scientific computing with Python. It contains: * a powerful N-dimensional array object * sophisticated (broadcasting) functions * basic linear algebra functions * basic Fourier transforms * sophisticated random number capabilities * tools for integrating Fortran code. Besides it's obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide-variety of databases. This is largely a bug fix release with one notable improvements: - The NumPy and SciPy developers have decided to adopt the Python naming convention for classes. So as of this release, TestCase classes may now be prefixed with either 'test' or 'Test'. This will allow us to write TestCase classes using the CapCase words, while still accepting the old style names. For information, please see the release notes: http://sourceforge.net/project/shownotes.php?release_id=552568&group_id=1369 Thank you to everybody who contributed to the recent release. Enjoy, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

1 0

[f2py] mailing lists, bug, and .f2py_f2cmap, Oh My!
by Peter Schmitt 07 Nov '07

07 Nov '07

Hi, I have one short question, a possible bug report, and one longer question: 1. (short question) Which mailing list is appropriate for f2py discussion: numpy-discussion or f2py-users? 2. (bug report?) I'm using numpy from svn (revision 4410). When using f2py to compile, I got an error: File "/home/pschmitt/usr/local/lib/python2.5/site-packages/numpy/f2py/rules.py", line 1228, in buildmodule for l in '\n\n'.join(funcwrappers2)+'\n'.split('\n'): TypeError: cannot concatenate 'str' and 'list' objects I made what I think should be a fix: $ svn diff numpy/numpy/f2py/rules.py Index: numpy/numpy/f2py/rules.py =================================================================== --- numpy/numpy/f2py/rules.py (revision 4410) +++ numpy/numpy/f2py/rules.py (working copy) @@ -1219,7 +1219,7 @@ f.write('! This file is autogenerated with f2py (version:%s)\n'%(f2py_version)) f.write('! It contains Fortran 90 wrappers to fortran functions.\n') lines = [] - for l in '\n\n'.join(funcwrappers2)+'\n'.split('\n'): + for l in ('\n\n'.join(funcwrappers2)+'\n').split('\n'): if len(l)>72 and l[0]==' ': lines.append(l[:72]+'&\n &') l = l[72:] but I'm not sure if this is what Pearu wanted with that block of code.... Should I submit a bug report??? 3. (Longer question) I have 3 source files: a) types_QG.f90, which specifies the "types_QG" module: "integer, parameter :: WP=4, intdim=selected_int_kind(8)" b) fftpack.f which is a f77 program to perform fourier transforms in 1D c) fft_lib.f90 which is a f90 program (a f90 module, actually) that uses fftpack.f to do 1,2 or 3D fourier transforms. I want access to access the functions in fft_lib.f90 via Python... So I use f2py! :) I just learned about ".f2py_f2cmap" to handle fortran90 <type spec> (kind=KIND(..)) statements and I'm trying to implement it into my code [http://cens.ioc.ee/projects/f2py2e/FAQ.html#q-what-if-fortran-90-code-uses-…]. f2py successfully applies the f2py_f2cmap: make fft gfortran -fPIC -c types_QG.f90 gfortran -fPIC -c fftpack.f f2py2.5 --fcompiler=gfortran -m fft_lib -c types_QG.o fftpack.o fft_lib.f90 Reading .f2py_f2cmap ... Mapping "real(kind=wp)" to "float" Mapping "real(kind=WP)" to "float" Mapping "integer(kind=intdim)" to "int" Succesfully applied user defined changes from .f2py_f2cmap ... However, f2py complains about the "kind=" syntax in every single function I have defined in the module within fft_lib.f90. The errors look something like this: > /tmp/tmpeP_IaX/src.linux-i686-2.5/fft_lib-f2pywrappers2.f90:9.16: > > real(kind=wp) a(2,2,2) > 1 > Error: Parameter 'wp' at (1) has not been declared or is a variable, which does not reduce to a constant expression If I remove every function from the fft_lib.f90 source code, f2py succesfully compiles the Python module, and I can call all the subroutines in fft_lib. Any ideas on what's going on with f2py, the "kind" statement, .f2py_f2cmap, and functions within a f90 module? Thanks, Pete

1 1

numpy.scons: 2nd alpha
by David Cournapeau 07 Nov '07

07 Nov '07

Hi, I have just finished to implement everything I wanted to do for a 2nd alpha for numpy.scons. Most of the work since 1st alpha was infrastructure polishing (performance library, fortran): now, most common performance libraries are fully supported (MKL and ATLAS on linux, sunperf on solaris, vecLib/Accelerate on Mac OS X). I would appreciate people testing the branch http://projects.scipy.org/scipy/numpy/wiki/NumpyScons#Gettingthecode There are minor problem preventing the thing to build on windows for some reasons, but this will be fixed soon. Basically, the whole design is now done, and it should be able to support all the platforms (and more) currently supported by numpy.distutils. I would really like to hear about people intimated with numpy.distutils, and package developers for some things I may have missed. I think numpy.scons is now much better and easier as a build system, but I am obviously biased. Most significant changes since 1st alpha: Documentation ============= I started to put more documentation on the numpy wiki: http://projects.scipy.org/scipy/numpy/wiki/NumpyScons There is also some basic description on the design: http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons/numpy/di… Configuration check =================== - performance libraries are globally supported (default and site.cfg configuration): MKL, ATLAS, Accelerate/vecLib and Sunperf are implemented. Adding a new performance library can be as easy as a 5 lines code. - 'meta' checkers are now reusing code from performance libraries: Both CBLAS and LAPACK are implemented. This means that numpy can now be entirely built with scons, using optimized libraries on the following platforms: - solaris with sunperf and sunstudio (this one has not been extensively tested: it looks like every solaris machine I have access to behaves really differently). - mac os X and Gnu toolchain - linux + ATLAS/MKL + gnu toolchain I am now relatively happy with the code for BLAS/LAPACK and performance libraries implementation. It is easily extensible and quite flexible. Generci library check ===================== I have updated and improved the basic checker, NumpyCheckLibAndHeader. This should cover most basic needs (check for a library with some functions and some headers), and should be able to replace system_info for many cases. For example, the following will check for the function sf_open in the library sndfile, with header sndfile.h: config.NumpyCheckLibAndHeader(*'sndfile'*, *'sf_open'*, *'sndfile.h'*, section = *'sndfile'*) section is optional, and tells the checker to use sndfile section in site.cfg if found. Fortran support =============== I have polished fortran support: in perticular, name mangling for F77 and F90 is implemented and has been tested with g77/gfortran/ifort/sunfort on various platforms. It also defines variables usable by f2py (http://projects.scipy.org/scipy/numpy/wiki/NumpySconsExtExamples#UsingFortr…). cheers, David

1 0

Making a minimalist NumPy
by Benjamin M. Schwartz 07 Nov '07

07 Nov '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 NumPy is included in the OLPC operating system, which is very constrained in space. Therefore, it would be nice to remove some subpackages to save a few megabytes. For example, the system does not include any Fortran code or compiler, so f2py (3.6 MB) seems superfluous. I also think the distutils subpackage (1.9M) is probably not necessary. Therefore, I have two questions. 1. Which packages do you think are necessary to have a functioning NumPy? 2. What is the easiest way to make (or get) a minimal NumPy installation? For example, would the scons/autoconf branch make this easier? - --Ben Schwartz OLPC volunteer -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHLjDPUJT6e6HFtqQRAnG/AKCYfJ/RZwg3+5i6ofHt6PpwhgusiQCfRkZa QYvFDDRXuLyy4lxt7VswnOs= =uzX/ -----END PGP SIGNATURE-----

3 2

Assessing the use of packages
by Pierre GM 06 Nov '07

06 Nov '07

All, It's evaluation time in my department (Bio. & Ag. Engng, UGA), and I'd need to document the impact of my Python contributions on the scientific community at large, or more realistically on the numpy/scipy user community... * Is there a way to estimate how many people installed one particular package from the SVN ? * if it's too tricky, would anybody using maskedarray and/or timeseries and/or pyloess (the SVN packages I helped implementing) mind dropping me a line off-list, with a very short description of the main field of research/use ? I'd basically need some kind of numbers to give to the People-In-Charge. Thanks a lot in advance for your time. P. PS: Sorry for the cross-posting..

2 1

boolean masks & lists
by John Hunter 06 Nov '07

06 Nov '07

A colleague of mine just asked for help with a pesky bug that turned out to be caused by his use of a list of booleans rather than an array of booleans as his logical indexing mask. I assume this is a feature and not a bug, but it certainly surprised him: In [58]: mask = [True, False, False, False, True] In [59]: maska = n.array(mask, n.bool) In [60]: x = arange(5) In [61]: x[mask] Out[61]: array([1, 0, 0, 0, 1]) In [62]: x[maska] Out[62]: array([0, 4])

4 4

Round 2 with Leopard+Python
by Brian Granger 05 Nov '07

05 Nov '07

Hi, In the process of working through the issues with sys.path on Leopard, I have found another potential Leopard bug that is particularly nasty. In Tiger, sudo preserves environment variables: $ export FOO=/tmp $ python -c "import os; print os.environ['FOO']" /tmp $ sudo python -c "import os; print os.environ['FOO']" /tmp But, in Leopard, sudo does not perserve environment variables: $ export FOO=/tmp $ python -c "import os; print os.environ['FOO']" /tmp $ sudo python -c "import os; print os.environ['FOO']" Password: Traceback (most recent call last): File "<string>", line 1, in <module> File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/UserDict.py", line 22, in __getitem__ raise KeyError(key) KeyError: 'FOO' This is a big problem. First, if you have set PYTHONPATH to point sys.path at the site-packages in /Library, this setting will be lost when you do: sudo python setup.py install On another package. I encountered this in building pytables, which requires numpy >= 1.0.3. I had installed numpy 1.0.4, and set my PYTHONPATH to point to it. But, the pytables setup.py script failts because PYTHONPATH is lost and it only sees the older (1.0.1) builtin numpy. Second, some setup.py scripts use environment variables to determine how things are built, find other dependencies, etc. Currently, this will fail on Leopard if such packages are installed into locations that require sudo. I haven't tried it yet, but I expect that this will also hold true for other python installations. The behavior also shows up with ruby on Leopard. The solution currently is to install all packages to locations that don't require sudo to write to. I will file a bug report, but until the bug is fixed, we should explore putting a note on the numpy/scipy site - and even possibly on the python.org site to describe the problem and its workaround. Brian

2 1