Mailman 3 October 2007 - NumPy-Discussion

Use of PyArray_FromAny() with a PyArrayInterface object
by Matthieu Brucher 12 Oct '07

12 Oct '07

Hi, I'm trying to understand (but perhaps everything is in the numpy book in which case I'd rather buy the book immediately) how to use the PyArray_FromAny() function. Suppose I have a C object (created by Pyrex or SWIG or Boost.Python) that has the __array_struct__ attribute. Can I pass this object to the PyArray_FromAny() function to get a PyArrayObject ? If it is not the case, if I have a PyCObject created with a PyArrayInterface, is it possible ? I read on a mailing-list that there is the context argument that can be passed to an __array__ method. What is the purpose of this __array__ method ? I don't recollect anything regarding this method :( Matthieu

4 5

For review: first milestone of scons support in numpy
by David Cournapeau 12 Oct '07

12 Oct '07

Hi, (if you are not interested in numpy developement, you can stop now :) ). Following the discussion a few days ago on using scons to build extensions in numpy, I have reached a somewhat usable milestone, in the numpy.scons branch of numpy, and would like to hear some comments, remarks, critics, etc...: Where to get/see: ----------------- svn repository : http://svn.scipy.org/svn/numpy/branches/numpy.scons looking at the code: http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons Examples: --------- To see how it feels from the package developer point of view, I have put three really simple examples: - Building a python extension: http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons/numpy/sc… - Building a ctypes-based package: http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons/numpy/sc… - An example on how to check for libraries and symbols in them: http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons/numpy/sc… For the numpy user, this should be totally transparent (no difference when building/installing). What: ----- This first milestone implements the following: - adding a scons command to numpy.distutils - adding an add_sconscript function to numpy.distutils setup, for packages willing to use scons - two builders: one for ctypes extension, and one for python extension - a basic implementation to check for libraries (the paths can be overwritten exactly like with distutils, using site.cfg; I have not yet implemented overwriting with environment variables). I have been testing this on the following platforms: - linux with gcc - linux with icc - linux with suncc - windows with MS toolikit 2003 - solaris studio express with suncc - mac os X (tiger, x86) And now ? --------- As discussed previously, I think numpy would benefit from exclusively using scons to build compiled extensions. I have started working on fortran support for scons (separate project, since this may be useful to all scons users, not just numpy): https://launchpad.net/numpy.scons.support and I can already do some non trivial things, not possible with numpy.distutils (automatically figuring out fortran mangling, flags for linking with C, blas/lapack flags). As expected, this is much more robust than distutils approach of hardcoding everything: although I used g77 for development, it worked without any change with ifort, gfortran and sun fortran compiler (on linux). There are still some issues for sure, but I don't see big problems. I don't want to do the work for nothing, though, so I would like to know the feeling of numpy developers first on this direction, in particular which platforms should work before merging consideration, etc... cheers, David

3 9

Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
by Ray S 11 Oct '07

11 Oct '07

Thanks all: At 10:00 AM 10/10/2007, Robert Kern wrote: >Something like the following should suffice (untested, though I've >done similar things with ctypes before): I tested, successfully: """ nFromAddress.py """ def fromaddress(address, dtype, shape, strides=None): """ Create a numpy array from an integer address, a dtype or dtype string, a shape tuple, and possibly strides. """ import numpy # Make sure our dtype is a dtype, not just "f" or whatever. dtype = numpy.dtype(dtype) class Dummy(object): pass d = Dummy() d.__array_interface__ = dict( data = (address, False), typestr = dtype.str, descr = dtype.descr, shape = shape, strides = strides, version = 3, ) return numpy.asarray(d) ## Numeric example, with address kludge import Numeric, numpy, ctypes, string a0 = Numeric.zeros((10000), Numeric.Int16) nAddress = int(string.split(repr(a0.__copy__))[-1][:-1], 16) tmp=(ctypes.c_long*1)(0) ctypes.memmove(tmp, nAddress+8, 4) nAddress = tmp[0] a1 = fromaddress(nAddress, numpy.int16, (10000,)) ## explicit type a0[0] = 5 print a1[0] ## numpy example a2 = numpy.zeros(10000, numpy.int16) nAddress = a2.__array_interface__['data'][0] nType = a2.__array_interface__['typestr'] nShape = a2.__array_interface__['shape'] a3 = fromaddress(nAddress, nType, nShape) a2[0] = 5 print a3[0] So, now with little effort the relevant info can be passed over pipes, shared memory, etc. and shares/views created in other processes, since they are not objects but ints and strings. >David Cournapeau Wrote: >Basically, you cannot expect file descriptors (or even file handles: >the >standard ones from C library fopen) to cross dll boundaries if the >dll do not have exactly the same runtime. It sounds like there is a general dilemma: no one with Python 2.4 or 2.5 can reliably expect to compile extensions/modules if they did not install the 7.1 compiler in time. The 2.6 seems to use VC 2005 Express, I don't know about py3000(?), with associated upgrade issues. It would be nice if the build bots could also compile suggested modules/extentions. Thanks again, Ray

3 5

numpy.distutils.cpuinfo bugs?
by Alan G Isaac 11 Oct '07

11 Oct '07

Two problems noted:: >>> from numpy.distutils import cpuinfo 0 >>> cpu = cpuinfo.cpuinfo() >>> print "Has SSE2? %s"%cpu._has_sse2() Has SSE2? False >>> 1. The printed '0' traces to an undesirable print statement. (I've reported this before.) 2. The 'False" report is on a Pentium M. Should that not be True? <URL:http://en.wikipedia.org/wiki/Pentium_M> Or am I misusing the test? Thank you, Alan Isaac

4 6

diff of find of diff trick
by Robin 11 Oct '07

11 Oct '07

Hello, As a converting MATLAB user I am bringing over some of my code. A core function I need is to sample the probability distribution of a given set of data. Sometimes these sets are large so I would like to make it as efficient as possible. (the data are integers representing members of a discrete space) In MATLAB the best way I found was the "diff of find of diff" trick which resulted in the completely vectorised solution (below). Does it make sense to translate this into numpy? I don't have a feel yet for what is fast/slow - are the functions below built in and so quick (I thought diff might not be). Is there a better/more pythonic way to do it? -------- function Pr=prob(data, nR) Pr = zeros(nR,1); % diff of find of diff trick for counting number of elements temp = sort(data(data>0)); % want vector excluding P(0) dtemp = diff([temp;max(temp)+1]); count = diff(find([1;dtemp])); indx = temp(dtemp>0); Pr(indx)= count ./ numel(data); % probability -------- Thanks Robin

2 1

fortran 90 compiler problem
by humufr＠yahoo.fr 11 Oct '07

11 Oct '07

Hello, I'm using the last svn version of numpy (. I tried the solution found on the mail list: python setup.py config_fc --fcompiler=gnu95 build the two interesting part (at least for me) in the building process are: customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config and later: customize G95FCompiler Found executable /usr/local/bin/g95 but I still have the same answer when I'm trying to do: f2py -c -m hello hello.f90 customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found and so: error: f90 not supported by GnuFCompiler needed for hello.f90 I can't compile my fortran module because of this. I'll attach the file obtain with python setup.py -v config_fc --fcompiler=gnu95 build And I have the same result for g95. thank you for numpy and f2py. N.

5 8

How does Python treat commons in fortran code?
by Guillem Borrell i Nogueras 10 Oct '07

10 Oct '07

Hello all This afternoon I introduced Python to my boss and the rest of my colleagues in the CFD laboratory (universidad politecnica de madrid, school of aeronautics). My boss wanted to know if it would be helpful to manage a quite complex parallel algorithm while keeping its performance. He has a long background in high performance computing, mainly in fortran. He asked me something i could not answer, in fact I just have no clue about it. We have some very tuned parallel subroutines to perform FFTs that are written in fortran 77; they have lots of _common_ structures that are used to manage large chunks of memory. Those subroutines have shown to be scalable to 2000 processors so it is very important that they perform as expected. Of course our intention is to use them and I had the idea of building a wrapper using f2py. His question was. ¿How does ctypes or f2py handle the _common_ structures present in fortran 77? ¿Are all they allocated at load? ¿Do they work exactly as they were called from a fortran executable and the main program is aware of all the _commons_ and allocates the storage when it is asked for? I told him that I see no difference between the python interpreter and any other executable but my knowledge of python is not that deep. I could not explain him successfully exactly what is stored in the interpreter stack and what is not neither. He has been programming for more than 40 years and I could not get him much into object oriented programing, however he found python+numpy+scipy a very promising tool. guillem

2 1

Re: [Numpy-discussion] fortran 90 compiler problem
by humufr＠yahoo.fr 10 Oct '07

10 Oct '07

I have already gfortran and g95 installed and working fine for all the system but numpy and scipy. By the way I'm using linux not MacOSX. My problem is that when I'm installing numpy gfortran (or g95) is detected but not at all the step and so f2py seem to be unable to use it. look at: line 92 in my attach file: find_executable('g77') Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found but line 469: customize Gnu95FCompiler find_executable('gfortran') Found executable /usr/local/bin/gfortran I don't understand the problem. Thanks N.

1 0

Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
by Ray Schumacher 09 Oct '07

09 Oct '07

At 05:22 AM 10/9/2007, David Cournapeau wrote: >Could not this be because you compiled the posh sources with a >compiler/runtime which is different than the other extensions and python >interpreter ? It definitely was - since my 2.4 wanted the free 7.1 compiler, I (and anyone else who didn't download it in time) are now seemingly SOL since it is no longer available. I saw much discussion of this as well, but even 2.5 is now "fixed" on 7.1 and reports of compiling distutil modules with the new MS SDK and having them work at all with 2.4 were very mixed. I also tried GCC and had a litany of other errors with the posh. >Sebastian Haase added: >I was in fact experimenting with this. The solution seemed to lie in >"simple" memmap as it is implemented in Windows: <snip> I had just found and started to write some tests with that MS function. If I can truly write to the array in one process and instantly read it in the other I'll be happy. Did you find that locks or semaphores were needed? >(( I have to mention, that I could crash a process while testing this ... )) That was one of my first results! I also found that using ctypes to create arrays from the other process's address and laying a numpy array on top was prone to that in experimentation. But I had the same issue as Mark Heslep http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3192422 of creating a numpy array from a raw address (not a c_array). Thanks, Ray Schumacher

3 2

Re: [Numpy-discussion] numpy/Windows shared arrays between processes? CODE...
by Ray S 09 Oct '07

09 Oct '07

# numpyShared.py import numpy as N def arrSharedMemory(shape, dtype, tag="PySharedMemory", access=None): """ Windows only ! share memory between different processes if same `tag` is used. """ itemsize = N.dtype(dtype).itemsize count = N.product(shape) size = count * itemsize import mmap sharedmem = mmap.mmap(0, size, tag, access) a=N.frombuffer(sharedmem, dtype, count) a.shape = shape return a if __name__ == '__main__': ## test it ## produce a mish-mash of output on one shell screen import random import time import subprocess, sys import mmap if len(sys.argv)<2: pid = subprocess.Popen([r'C:\python24\python.exe', ['numpyShared.py rjs100']]).pid ## the parent process a1 = arrSharedMemory((10000,), N.int32, tag='rjs100', access=mmap.ACCESS_WRITE) time.sleep(.7) now=time.time(); while time.time()<now+3: tmp = a1[0]**.5 a1[0]=random.random()*10 #print '\np%d' % a1[0] else: tag = sys.argv[0][1] a2 = arrSharedMemory((10000,), N.int32, tag='rjs100', access=mmap.ACCESS_READ) now=time.time() while time.time()<now+3: tmp = a2[0]**.5 #print '\nc%d' % a2[0]

1 0