Mailman 3 June 2006 - NumPy-Discussion

Building shared libraries with numpy.distutils
by Albert Strasheim June 10, 2006

June 10, 2006

Hello all For my Summer of Code project, I'm adding Support Vector Machine code to SciPy. Underneath, I'm currently using libsvm. Thus far, I've been compiling libsvm as a shared library (DLL on Windows) using SCons and doing the wrapping with ctypes. Now, I would like to integrate my code into the SciPy build. Unfortunately, it doesn't seem as if numpy.distutils or distutils proper knows about building shared libraries. Building shared libraries across multiple platforms is tricky to say the least so I don't know if implementing this functionality again is something worth doing. The alternative -- never using shared libraries, doesn't seem very appealing either. Is anybody building shared libraries? Any code or comments? Regards, Albert

2 1

Re: [Numpy-discussion] Suggestions for NumPy
by Joe Harrington June 9, 2006

June 9, 2006

> Date: Fri, 2 Jun 2006 18:04:32 -0400 > From: "Jonathan Taylor" <jonathan.taylor(a)utoronto.ca> > Subject: Re: [Numpy-discussion] Suggestions for NumPy > To: numpy-discussion(a)lists.sourceforge.net > Message-ID: > <463e11f90606021504h742e92e4t5ff418d1e29e426(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > My suggestion would be to have both numpy.org and scipy.org be the > exact same page, but make it extremely clear that there are two > different projects on the front page. > Cheers. > Jon. The goal of the web is to make information easy to find. The easiest and most successful way of doing that is to answer many needs in one place, hence the existence of "portal" pages, which scipy.org bills itself as. The relationship between scipy and numpy is laid out in its front page text. With two (actually many more) packages distributed separately, there will always be confused people, but having one main site that tells the whole story and provides comprehensive information will be the quickest way to deconfuse them. Conversely, a plethora of pages is a poor marketing strategy, as we have been learning with the zoo that's out there already. My suggestion is that all the other pages be automatic redirects to the scipy.org page or subpages thereof. I know that will probably make some people feel their toes have been stepped on. We could consider a website name change to avoid that, but I hope we don't have to. Unite and conquer... --jh--

3 3

Getting rid of annoying weave nag
by Fernando Perez June 9, 2006

June 9, 2006

Hi all, the following warning about strict-prototypes in weave drives me crazy: longs[~]> python wbuild.py <weave: compiling> cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ since I use weave on auto-generated code, I get it lots of times and I find spurious warnings to be very distracting. Anyone object to this patch against current numpy SVN to get rid of this thing? (tracking where the hell that thing was coming from was all kinds of fun) Index: ccompiler.py =================================================================== --- ccompiler.py (revision 2588) +++ ccompiler.py (working copy) @@ -191,6 +191,19 @@ log.info('customize %s' % (self.__class__.__name__)) customize_compiler(self) if need_cxx: + # In general, distutils uses -Wstrict-prototypes, but this option is + # not valid for C++ code, only for C. Remove it if it's there to + # avoid a spurious warning on every compilation. All the default + # options used by distutils can be extracted with: + + # from distutils import sysconfig + # sysconfig.get_config_vars('CC', 'CXX', 'OPT', 'BASECFLAGS', + # 'CCSHARED', 'LDSHARED', 'SO') + try: + self.compiler_so.remove('-Wstrict-prototypes') + except ValueError: + pass + if hasattr(self,'compiler') and self.compiler[0].find('gcc')>=0: if sys.version[:3]>='2.3': if not self.compiler_cxx: ### EOF Cheers, f

3 5

Re: [Numpy-discussion] Array Protocol change for Python 2.6
by Andrew Straw June 9, 2006

June 9, 2006

Travis Oliphant wrote: > Andrew Straw wrote: > >> On the one hand, I feel we should keep __array_struct__ behaving >> exactly as it is now. There's already lots of code that uses it, and >> it's tremendously useful despite (because of?) it's simplicity. For >> these of use cases, the __array_descr__ information has already >> proven unnecessary. I must say that I, and probably others, thought >> that __array_struct__ would be future-proof. Although the magnitude >> of the proposed change to add this information to the C-struct >> PyArrayInterface is minor, it still breaks code in the wild. >> > I don't see how it breaks any code in the wild to add an additional > member to the C-struct. We could easily handle it in new code with a > flag setting (like Python uses). The only possible problem is > looking for it when it is not there. Ahh, thanks for clarifying. Let me paraphrase to make sure I got it right: given a C-struct "inter" of type PyArrayInterface, if and only if ((inter.flags & HAS_ARRAY_DESCR) == HAS_ARRAY_DESCR) inter could safely be cast as PyArrayInterfaceWithArrayDescr and thus expose a new member. This does seem to avoid all the issues and maintain backwards compatibility. I guess the only potential complaint is that it's a little C trick which might be unpalatable to the core Python devs, but it doesn't seem egregious to me. If I do understand this issue, I'm +1 for the above scheme provided the core Python devs don't mind. Cheers! Andrew

1 0

Build questions, atlas, lapack...
by Fernando Perez June 9, 2006

June 9, 2006

Hi all, I'm starting the transition of a large code from Numeric to numpy, so I am now doing a fresh build with a lot more care than before, actually reading all the intermediate messages. I am a bit puzzled and could use some help. This is all on an ubuntu dapper box with the atlas-sse2 packages (and everything else recommended installed). By running as suggested in the scipy readme: python ~/tmp/local/lib/python2.4/site-packages/numpy/distutils/system_info.py I get the following message at some point: ==================================== atlas_info: ( library_dirs = /usr/local/lib:/usr/lib ) ( paths: /usr/lib/atlas,/usr/lib/sse2 ) looking libraries f77blas,cblas,atlas in /usr/local/lib but found None looking libraries f77blas,cblas,atlas in /usr/local/lib but found None looking libraries lapack_atlas in /usr/local/lib but found None looking libraries lapack_atlas in /usr/local/lib but found None looking libraries f77blas,cblas,atlas in /usr/lib/atlas but found None looking libraries f77blas,cblas,atlas in /usr/lib/atlas but found None looking libraries lapack_atlas in /usr/lib/atlas but found None looking libraries lapack_atlas in /usr/lib/atlas but found None ( paths: /usr/lib/sse2/libf77blas.so ) ( paths: /usr/lib/sse2/libcblas.so ) ( paths: /usr/lib/sse2/libatlas.so ) ( paths: /usr/lib/sse2/liblapack_atlas.so ) looking libraries lapack in /usr/lib/sse2 but found None looking libraries lapack in /usr/lib/sse2 but found None looking libraries f77blas,cblas,atlas in /usr/lib but found None looking libraries f77blas,cblas,atlas in /usr/lib but found None looking libraries lapack_atlas in /usr/lib but found None looking libraries lapack_atlas in /usr/lib but found None system_info.atlas_info ( include_dirs = /usr/local/include:/usr/include ) ( paths: /usr/include/atlas_misc.h,/usr/include/atlas_enum.h,/usr/include/atlas_aux.h,/usr/include/atlas_type.h ) /usr/local/installers/src/scipy/numpy/numpy/distutils/system_info.py:870: UserWarning: ********************************************************************* Could not find lapack library within the ATLAS installation. ********************************************************************* warnings.warn(message) ( library_dirs = /usr/local/lib:/usr/lib ) ( paths: /usr/lib/atlas,/usr/lib/sse2 ) FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib/sse2'] language = c define_macros = [('ATLAS_WITHOUT_LAPACK', None)] ==================================== What I find very puzzling here is that later on, the following goes by: lapack_atlas_info: ( library_dirs = /usr/local/lib:/usr/lib ) ( paths: /usr/lib/atlas,/usr/lib/sse2 ) looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/local/lib but found None looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/local/lib but found None looking libraries lapack_atlas in /usr/local/lib but found None looking libraries lapack_atlas in /usr/local/lib but found None looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/lib/atlas but found None looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/lib/atlas but found None looking libraries lapack_atlas in /usr/lib/atlas but found None looking libraries lapack_atlas in /usr/lib/atlas but found None ( paths: /usr/lib/sse2/liblapack_atlas.so ) ( paths: /usr/lib/sse2/libf77blas.so ) ( paths: /usr/lib/sse2/libcblas.so ) ( paths: /usr/lib/sse2/libatlas.so ) ( paths: /usr/lib/sse2/liblapack_atlas.so ) looking libraries lapack in /usr/lib/sse2 but found None looking libraries lapack in /usr/lib/sse2 but found None looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/lib but found None looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/lib but found None looking libraries lapack_atlas in /usr/lib but found None looking libraries lapack_atlas in /usr/lib but found None system_info.lapack_atlas_info ( include_dirs = /usr/local/include:/usr/include ) ( paths: /usr/include/atlas_misc.h,/usr/include/atlas_enum.h,/usr/include/atlas_aux.h,/usr/include/atlas_type.h ) ( library_dirs = /usr/local/lib:/usr/lib ) ( paths: /usr/lib/atlas,/usr/lib/sse2 ) FOUND: libraries = ['lapack_atlas', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib/sse2'] language = c define_macros = [('ATLAS_WITH_LAPACK_ATLAS', None)] ============================================== Does the second mean that it /is/ finding the right libraries? Since the first search in atlas_info is also printing ( paths: /usr/lib/sse2/liblapack_atlas.so ) I don't quite understand why it then reports the warning. For reference, here's the content of the relevant directories on my system: ============================================== longs[sse2]> ls /usr/lib/sse2 libatlas.a libcblas.a libf77blas.a liblapack_atlas.a libatlas.so@ libcblas.so@ libf77blas.so@ liblapack_atlas.so@ libatlas.so.3@ libcblas.so.3@ libf77blas.so.3@ liblapack_atlas.so.3@ libatlas.so.3.0 libcblas.so.3.0 libf77blas.so.3.0 liblapack_atlas.so.3.0 longs[sse2]> ls /usr/lib/atlas/sse2/ libblas.a libblas.so.3@ liblapack.a liblapack.so.3@ libblas.so@ libblas.so.3.0 liblapack.so@ liblapack.so.3.0 ============================================== In summary, I don't really know if this is actually finding what it wants or not, given the two messages. Cheers, f ps - it's worth mentioning that the sequence: python ~/tmp/local/lib/python2.4/site-packages/numpy/distutils/system_info.py gets itself into a nasty recursion where it fires the interactive session 3 times in a row. And in doing so, it splits its own output in a funny way: [...] blas_opt_info: ======================================================================== Starting interactive session ------------------------------------------------------------------------ Tasks: i - Show python/platform/machine information ie - Show environment information c - Show C compilers information c<name> - Set C compiler (current:None) f - Show Fortran compilers information f<name> - Set Fortran compiler (current:None) e - Edit proposed sys.argv[1:]. Task aliases: 0 - Configure 1 - Build 2 - Install 2<prefix> - Install with prefix. 3 - Inplace build 4 - Source distribution 5 - Binary distribution Proposed sys.argv = ['/home/fperez/tmp/local/lib/python2.4/site-packages/numpy/distutils/system_info.py'] Choose a task (^D to quit, Enter to continue with setup): ##### msg: ( library_dirs = /usr/local/lib:/usr/lib ) FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib/sse2'] language = c define_macros = [('NO_ATLAS_INFO', 2)] ================= I tried to fix it, but the call sequence in that code is convoluted enough that after a few 'import traceback;traceback.print_stack()' tries I sort of gave up. That code is rather (how can I say this nicely) pasta-like :), and thoroughly uncommented, so I'm afraid I won't be able to contribute a cleanup here. I think this tool should run by default in a mode with NO attempt to fire a command-line subsystem of its own, so users can simply run python /path/to/system_info > system_info.log for further analysis.

5 7

Building statically-linked Numpy causes problems with f2py extensions
by listservs＠mac.com June 8, 2006

June 8, 2006

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Because of complaints of linking errors from some OS X users, I am trying to build and distribute statically-linked versions. To do this, I have taken the important libraries (e.g. freetype, libg2c), and put them in a directory called staticlibs, then built numpy by: python setup.py build_clib build_ext -L../staticlibs build bdist_mpkg It builds, installs and runs fine. However, when I go to build and run f2py extensions, I now get the following (from my PyMC code): /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/PyMC/MCMC.py 37 _randint = random.randint 38 rexponential = random.exponential - ---> 39 from flib import categor as _categorical global flib = undefined global categor = undefined global as = undefined _categorical = undefined 40 from flib import rcat as rcategorical 41 from flib import binomial as _binomial ImportError: Loaded module does not contain symbol _initflib Here, flib is the f2py extension that is built in the PyMC setup file according to: from numpy.distutils.core import setup, Extension flib = Extension(name='PyMC.flib',sources=['PyMC/flib.f']) version = "1.0" distrib = setup( version=version, author="Chris Fonnesbeck", author_email="fonnesbeck(a)mac.com", description="Version %s of PyMC" % version, license="Academic Free License", name="PyMC", url="pymc.sourceforge.net", packages=["PyMC"], ext_modules = [flib] ) This worked fine before my attempts to statically link numpy. Any ideas regarding a solution? Thanks, Chris - -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iD8DBQFEiLY+keka2iCbE4wRAi1/AJ90K7LIkF7Y+ti65cVxLB1KCA+MNgCggj2p I1jzals7IoBeYX0cWfmlbcI= =bY3a -----END PGP SIGNATURE-----

1 0

.debs of numpy-0.9.8 available for Ubuntu Dapper
by Andrew Straw June 8, 2006

June 8, 2006

I've put together some .debs for numpy-0.9.8. There are binaries compiled for amd64 and i386 architectures of Ubuntu Dapper, and I suspect these will build from source for just about any Debian-based distro and architecture. The URL is http://sefton.astraw.com/ubuntu/dapper and you would add the following lines to your /etc/apt/sources.list: deb http://sefton.astraw.com/ubuntu/ dapper/ deb-src http://sefton.astraw.com/ubuntu/ dapper/ Although this is the culmination of my first serious attempt Debianizing something, I've attempted to build these "properly" (using inspiration from Matthias Klose's Numeric and numarray packages for Debian and Ubuntu, although I've updated the build system to use CDBS). The numpy source has a build dependency on setuptools (0.6b2), which is also available at the repository. Numpy doesn't get installed as an .egg, but it carries along .egg-info, which means that numpy can be part of a setuptools dependency specification. This was done using the --single-version-externally-managed command for setuptools. I'm building this repository to serve some of my needs at work, and I hope to add recent versions of several other projects including matplotlib and scipy in the coming days. I hope to be able to keep the repository up-to-date over time and to respond to bug reports and questions, although the amount of time I have to devote to this sort of stuff is unfortunately often near zero. If I get some positive feedback, I'm likely to add this to the scipy.org download page. Also, I hope the official Debian and Ubuntu distros pick up numpy soon, and perhaps this will speed them along.

4 7

Intel OSX test failure
by Gennan Chen June 8, 2006

June 8, 2006

Hi! I just got an MacBook Pro and tried to install numpy+scipy on that. I successfully installed ipython+matplotlib+python 2.4 through darwinports. Then I svn co a copy of numpy +scipy. Compilation (gcc 4.0.1 + gfortran) seems working fine for numpy. After I installed it and run numpy.test() in ipython, it failed. And the error is: In [4]: numpy.test() Found 3 tests for numpy.lib.getlimits Found 30 tests for numpy.core.numerictypes Found 13 tests for numpy.core.umath Found 3 tests for numpy.core.scalarmath Found 8 tests for numpy.lib.arraysetops Found 42 tests for numpy.lib.type_check Found 95 tests for numpy.core.multiarray Found 3 tests for numpy.dft.helper Found 36 tests for numpy.core.ma Found 2 tests for numpy.core.oldnumeric Found 9 tests for numpy.lib.twodim_base Found 9 tests for numpy.core.defmatrix Found 1 tests for numpy.lib.ufunclike Found 35 tests for numpy.lib.function_base Found 1 tests for numpy.lib.polynomial Found 6 tests for numpy.core.records Found 19 tests for numpy.core.numeric Found 5 tests for numpy.distutils.misc_util Found 4 tests for numpy.lib.index_tricks Found 46 tests for numpy.lib.shape_base Found 0 tests for __main__ ..............................................F......................... ........................................................................ ........................................................................ ........................................................................ ........................................................................ .......... ====================================================================== FAIL: check_large_types (numpy.core.tests.test_scalarmath.test_power) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ test_scalarmath.py", line 42, in check_large_types assert b == 6765201, "error with %r: got %r" % (t,b) AssertionError: error with <type 'float128scalar'>: got 0.0 ---------------------------------------------------------------------- Ran 370 tests in 0.510s FAILED (failures=1) Out[4]: <unittest.TextTestRunner object at 0x1581fd0> Anyone has any idea?? or Anyone ever successfully did that? Gen-Nan Chen, PhD Chief Scientist Research and Development Group CorTechs Labs Inc (www.cortechs.net) 1020 Prospect St., #304, La Jolla, CA, 92037 Tel: 1-858-459-9700 ext 16 Fax: 1-858-459-9705 Email: gnchen(a)cortechs.net

2 2

remaining matrix-non-preserving functions
by Sven Schreiber June 8, 2006

June 8, 2006

Hi all, I just discovered that the diff function returns a numpy-array even for matrix inputs. Since I'm a card-carrying matrix fanatic, I hope that behavior qualifies as a bug. Then I went through some (most?) other functions/methods for which IMO it's best to return matrices if the input is also a matrix-type. I found that the following functions share the problem of diff (see below for illustrations): vstack and hstack (although I always use r_ and c_ and they work fine with matrices) outer msort Should I open new tickets? (Or has this already been fixed since 0.9.8, which I used because this time building the svn version failed for me?) Cheers, Sven >>> n.__version__ '0.9.8' >>> a matrix([[1, 0, 0], [0, 1, 0], [0, 0, 1]]) >>> b matrix([[0, 0, 0], [0, 0, 0]]) >>> n.diff(a) array([[-1, 0], [ 1, -1], [ 0, 1]]) >>> n.outer(a,b) array([[0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]) >>> n.msort(a) array([[0, 0, 0], [0, 0, 0], [1, 1, 1]]) >>> n.vstack([a,b]) array([[1, 0, 0], [0, 1, 0], [0, 0, 1], [0, 0, 0], [0, 0, 0]]) >>> n.hstack([a,b.T]) array([[1, 0, 0, 0, 0], [0, 1, 0, 0, 0], [0, 0, 1, 0, 0]]) >>>

2 2

argsort question
by Robert Cimrman June 8, 2006

June 8, 2006

Hi all, I have just lost some time to find a bug related to the fact, that argsort does not preserve the order of an array that is already sorted, see the example below. For me, it would be sufficient to mention this fact in the docstring, although having order preserving argsort is also an option :). What do the developers think? In [33]:a = nm.zeros( 10000 ) In [34]:b = nm.arange( 10000 ) In [35]:nm.alltrue( nm.argsort( a ) == b ) Out[35]:False r.

3 5