Mailman 3 April 2008 - NumPy-Discussion

Re: [Numpy-discussion] powerpc yellow dog linux port of numpy
by Vincent Broman 21 Apr '08

21 Apr '08

I succeeded in working around the other Yellow Dog Linux porting problem connected with the floating point exception calls. It turns out that a problematic #include <bits/fenvinline.h> was protected by a "#ifdef __OPTIMIZE__" so my preprocessing with "gcc -E" never saw its effect. So, by avoiding optimization of umath, I was able to get a slower, but working copy of numpy compiled, which passed the numpy.test() suite. The one-line patch avoiding my two compile problems adds to numpy/core/setup.py after line 270 the following line. extra_compile_args = ["-fno-builtin -O0"] I don't know enough to say whether the quoted args should be split(). I do want to find real fixes, though, not just bandaids. In numpy/core/include/numpy/ndarrayobject.h is code which causes me one problem. #if NPY_SIZEOF_LONGDOUBLE == NPY_SIZEOF_DOUBLE typedef double npy_longdouble; #define NPY_LONGDOUBLE_FMT "g" #else typedef long double npy_longdouble; #define NPY_LONGDOUBLE_FMT "Lg" #endif I do not see why the size is critical. Couldn't we just use long double for any compiler that supports long double? If the double and long double have the same format, why do we prefer double? My other assembler code problem is caused, when optimization is enabled, by code using the following definition in /usr/include/bits/fenvinline.h, which I do not understand, never having done ppc assembler. ------------------quote------------------------------------------------- /* The weird 'i#*X' constraints on the following suppress a gcc warning when __excepts is not a constant. Otherwise, they mean the same as just plain 'i'. */ /* Inline definition for feraiseexcept. */ # define feraiseexcept(__excepts) \ ((__builtin_constant_p (__excepts) \ && ((__excepts) & ((__excepts)-1) == 0 \ && (__excepts) != FE_INVALID) \ ? ((__excepts) != 0 \ ? (__extension__ ({ __asm__ __volatile__ \ ("mtfsb1 %s0" \ : : "i#*X"(__builtin_ffs (__excepts))); \ 0; })) \ : 0) \ : (feraiseexcept) (__excepts)) ----------------endquote--------------------------------------------------- This definition causes all the occurences in umathmodule.c of feraiseexcept( FE_DIVBYZERO) to make gcc complain of "inconsistent operand constraints in an `asm'". Is there anyone who can parse that hornet's nest of a definition? Vincent Broman

1 0

API scan
by Charles R Harris 21 Apr '08

21 Apr '08

Hi All, I would like to scan generated files for API functions, but it seems that the API scan runs on files in the numpy/core/src directory, not in the build directory where the generated files end up. Is there an easy way to fix this? Chuck

2 2

Speed of matrix multiplication
by Keith Goodman 21 Apr '08

21 Apr '08

Why is a.T*b slower than M.dot(a.T, b)? Does it take longer to parse or something? >> a = M.randn(500,1) >> b = M.randn(500,1) >> timeit a.T*b 100000 loops, best of 3: 11 µs per loop >> timeit M.dot(a.T, b) 100000 loops, best of 3: 6.72 µs per loop >> a = M.randn(5,1) >> b = M.randn(5,1) >> timeit a.T*b 100000 loops, best of 3: 10.1 µs per loop >> timeit M.dot(a.T, b) 100000 loops, best of 3: 6.16 µs per loop I use matrices, but for inner loops I'll convert to arrays: >> a = a.A >> b = b.A >> dot = M.dot >> timeit dot(a.T, b) 1000000 loops, best of 3: 1.82 µs per loop >> a = a.T >> timeit dot(a, b) 1000000 loops, best of 3: 1.44 µs per loop

3 2

f2py on windows cant find gfortran because split_quoted() removed from ccompiler.py
by Jarrod Millman 21 Apr '08

21 Apr '08

Hey Pearu, Sorry about messing up distutils; I have been warned that it can be hairy. Anyway for those who haven't been following it, last November I emailed the list a couple of times (obviously I should have checked with Pearu and could have avoided this) asking if I could remove some code from numpy/distutils/ccompiler.py. Basically, we had our own version of distutils.util.split_quoted() in numpy/distutils/ccompiler.py, which made a minor change to distutils default behavior (the fix keeps quotes when a quoted word doesn't contain whitespace). I created a ticket and ended up removing the code: http://scipy.org/scipy/numpy/ticket/619 Now it appears that this actually is breaking f2py on Windows: http://scipy.org/scipy/numpy/ticket/723 Since my change may have broken something, I went ahead and reverted it in r5054: http://projects.scipy.org/scipy/numpy/changeset/5054 Hopefully, this will fix the problem with gfortran not being found and won't introduce new problems. Now we have come full circle and are back where we were when I originally brought this issue to the list. Our version of split_quoted() was added several years ago and should I think be submitted for inclusion upstream. I know that distutils isn't exactly being maintained, but--Pearu--is it possible that we could get this upstream? Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

2 1

[numscons] 0.6.1 release: it build scipy, and on windows !
by David Cournapeau 21 Apr '08

21 Apr '08

Hi, I am pleased to announce the 0.6.1 release of numscons. You can get tarballs, eggs for python 2.4/2.5 and windows installers on launchpad: https://launchpad.net/numpy.scons.support/0.6/0.6.1 I did not announce any release of numscons for some time, but this one is a significant milestone: it can build both numpy and scipy on linux, mac os X and windows (mingw only for now), and all tests pass on those three platforms (that is, all tests which pass on distutils build). Here is a full changelog since the last announced release: - Update the local copy of scons to 0.98. All internal modifications to scons needed for numscons are now integrated upstream, which means it is theoretically possible to use an external scons - f2py scons tool is now executed in its own process to avoid race issues with parallel builds. Concretely, this means building scipy with the --jobs options to use multi core should work. - f2py has been almost entirely rewritten: it can now scan the module name automatically, and should be much more reliable. - Msvc runtime issues are now fixed: the PythonExtension builder does not link the msvc runtime if its sources contain some fortran files - Added initial support for C++ flags customization cheers, David

3 3

Buildbot is down
by Charles R Harris 21 Apr '08

21 Apr '08

The buildbot seems to be off line these days. Is that planned? Chuck

2 1

sparse matrices -> numpy ?
by Soeren Sonnenburg 20 Apr '08

20 Apr '08

Dear all, is there any schedule when sparse matrices will arrive in the numpy core? Best, Soeren

2 1

Shogun - A Large Scale Machine Learning Toolbox
by Soeren Sonnenburg 20 Apr '08

20 Apr '08

SHOGUN ( http://www.shogun-toolbox.org ) is a large scale machine learning toolbox with focus on especially Support Vector Machines (SVM). * It provides a generic SVM object interfacing to several different SVM implementations, among them the state of the art LibSVM, SVMLight, SVMLin and GPDT. Each of the SVMs can be combined with a variety of kernels. The toolbox not only provides efficient implementations of the most common kernels, like the Linear, Polynomial, Gaussian and Sigmoid Kernel but also comes with a number of recent string kernels as e.g. the Locality Improved, Fischer, TOP, Spectrum, Weighted Degree Kernel (with shifts). For the latter the efficient LINADD optimizations are implemented. * Also SHOGUN offers the freedom of working with custom pre-computed kernels. One of its key features is the combined kernel which can be constructed by a weighted linear combination of a number of sub-kernels, each of which not necessarily working on the same domain. An optimal sub-kernel weighting can be learned using Multiple Kernel Learning. Currently SVM 2-class classification and regression problems can be dealt with. * SHOGUN also implements a number of linear methods like Linear Discriminant Analysis (LDA), Linear Programming Machine (LPM), (Kernel) Perceptrons and features algorithms to train hidden markov models. * The input feature-objects can be dense, sparse or strings and of type int/short/double/char and can be converted into different feature types. Chains of preprocessors (e.g. substracting the mean) can be attached to each feature object allowing for on-the-fly pre-processing. * SHOGUN is implemented in C++ and interfaces to Matlab(tm), R, Octave and Python and is proudly released as Machine Learning Open Source Software ( http://mloss.org/ ) under the GPLv3+.

1 0

__array_interface__ questions
by Matthias Baas 20 Apr '08

20 Apr '08

Hi, I would like to make use of the __array_interface__ to be able to get large data sets into and out of C/C++ functions without having to touch each item individually by Python code. I was reading through the __array_interface__ description and now I have a couple of questions to clarify things: - There are two attributes, __array_interface__ and __array_struct__, and the spec says that an array has to implement only one of the two and an object processing an array also has only to check for one those attributes. But this means both sides can be compliant to the spec but they still can't be used with each other. Or is this meant to be two separate specs that just serve the same purpose? - There is a version attribute specifying the version of the interface. If I support version 3, what am I supposed to do when I encounter a higher version number? Is it always guaranteed that higher versions are backwards compatible? (the last sentence in the description seems to suggest that but can this really be guaranteed?). Shouldn't there be something like a major and minor version where different major versions are incompatible to each other? Or maybe a current version number and a compatibility version number or something like that? - There are several variants how the "data" attribute can look like and in some cases it is necessary to use the Python buffer interface. Is there an example how to use that interface (as I haven't done this before)? Or even better, is there a reference implementation for module authors that they could just use and that provides a simple to use API for obtaining the internal buffer pointer and layout information given an arbitrary Python object? I realized that writing a correct implementation of this myself is not really a trivial task (mainly because of all the variations that are allowed by the spec). - Is there anything particular I have to know to make sure my module also works on 64bit systems? What types should I use/assume when reading the pointer from the "data" tuple? - I had a look at PIL 1.1.6 and the __array_interface__ dict that the image object provides. The first thing I noticed is that it doesn't have a "version" attribute even though the spec says this one is required. The second thing is that the "data" attribute is a string, so I would have to use the buffer interface to access the data. Is there a way to get at the data pointer from Python so that I can pass it to a C function via ctypes? - Using the array interface for reading data is fine, but what is the recommended way of writing data into a buffer when the number of items is not known in advance? (so what would be the equivalent of the C++ vector::push_back() call?) - What's the status on getting this interface into the Python core? Thanks, - Matthias -

1 0

OWNDATA flag and reshape() -- views vs. copies
by Guillaume Desjardins 19 Apr '08

19 Apr '08

I'm pretty new to Python and numpy (longtime c / matlab programmer), but after a read through some of the past threads and Travis' "Guide to Numpy", I think I have a fairly good understanding of how the reshape() function / methods work, with regards to views and copies. For what its worth (and to make things clearer), my understanding is that: * Function reshape: create a view, or if it can't (i.e. resulting array cannot be expressed with constant stride parameters), a copy * Method obj.reshape(): creates a view, or throws an exception if it can't Now assuming that is the case, I do not understand the behaviour of the following code. a = array([[1,2,3,4],[5,6,7,8],[9,10,11,12]]); a.flags C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False b = a[1:,1:3] # creates view of a >> array([[ 6, 7], [10, 11]]) b.flags C_CONTIGUOUS : False F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False c = reshape(b,(4,1)) # RESHAPE FUNCTION: should copy, as view of this is impossible (? isn't it ?) c >> array([[ 6], [ 7], [10], [11]]) c.flags C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False First question: if c is a copy of b, then why isn't the OWNDATA set to true ? To prove that it is indeed a copy, >> c[0] = 600 modifies the values of c but not of b and a (while modifying b - which is clearly a view - also modifies a) d = b.reshape((4,1)) # RESHAPE METHOD: same thing but with method... shouldn't this give an error ? d.flags C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False Second question: shouldn't b.reshape((4,1)) raise a ValueError since it can't (as far as I see it) create a view ? Once again, modifying d does not affect b or a (hence it is a copy). Thanks for clearing up this issue. It is imperative that I know exactly when numpy creates views vs. copies (and that I understand all the intricacies), as I'm planning on doing A LOT of reshape() with inplace modifications. Hope I'm not polluting the mailing list with a newbie question. Thanks in advance, -- Guillaume ps: sorry to the moderators for having submitted this twice (once as a non-member, and after signing up)

2 1