Mailman 3 April 2009 - NumPy-Discussion

bug in stats.randint
by Flavio Coelho 23 Apr '09

23 Apr '09

Hi, I stumbled upon something I think is a bug in scipy: In [4]: stats.randint(1.,15.).ppf([.1,.2,.3,.4,.5]) Out[4]: array([ 2., 3., 5., 6., 7.]) When you pass float arguments to stats.randint and then call the ppf method, you get an array of floats, which clearly wrong. The rvs method doesn't display this bug so I think is a matter of poor type checking in the ppf implementation... -- --- Flávio Codeço Coelho

2 4

NaN as dictionary key?
by Wes McKinney 23 Apr '09

23 Apr '09

I assume that, because NaN != NaN, even though both have the same hash value (hash(NaN) == -32768), that Python treats any NaN double as a distinct key in a dictionary. In [76]: a = np.repeat(nan, 10) In [77]: d = {} In [78]: for i, v in enumerate(a): ....: d[v] = i ....: ....: In [79]: d Out[79]: {nan: 0, nan: 1, nan: 6, nan: 4, nan: 3, nan: 9, nan: 7, nan: 2, nan: 8, nan: 5} I'm not sure if this ever worked in a past version of NumPy, however, I have code which does a "group by value" and occasionally in the real world those values are NaN. Any ideas or a way around this problem? Thanks, Wes

4 3

String manipulation
by Nils Wagner 23 Apr '09

23 Apr '09

Hi all, How can I obtain the position of the minus sign within the following string ? >>> liste[1] '1.5-te' Nils

2 1

buggy fortran array reshape ?
by Fabrice Pardo 23 Apr '09

23 Apr '09

After reshaping a Fortran array, the new array doesn't share data with original array. I will be glad if someone can explain the strange behaviour of this program. Is it a numpy bug ? #vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv def check_bug(order): a = numpy.ndarray((3,2),order=order,dtype=int) a[0,0] = 1 b = a.reshape((6,)) a[0,0] = 2 print b[0] check_bug('C') # 2, good check_bug('F') # 1, wrong ??? print(numpy.version.version) # 1.2.1 #^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- FP

4 7

performance issue (again)
by Mathew Yeates 22 Apr '09

22 Apr '09

Hi I posted something about this earlier Say I have 2 arrays X and Y with shapes (N,3) where N is large I am doing the following for row in range(N): result=polyfit(X[row,:],Y[row,:],1,full=True) # fit 3 points with a line This takes forever and I was hoping to find a way to speed things up. But now I'm starting to wonder if this pointless. If the routine "poly fit takes a long time, when compared with the time for a Python function call, then things can't be sped up. Any comments? Mathew

3 10

Linker script, smaller source files and symbol visibility
by David Cournapeau 22 Apr '09

22 Apr '09

Hi, For quite a long time I have been bothered by the very large files needed for python extensions. In particular for numpy.core, which consists in a few files which are ~ 1 Mb, I find this a pretty high barrier of entry for newcomers, and it has quite a big impact on the code organization. I think I have found a way to split things on common platforms (this includes at least windows, mac os x, linux and solaris), without impacting other potentially less capable platforms, or static linking of numpy. Assuming my idea is technically sound and that I can demonstrate it works on say Linux without impacting other platforms (see example below), would that be considered useful ? cheers, David Technical details ================== The rationale for doing things as they are is a C limitation related to symbol visibility being limited to file scope, i.e. if you want to share a function into several files without making it public in the binary, you have to tag the function static, and include all .c files which use this function into one giant .c file. That's how we do it in numpy. Many binary format (elf, coff and Mach-O) have a mechanism to limit the symbol visibility, so that we can explicitly set the functions we do want to export. With a couple of defines, we could either include every files and tag the implementation functions as static, or link every file together and limit symbol visibility with some linker magic. Example ------- I use the spam example from the official python doc, with one function PySpam_System which is exported in a C API, and the actual implementation is _pyspam_system. * spammodule.c: define the interface available from python interpreter: #include <Python.h> #include <stdio.h> #define SPAM_MODULE #include "spammodule.h" #include "spammodule_imp.h" /* if we don't know how to deal with symbol visibility on the platform, just include everything in one file */ #ifdef SYMBOL_SCRIPT_UNSUPPORTED #include "spammodule_imp.c" #endif /* C API for spam module */ static int PySpam_System(const char *command) { _pyspam_implementation(command); return 0; } * spammodule_imp.h: declares the implementation, should only be included by spammodule.c and spammodule_imp.c which implements the actual function #ifndef _IMP_H_ #define _IMP_H_ #ifndef SPAM_MODULE #error this should not be included unless you really know what you are doing #endif #ifdef SYMBOL_SCRIPT_UNSUPPORTED #define SPAM_PRIVATE static #else #define SPAM_PRIVATE #endif SPAM_PRIVATE int _pyspam_implementation(const char *command); #endif For supported platforms (where SYMBOL_SCRIPT_UNSUPPORTED is not defined), _pyspam_implementation would not be visible because we would have a list of functions to export (only initspam in this case). Advantages ---------- This has several advantages on platforms where this is supported - code more amenable: source code which are thousand of lines are difficult to follow - faster compilation times: in my experience, compilation time doesn't scale linearly with the amount of code. - compilation can be better parallelized - changing one file does not force a whole multiarray/ufunc module recompilation (which can be pretty long when you chase bugs in it) Another advantage is related to namespace pollution. Since library extensions are static libraries for now, any symbol frome those libraries used by any extension is publicly available. For example, now that multiarray.so uses the npy_math library, every symbol in npy_math is in the public namespace. That's also true for every scipy extensions (for example, _fftpack.so exports the whole dfftpack public API). If we want to go further down the road of making core computational code publicly available, I think we should improve this first. Disadvantage ------------ We need to code it. There are two parts: - numpy.distutils support: I have already something working in for linux. Once we have one platform working, adding others should not be a problem - changing the C code: we could at first splitting things in .c files but still including everything, and then starting the conversion.

3 7

numpy Mac binary for Python 2.6
by Russell E. Owen 22 Apr '09

22 Apr '09

Does anyone have a binary installer for numpy 1.3.0 and Python 2.6? I've been able to install from source and all tests passed, but I prefer official binaries because I have some confidence that there are no hidden dependencies (important for distributing self-contained apps). I tried to build a binary myself with bdist_mpkg, but it failed with the following (I appended the full traceback, this is just a summary): "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac kages/bdist_mpkg/tools.py", line 90, in get_gid raise ValueError('group %s not found' % (name,)) ValueError: group admin not found This is with bdist_mpkg 0.4.3 (official release), machllib and py2app from svn trunk (but same error with the official releases), svn 1.6.1, MacOS X 10.5.6 and export MACOSX_DEPLOYMENT_TARGET=10.4. Not sure why get_gid would be failing. The admin group certainly appears to exist and is the owner of /Library/Frameworks. -- Russell /usr/bin/mkbom build/bdist.macosx-10.3-fat/mpkg/platlib/Library/Frameworks/Python.framew ork/Versions/2.6/lib/python2.6/site-packages dist/numpy-1.3.0-py2.6-macosx10.5.mpkg/./Contents/Packages/numpy-platlib- 1.3.0-py2.6-macosx10.5.pkg/Contents/Archive.bom sh: /usr/bin/nidump: No such file or directory Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.6/bin/bdist_mpkg", line 8, in <module> load_entry_point('bdist-mpkg==0.4.3', 'console_scripts', 'bdist_mpkg')() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac kages/bdist_mpkg/script_bdist_mpkg.py", line 27, in main execfile(sys.argv[0], g, g) File "setup.py", line 172, in <module> setup_package() File "setup.py", line 165, in setup_package configuration=configuration ) File "/Users/rowen/Archives/PythonPackages/numpy-1.3.0/numpy/distutils/core.p y", line 184, in setup return old_setup(**new_attr) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutil s/core.py", line 152, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutil s/dist.py", line 975, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutil s/dist.py", line 995, in run_command cmd_obj.run() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac kages/bdist_mpkg/cmd_bdist_mpkg.py", line 443, in run self.make_scheme_package(scheme) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac kages/bdist_mpkg/cmd_bdist_mpkg.py", line 385, in make_scheme_package self.get_scheme_description(scheme), File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac kages/bdist_mpkg/pkg.py", line 157, in make_package admin = tools.admin_writable(prefix) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac kages/bdist_mpkg/tools.py", line 106, in admin_writable gid = get_gid('admin') File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac kages/bdist_mpkg/tools.py", line 90, in get_gid raise ValueError('group %s not found' % (name,)) ValueError: group admin not found

3 6

Re: [Numpy-discussion] need some help with python scientific packages
by Emmanuelle Gouillart 22 Apr '09

22 Apr '09

Hi Nacer, > I read your message on the african python tour mailing list. I am > looking for someone able to help me find the right scientific package > for parallel computing (data mining of frequent patterns). I am > involve in a DEA course and need such a package to illustrate the > topics I am developping in my "memoire". Could you please point me > some resources on the subject? first of all, I suggest you to ask such questions on the mailing-lists dedicated to scientific Python, among which numpy-discussion(a)scipy.org (Cced in this e-mail) or SciPy Users List scipy-user(a)scipy.org. These are very active mailing-lists, where you may get answers from a large pool of experts in scientific Python, and it is always faster and more efficient than asking a single person -- all the more than I'm not an expert of frequent pattern mining or parallel computing at all! You have to suscribe first to these lists in order to get the answers to your questions. Did you take a look at the page dedicated to parallel computing with Python http://wiki.python.org/moin/ParallelProcessing ? (You don't mention explicitely in your message that you're using Python for data processing, can you first confirm this?). You may see from this page that there is a great deal of possibilities. It is difficult for me to give you more information without knowing some further details. Could you please tell us: * which kind of parallel computing do you want to perform? And on which architecture? Do you want to run jobs on a multicore computer with shared memory or to perform distributed computing on a cluster? The solutions may be quite different depending on the answer. * do you wish to perform fine-grained parallel computing (e.g. do you need to separate your data objects into different parts that will be processed on different cores and "glue together" the output of the different tasks), or do you have independent high-level operations that could be easily parallelized (e.g. independent processing of different datasets) * and in general, can you tell us a little more about what you want to do? However, once again, I don't know much about such subjects, you might get far more interesting answers from the list. Cheers (and good luck with you Master thesis :D), Emmanuelle

1 0

Numpy float precision vs Python list float issue
by Ruben Salvador 21 Apr '09

21 Apr '09

Hi everybody! First of all I should say I am a newbie with Python/Scipy. Have been searching a little bit (google and lists) and haven't found a helpful answer...so I'm posting. I'm using Scipy/Numpy to do image wavelet transforms via the lifting scheme. I grabbed some code implementing the transforms with Python lists (float type). This code works perfectly, but slow for my needs (I'll be doing some genetic algorithms to evolve coefficients of the filters and the forward and inverse transform will be done many times). It's just implemented by looping in the lists and making computations this way. Reconstructed image after doing a forward and inverse transform is perfect, this is, original and reconstructed images difference is 0. With Scipy/Numpy float arrays slicing this code is much faster as you know. But the reconstructed image is not perfect. The image difference maximum and minimum values returns: maximum difference => 3.5527136788e-15 minimum difference => -3.5527136788e-15 Is this behavior expected? Because it seems sooo weird to me. If expected, anyway to override it? I include some test code for you to reproduce. It's part of a transform over a 8x8 2D signal (for simplicity). It's not complete (again for simplicity), but enough to reproduce. It does part of a forward and inverse transform both with lists and arrays, printing the differences (there is commented code showing some plots with the results as I used when transforming real images, but for the purpose, is enough with the return results I think). Code is simple (maybe long but it's always the same). Instead of using the faster array slicing as commented above, I am using here array looping, so that the math code is exactly the same as in the case list. This happens in the next three system/platforms. * System 1 (laptop): --------------------------- 64 bit processor running Kubuntu 8.04 32 bits Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52 Numpy version: 1:1.0.4-6ubuntu3 Scipy version: 0.6.0-8ubuntu1 * System 2 (PC): -------------------------- Windows Xp on 64 bit processor Enthought Python distribution (EPD Py25 v4.1.30101). This is a Python 2.5.2 with Numpy 1.1.1 and Scipy 0.6.0 * System 3 (same PC as 2): -------------------------------------- Debian Lenny 64 bit on 64 bit processor Not sure about versions here, but doesn't mind because behavior is prety much the same in the 3 systems Thanks everybody in advance for the help! Rubén Salvador PhD Student Industrial Electronics Center Universidad Politécnica de Madrid

11 16

help getting started
by John Seales 21 Apr '09

21 Apr '09

Intel mac. Python 2.6. I'm trying to start using numpy and scipy, and am having trouble. I'm trying to install numpy following the instructions on http://scipy.org/Installing_SciPy/Mac_OS_X . when I give the python setup.py build command, it doesn't work. Here's my interaction at the terminal: john-seales-computer:/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy johnseales$ python setup.py build This is the wrong setup.py file to run john-seales-computer:/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy Anybody have any ideas how I can get up and runnng? _________________________________________________________________ Rediscover Hotmail®: Now available on your iPhone or BlackBerry http://windowslive.com/RediscoverHotmail?ocid=TXT_TAGLM_WL_HM_Rediscover_Mo…

4 9