Mailman 3 May 2010 - NumPy-Discussion

zeros_like and friends shouldn't use ndarray.__new__(type(a), ...)
by Jim Porter May 27, 2010

May 27, 2010

Ok, let's try sending this message again, since it looks like I can't send from gmane... (See discussion on python-list at http://permalink.gmane.org/gmane.comp.python.general/661328 for context) numpy.zeros_like contains the following code: def zeros_like(a): if isinstance(a, ndarray): res = ndarray.__new__(type(a), a.shape, a.dtype, order=a.flags.fnc) res.fill(0) return res ... This is a problem because basetype.__new__(subtype, ...) raises an exception when subtype is defined from C (specifically, when Py_TPFLAGS_HEAPTYPE is not set). There's a check in Objects/typeobject.c in tp_new_wrapper that disallows this (you can grep for "is not safe" to find there the exception is raised). The end result is that it's impossible to use zeros_like, ones_like or empty_like with ndarray subtypes defined in C. While I'm still not sure why Python needs this check in general, Robert Kern pointed out that the problem can be fixed pretty easily in NumPy by changing zeros_like and friends to something like this (with some modifications from me): def zeros_like(a): if isinstance(a, ndarray): res = numpy.zeros(a.shape, a.dtype, order=a.flags.fnc) res = res.view(type(a)) res.__array_finalize__(a) return res ... - Jim

2 1

Bug in frompyfunc starting at 10000 elements?
by James Bergstra May 26, 2010

May 26, 2010

Hi all, I'm wondering if this is a bug... Something strange happens with my ufunc as soon as I use 10000 elements. As the test shows, the ufunc computes the correct result for either the first or last 9999 elements, but both at the same time is no good. Turns out I'm only running numpy 1.3.0 with Python 2.6.4... could someone with a more recent installation maybe check to see if this has been fixed? Thanks, def test_ufunc(): np = numpy rng = np.random.RandomState(2342) a = rng.randn(10000, 2) b = rng.randn(10000, 1) f = lambda x,y:x*y ufunc = np.frompyfunc(lambda *x:numpy.prod(x), 2, 1) def g(x,y): return np.asarray(ufunc(x,y), dtype='float64') assert numpy.allclose(f(a[:-1],b[:-1]), g(a[:-1],b[:-1])) # PASS assert numpy.allclose(f(a[1:],b[1:]), g(a[1:],b[1:])) # PASS assert numpy.allclose(f(a,b), g(a,b)) # FAIL -- http://www-etud.iro.umontreal.ca/~bergstrj

3 2

numpy and the Google App Engine
by Christopher Hanley May 26, 2010

May 26, 2010

Greetings, Google provides a product called App Engine. The description from their site follows, "Google App Engine enables you to build and host web apps on the same systems that power Google applications. App Engine offers fast development and deployment; simple administration, with no need to worry about hardware, patches or backups; and effortless scalability. " You can deploy applications written in either Python or JAVA. There are free and paid versions of the service. The Google App Engine would appear to be a powerful source of CPU cycles for scientific computing. Unfortunately this is currently not the case because numpy is not one of the supported libraries. The Python App Engine allows only the installation of user supplied pure Python code. I have recently returned from attending the Google I/O conference in San Francisco. While there I inquired into the possibility of getting numpy added. The basic response was that there doesn't appear to be much interest from the community given the amount of work it would take to vet and add numpy. I would like to ask your help in changing this perception. The quickest and easiest thing you can do would be to add your "me too" to this feature request (item #190) on the support site: http://code.google.com/p/googleappengine/issues/detail?id=190 If this issue is important to you could also consider raising this issue in the related Google Group: http://groups.google.com/group/google-appengine Letting Google know how you will use numpy would be helpful. If you or your institution would be willing to pay for service if you could deploy cloud applications that required numpy would be helpful to let them know as well. Finally, if you run into any App Engine developers (Guido included) let them know that you would like to see numpy added. Thank you for your time and consideration. Chris -- Christopher Hanley Senior Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338

4 5

[Patch] Fix memmap pickling
by Gael Varoquaux May 26, 2010

May 26, 2010

Memmapped arrays don't pickle right. I know that to get them to really pickle and restore identically, we would need some effort. However, in the current status, pickling and restoring a memmapped array leads to tracebacks that seem like they could be avoided. I am attaching a patch with a test that shows the problem, and a fix. Should I create a ticket, or is this light-enough to be applied immediatly? Cheers, Gaël

2 3

FW: Numpy python build
by Padma TAN May 26, 2010

May 26, 2010

Hi, Can I just install numpy and scipy without ATLAS? And what does this means " gnu: no Fortran 90 compiler found"? Im installing on RHEL Thanks in advance! [root@giswk002 numpy-1.3.0]# python setup.py build Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /usr/local/Python-2.6.2/lib libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/Python-2.6.2/lib libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2 libraries ptf77blas,ptcblas,atlas not found in /usr/lib NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in /usr/local/Python-2.6.2/lib libraries f77blas,cblas,atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib/sse2 libraries f77blas,cblas,atlas not found in /usr/lib NOT AVAILABLE /usr/local/numpy-1.3.0/numpy/distutils/system_info.py:1383: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) blas_info: libraries blas not found in /usr/local/Python-2.6.2/lib libraries blas not found in /usr/local/lib FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /usr/local/Python-2.6.2/lib libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/Python-2.6.2/lib libraries lapack_atlas not found in /usr/local/Python-2.6.2/lib libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2 libraries lapack_atlas not found in /usr/lib/sse2 libraries ptf77blas,ptcblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in /usr/local/Python-2.6.2/lib libraries lapack_atlas not found in /usr/local/Python-2.6.2/lib libraries f77blas,cblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib/sse2 libraries lapack_atlas not found in /usr/lib/sse2 libraries f77blas,cblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_info NOT AVAILABLE /usr/local/numpy-1.3.0/numpy/distutils/system_info.py:1290: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: libraries lapack not found in /usr/local/Python-2.6.2/lib libraries lapack not found in /usr/local/lib FOUND: libraries = ['lapack'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['lapack', 'blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src building py_modules sources building library "npymath" sources building extension "numpy.core._sort" sources adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/config.h' to sources. adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h' to sources. numpy/core/code_generators/genapi.py:9: DeprecationWarning: the md5 module is deprecated; use hashlib instead import md5 executing numpy/core/code_generators/generate_numpy_api.py adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/__multiarray_api.h' to sources. numpy.core - nothing done with h_files = ['build/src.linux-i686-2.6/numpy/core/include/numpy/config.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/__multiarray_api.h'] building extension "numpy.core.multiarray" sources adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/config.h' to sources. adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h' to sources. executing numpy/core/code_generators/generate_numpy_api.py adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/__multiarray_api.h' to sources. adding 'build/src.linux-i686-2.6/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files = ['build/src.linux-i686-2.6/numpy/core/src/scalartypes.inc', 'build/src.linux-i686-2.6/numpy/core/src/arraytypes.inc', 'build/src.linux-i686-2.6/numpy/core/include/numpy/config.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/__multiarray_api.h'] building extension "numpy.core.umath" sources adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/config.h' to sources. adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h' to sources. executing numpy/core/code_generators/generate_ufunc_api.py adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/__ufunc_api.h' to sources. adding 'build/src.linux-i686-2.6/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files = ['build/src.linux-i686-2.6/numpy/core/src/scalartypes.inc', 'build/src.linux-i686-2.6/numpy/core/src/arraytypes.inc', 'build/src.linux-i686-2.6/numpy/core/src/umath_funcs.inc', 'build/src.linux-i686-2.6/numpy/core/src/umath_loops.inc', 'build/src.linux-i686-2.6/numpy/core/include/numpy/config.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/__ufunc_api.h'] building extension "numpy.core.scalarmath" sources adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/config.h' to sources. adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h' to sources. executing numpy/core/code_generators/generate_numpy_api.py adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/__multiarray_api.h' to sources. executing numpy/core/code_generators/generate_ufunc_api.py adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/__ufunc_api.h' to sources. numpy.core - nothing done with h_files = ['build/src.linux-i686-2.6/numpy/core/include/numpy/config.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/__multiarray_api.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/__ufunc_api.h'] building extension "numpy.core._dotblas" sources building extension "numpy.core.umath_tests" sources building extension "numpy.lib._compiled_base" sources building extension "numpy.numarray._capi" sources building extension "numpy.fft.fftpack_lite" sources building extension "numpy.linalg.lapack_lite" sources adding 'numpy/linalg/lapack_litemodule.c' to sources. adding 'numpy/linalg/python_xerbla.c' to sources. building extension "numpy.random.mtrand" sources /usr/local/numpy-1.3.0/numpy/distutils/command/config.py:39: DeprecationWarning: +++++++++++++++++++++++++++++++++++++++++++++++++ Usage of try_run is deprecated: please do not use it anymore, and avoid configuration checks involving running executable on the target machine. +++++++++++++++++++++++++++++++++++++++++++++++++ DeprecationWarning) customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/local/Python-2.6.2/include/python2.6 -c' gcc: _configtest.c gcc -pthread _configtest.o -o _configtest _configtest failure. removing: _configtest.c _configtest.o _configtest building data_files sources running build_py copying numpy/version.py -> build/lib.linux-i686-2.6/numpy copying build/src.linux-i686-2.6/numpy/__config__.py -> build/lib.linux-i686-2.6/numpy copying build/src.linux-i686-2.6/numpy/distutils/__config__.py -> build/lib.linux-i686-2.6/numpy/distutils running build_clib customize UnixCCompiler customize UnixCCompiler using build_clib running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using build_ext running scons running build_scripts adding 'build/scripts.linux-i686-2.6/f2py' to scripts ------ End of Forwarded Message

2 1

__eq__ with str and object
by Keith Goodman May 26, 2010

May 26, 2010

I don't understand this: >> a1 = np.array(['a', 'b'], dtype=object) >> a2 = np.array(['a', 'b']) >> >> a1 == a2 array([ True, True], dtype=bool) # Looks good >> a2 == a1 False # Should I have expected this? This works like I expected: >> a1 = np.array([1, 2], dtype=object) >> a2 = np.array([1, 2]) >> >> a1 == a2 array([ True, True], dtype=bool) >> a2 == a1 array([ True, True], dtype=bool)

3 3

Reading and writing binary data to/from file objects
by Christoph Bersch May 26, 2010

May 26, 2010

Hi, I want to read binary data from a file using fromfile. This works as long as I use a file name as argument to fromfile. With a file object the data is wrong! Consider the following example: from numpy import * fname='file.bin' fname2='file2.bin' a = arange(1, 30) print type(a[0]) print "\noriginal data" print a # write to file name a.tofile(fname) # write to file object f = open(fname2, 'w') a.tofile(f) f.close() print "\nWritten to file name, read from file name" b = fromfile(fname, dtype=int32) print b print "\nWritten to file name, read from file object" f = open(fname, 'r') b = fromfile(f, dtype=int32) f.close() print b print "\nWritten to file object, read from file name" b = fromfile(fname2, dtype=int32) print b print "\nWritten to file object, read from file object" f = open(fname2, 'r') b = fromfile(f, dtype=int32) f.close() print b This prints: <type 'numpy.int32'> original data [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29] Written to file name, read from file name [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29] Written to file name, read from file object [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 0 0 0 0] Written to file object, read from file name [ 1 2 3 4 5 6 7 8 9 2573 2816 3072 3328 3584 3840 4096 4352 4608 4864 5120 5376 5632 5888 6144 6400 6656 6912 7168 7424] Written to file object, read from file object [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 0 0 0 0] Size of 'file.bin' (written to file name) is 116 Bytes, size of 'file2.bin' (written to file object) is 117 Bytes! So the only correct data is when writing to a file name and reading from a file name. All other variants yield wrong data! This happens with Numpy 1.4.1 with Python 2.6 under Windows XP. For comparison I only have a 1.1.0 Numpy with Python 2.5 under Linux Debian where in all cases I get the same, correct arrays! Am I doing something substantially wrong or is this a bug? Christoph

2 2

Geometric, negative binomial and poisson fail for extreme arguments
by Charles R Harris May 26, 2010

May 26, 2010

Josef, This is ticket #896 from two years ago. IIRC, there was some more recent discussion on the list of some of these. Do you know what the current state of these distributions is? Chuck

2 2

Re: [Numpy-discussion] Introduction to Scott, Jason, and (possibly) others from Enthought
by Joe Harrington May 25, 2010

May 25, 2010

On Tue, 25 May 2010 15:54:26 -0500, Travis Oliphant <oliphant(a)enthought.com> wrote: >On May 25, 2010, at 2:50 PM, Charles R Harris wrote: > >> >> >> On Tue, May 25, 2010 at 1:37 PM, Travis Oliphant <oliphant(a)enthought.com >> > wrote: >> >> Hi everyone, >> >> There has been some talk about re-factoring NumPy to separate out the >> Python C-API layer and make NumPy closer to a C-library. I know >> there are a few different ideas about what this means, and also that >> people are very busy. I also know there is a NumPy 2.0 release that >> is in the works. >> >> I'm excited to let everyone know that we (at Enthought) have been able >> to find resources (about 3 man months) to work on this re-factoring >> project and Scott and Jason (both very experienced C and Python >> programmers) are actively pursuing it. My hope is that NumPy 2.0 >> will contain this re-factoring (which should be finished just after >> SciPy 2010 --- where I'm going to organize a Sprint on NumPy which >> will include at least date-time improvements and re-factoring work). >> >> While we have specific goals for the re-factoring, we want this >> activity to be fully integrated with the NumPy community and Scott and >> Jason want to interact with the community as much as feasible as they >> suggest re-factoring changes (though they both have more experience >> with phone-conversations to resolve concerns than email chains and so >> some patience from everybody will be appreciated). >> >> Because Jason and Scott are new to this mailing list (but not new to >> NumPy), I wanted to introduce them so they would feel more >> comfortable posting questions and people would have some context as to >> what they were trying to do. >> >> Scott and Jason are both very proficient and skilled programmers and I >> have full confidence in their abilities. That said, we very much >> want the input of as many people as possible as we pursue the goal of >> grouping together more tightly the Python C-API interface layer to >> NumPy. >> >> I will be involved in some of the discussions, but am currently on a >> different project which has tight schedules and so I will only be able >> to provide limited "mailing-list" visibility. >> >> >> I think 2.0 would be a bit early for this. Is there any reason it >> couldn't be done in 2.1? What is the planned policy with regards to >> the visible interface for extensions? It would also be nice to have >> a rough idea of how the resulting code would be layered, i.e., what >> is the design for this re-factoring. Simply having a design would be >> a major step forward. > >The problem with doing it in 2.1 is that this re-factoring will >require extensions to be re-built. The visible interface to >extensions will not change, but there will likely be ABI >incompatibility. It seems prudent to do this in NumPy 2.0. >Perhaps we can also put in place the ABI-protecting indirection >approaches that David C. was suggesting earlier. > >Some aspects of the design are still being fleshed out, but the basic >idea is to separate out a core library that is as independent of the >Python C-API as possible. There will likely be at least some >dependency on the Python C-API (reference counting and error handling >and possibly others) which any interface would have to provide in a >very simple Python.h -- equivalent, for example. > >Our purpose is to allow NumPy to be integrated with other languages or >other frameworks systems without explicitly relying on CPython. >There are a lot of questions as to how this will work, and so much of >that is being worked out. Part of the reason for this mail is to >help ensure that as much of this discussion as possible takes place in >public. > >-Travis > > >> >> In any case, I think the primary goal for 2.0 should remain the >> python3k port. >> >> Chuck >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion(a)scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >-- >Travis Oliphant >Enthought Inc. >1-512-536-1057 >http://www.enthought.com >oliphant(a)enthought.com > Chuck wants to make 2.0 be the py3k release. I expressed a desire some time ago to hold 2.0 until we had reviewed docs, on the basis that this should have been done for 1.0, and 2.0 is our "real 1.0" for this and other reasons. Now Travis wants to refactor the C code in a way that breaks extensions (another "real 1.0" item). But Chuck wants to get his project out the door soon, and not without justification. NumPy is not a small project anymore and it would make sense not only to have an open community process (which we sort-of do, except when we don't), but to define that process and to conduct more of it in plain view, on the web site, rather than hiding it here in the list archives where only subscribers are likely to see it. There are many interested parties (users) who are not on the lists. It's fine (even good) not to commit to specific dates until we know them for sure, but the process and rough timeline should be laid out publicly. I hope we can agree on one item, which is that the 2.0 release should be one that we are not retroactively embarrassed about. This means it should be well tested and that the docs should be up to date for the new stuff, including examples. Whether the docs need to be reviewed or not is a different question; they should at least not be wrong. It's May 25. SciPy 2010 starts 28 June, just one month out. Chuck and Travis, would you be amenable to making a 2.0 release page on the web site and share there your timelines for making these major changes, including testing and documenting them? It would also be good to see any other planned changes listed out, the 2.0 release cutoff date, and a timeline for the release testing process. Thanks, --jh--

1 0

ensuring docstrings in new code
by Joe Harrington May 25, 2010

May 25, 2010

Over on [Numpy-discussion] Extending documentation to c code, David G. gave voice to a frustration he and I share about the status of documentation in the new-code development process. I don't want to paint with a broad brush, yet in recent months there have been a number of checkins, unanimously passed off by the entire core development group/Steering Committee, without a single mention that this code entered SVN with deficient or even completely absent docstrings. I'm not pointing out the offenses simply because I don't think public embarrassment is a good policy, but David has a list that we can share with developers privately. It is really hard to do much without docs on anything you didn't write yourself. When I learned how to program, it was beaten into me to document as I went along, that good projects always did, and that major problems up to and including a complete implosion and abandonment of the software could and sometimes did result if docs were not kept current. The Doc Project is not the writing arm of the developers. So far, we have only worked on the existing software, not anything new, and most of us don't even have SVN access. Rather, the Doc Project is cleaning up what others, in their haste, did not do, and it frustrates us to see others still not doing it. What change of procedures would ensure that no code, possibly including bug fixes, enters SVN without a ready-for-review docstring? Perhaps new code could be reviewed according to a checklist that includes "has a ready-for-review docstring". Or perhaps announcements to the lists of pending code checkins should include [CHECKIN] in the subject, to attract the attention of the Doc Project's Editor (i.e., David), so that he can act as an outside cop. Or maybe the SVN can somehow be made to check for docstrings, or to ask for them on checkin. Or maybe something in the build could make such a list and compare it to the last list and complain if there are new routines on it. We've called for a rule of no new code without docs many times. I'm not sure of the best solution, but we need something or things won't change. It's not like these are hard to write. The lack of a ready-for-review docstring is a major bug in itself. --jh--

1 0