NumPy-Discussion September 2011

numpy-discussion@python.org

88 participants
79 discussions

replacing the mechanism for dispatching ufuncs

by Mark Wiebe

NumPy has a mechanism built in to allow subclasses to adjust or override aspects of the ufunc behavior. While this goal is important, this mechanism only allows for very limited customization, making for instance the masked arrays unable to work with the native ufuncs in a full and proper way. I would like to deprecate the current mechanism, in particular __array_prepare__ and __array_wrap__, and introduce a new method I will describe below. If you've ever used these mechanisms, please review this design to see if it meets your needs. Any class type which would like to override its behavior in ufuncs would define a method called _numpy_ufunc_, and optionally an attribute __array_priority__ as can already be done. The class which wins the priority battle gets its _numpy_ufunc_ function called as follows: return arr._numpy_ufunc_(current_ufunc, *args, **kwargs) To support this overloading, the ufunc would get a new support method, result_type, and there would be a new global function, broadcast_empty_like. The function ufunc.empty_like behaves like the global np.result_type, but produces the output type or a tuple of output types specific to the ufunc, which may follow a different convention than regular arithmetic type promotion. This allows for a class to create an output array of the correct type to pass to the ufunc if it needs to be different than the default. The function broadcast_empty_like is just like empty_like, but takes a list or tuple of arrays which are to be broadcast together for producing the output, instead of just one. Thanks, Mark A simple class which overrides the ufuncs might look as follows: def sin(ufunc, *args, **kwargs): # Convert degrees to radians args[0] = np.deg2rad(args[0]) # Return a regular array, since the result is not in degrees return ufunc(*args, **kwargs) class MyDegreesClass: """Array-like object with a degrees unit""" def __init__(arr): self.arr = arr def _numpy_ufunc_(ufunc, *args, **kwargs): override = globals().get(ufunc.name) if override: return override(ufunc, *args, **kwargs) else: raise TypeError, 'ufunc %s incompatible with MyDegreesClass' % ufunc.name A more complex example will be something like this: def my_general_ufunc(ufunc, *args, **kwargs): # Extract the 'out' argument. This only supports ufuncs with # one output, currently. out = kwargs.get('out') if len(args) > ufunc.nin: if out is None: out = args[ufunc.nin] else: raise ValueError, "'out' given as both a position and keyword argument" # Just want the inputs from here on args = args[:ufunc.nin] # Strip out MyArrayClass, but allow operations with regular ndarrays raw_in = [] for a in args: if isinstance(a, MyArrayClass): raw_in.append(a.arr) else: raw_in.append(a) # Allocate the output array if not out is None: if isinstance(out, MyArrayClass): raise TypeError, "'out' must have type MyArrayClass" else: # Create the output array, obeying the 'order' parameter, # but disallowing subclasses out = np.broadcast_empty_like([args, order=kwargs.get('order'), dtype=ufunc.result_type(args), subok=False) # Override the output argument kwargs['out'] = out.arr # Call the ufunc ufunc(*args, **kwargs) # Return the output return out class MyArrayClass: def __init__(arr): self.arr = arr def _numpy_ufunc_(ufunc, *args, **kwargs): override = globals().get(ufunc.name) if override: return override(ufunc, *args, **kwargs) else: return my_general_ufunc(ufunc, *args, **kwargs)

10 years, 3 months

factorization updates in numpy

by Jaroslav Hajek

hello, I've been looking into options to do fast low-rank dense matrix factorization updates (aka Matlab's qrupdate) with NumPy, but couldn't find basically anything. Did I miss something? A few years ago I wrote the qrupdate library http://sourceforge.net/projects/qrupdate/, and subsequently wrapped it for Octave, giving it a broad spectrum of efficient factorization updating capabilities - see http://www.gnu.org/software/octave/doc/interpreter/Matrix-Factorizations.ht…, functions qrupdate, qrinsert, qrdelete, qrshift, cholupdate, cholinsert, choldelete, cholshift, luupdate. Would there be interest in having similar functions in NumPy? Would NumPy or SciPy be a better place? best regards -- Jaroslav Hajek Site Reliability Engineer Google Switzerland GmbH

10 years, 3 months

[ANN] ODE dy/dt = f(t) solver with guaranteed speficiable accuracy

by Dmitrey

hi all, now free solver interalg from OpenOpt framework (based on interval analysis) can solve ODE dy/dt = f(t) with guaranteed specifiable accuracy. See the ODE webpage for more details, there is an example of comparison with scipy.integrate.odeint, where latter fails to solve a problem. Future plans include solving of some general ODE systems dy/dt = f(y, t). Regards, D.

10 years, 3 months

Re: [Numpy-discussion] Trouble installing numpy

by The Helmbolds

Thanks. The error message did not refer to the difference between 32-bit and 64-bit, only to the supposed absence of Python 2.7. And Idle says it's using Python 2.7.2 64 bit AMD64 on Win 32, which confused me. However, I see that the suggested web page contains the following four options, each of which apparently offers a NumPy-like Windows installation package for 64-bit windows machines: Bottleneck-0.5.0.win-amd64-py2.7.exe numpy-MKL-1.6.1.win-amd64-py2.7.exe [*-see comment below] numpy-unoptimized-1.6.1.win-amd64-py2.7.exe numscons-012.0.win-amd64-py2.7.exe [*-coment] This item is described as "not compatible with the official SciPy distributions." Any comments on which ones I should try first? The Helmbolds 2645 E Southern Ave A241 Tempe AZ 85282 Email: helmrp(a)yahoo.com VOX: 480-831-3611 CELL: 602-568-6948 (but not often turned on) >________________________________ >From: "numpy-discussion-request(a)scipy.org" <numpy-discussion-request(a)scipy.org> >To: numpy-discussion(a)scipy.org >Sent: Monday, September 26, 2011 10:00 AM >Subject: NumPy-Discussion Digest, Vol 60, Issue 55 > >Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)scipy.org > >To subscribe or unsubscribe via the World Wide Web, visit > http://mail.scipy.org/mailman/listinfo/numpy-discussion >or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org > >You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org > >When replying, please edit your Subject line so it is more specific >than "Re: Contents of NumPy-Discussion digest..." > > >Today's Topics: > > 1. Re: Trouble installing numpy (Charles R Harris) > 2. Re: Trouble installing numpy (Olivier Delalleau) > > >---------------------------------------------------------------------- > >Message: 1 >Date: Mon, 26 Sep 2011 10:14:16 -0600 >From: Charles R Harris <charlesr.harris(a)gmail.com> >Subject: Re: [Numpy-discussion] Trouble installing numpy >To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> >Message-ID: > <CAB6mnx+rC01d7mAtdEgvGxT=SEh+vRLo5P-HSvGvG16XreN2Hw(a)mail.gmail.com> >Content-Type: text/plain; charset="iso-8859-1" > >On Mon, Sep 26, 2011 at 9:43 AM, The Helmbolds <helmrp(a)yahoo.com> wrote: > >> Using Source Forge download of NumPy installer package: >> numpy-1.6.1-win32-superpack-python 2.7.exe. >> >> Installation Wizard starts, but then installation fails with error message: >> Python version 2.7 required, which was not found in the registry >> >> Idle says it's using: >> Python 2.7.2 64 bit AMD64 on Win 32 >> >> So what's holding up the installation, and what do I need to do to install >> numpy? >> >> > >Your Python is 64 bits, the numpy package is 32 bits and needs 32 bit >Python. If you need free 64 bit numpy on windows your best bet is probably >here <http://www.lfd.uci.edu/%7Egohlke/pythonlibs/>. > >Chuck >

10 years, 3 months

Trouble installing numpy

by The Helmbolds

Using Source Forge download of NumPy installer package: numpy-1.6.1-win32-superpack-python 2.7.exe. Installation Wizard starts, but then installation fails with error message: Python version 2.7 required, which was not found in the registry Idle says it's using: Python 2.7.2 64 bit AMD64 on Win 32 So what's holding up the installation, and what do I need to do to install numpy? The Helmbolds 2645 E Southern Ave A241 Tempe AZ 85282 Email: helmrp(a)yahoo.com VOX: 480-831-3611 CELL: 602-568-6948 (but not often turned on)

10 years, 3 months

Ticket review process

by Hrvoje Nikšić

Hello! About a month ago I filed ticket #1943 (http://projects.scipy.org/numpy/ticket/1943), which describes a remaining unfixed item in #1388. Several days ago I also attached a patch and moved the ticket to the "needs review" state. Is there anything else I should or could do to get the patch reviewed and, if passing, applied? The bug can be worked around, but requires workarounds in all code that needs to use frombuffer with potentially empty buffers.

10 years, 3 months

How to plot heatmap in matplotlib?

by fdu.xiaojf＠gmail.com

Dear all, Heatmap (like those on the page http://www2.warwick.ac.uk/fac/sci/moac/students/peter_cock/r/heatmap/) is a frequently used type of image in microarray data analysis. However, it seems there are no convenient functions in matplotlib to plot heatmap (please correct me if I was wrong), so I'm planning to write my own. Let me take the heatmap by the link http://www2.warwick.ac.uk/fac/sci/moac/students/peter_cock/r/heatmap/scaled… as an example, which is produced by R. With my limited knowledge and expertise of matplotlib, I have the following questions and I hope you guys could help me. 1) I tend to use pcolor to draw the colormap in the central area. However, I've seen a lot of examples draw colormap with imshow. What's the difference between pcolor and imshow? Shall I use pcolor or imshow to produce the heatmap in the link above? 2) How to draw the dendrograms on the top and left of the colormap? I got hints from http://matplotlib.sourceforge.net/examples/axes_grid/scatter_hist.html on how to append axes to current plot, but I still have now idea how to draw the dengrograms. 3) How to draw the column side colormap (the smaller one) between the top dendrogram and the big colormap? 4) I can use colorbar to draw a colorbar, but how to place the colorbar on the topleft of the image just as the R heatmap does? 5) Any other suggestions on how to draw the heatmap? Thanks and any help will be greatly appreciated. Regards, Jianfeng

10 years, 3 months

ANN: PyTables 2.3 released

by Antonio Valentino

=========================== Announcing PyTables 2.3 =========================== We are happy to announce PyTables 2.3. This release comes after about 10 months of development and after that Francesc Altet, the creator of PyTables, ceased activities with the project. Thank you Francesc. Also the project has been moved to GitHub: http://github.com/PyTables/PyTables. What's new ========== The main new features in 2.3 series are: * PyTables now includes the codebase of PyTables Pro (now release under open source license) gaining a lot of performance improvements and some new features like: - the new and powerful indexing engine: OPSI - a fine-tuned LRU cache for both metadata (nodes) and regular data * The entire documentation set has been converted to ReStructuredTest and Sphinx As always, a large amount of bugs have been addressed and squashed too. In case you want to know more in detail what has changed in this version, have a look at: http://pytables.github.com/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/2.3 For an on-line version of the manual, visit: http://pytables.github.com/usersguide/index.html What it is? =========== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than 1 tenth of a second. Resources ========= About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team

10 years, 3 months

Debug memory block at address p=0x2ba5928: API 'm'

by Sandro Tosi

Hello, I'm running tests for numpy 1.6.1 on 2.7 -dbg flavor and I got this: .........................Debug memory block at address p=0x2ba5928: API 'm' 8 bytes originally requested The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. The 8 pad bytes at tail=0x2ba5930 are FORBIDDENBYTE, as expected. The block was made by call #1253108 to debug malloc/realloc. Data at p: a3 03 00 00 00 00 00 00 Fatal Python error: bad ID: Allocated using API 'm', verified using API 'o' Running unit tests for numpy NumPy version 1.6.1 and it got aborted. Running the tests with 2.6-dbg doesn't show this behavior. How can I help you debug it? Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi

10 years, 3 months

ImportError: libptf77blas.so.3: cannot open shared object file: No such file or directory

by Marijn Verkerk

Dear all, Maybe someone has experienced this problem before. I was able to build and install numpy in a seperate python 2.7 folder on our cluster. When I try to import numpy I get the following error: $ /software/Python-2.7.2/bin/python -c 'import numpy; numpy.test()' Traceback (most recent call last): File "<string>", line 1, in <module> File "/illumina/software/Python-2.7.2/lib/python2.7/site-packages/numpy/__init__.py", line 137, in <module> import add_newdocs File "/illumina/software/Python-2.7.2/lib/python2.7/site-packages/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/illumina/software/Python-2.7.2/lib/python2.7/site-packages/numpy/lib/__init__.py", line 13, in <module> from polynomial import * File "/illumina/software/Python-2.7.2/lib/python2.7/site-packages/numpy/lib/polynomial.py", line 17, in <module> from numpy.linalg import eigvals, lstsq File "/illumina/software/Python-2.7.2/lib/python2.7/site-packages/numpy/linalg/__init__.py", line 48, in <module> from linalg import * File "/illumina/software/Python-2.7.2/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 23, in <module> from numpy.linalg import lapack_lite ImportError: libptf77blas.so.3: cannot open shared object file: No such file or directory In the __config.py file the folder where libptf77 should be is present. Any suggestions? Regards, Marijn

10 years, 3 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion September 2011