Mailman 3 July 2012 - NumPy-Discussion

Multivariate hypergeometric distribution?
by Fernando Perez 03 Jul '12

03 Jul '12

Hi all, in recent work with a colleague, the need came up for a multivariate hypergeometric sampler; I had a look in the numpy code and saw we have the bivariate version, but not the multivariate one. I had a look at the code in scipy.stats.distributions, and it doesn't look too difficult to add a proper multivariate hypergeometric by extending the bivariate code, with one important caveat: the hard part is the implementation of the actual discrete hypergeometric sampler, which lives inside of numpy/random/mtrand/distributions.c: https://github.com/numpy/numpy/blob/master/numpy/random/mtrand/distribution… That code is hand-written C, and it only works for the bivariate case right now. It doesn't look terribly difficult to extend, but it will certainly take a bit of care and testing to ensure all edge cases are handled correctly. Does anyone happen to have that implemented lying around, in a form that would be easy to merge to add this capability to numpy? Thanks, f

3 6

Combined versus separate build
by Nathaniel Smith 03 Jul '12

03 Jul '12

Currently the numpy build system(s) support two ways of building numpy: either by compiling a giant concatenated C file, or by the more conventional route of first compiling each .c file to a .o file, and then linking those together. I gather from comments in the source code that the former is the traditional method, and the latter is the newer "experimental" approach. It's easy to break one of these builds without breaking the other (I just did this with the NA branch, and David had to clean up after me), and I don't see what value we really get from having both options -- it seems to just double the size of the test matrix without adding value. Now that the separate build seems to be fully supported, maybe it's time to finish the "experiment" and pick one approach to support going forward? I guess the arguments for each would be: - The monolithic build in principle allows for some extra intra-procedural optimization. I won't believe this until I see benchmarks, though; numpy doesn't have a lot of tiny inline-able function calls or anything like that. - The separate build is probably more convenient for developers, allowing faster rebuilds. Numpy builds fast enough for me that I'm not too worried about which approach we use, but it definitely seems worthwhile to reduce the number of configurations we have to support one way or the other. -N

3 13

Fwd: an interesting single-file, cross-platform Python deployment tool.
by Fernando Perez 03 Jul '12

03 Jul '12

Hi all, sorry for the slightly off-topic post, but I know that in our community many people often struggle with deployment issues (to colleagues, to experimental/hardware control machines, to one-off test machines, ...). I just stumbled upon this announcement by accident, and figured it might prove useful. If anyone happens to try it out and has any feedback on whether it works well or not, I suspect I wouldn't be the only one interested. Cheers, f On Mon, 02 Jul 2012 15:52:20 +0200, eGenix Team: M.-A. Lemburg wrote: > ________________________________________________________________________ > ANNOUNCING > > eGenix PyRun - One file Python Runtime > > Version 1.0.0 > > > An easy-to-use single file relocatable Python run-time - > available for Windows, Mac OS X and Unix platforms > > > This announcement is also available on our web-site for online reading: > http://www.egenix.com/company/news/eGenix-PyRun-1.0.0.html > > ________________________________________________________________________ > INTRODUCTION > > Our new eGenix PyRun combines a Python interpreter with an almost > complete Python standard library into a single easy-to-use executable, > that does not require a system wide installation and is fully > relocatable. > > eGenix PyRun's executable only needs 12MB, but still supports most > Python application and scripts - and it can be further compressed to > 3-4MB using gzexe or upx. > > Compared to a regular Python installation of typically 100MB on disk, > this makes eGenix PyRun ideal for applications and scripts that need to > be distributed to many target machines, client installations or > customers. > > It makes "installing" Python on a Unix based system as simple as copying > a single file. > > http://www.egenix.com/products/python/PyRun/ > > ________________________________________________________________________ > NEWS > > This is the first public release of eGenix PyRun. We have been using the > product internally in our mxODBC Connect Server since 2008 with great > success and have now extracted it into a stand-alone open-source > product. > > We provide both the source archive to build your own eGenix PyRun, as > well as provide pre-compiled binaries for Linux, > FreeBSD and Mac OS X, for the resp. 32- and 64-bit platforms. > > Presentation at EuroPython 2012 ------------------------------- > > Marc-André, CEO of eGenix, will be giving a presentation about eGenix > PyRun at EuroPython 2012 in Florence, Italy on Wednesday, > July 4th in the Room Tagliatelle. > > He will also be available during the conference to answer questions. > > ________________________________________________________________________ > DOWNLOADS > > The download archives and instructions for installing the product can be > found at: > > http://www.egenix.com/products/python/PyRun/ > > _______________________________________________________________________ > SUPPORT > > Commercial support for these packages is available from eGenix.com. > Please see > > http://www.egenix.com/services/support/ > > for details about our support offerings. > > ________________________________________________________________________ > MORE INFORMATION > > For more information about eGenix PyRun, licensing and download > instructions, please visit our web-site or write to sales(a)egenix.com. > > Enjoy, > -- > Marc-Andre Lemburg eGenix.com > > Professional Python Services directly from the Source >>>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/

3 2

Change in memmap behaviour
by Sveinung Gundersen 02 Jul '12

02 Jul '12

Hi, We are developing a large project for genome analysis (http://hyperbrowser.uio.no), where we use memmap vectors as the basic data structure for storage. The stored data are accessed in slices, and used as basis for calculations. As the stored data may be large (up to 24 GB), the memory footprint is important. We experienced a problem with 64-bit addressing for the function concatenate (using quite old numpy version 1.5.1rc), and have thus updated the version of numpy to 1.7.0.dev-651ef74, where the problem has been fixed. We have, however, experienced another problem connected to a change in memmap behaviour. This change seems to have come with the 1.6 release. Before (1.5.1rc1): >>> import platform; print platform.python_version() 2.7.0 >>> import numpy as np >>> np.version.version '1.5.1rc1' >>> a = np.memmap('testmemmap', 'int32', 'w+', shape=20) >>> a[:] = 2 >>> a[0:2] memmap([2, 2], dtype=int32) >>> a[0:2]._mmap <mmap.mmap object at 0x3c246f8> >>> a.sum() 40 >>> a.sum()._mmap Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'numpy.int64' object has no attribute '_mmap' After (1.6.2): >>> import platform; print platform.python_version() 2.7.0 >>> import numpy as np >>> np.version.version '1.6.2' >>> a = np.memmap('testmemmap', 'int32', 'w+', shape=20) >>> a[:] = 2 >>> a[0:2] memmap([2, 2], dtype=int32) >>> a[0:2]._mmap <mmap.mmap object at 0x1b82ed50> >>> a.sum() memmap(40) >>> a.sum()._mmap <mmap.mmap object at 0x1b82ed50> The problem is then that doing calculations of memmap objects, resulting in scalar results, previously returned a numpy scalar, with no reference to the memmap object. We could then just keep the result, and mark the memmap for garbage collection. Now, the memory usage of the system has increased dramatically, as we now longer have this option. So, the question is twofold: 1) What is the reason behind this change? It makes sense to keep the reference to the mmap when slicing, but to go from a scalar value to the mmap does not seem very useful. Is there a possibility to return to the old solution? 2) If not, do you have any advice how we can retain the old solution without rewriting the system. We could cast the results of all functions on the memmap, but these are scattered throughout the system and would probably cause much headache. So we would rather implement a general solution, for instance wrapping the memmap object somehow. Do you have any ideas? Connected to this is the rather puzzling fact that the 'new' memmap scalar object has an __iter__ method, but no length. Should not the __iter__ method be removed, as this signals that the object is iterable? Before (1.5.1rc1): >>> a[0:2].__iter__() <iterator object at 0x3c22b10> >>> len(a[0:2]) 2 >>> a.sum().__iter__ Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'numpy.int64' object has no attribute '__iter__' >>> len(a.sum()) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: object of type 'numpy.int64' has no len() After (1.6.2): >>> a[0:2].__iter__() <iterator object at 0x1b7befd0> >>> len(a[0:2]) 2 >>> a.sum().__iter__ <method-wrapper '__iter__' of memmap object at 0x1b7cab18> >>> len(a.sum()) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: len() of unsized object >>> [x for x in a.sum()] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: iteration over a 0-d array Regards, Sveinung Gundersen

2 1

running a development branch?
by Henry Gomersall 02 Jul '12

02 Jul '12

Forgive me for what seems to me should be an obvious question. How do people run development code without the need to build an entire source distribution each time? My current strategy is to develop in a virtualenv and then copy the changes to my numpy fork when done, but there are lots of obvious problems with that. Trying to run modules in the numpy branch gives me the error about trying to import numpy from its source tree. Cheers, Henry

2 1

Bug in pickling an ndarray?
by Daniel Hyams 01 Jul '12

01 Jul '12

I am having trouble pickling (and then unpickling) an ndarray. Upon unpickling, the "base" attribute of the ndarray is set to some very strange string ("base" was None when the ndarray was pickled, so it should remain None). I have tried on various platforms and versions of numpy, with inconclusive results: # tested: Linux (Suse 11.1), numpy 1.5.1 BUG # Linux (Suse 11,0), numpy 1.6.1 OK # Linux (Mint Debian), numpy 1.6.1 BUG # Linux (Mint Debian), numpy 1.6.2 BUG # OSX (Snow Leopard), numpy 1.5.1rc1 BUG # OSX (Snow Leopard), numpy 1.6.2 BUG # Windows 7, numpy 1.4.1 OK I have attached a script below that can be used to check for the problem; I suppose that this is a bug report, unless I'm doing something terribly wrong or my expectations for the base attribute are off. ---------------- cut here --------------------------------- # this little demo shows a problem with the base attribute of an ndarray, when # pickling. Before pickling, dset.base is None, but after pickling, it is some # strange string. import cPickle as pickle import numpy print numpy.__version__ #import pickle dset = numpy.ones((2,2)) print "BEFORE PICKLING" print dset print "base = ",dset.base print dset.flags # pickle. s = pickle.dumps(dset) # now unpickle. dset = pickle.loads(s) print "AFTER PICKLING AND THEN IMMEDIATELY UNPICKLING" print dset print "base = ",dset.base print dset.flags -- Daniel Hyams dhyams(a)gmail.com

4 5