Mailman 3 July 2009 - NumPy-Discussion

SciPy Foundation
by Joe Harrington 31 Jul '09

31 Jul '09

About sixteen months ago, I launched the SciPy Documentation Project and its Marathon. Dozens pitched in and now numpy docs are rapidly approaching a professional level. The "pink wave" ("Needs Review" status) is at 56% today! There is consensus among doc writers that much of the rest can be labeled in the "unimportant" category, so we're close to starting the review push (hold your fire, there is a web site mod to be done first). We're also nearing the end of the summer, and it's time to look ahead. The path for docs is clear, but the path for SciPy is not. I think our weakest area right now is organization of the project. There is no consensus-based plan for improvement of the whole toward a stated goal, no centralized coordination of work, and no funded work focused on many of our weaknesses, notwithstanding my doc effort and what Enthought does for code. I define success as popular adoption in preference to commercial packages. I believe in vote-with-your-feet: this goal will not be reached until all aspects of the package and its presentation to the world exceed those of our commercial competition. Scipy is now a grass roots effort, but that takes it only so far. Other projects, such as OpenOffice and Sage, don't follow this model and do produce quality products that compete with commercial offerings, at least on open-source platforms. Before we can even hope for that, we have to do the following: - Docs - Rest of numpy reference pages reviewed and proofed or marked unimportant - Scipy reference pages - User manual for the whole toolstack - Multiple commercial books - Packaging - Personal Package Archive or equivalent for every release of every OS for the full toolstack (There are tools that do this but we don't use them. NSF requires Metronome - http://nmi.cs.wisc.edu/ - for funding most development grants, so right now we're not even on NSF's radar.) - Track record of having the whole toolstack installation "just work" in a few command lines or clicks for *everyone* - Regular, scheduled releases of numpy and scipy - Coordinated releases of numpy, scipy, and stable scikits into PPA system - Public communication - A real marketing plan - Executing on that plan - Web site geared toward multiple audiences, run by experts at that kind of communication - More webinars, conference booths, training, aimed at all levels - Demos, testimonials, topical forums, all showcased - Code - A full design review for numpy 2.0 - No more inconsistencies like median(), lacking "out", degrees option for angle functions? - Trimming of financial functions, maybe others, from numpy? - Package structure review (eliminate "fromnumeric"?) - Goal that this be the last breakage for numpy API (the real 1.0) - Scipy - Is it maintainable? should it be broken up? - Clear code addition path (or decide never to add more) - Docs (see above) - Add-on packages - Both existence of and good indexing/integration/support for field-specific packages - Clearer development path for new packages - Central hosting system for packages (svn, mailing lists, web, build integration, etc.) - Simultaneous releases of stable packages along with numpy/scipy I posted a basic improvement plan some years back. The core ideas have not changed; it is linked from the bottom of http://scipy.org/Developer_Zone. I chose our major weakness to begin with and started the doc project, using some money I could justify spending simply for the utility of docs for my own research. I funded the work of two doc coordinators, one each this summer and last. Looking at http://docs.scipy.org/numpy/stats/, you can see that when a doc coordinator was being paid (summers), work got done. When not, then not. Without publicly announcing what these guys made, I'll be the first to admit that it wasn't a lot. Yet, those small sums bought a huge contribution to numpy through the work of several dozen volunteers and the major contributions of a few. My conclusion is that active and constant coordination is central to motivating volunteer work, and that without a salary we cannot depend on coordination remaining active. On the other hand, I have heard Enthought's leaders bemoan the high cost of devoting employee time to this project, and the low returns available from selling support to universities and non-profit research institutes. Their leadership has moved us forward, particularly in the area of code, but has not provided the momentum necessary to carry us forward on all fronts. It is time for the public and education sectors to kick in some resources and organizational leadership. We are, after all, benefitting immensely. Since the cost of employee time is not so high for us in the public and education sectors, I propose to continue hiring people like Stefan and David as UCF employees or contractors, and to expand to hiring others in areas like packaging and marketing, provided that funding for those hires can be found. However, my grant situation is no longer as rich as it has been the past two years, and the needs going forward are greater than in the past if we're now to tackle all the points above. So, I will not be hiring another doc guru from my research grants next year. I am confident that others are willing to pitch in financially, but few will pitch in a full FTE, and we need several. We can (and will) set up a donations site, but donation sites tend to receive pizza money unless a sugar daddy comes along. Those benefitting most from the software, notably education, non-profit research, and government institutions, are *forbidden* from making donations by the terms of their grants. NSF doesn't give you money so you can give it away. We need to provide services they can buy on subcontract and a means for handling payments from them. Selling support does not solve the problem, as that requires spending most of the income on servicing that particular client. Rather, we need to sell a chunk of documentation or the packaging of a particular release, and then provide the product not just to that client but to everyone. We can also propose directly for federal and corporate grant funds. I have spoken with several NASA and NSF program managers and with Google's Federal Accounts Representative, and the possibilities for funding are good. But, I am not going to do this alone. We need a strong proposal team to be credible. So, I am seeking a group that is willing to work with me to put up the infrastructure of a funded project, to write grant proposals, and to coordinate a financial effort. Members of this group must have a track record of funded grants, business success, foundation support, etc. We might call it the SciPy Foundation. It could be based at UCF, which has a low overhead rate and has infrastructure (like an HR staff), or it might be independent if we can find a good director willing to devote significant time for relatively low pay compared to what they can likely make elsewhere. I would envision hiring permanent coordinators for docs, packaging, and marketing communications. Enthought appears to have code covered by virtue of having hired Travis, Robert, etc.; how to integrate that with this effort is an open question but not a difficult one, I think, as code is our strongest asset at this point. I invite discussion of this approach and the task list above on the scipy-dev(a)scipy.org mailing list. If you are seeing this post elsewhere, please reply only on scipy-dev(a)scipy.org. If you are eligible to lead funding proposals and are interested in participating in grant writing and management activities related to work in our weak areas, please contact me directly. Thanks, --jh-- Prof. Joseph Harrington Planetary Sciences Group Department of Physics MAP 414 4000 Central Florida Blvd. University of Central Florida Orlando, FL 32816-2385 jh(a)physics.ucf.edu planets.ucf.edu

1 0

Python 3.n and Scipy Numpy
by BBands 31 Jul '09

31 Jul '09

Could someone point me toward some information on Scipy/Numpy and Python 3.1? I'd like to upgrade, but can't seem to find the path. jab

3 4

Memory layout of record arrays
by Nicolas Rougier 31 Jul '09

31 Jul '09

Hello, I've been using record arrays to create arrays with different types and since I'm doing a lot of computation on each of the different fields, the default memory layout does not serve my computations. Ideally, I would like to have record arrays where each field is a contiguous block of memory. I searched the dtype documentation but did not find anything about it. In the meantime, I wrote a class that emulates this behavior, but it does not inherit from nd.array and I need inheritance. To sum up, I would like to have: >>> Z = np.zeros(3, dtype=[('x',np.float32), ('y',np.int)]) >>> print Z['x'].flags[['C_CONTIGUOUS'] False <= should be True >>> print Z['y'].flags[['C_CONTIGUOUS'] False <= should be True >>> Z.shape == Z['x'].shape == Z['y'].shape True Is there any obvious solution that I missed ? Nicolas

2 3

Timeline for 1.4.0 and installer for Windows 64bit ?
by Klaus Noekel 31 Jul '09

31 Jul '09

Dear folks, just over a month ago there was a thread about plans for numpy, and IIRC somebody had volunteered to try and put together a working AMD64 version with an installer. Since then I have not heard about the issue again - but I may have missed part of the discussion. Have the plans firmed up by now? Is there a tentative date for a beta or RC? In particular: how much hope is there for a reasonably usable AMD64 numpy under Windows? Thanks for any update! Klaus Nökel

4 4

EPD Webinar August 7: Speed up Python (& wait list for non-subscribers!)
by Amenity Applewhite 30 Jul '09

30 Jul '09

Friday, August 7th 1pm CDT How do I...speed up Python? Exploration of Weave and Cython We wanted to let you know that next week we'll host another installment of our popular EPD webinar series. Although only EPD Basic or above subscribers are guaranteed seats at EPD webinars, we invite non-subscribers to add their names to the wating list for each event. If there are available seats, you will be notified by next Thursday and given access to the webinar. Links to the waiting lists and upcoming topics are posted here: http://enthought.com/training/webinars.php These events feature detailed demonstrations of powerful Python techniques that Enthought developers use to enhance our applications or development process. Participants are often invited to participate in the demonstration, and are welcome to join the interactive VOIP discussion later in the session. This is a great opportunity to learn new methods and interact with our expert developers. If you have topics you'd like to see addressed during the webinar, feel free to let us know at media(a)enthought.com. How do I...speed up Python: Exploration of Weave and Cython Python excels as a language for rapid development and interactive technical computing. Sometimes, however, the full speed of the machine is needed. Fortunately, Python is also an excellent "glue" language that allows simple integration to compiled languages. In this webinar, we will present a few of the methods useful for speeding up sections of code. In particular we will focus on weave and Cython. Weave is a sub-package of SciPy (scipy.weave) and allows seamless integration with C/C++. Sections of code that need to be sped up can be re-coded in C and seamlessly integrated into the remaining Python program using weave. Cython is a newer Python-like language which is compiled to an extension module that can be dynamically loaded into Python just like any other Python module. Cython also supports writing new built-in Python types and tries to compile loops to C-speeds when variable types are indicated. Both weave and Cython integrate well with NumPy which makes them ideal whenever you need to iterate and calculate with large amounts of data and can't find a built-in (vectorized) approach. Once again, to add your name to the wait-list: http://enthought.com/training/webinars.php We hope to see you there! Thanks, Enthought Media Quick Links... Enthought.com Enthought Python Distribution (EPD) Enthought Webinars @Facebook @Twitter -- Amenity Applewhite Enthought, Inc. Scientific Computing Solutions www.enthought.com

1 0

Internal Math Library of Numpy
by Nanime Puloski 30 Jul '09

30 Jul '09

Does Numpy use Python's standard math library when calculating elementary functions such as exp(x) and acos(x)? Also, what is the internal library of Numpy and Python's standard math library? Are they platform independent?

2 1

question about a data structure
by Ernest Adrogué 30 Jul '09

30 Jul '09

Hello, Suppose I want to store some data (a non-negative integer) related to an "event" involving two entities. It occurs to me that the way to do it is using a 2-d array, like a table: For example: a b c a 1 b 2 0 c 5 in the event 'b-a' the data is 2, and in the event 'c-a' is 5, etc. Now, I need to be able to differentiate between 0 and 'no data'. I know I could use a number like -1, but this would affect arithmetic operations on the array, and also I need that indexing one such location with no data returns an empty array. Is it possible to do this with the standard array class? Thanks. Ernest

3 3

Reductions and binary ops on recarrays...
by Fernando Perez 30 Jul '09

30 Jul '09

Howdy, we recently had a discussion about being able to do some common things like reductions and binary operations on recarrays, and there didn't seem to be much consensus on it being needed in the core of numpy. Since we do actually need this quite pressingly for everyday tasks, we wrote a very simple version of this today, I'm attaching it here in case it proves useful to others. Basically it lets you do reductions and binary operations on record arrays whose dtype is a simple composite of native ones (example at the end). For our needs it's quite useful, so it may also be to others. Cheers, f Example: >>> import recarrutil as ru >>> dt = np.dtype(dict(names=['x','y'],formats=[float,float])) >>> x = np.arange(6,dtype=float).reshape(2,3) >>> y = np.arange(10,16,dtype=float).reshape(2,3) >>> z = np.empty( (2,3), dt).view(np.recarray) >>> z.x = x >>> z.y = y >>> z rec.array([[(0.0, 10.0), (1.0, 11.0), (2.0, 12.0)], [(3.0, 13.0), (4.0, 14.0), (5.0, 15.0)]], dtype=[('x', '<f8'), ('y', '<f8')]) >>> ru.mean(z) rec.array((2.5, 12.5), dtype=[('x', '<f8'), ('y', '<f8')]) >>> ru.mean(z,0) rec.array([(1.5, 11.5), (2.5, 12.5), (3.5, 13.5)], dtype=[('x', '<f8'), ('y', '<f8')]) >>> ru.mean(z,1) rec.array([(1.0, 11.0), (4.0, 14.0)], dtype=[('x', '<f8'), ('y', '<f8')]) >>> ru.add(z,z) rec.array([[(0.0, 20.0), (2.0, 22.0), (4.0, 24.0)], [(6.0, 26.0), (8.0, 28.0), (10.0, 30.0)]], dtype=[('x', '<f8'), ('y', '<f8')]) >>> ru.subtract(z,z) rec.array([[(0.0, 0.0), (0.0, 0.0), (0.0, 0.0)], [(0.0, 0.0), (0.0, 0.0), (0.0, 0.0)]], dtype=[('x', '<f8'), ('y', '<f8')])

3 5

Agenda items for steering committee meeting, 2009/08
by Stéfan van der Walt 30 Jul '09

30 Jul '09

Hi all, We are soliciting agenda items for the SciPy Steering Committee meeting taking place during the SciPy09 conference. Please view the current list at http://www.scipy.org/SteeringCommittee_08_2009 and post new items to this thread. Kind regards Stéfan

1 0

how to configure f2py on windows XP?
by ian 30 Jul '09

30 Jul '09

Hi,everyone！ I installed MSVS2005 + Intel visual fortran 9.1 + Python 2.5 + numpy-1.3.0-win32-superpack-python2.5.exe on my computer. The numpy works well: Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32 Type "copyright", "credits" or "license()" for more information. **************************************************************** Personal firewall software may warn about the connection IDLE makes to its subprocess using this computer's internal loopback interface. This connection is not visible on any external interface and no data is sent to or received from the Internet. **************************************************************** IDLE 1.2 >>> import numpy >>> but when I use : python c:\python25\scripts\f2py.py --fcompiler=intel -c hello.f -m hello , the result is: running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src building extension "hello" sources f2py options: [] f2py:> c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5\hellomodule.c creating c:\docume~1\admini~1\locals~1\temp\tmpwktrfc creating c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5 Reading fortran codes... Reading file 'hello.f' (format:fix,strict) Post-processing... Block: hello Block: foo Post-processing (stage 2)... Building modules... Building module "hello"... Constructing wrapper function "foo"... foo(a) Wrote C/API module "hello" to file "c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5/hellomodule.c" adding 'c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5\fortranobject.c' to sources. adding 'c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5' to include_dirs. copying C:\Python25\lib\site-packages\numpy\f2py\src\fortranobject.c -> c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5 copying C:\Python25\lib\site-packages\numpy\f2py\src\fortranobject.h -> c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5 running build_ext No module named msvccompiler in numpy.distutils; trying from distutils error: python was built with Visula Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py. who could give me some suggestions on how to configure f2py? Thanks in advance. ian liu.ian0571(a)gmail.com 2009-07-30

3 3