Mailman 3 February 2012 - NumPy-Discussion

Re: [Numpy-discussion] ndarray and lazy evaluation
by James Bergstra Feb. 20, 2012

Feb. 20, 2012

On Mon, Feb 20, 2012 at 1:01 PM, James Bergstra <james.bergstra(a)gmail.com>wrote: > On Mon, Feb 20, 2012 at 12:28 PM, Francesc Alted <francesc(a)continuum.io>wrote: > >> On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote: >> > You need at least a slightly different Python API to get anywhere, so >> > numexpr/Theano is the right place to work on an implementation of this >> > idea. Of course it would be nice if numexpr/Theano offered something as >> > convenient as >> > >> > with lazy: >> > arr = A + B + C # with all of these NumPy arrays >> > # compute upon exiting… >> >> Hmm, that would be cute indeed. Do you have an idea on how the code in >> the with context could be passed to the Python AST compiler (à la >> numexpr.evaluate("A + B + C"))? >> >> > The biggest problem with the numexpr approach (e.g. evaluate("A + B + C")) > whether the programmer has to type the quotes or not, is that the > sub-program has to be completely expressed in the sub-language. > > If I write > > >>> def f(x): return x[:3] > >>> numexpr.evaluate("A + B + f(C)") > > I would like that to be fast, but it's not obvious at all how that would > work. We would be asking numexpr to introspect arbitrary callable python > objects, and recompile arbitrary Python code, effectively setting up the > expectation in the user's mind that numexpr is re-implementing an entire > compiler. That can be fast obviously, but it seems to me to represent > significant departure from numpy's focus, which I always thought was the > data-container rather than the expression evaluation (though maybe this > firestorm of discussion is aimed at changing this?) > > Theano went with another option which was to replace the A, B, and C > variables with objects that have a modified __add__. Theano's back-end can > be slow at times and the codebase can feel like a heavy dependency, but my > feeling is still that this is a great approach to getting really fast > implementations of compound expressions. > > The context syntax you suggest using is a little ambiguous in that the > indented block of a with statement block includes *statements* whereas what > you mean to build in the indented block is a *single expression* graph. > You could maybe get the right effect with something like > > A, B, C = np.random.rand(3, 5) > > expr = np.compound_expression() > with np.expression_builder(expr) as foo: > arr = A + B + C > brr = A + B * C > foo.return((arr, brr)) > > # compute arr and brr as quickly as possible > a, b = expr.run() > > # modify one of the arrays that the expression was compiled to use > A[:] += 1 > > # re-run the compiled expression on the new value > a, b = expr.run() > > - JB > I should add that the biggest benefit of expressing things as compound expressions in this way is not in saving temporaries (though that is nice) it's being able to express enough computation work at a time that it offsets the time required to ship the arguments off to a GPU for evaluation! This has been a *huge* win reaped by the Theano approach, it works really well. The abstraction boundary offered by this sort of expression graph has been really effective. This speaks even more to the importance of distinguishing between the data container (e.g. numpy, Theano's internal ones, PyOpenCL's one, PyCUDA's one) and the expression compilation and evaluation infrastructures (e.g. Theano, numexpr, cython). The goal should be as much as possible to separate these two so that programs can be expressed in a natural way, and then evaluated using containers that are suited to the program. - JB -- James Bergstra, Ph.D. Research Scientist Rowland Institute, Harvard University

2 2

Re: [Numpy-discussion] ndarray and lazy evaluation
by James Bergstra Feb. 20, 2012

Feb. 20, 2012

On Mon, Feb 20, 2012 at 12:28 PM, Francesc Alted <francesc(a)continuum.io>wrote: > On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote: > > You need at least a slightly different Python API to get anywhere, so > > numexpr/Theano is the right place to work on an implementation of this > > idea. Of course it would be nice if numexpr/Theano offered something as > > convenient as > > > > with lazy: > > arr = A + B + C # with all of these NumPy arrays > > # compute upon exiting… > > Hmm, that would be cute indeed. Do you have an idea on how the code in > the with context could be passed to the Python AST compiler (à la > numexpr.evaluate("A + B + C"))? > > The biggest problem with the numexpr approach (e.g. evaluate("A + B + C")) whether the programmer has to type the quotes or not, is that the sub-program has to be completely expressed in the sub-language. If I write >>> def f(x): return x[:3] >>> numexpr.evaluate("A + B + f(C)") I would like that to be fast, but it's not obvious at all how that would work. We would be asking numexpr to introspect arbitrary callable python objects, and recompile arbitrary Python code, effectively setting up the expectation in the user's mind that numexpr is re-implementing an entire compiler. That can be fast obviously, but it seems to me to represent significant departure from numpy's focus, which I always thought was the data-container rather than the expression evaluation (though maybe this firestorm of discussion is aimed at changing this?) Theano went with another option which was to replace the A, B, and C variables with objects that have a modified __add__. Theano's back-end can be slow at times and the codebase can feel like a heavy dependency, but my feeling is still that this is a great approach to getting really fast implementations of compound expressions. The context syntax you suggest using is a little ambiguous in that the indented block of a with statement block includes *statements* whereas what you mean to build in the indented block is a *single expression* graph. You could maybe get the right effect with something like A, B, C = np.random.rand(3, 5) expr = np.compound_expression() with np.expression_builder(expr) as foo: arr = A + B + C brr = A + B * C foo.return((arr, brr)) # compute arr and brr as quickly as possible a, b = expr.run() # modify one of the arrays that the expression was compiled to use A[:] += 1 # re-run the compiled expression on the new value a, b = expr.run() - JB -- James Bergstra, Ph.D. Research Scientist Rowland Institute, Harvard University

1 0

Forbidden charcter in the "names" argument of genfromtxt?
by Adam Hughes Feb. 20, 2012

Feb. 20, 2012

Hey everyone, I have timeseries data in which the column label is simply a filename from which the original data was taken. Here's some sample data: name1.txt name2.txt name3.txt 32 34 953 32 03 402 I've noticed that the standard genfromtxt() method works great; however, the names aren't written correctly. That is, if I use the command: print data['name1.txt'] Nothing happens. However, when I remove the file extension, Eg: name1 name2 name3 32 34 953 32 03 402 Then print data['name1'] return (32, 32) as expected. It seems that the period in the name isn't compatible with the genfromtxt() names attribute. Is there a workaround, or do I need to restructure my program to get the extension removed? I'd rather not do this if possible for reasons that aren't important for the discussion at hand. Thanks.

3 4

ndarray and lazy evaluation (was: Proposed Rodmap Overview)
by Dag Sverre Seljebotn Feb. 20, 2012

Feb. 20, 2012

On 02/20/2012 09:24 AM, Olivier Delalleau wrote: > Hi Dag, > > Would you mind elaborating a bit on that example you mentioned at the > end of your email? I don't quite understand what behavior you would like > to achieve Sure, see below. I think we should continue discussion on numpy-discuss. I wrote: > You need at least a slightly different Python API to get anywhere, so > numexpr/Theano is the right place to work on an implementation of this > idea. Of course it would be nice if numexpr/Theano offered something as > convenient as > > with lazy: > arr = A + B + C # with all of these NumPy arrays > # compute upon exiting... More information: The disadvantage today of using Theano (or numexpr) is that they require using a different API, so that one has to learn and use Theano "from the ground up", rather than just slap it on in an optimization phase. The alternative would require extensive changes to NumPy, so I guess Theano authors or Francesc would need to push for this. The alternative would be (with A, B, C ndarray instances): with theano.lazy: arr = A + B + C On __enter__, the context manager would hook into NumPy to override it's arithmetic operators. Then it would build a Theano symbolic tree instead of performing computations right away. In addition to providing support for overriding arithmetic operators, slicing etc., it would be necesarry for "arr" to be an ndarray instance which is "not yet computed" (data-pointer set to NULL, and store a compute-me callback and some context information). Finally, the __exit__ would trigger computation. For other operations which need the data pointer (e.g., single element lookup) one could either raise an exception or trigger computation. This is just a rough sketch. It is not difficult "in principle", but of course there's really a massive amount of work involved to work support for this into the NumPy APIs. Probably, we're talking a NumPy 3.0 thing, after the current round of refactorings have settled... Please: Before discussing this further one should figure out if there's manpower available for it; no sense in hashing out a castle in the sky in details. Also it would be better to talk in person about this if possible (I'm in Berkeley now and will attend PyData and PyCon). Dag

2 2

Re: [Numpy-discussion] ndarray and lazy evaluation (was: Proposed Rodmap Overview)
by Olivier Delalleau Feb. 20, 2012

Feb. 20, 2012

Never mind. The link Francesc posted answered my question :) -=- Olivier Le 20 février 2012 12:54, Olivier Delalleau <delallea(a)iro.umontreal.ca> a écrit : > Le 20 février 2012 12:46, Dag Sverre Seljebotn <d.s.seljebotn(a)astro.uio.no > > a écrit : > > On 02/20/2012 09:24 AM, Olivier Delalleau wrote: >> > Hi Dag, >> > >> > Would you mind elaborating a bit on that example you mentioned at the >> > end of your email? I don't quite understand what behavior you would like >> > to achieve >> >> Sure, see below. I think we should continue discussion on numpy-discuss. >> >> I wrote: >> >> > You need at least a slightly different Python API to get anywhere, so >> > numexpr/Theano is the right place to work on an implementation of this >> > idea. Of course it would be nice if numexpr/Theano offered something as >> > convenient as >> > >> > with lazy: >> > arr = A + B + C # with all of these NumPy arrays >> > # compute upon exiting... >> >> More information: >> >> The disadvantage today of using Theano (or numexpr) is that they require >> using a different API, so that one has to learn and use Theano "from the >> ground up", rather than just slap it on in an optimization phase. >> >> The alternative would require extensive changes to NumPy, so I guess >> Theano authors or Francesc would need to push for this. >> >> The alternative would be (with A, B, C ndarray instances): >> >> with theano.lazy: >> arr = A + B + C >> >> On __enter__, the context manager would hook into NumPy to override it's >> arithmetic operators. Then it would build a Theano symbolic tree instead >> of performing computations right away. >> >> In addition to providing support for overriding arithmetic operators, >> slicing etc., it would be necesarry for "arr" to be an ndarray instance >> which is "not yet computed" (data-pointer set to NULL, and store a >> compute-me callback and some context information). >> >> Finally, the __exit__ would trigger computation. For other operations >> which need the data pointer (e.g., single element lookup) one could >> either raise an exception or trigger computation. >> >> This is just a rough sketch. It is not difficult "in principle", but of >> course there's really a massive amount of work involved to work support >> for this into the NumPy APIs. >> >> Probably, we're talking a NumPy 3.0 thing, after the current round of >> refactorings have settled... >> >> Please: Before discussing this further one should figure out if there's >> manpower available for it; no sense in hashing out a castle in the sky >> in details. Also it would be better to talk in person about this if >> possible (I'm in Berkeley now and will attend PyData and PyCon). >> >> Dag >> > > Thanks for the additional details. > > I feel like this must be a stupid question, but I have to ask: what is the > point of being lazy here, since the computation is performed on exit anyway? > > -=- Olivier >

1 0

NumPy in PyPy ?
by xavier.gnata＠gmail.com Feb. 19, 2012

Feb. 19, 2012

Hi, I'm trying to understand what's going on with : http://morepypy.blogspot.com/2012/01/numpypy-status-update.html What's your opinion on such a numpy rewrite?? Thanks, Xavier

2 2

Re: [Numpy-discussion] How a transition to C++ could work
by Ben Walsh Feb. 19, 2012

Feb. 19, 2012

> Date: Sun, 19 Feb 2012 01:18:20 -0600 > From: Mark Wiebe <mwwiebe(a)gmail.com> > Subject: [Numpy-discussion] How a transition to C++ could work > To: Discussion of Numerical Python <NumPy-Discussion(a)scipy.org> > Message-ID: > <CAMRnEmpVTmt=KduRpZKtgUi516oQtqD4vAzm746HmpqgpFXNqQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > The suggestion of transitioning the NumPy core code from C to C++ has > sparked a vigorous debate, and I thought I'd start a new thread to give my > perspective on some of the issues raised, and describe how such a > transition could occur. > > First, I'd like to reiterate the gcc rationale for their choice to switch: > http://gcc.gnu.org/wiki/gcc-in-cxx#Rationale > > In particular, these points deserve emphasis: > > - The C subset of C++ is just as efficient as C. > - C++ supports cleaner code in several significant cases. > - C++ makes it easier to write cleaner interfaces by making it harder to > break interface boundaries. > - C++ never requires uglier code. > I think they're trying to solve a different problem. I thought the problem that numpy was trying to solve is "make inner loops of numerical algorithms very fast". C is great for this because you can write C code and picture precisely what assembly code will be generated. C++ removes some of this advantage -- now there is extra code generated by the compiler to handle constructors, destructors, operators etc which can make a material difference to fast inner loops. So you end up just writing "C-style" anyway. On the other hand, if your problem really is "write lots of OO code with virtual methods and have it turned into machine code" (probably like the GCC guys) then maybe C++ is the way to go. Some more opinions on C++: http://gigamonkeys.wordpress.com/2009/10/16/coders-c-plus-plus/ Sorry if this all seems a bit negative about C++. It's just been my experience that C++ adds complexity while C keeps things nice and simple. Looking forward to seeing some more concrete examples. Cheers Ben

8 11

How a transition to C++ could work
by Mark Wiebe Feb. 19, 2012

Feb. 19, 2012

The suggestion of transitioning the NumPy core code from C to C++ has sparked a vigorous debate, and I thought I'd start a new thread to give my perspective on some of the issues raised, and describe how such a transition could occur. First, I'd like to reiterate the gcc rationale for their choice to switch: http://gcc.gnu.org/wiki/gcc-in-cxx#Rationale In particular, these points deserve emphasis: - The C subset of C++ is just as efficient as C. - C++ supports cleaner code in several significant cases. - C++ makes it easier to write cleaner interfaces by making it harder to break interface boundaries. - C++ never requires uglier code. Some people have pointed out that the Python templating preprocessor used in NumPy is suggestive of C++ templates. A nice advantage of using C++ templates instead of this preprocessor is that third party tools to improve software quality, like static analysis tools, will be able to run directly on the NumPy source code. Additionally, IDEs like XCode and Visual C++ will be able to provide the full suite of tab-completion/intellisense features that programmers working in those environments are accustomed to. There are concerns about ABI/API interoperability and interactions with C++ exceptions. I've dealt with these types of issues on enough platforms to know that while they're important, they're a lot easier to handle than the issues with Fortran, BLAS, and LAPACK in SciPy. My experience has been that providing a C API from a C++ library is no harder than providing a C API from a C library. It's worth comparing the possibility of C++ versus the possibility of other languages, and the ones that have been suggested for consideration are D, Cython, Rust, Fortran 2003, Go, RPython, C# and Java. The target language has to interact naturally with the CPython API. It needs to provide direct access to all the various sizes of signed int, unsigned int, and float. It needs to have mature compiler support wherever we want to deploy NumPy. Taken together, these requirements eliminate a majority of these possibilities. From these criteria, the only languages which seem to have a clear possibility for the implementation of Numpy are C, C++, and D. For D, I suspect the tooling is not mature enough, but I'm not 100% certain of that. The biggest question for any of these possibilities is how do you get the code from its current state to a state which fully utilizes the target language. C++, being nearly a superset of C, offers a strategy to gradually absorb C++ features. Any of the other language choices requires a rewrite, which would be quite disruptive. Because of all these reasons taken together, I believe the only realistic language to use, other than sticking with C, is C++. Finally, here's what I think is the best strategy for transitioning to C++. First, let's consider what we do if 1.7 becomes an LTS release. 1) Immediately after branching for 1.7, we minimally patch all the .c files so that they can build with a C++ compiler and with a C compiler at the same time. Then we rename all .c -> .cpp, and update the build systems for C++. 2) During the 1.8 development cycle, we heavily restrict C++ feature usage. But, where a feature implementation would be arguably easier and less error-prone with C++, we allow it. This is a period for learning about C++ and how it can benefit NumPy. 3) After the 1.8 release, the community will have developed more experience with C++, and will be in a better position to discuss a way forward. If, for some reason, a 1.7 LTS is unacceptable, it might be a good idea to restrict the 1.8 release to the subset of both C and C++. I would much prefer using the 1.8 development cycle to dip our toes into the C++ world to get some of the low-hanging benefits without doing anything disruptive. A really important point to emphasize is that C++ allows for a strategy where we gradually evolve the codebase to better incorporate its language features. This is what I'm advocating. No massive rewrite, no disruptive changes. Gradual code evolution, with ABI and API compatibility comparable to what we've delivered in 1.6 and the upcoming 1.7 releases. Thanks, Mark

6 14

Flag this message Re: [Matplotlib-users] Discussion with Guido van Rossum and (hopefully) core python-dev on scientific Python and Python3
by Michiel de Hoon Feb. 19, 2012

Feb. 19, 2012

> While a number of scientific Python packages are already available for > Python 3 (either in released form or in their master git branches), > it's fair to say that there hasn't been a major transition of the > scientific community to Python3. Since there is no more development > being done on the Python2 series, eventually we will all want to find > ways to make this transition, and we think that this is an excellent > time to engage the core python development team and consider ideas > that would make Python3 generally a more appealing language for > scientific work. For scientific visualization, in particular for matplotlib, it would be really good if Python3 had event loop support. In Python2 extension modules such as Tkinter, PyGTK, PyQT each implement their own event loop, leading to incompatibilities. For reliable event handling, Python should be in charge of the event loop, rather than the extension modules. For comparison, see the event loop support provided by Tcl / Tk. -Michiel.

1 0

change the mask state of one element in a masked array
by Chao YUE Feb. 19, 2012

Feb. 19, 2012

Dear all, I built a new empty masked array: In [91]: a=np.ma.empty((2,5)) In [92]: a Out[92]: masked_array(data = [[ 1.20569155e-312 3.34730819e-316 1.13580079e-316 1.11459945e-316 9.69610549e-317] [ 6.94900258e-310 8.48292532e-317 6.94900258e-310 9.76397825e-317 6.94900258e-310]], mask = False, fill_value = 1e+20) as you see, the mask for all the elements are false. so how can I set for some elements to masked elements (mask state as true)? let's say, I want a[0,0] to be masked. thanks & cheers, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

3 3