From fperez at colorado.edu Tue Sep 2 17:28:00 2003 From: fperez at colorado.edu (Fernando Perez) Date: Tue, 02 Sep 2003 15:28:00 -0600 Subject: [SciPy-dev] Low-level code integration discussion at scipy'03? Message-ID: <3F550B60.5060108@colorado.edu> Hi everyone, as I mentioned before, I think it would be great to have a bit of an open discussion on methods and techniques for plugging low-level code into python at Scipy'03. I don't mean to teach people how to write an extension module, but rather what the best/easiest approaches are for writing time-critical code which manipulates Numpy/numarray arrays. A brief summary of the alternatives: * Scipy's weave relies on blitz, which is extremely nice and maps surprisingly well to python arrays. Blitz arrays have slices, ranges, overloaded arithmetic operators and math functions, etc. In fact, writing blitz C++ feels almost like writing python code, with the caveats of handling compile-time type information. But blitz development seems to be rather slow these days, and the issue of access to Blas/lapack from blitz isn't fully settled (i.e., it's not out-of-the box easy). * The ublas library (http://www.boost.org/libs/numeric/ublas/doc/overview.htm). It has direct blas support, but it seems to focus strictly on vectors and matrices, and as far as I see it doesn't support higher rank objects (I could be wrong, though: I've been using blitz for 'real' and only looked at the ublas webpage). * Writing low-level code in fortran and using Pearu's f2py. I love f2py, but I'd prefer my low-level code to have the ability to also use more information from the python side of things if needed. I think f2py is a great tool for wrapping existing fortran codes, but not so much for extending a python program with a few low-level routines. And there is always the potential need for transposition operations when going to Fortran because of the row/column memory ordering discrepancy between C and Fortran. * Complete roll-your-own solutions using the Numarray C api. This is doable, but after using blitz arrays, it feels really low-level. Call me lazy, but after using python I've become fully enamored of writing code which reads as cleanly as possible. What I'd like to know is if there is any interest in organizing a bit of a discussion on this. If an hour can be squeezed in for this in the schedule, great (and I volunteer to moderate if nobody else wants to). If not, perhaps some of people involved with these topics (Eric, Travis, Konrad, Todd and Perry at least, plus whoever else is interested) might be willing to commit to a lunch/dinner to be spent talking about this. It would be nice to know something in advance, because one of blitz's current developers (Julian Cummings) is at Caltech. Perhaps we could ask him to come in and answer some questions about Blitz's future. Blitz feels like the lowest-impedance fit to python's semantics, but I'd like to know more about its long-term feasibility before committing to use it too much. If we set any kind of schedule for this discussion, I can try to contact Julian. Regards, Fernando. From patmiller at llnl.gov Tue Sep 2 17:55:02 2003 From: patmiller at llnl.gov (Pat Miller) Date: Tue, 02 Sep 2003 14:55:02 -0700 Subject: [SciPy-dev] Low-level code integration discussion at scipy'03? In-Reply-To: <3F550B60.5060108@colorado.edu> References: <3F550B60.5060108@colorado.edu> Message-ID: <3F5511B6.3050900@llnl.gov> I've been working on inline compilation that grabs values out of Python and slams them into C. This is basically a rework of ideas that Eric had for weave. Part of the win is noticing variables that exist in Python and importing them. So, from compilation import inline def f(x,y): z = 0.0 inline(r''' printf("x: %g and y: %g\n",x,y); z = x+y; ''') return z print f(11.1,22.2) # Will print 33.3 This is pretty clean since in notices that X, Y, and Z are double and builds the extension appropriately. * * * * * Now we jump into vaporware * * * * * Suppose we had def mm(A,x,Z): "A is a matrix, x is a column vector, Z is just some double" inline.fortran(r'''' CALL MYMATMULT(A,DIM1(A),DIM2(A),x,SIZE(x)) Z = MAX(x) ''', library='mystuff') where libmystuff.so contains all the nifty stuff you want to interface too (not python libs, just the FORTRAN). The code generator (once, ala weave) builds an extension that gets builds the code you want with a shell that is called from python. SUBROUTINE GENERATED(A,DIM1_A,DIM2_A,x,SIZE_x,Z) CALL MYMATMULT(...) END SUBROUTINE and in C PyObject* somegenerated_name(PyObject* self, PyObject* args) { double* A = << get base pointer >> int DIM1_A = << get size info >> double Z = .... _GENERATED(A,DIM1,..., & Z); << Code to flush Z back into python >> return Py_None; } -- Patrick Miller | (925) 423-0309 | http://www.llnl.gov/CASC/people/pmiller When I was a boy I was told that anybody could become President. Now I'm beginning to believe it. -- Clarence Darrow, lawyer and author (1857-1938) From dmorrill at enthought.com Tue Sep 2 20:01:21 2003 From: dmorrill at enthought.com (David C. Morrill) Date: Tue, 2 Sep 2003 19:01:21 -0500 Subject: [SciPy-dev] Announce: Python 2.3 for Windows (Enthought Edition) now available Message-ID: <002101c371ae$826dd4b0$8201a8c0@dellbert> Just a quick heads up that Python 2.3 for Windows (Enthought Edition) is now available at: http://www.enthought.com This is in addition to the Python 2.2.3 for Windows (Enthought Edition) that was previously announced. Besides the obvious feature of being based on Python 2.3, this new version also includes updates to several other packages, such as wxPython So if you've been putting off giving Python 2.3 a spin, now might be a good time to upgrade. And if you do decide to give it a try, we'd also appreciate hearing from you either on the mailing lists or on the bug tracker concerning what you think about the new Python distribution. Enjoy! The Python guys at Enthought, Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pajer at iname.com Tue Sep 2 23:07:33 2003 From: pajer at iname.com (Gary Pajer) Date: Tue, 2 Sep 2003 23:07:33 -0400 Subject: [SciPy-dev] Re: [SciPy-user] Announce: Python 2.3 for Windows (Enthought Edition) now available References: <002101c371ae$826dd4b0$8201a8c0@dellbert> Message-ID: <000801c371c8$8541b660$01fd5644@playroom> Just gave it a whirl. Some problems. The first thing I did was, from a DOS window python session: from scipy import plt plt.plot((1,2,3)) pythonw.exe crash, asking me if I want to send a note to Microsoft. Then tried PyCrust shell. from scipy import plt plt.plot((1,2,3)) This time the plot appeared, along with an error dialog: C:\Python23\lib\site-packages\wxPython\gdi.py:600: DeprecationWarning: integer argument expected, got float val = gdic.wxDC_DrawEllipse(self, *_args, **_kwargs) This looks like it might be a wxPython warning. Nonetheless... Finally from chaco import wxplt wxplt.plot((1,2,3)) Plots the data, but again an error dialog: C:\Python23\lib\site-packages\kiva\agg\agg.py:611: DeprecationWarning: integer argument expected, got float def draw_glyphs(*args): return apply(_agg._GraphicsContextArray_draw_glyphs,args) Again, it looks like a warning, but ... so far I'm 0 for 3. bugs? or am I buggy? -gary ----- Original Message ----- From: "David C. Morrill" To: ; ; Sent: Tuesday, September 02, 2003 8:01 PM Subject: [SciPy-user] Announce: Python 2.3 for Windows (Enthought Edition) now available Just a quick heads up that Python 2.3 for Windows (Enthought Edition) is now available at: http://www.enthought.com This is in addition to the Python 2.2.3 for Windows (Enthought Edition) that was previously announced. Besides the obvious feature of being based on Python 2.3, this new version also includes updates to several other packages, such as wxPython So if you've been putting off giving Python 2.3 a spin, now might be a good time to upgrade. And if you do decide to give it a try, we'd also appreciate hearing from you either on the mailing lists or on the bug tracker concerning what you think about the new Python distribution. Enjoy! The Python guys at Enthought, Inc. From prabhu at aero.iitm.ernet.in Wed Sep 3 02:25:16 2003 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Wed, 3 Sep 2003 11:55:16 +0530 Subject: [SciPy-dev] Re: [SciPy-user] Low-level code integration discussion at scipy'03? In-Reply-To: <3F550B60.5060108@colorado.edu> References: <3F550B60.5060108@colorado.edu> Message-ID: <16213.35148.87968.555025@monster.linux.in> Hi, I have a few opinions on the subject. >>>>> "FP" == Fernando Perez writes: [snip] FP> A brief summary of the alternatives: FP> * Scipy's weave relies on blitz, which is extremely nice and FP> maps surprisingly FP> well to python arrays. Blitz arrays have slices, ranges, FP> overloaded arithmetic operators and math functions, etc. In FP> fact, writing blitz C++ feels almost like writing python code, FP> with the caveats of handling compile-time type information. FP> But blitz development seems to be rather slow these days, and FP> the issue of access to Blas/lapack from blitz isn't fully FP> settled (i.e., it's not out-of-the box easy). AFAIK, the other problems with blitz++ and weave are: 1. Blitz is not as fast as a hand coded inline C version. More on ugliness of code later. 2. Blitz compile times are large. 3. An extra layer of complexity for weave, one more package to keep track of -- the blitz sources are bundled with weave. 4. With all due respect, not many C++ programmers (forget Python programmers) would be able to understand/code anything with the techniques used inside Blitz. Agreed, this is transparent to the user in the present context but its still worth noting. [snip] FP> * Complete roll-your-own solutions using the Numarray C api. FP> This is doable, FP> but after using blitz arrays, it feels really low-level. Call FP> me lazy, but after using python I've become fully enamored of FP> writing code which reads as cleanly as possible. Here is my take. I've always felt that there should be a way to seamlessly handle Python/C/C++ Numeric arrays using a standard API for each language. I don't believe I've mentioned it anywhere though and I am not sure if such a thing exists or not. Numeric/numarray offer the most commonly used, maintained and popular API for arrays for numerics in Python. It would be best if a clean C/C++ API would exist. I'm thinking of a higher level wrapper like class that gives the C/C++ user a nice API. No re-implementation of the Numeric/numarray API should be neccessary. If this were done, the following becomes easily possible: 1. Its easy for C/C++ programmers to use the API and not have to worry about translating their C code to Python. The translation should be easy since they are using the standard wrapper API that uses the raw numeric API underneath. 2. Performance is not too much of an issue since there is no copying of data back and forth between C/Python and the C++ layer would most probably be entirely inlined. 3. Writing weave.inline code should be almost as easy as the blitz version. Agreed blitz is extremely easy to use but if we had a fairly powerful wrapper API most of the common stuff could be easily done via this API. Agreed, getting slices to work would be some work but if there is significant interest, this should also be possible. In any case the API could be made sufficiently useable without the need for blitz conversions. 4. Compile times should drop since we aren't really looking to redo blitz but simply provide a clean Numeric API for C/C++. >From what I can tell this should solve most problems satisfactorily and should be useful outside of weave. cheers, prabhu From fperez at colorado.edu Wed Sep 3 10:15:35 2003 From: fperez at colorado.edu (Fernando Perez) Date: Wed, 03 Sep 2003 08:15:35 -0600 Subject: [SciPy-dev] Re: [SciPy-user] Low-level code integration discussion at scipy'03? In-Reply-To: <16213.35148.87968.555025@monster.linux.in> References: <3F550B60.5060108@colorado.edu> <16213.35148.87968.555025@monster.linux.in> Message-ID: <3F55F787.3080809@colorado.edu> Prabhu Ramachandran wrote: > AFAIK, the other problems with blitz++ and weave are: > > 1. Blitz is not as fast as a hand coded inline C version. More on > ugliness of code later. From my experience, this is definitely true. Depending on how much one uses Blitz's abstractions, the overhead can be rather high. This is one of the things I'd like to discuss with J. Cummings at Caltech, since he works on the blitz internals. > 2. Blitz compile times are large. True. > 3. An extra layer of complexity for weave, one more package to keep > track of -- the blitz sources are bundled with weave. > 4. With all due respect, not many C++ programmers (forget Python > programmers) would be able to understand/code anything with the > techniques used inside Blitz. Agreed, this is transparent to the > user in the present context but its still worth noting. True. Using it, however, is a dream :) > Here is my take. I've always felt that there should be a way to > seamlessly handle Python/C/C++ Numeric arrays using a standard API for > each language. I don't believe I've mentioned it anywhere though and > I am not sure if such a thing exists or not. Numeric/numarray offer > the most commonly used, maintained and popular API for arrays for > numerics in Python. It would be best if a clean C/C++ API would > exist. I'm thinking of a higher level wrapper like class that gives > the C/C++ user a nice API. No re-implementation of the > Numeric/numarray API should be neccessary. If this were done, the > following becomes easily possible: > > 1. Its easy for C/C++ programmers to use the API and not have to > worry about translating their C code to Python. The translation > should be easy since they are using the standard wrapper API that > uses the raw numeric API underneath. > 2. Performance is not too much of an issue since there is no copying > of data back and forth between C/Python and the C++ layer would > most probably be entirely inlined. > 3. Writing weave.inline code should be almost as easy as the blitz > version. Agreed blitz is extremely easy to use but if we had a > fairly powerful wrapper API most of the common stuff could be > easily done via this API. Agreed, getting slices to work would be > some work but if there is significant interest, this should also > be possible. In any case the API could be made sufficiently > useable without the need for blitz conversions. > 4. Compile times should drop since we aren't really looking to redo > blitz but simply provide a clean Numeric API for C/C++. Well, I'm quite afraid that in order to get something like this done right, one would probably end up quite close to blitz (or something similar). The big problem with getting an efficient, low-level class representation for arrays is avoiding temporaries and extra copies in array expressions. As far as I know, the only way to achieve that is to use expression templates, a la blitz. This is just hard. So yes, perhaps there is a way to do this without blitz/ublas. But if one ends up faced with the prospect of writing a full expression template system, might as well use one of the existing ones. I am not saying that blitz has to be used. It's just that from what I've seen, it seems to satisfy the following requirements: - trivial to get a blitz array from a numarray -> no need to change anything in numarray's design to use it. - it tackled the problem of expression analysis to avoid temporaries and extra copying. The technique used was expression templates (very complex code), but I don't know of any simpler solution for this problem. So if you think that it is possible to obtain something similar with much less work, and lower compilation times/simpler code, great. Let's then have this discussion session at Scipy'03! (hint to the organizers: drop us a word about scheduling options, or we can plan for a lunch/dinner/beers time for this :) Best, f. From prabhu at aero.iitm.ernet.in Wed Sep 3 11:52:48 2003 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Wed, 3 Sep 2003 21:22:48 +0530 Subject: [SciPy-dev] Re: [SciPy-user] Low-level code integration discussion at scipy'03? In-Reply-To: <3F55F787.3080809@colorado.edu> References: <3F550B60.5060108@colorado.edu> <16213.35148.87968.555025@monster.linux.in> <3F55F787.3080809@colorado.edu> Message-ID: <16214.3664.675029.734813@monster.linux.in> >>>>> "FP" == Fernando Perez writes: [ PR on nice C++/C API for Numeric ] FP> Well, I'm quite afraid that in order to get something like FP> this done right, one would probably end up quite close to FP> blitz (or something similar). The big problem with getting an FP> efficient, low-level class representation for arrays is FP> avoiding temporaries and extra copies in array expressions. FP> As far as I know, the only way to achieve that is to use FP> expression templates, a la blitz. This is just hard. So yes, FP> perhaps there is a way to do this without blitz/ublas. But if FP> one ends up faced with the prospect of writing a full FP> expression template system, might as well use one of the FP> existing ones. FP> I am not saying that blitz has to be used. It's just that FP> from what I've seen, it seems to satisfy the following FP> requirements: Sure, if you want all the machinery that blitz provides you'll probably need all that power and complexity internally. However, I still think that having a simpler and easy to use C++ API will help tons. I'm not asking for all the expression template and curiously recursive templated definition stuff. If one can write C++ code that is a lot like Python or simple C and manipulate numeric arrays, it would be really nice. Other points: 1. I guess that weave code will usually be used to optimize the slowest loops. Typically eliminating 'for/while' loops. The trivial (and ancient) Laplacian examples on the SciPy pages demonstrate one such case. My guess is most other uses would be similar. In such cases merely having slices working is not good enough. You might also need to do something non-trivial to the array. At that point I'm not sure how much the niceties of blitz would be used (efficiently)? Could you provide a more illustrative example of what blitz does that could not be done by a simple C++ loop or something? 2. Given the above I don't see a C++ user really worrying about how long the code is so long as the speed advantage is there. 3. If plain old inlined C code does indeed perform better than the blitz equivalents, it sure would be useful to have the cleaner API anyway. Afterall performance is a key issue here. Having a cleaner API (without fancy expression templates but without having to convert between C++ and Python data) would really be useful in any case. Of course, if blitz could still work along with blas support etc. thats great too. :) [snip] FP> - it tackled the problem of expression analysis to avoid FP> temporaries and extra FP> copying. The technique used was expression templates (very FP> complex code), but I don't know of any simpler solution for FP> this problem. Well, I'm sure I would not mind writing this kind of loop (taken from here http://www.scipy.org/site_content/weave/python_performance.html): double tmp, err, diff; err = 0.0; for (int i=1; i times/simpler code, great. Let's then have this discussion FP> session at Scipy'03! (hint to the organizers: drop us a word Sure. cheers, prabhu From fperez at colorado.edu Wed Sep 3 12:42:28 2003 From: fperez at colorado.edu (Fernando Perez) Date: Wed, 03 Sep 2003 10:42:28 -0600 Subject: [SciPy-dev] Re: [SciPy-user] Low-level code integration discussion at scipy'03? In-Reply-To: <16214.3664.675029.734813@monster.linux.in> References: <3F550B60.5060108@colorado.edu> <16213.35148.87968.555025@monster.linux.in> <3F55F787.3080809@colorado.edu> <16214.3664.675029.734813@monster.linux.in> Message-ID: <3F5619F4.9060003@colorado.edu> Prabhu Ramachandran wrote: > 1. I guess that weave code will usually be used to optimize the > slowest loops. Typically eliminating 'for/while' loops. The > trivial (and ancient) Laplacian examples on the SciPy pages > demonstrate one such case. My guess is most other uses would be > similar. In such cases merely having slices working is not good > enough. You might also need to do something non-trivial to the > array. At that point I'm not sure how much the niceties of blitz > would be used (efficiently)? Could you provide a more > illustrative example of what blitz does that could not be done by > a simple C++ loop or something? It's not a matter of being impossible in loops, just that the blitz syntax is incredibly compact. Consider computing Numeric's innerproduct() operation for a rank-2 array M and a rank-6 one T, storing the result in U. In blitz, the code reads: // 6d version void mat_ten_inner(Array& M, Array& T, Array& U ) { firstIndex i1; secondIndex i2; thirdIndex i3; fourthIndex i4; fifthIndex i5; sixthIndex i6; seventhIndex j; U = sum(M(i1,j)*T(i2,i3,i4,i5,i6,j),j); } That's it. ONE line of actual code! A version of this done with hand loops requires obviously 7 nested for loops. Since I needed similar code for dimensions 1..6, I wrote preprocessor macros to generate all versions from a 'template' (meaning, another macro, not a C++ template<>). This is, of course, ugly. Having said that, the above one-liner performs about 3 times slower than the loop macro version for d=6. For d=1 they are about even, the blitz one-liner gets worse as d increases. I posted to the blitz list asking about this, but no response so far. So, in the end, I'm writing my own loops too :) This is just to point that I'm ok with writing loops too, even though I'd love blitz to handle things like the above better. What blitz DOES buy you is A(i,j,k,l...) type indexing of high-rank arrays, which is not trivial with the C Numeric API. And that is important. Writing loops isn't bad, but all the manual pointer arithmetic currently necessary with the existing APIs is rather annoying. It's a matter of balancing abstraction with performance. Blitz provides a lot of abstraction, but performance isn't very good. The subset of blitz which gives just indexing (you do the rest yourself) seems to maintain pretty good performance in many cases (though I need to do more precise benchmarks against fortran). > 3. If plain old inlined C code does indeed perform better than the > blitz equivalents, it sure would be useful to have the cleaner API > anyway. Afterall performance is a key issue here. Having a > cleaner API (without fancy expression templates but without having > to convert between C++ and Python data) would really be useful in > any case. This, we agree 100% on. In fact, if the C API offered a way to index high rank arrays without manually computing stride offsets on every access, that would probably satisfy a significant fraction of usage cases. I consider one-liners like the above icing on the cake, but A(i,j,k) indexing is really a significant change which makes development far more productive. > Well, I'm sure I would not mind writing this kind of loop (taken from > here http://www.scipy.org/site_content/weave/python_performance.html): > > double tmp, err, diff; > err = 0.0; > for (int i=1; i for (int j=1; j tmp = u(i,j); > u(i,j) = ((u(i-1,j) + u(i+1,j))*dy2 + > (u(i,j-1) + u(i,j+1))*dx2)*dnr_inv; > diff = u(i,j) - tmp; > err += diff*diff; > } > } I don't either. As I said, the main consideration for me is clean indexing of rank2+ arrays. Everything else I take as extra credit, and I can live without. For example, blitz has 'stencils' to do the above, and in 3d cases they do simplify writing the code a lot. But I haven't tested their performance, and after seeing my innerproduct() example above, I'd be at least a bit skeptical. So loops are fine with me. Best, f. From eric at enthought.com Wed Sep 3 14:16:11 2003 From: eric at enthought.com (eric jones) Date: Wed, 03 Sep 2003 12:16:11 -0600 Subject: [SciPy-dev] Re: [SciPy-user] Low-level code integration discussion at scipy'03? In-Reply-To: <3F5619F4.9060003@colorado.edu> References: <3F550B60.5060108@colorado.edu> <3F55F787.3080809@colorado.edu> <16214.3664.675029.734813@monster.linux.in> <3F5619F4.9060003@colorado.edu> Message-ID: <3F562FEB.5030706@enthought.com> Hey Guys, I'm interested in the conversation also. We can set up a BOF session on Thursday night for this. Fernando, would you be willing to chair it? There will be a lot of others available to chime in I'm sure. I don't have an "official" time yet, but my guess would be that it will start around 8:00 to 8:30 in the evening and run until people are tired of talking. Michael: Will it be any problem to get a couple of meeting rooms that evening? thanks, eric Fernando Perez wrote: > Prabhu Ramachandran wrote: > >> 1. I guess that weave code will usually be used to optimize the >> slowest loops. Typically eliminating 'for/while' loops. The >> trivial (and ancient) Laplacian examples on the SciPy pages >> demonstrate one such case. My guess is most other uses would be >> similar. In such cases merely having slices working is not good >> enough. You might also need to do something non-trivial to the >> array. At that point I'm not sure how much the niceties of blitz >> would be used (efficiently)? Could you provide a more >> illustrative example of what blitz does that could not be done by >> a simple C++ loop or something? > > > It's not a matter of being impossible in loops, just that the blitz > syntax is incredibly compact. Consider computing Numeric's > innerproduct() operation for a rank-2 array M and a rank-6 one T, > storing the result in U. In blitz, the code reads: > > // 6d version > void mat_ten_inner(Array& M, > Array& T, Array& U ) { > firstIndex i1; > secondIndex i2; > thirdIndex i3; > fourthIndex i4; > fifthIndex i5; > sixthIndex i6; > seventhIndex j; > > U = sum(M(i1,j)*T(i2,i3,i4,i5,i6,j),j); > } > > That's it. ONE line of actual code! A version of this done with hand > loops requires obviously 7 nested for loops. Since I needed similar > code for dimensions 1..6, I wrote preprocessor macros to generate all > versions from a 'template' (meaning, another macro, not a C++ > template<>). This is, of course, ugly. > > Having said that, the above one-liner performs about 3 times slower > than the loop macro version for d=6. For d=1 they are about even, the > blitz one-liner gets worse as d increases. I posted to the blitz list > asking about this, but no response so far. > > So, in the end, I'm writing my own loops too :) This is just to point > that I'm ok with writing loops too, even though I'd love blitz to > handle things like the above better. > > What blitz DOES buy you is A(i,j,k,l...) type indexing of high-rank > arrays, which is not trivial with the C Numeric API. And that is > important. Writing loops isn't bad, but all the manual pointer > arithmetic currently necessary with the existing APIs is rather annoying. > > It's a matter of balancing abstraction with performance. Blitz > provides a lot of abstraction, but performance isn't very good. The > subset of blitz which gives just indexing (you do the rest yourself) > seems to maintain pretty good performance in many cases (though I need > to do more precise benchmarks against fortran). > >> 3. If plain old inlined C code does indeed perform better than the >> blitz equivalents, it sure would be useful to have the cleaner API >> anyway. Afterall performance is a key issue here. Having a >> cleaner API (without fancy expression templates but without having >> to convert between C++ and Python data) would really be useful in >> any case. > > > This, we agree 100% on. In fact, if the C API offered a way to index > high rank arrays without manually computing stride offsets on every > access, that would probably satisfy a significant fraction of usage > cases. I consider one-liners like the above icing on the cake, but > A(i,j,k) indexing is really a significant change which makes > development far more productive. > >> Well, I'm sure I would not mind writing this kind of loop (taken from >> here http://www.scipy.org/site_content/weave/python_performance.html): >> >> double tmp, err, diff; >> err = 0.0; >> for (int i=1; i> for (int j=1; j> tmp = u(i,j); >> u(i,j) = ((u(i-1,j) + u(i+1,j))*dy2 + >> (u(i,j-1) + u(i,j+1))*dx2)*dnr_inv; >> diff = u(i,j) - tmp; >> err += diff*diff; >> } >> } > > > I don't either. As I said, the main consideration for me is clean > indexing of rank2+ arrays. Everything else I take as extra credit, > and I can live without. For example, blitz has 'stencils' to do the > above, and in 3d cases they do simplify writing the code a lot. But I > haven't tested their performance, and after seeing my innerproduct() > example above, I'd be at least a bit skeptical. So loops are fine > with me. > > Best, > > f. > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev From fperez at colorado.edu Wed Sep 3 13:25:59 2003 From: fperez at colorado.edu (Fernando Perez) Date: Wed, 03 Sep 2003 11:25:59 -0600 Subject: [SciPy-dev] Re: [SciPy-user] Low-level code integration discussion at scipy'03? In-Reply-To: <3F562FEB.5030706@enthought.com> References: <3F550B60.5060108@colorado.edu> <3F55F787.3080809@colorado.edu> <16214.3664.675029.734813@monster.linux.in> <3F5619F4.9060003@colorado.edu> <3F562FEB.5030706@enthought.com> Message-ID: <3F562427.6080604@colorado.edu> eric jones wrote: > Hey Guys, > > I'm interested in the conversation also. We can set up a BOF session on > Thursday night for this. Fernando, would you be willing to chair it? > There will be a lot of others available to chime in I'm sure. I don't > have an "official" time yet, but my guess would be that it will start > around 8:00 to 8:30 in the evening and run until people are tired of > talking. Great to hear from you, Eric. Yes, I'm fully willing to coordinate the discussion. I'd be glad to present a _brief_ (5-10 min) summary of the issues, so if anyone has anything they'd like me to include at the beginning, please drop it into this discussion. I'll just recap this thread for everyone's benefit, and then we'll go off into discussions. I'll also contact Julian Cummings to see if he's willing to stop by, since he seems to be the most active blitz developer at the moment. Best, Fernando. From loredo at astro.cornell.edu Wed Sep 3 13:40:59 2003 From: loredo at astro.cornell.edu (Tom Loredo) Date: Wed, 3 Sep 2003 13:40:59 -0400 (EDT) Subject: [SciPy-dev] [SciPy-user] Low-level code integration Message-ID: <200309031740.h83HexH07363@laplace.astro.cornell.edu> Hi folks- I'll be in the Bay area next week but unfortunately at another meeting, so I can't participate in this discussion. But I think the thread raises important issues. > This, we agree 100% on. In fact, if the C API offered a way to index high > rank arrays without manually computing stride offsets on every access, that > would probably satisfy a significant fraction of usage cases. I consider > one-liners like the above icing on the cake, but A(i,j,k) indexing is really a > significant change which makes development far more productive. This point of agreement between the two of you is indeed what would satisfy me most of the time. I write a lot of extensions with few-d arrays, and calculating the strides does indeed become a nuisance and makes the code harder to read and maintain. Another possible virtue of having this kind of intermediate API between C and numarray is that it might isolate the user to possible future changes in the numarray C API, though I don't know if there is a significant chance of such changes. Just thought I'd add a voice of agreement on the value of the kind of API being discussed, even if I'm not in a position to offer code.... -Tom Loredo From prabhu at aero.iitm.ernet.in Wed Sep 3 13:56:59 2003 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Wed, 3 Sep 2003 23:26:59 +0530 Subject: [SciPy-dev] Re: [SciPy-user] Low-level code integration discussion at scipy'03? In-Reply-To: <3F5619F4.9060003@colorado.edu> References: <3F550B60.5060108@colorado.edu> <16213.35148.87968.555025@monster.linux.in> <3F55F787.3080809@colorado.edu> <16214.3664.675029.734813@monster.linux.in> <3F5619F4.9060003@colorado.edu> Message-ID: <16214.11115.945651.476138@monster.linux.in> >>>>> "FP" == Fernando Perez writes: [ FP's nice 6d example ] FP> That's it. ONE line of actual code! A version of this done FP> with hand loops requires obviously 7 nested for loops. Since FP> I needed similar code for dimensions 1..6, I wrote FP> preprocessor macros to generate all versions from a 'template' FP> (meaning, another macro, not a C++ template<>). This is, of FP> course, ugly. Interesting example but I'd imagine that apart from you and some string theorists (are there any here?) most of us mere mortals would deal with 3 to 4 d data. ;-) Agreed a one liner is a lot nicer but 3 nested loops are quite normal. So indeed blitz does have lots of advantages. In fact its the fastest way to get a 3 fold increase over plain Python Numeric. FP> Having said that, the above one-liner performs about 3 times FP> slower than the loop macro version for d=6. For d=1 they are FP> about even, the blitz one-liner gets worse as d increases. I FP> posted to the blitz list asking about this, but no response so FP> far. Ahh. :) [snip] >> 3. If plain old inlined C code does indeed perform better than >> the >> blitz equivalents, it sure would be useful to have the cleaner >> API anyway. Afterall performance is a key issue here. Having >> a cleaner API (without fancy expression templates but without >> having to convert between C++ and Python data) would really be >> useful in any case. FP> This, we agree 100% on. In fact, if the C API offered a way FP> to index high rank arrays without manually computing stride FP> offsets on every access, that would probably satisfy a FP> significant fraction of usage cases. I consider one-liners FP> like the above icing on the cake, but A(i,j,k) indexing is FP> really a significant change which makes development far more FP> productive. Great! So we have three strong votes on this on this. cheers, prabhu From perry at stsci.edu Wed Sep 3 13:58:33 2003 From: perry at stsci.edu (Perry Greenfield) Date: Wed, 3 Sep 2003 13:58:33 -0400 Subject: [SciPy-dev] Re: [SciPy-user] Low-level code integrationdiscussion at scipy'03? In-Reply-To: <3F562427.6080604@colorado.edu> Message-ID: I'm planning to be part of the discussion as well. Perry From perry at stsci.edu Wed Sep 3 14:01:44 2003 From: perry at stsci.edu (Perry Greenfield) Date: Wed, 3 Sep 2003 14:01:44 -0400 Subject: [SciPy-dev] Re: [SciPy-user] Low-level code integrationdiscussion at scipy'03? In-Reply-To: <16214.11115.945651.476138@monster.linux.in> Message-ID: > > FP> This, we agree 100% on. In fact, if the C API offered a way > FP> to index high rank arrays without manually computing stride > FP> offsets on every access, that would probably satisfy a > FP> significant fraction of usage cases. I consider one-liners > FP> like the above icing on the cake, but A(i,j,k) indexing is > FP> really a significant change which makes development far more > FP> productive. > Are you just talking about C macros to do the indexing (including strides? Perry From fperez at colorado.edu Wed Sep 3 14:09:50 2003 From: fperez at colorado.edu (Fernando Perez) Date: Wed, 03 Sep 2003 12:09:50 -0600 Subject: [SciPy-dev] Re: [SciPy-user] Low-level code integrationdiscussion at scipy'03? In-Reply-To: References: Message-ID: <3F562E6E.7040704@colorado.edu> Perry Greenfield wrote: >> FP> This, we agree 100% on. In fact, if the C API offered a way >> FP> to index high rank arrays without manually computing stride >> FP> offsets on every access, that would probably satisfy a >> FP> significant fraction of usage cases. I consider one-liners >> FP> like the above icing on the cake, but A(i,j,k) indexing is >> FP> really a significant change which makes development far more >> FP> productive. >> > > Are you just talking about C macros to do the indexing (including strides? Well, anything which allows me to say in C/C++: A(i,j,k,l,m,n) = some_function_of(i,j,k,l,m,n); square brackets would also work, though A[i][j][k][l][m][n] starts to feel ugly real quick. There seems to be some consensus that in most cases, simplified indexing is the #1 wish. Everything else (array arithmetic, math functions over arrays, etc) would be nice, but not a first requirement. I'd also add that slicing is pretty high on my list, or some mechanism to say 'this part of this array is really a subset of that one' without data copying. Blitz supports this with semantics and syntax which are eerily similar to those in Numeric/Numarray at the python level. Best, f. From perry at stsci.edu Wed Sep 3 14:43:30 2003 From: perry at stsci.edu (Perry Greenfield) Date: Wed, 3 Sep 2003 14:43:30 -0400 Subject: [SciPy-dev] Re: [SciPy-user] Low-level code integrationdiscussion at scipy'03? In-Reply-To: <3F562E6E.7040704@colorado.edu> Message-ID: > Perry Greenfield wrote: > >> FP> This, we agree 100% on. In fact, if the C API offered a way > >> FP> to index high rank arrays without manually computing stride > >> FP> offsets on every access, that would probably satisfy a > >> FP> significant fraction of usage cases. I consider one-liners > >> FP> like the above icing on the cake, but A(i,j,k) indexing is > >> FP> really a significant change which makes development far more > >> FP> productive. > >> > > > > Are you just talking about C macros to do the indexing > (including strides? > > Well, anything which allows me to say in C/C++: > > A(i,j,k,l,m,n) = some_function_of(i,j,k,l,m,n); > > square brackets would also work, though > > A[i][j][k][l][m][n] > > starts to feel ugly real quick. > Well, since C is not blessed with convenient handling of multidimensional array indexing, it would have to be a macro or a function to do this. A function probably brings in about a factor of 3 penalty for data access (at least it did a couple years ago when we benchmarked that approach--hmmm, makes me wonder if blitz resorts to that for large numbers of dimensions). Mind you, for numarray instances where byteswapped or nonaligned data are involved, I think a function for indexing is a necessity. So given that, I (with my admittedly limited imagination :-) can't see how syntax such as A(i,j,k,l,m,n) can work in C (I hope someone can set me straight). Perhaps something like Index6d(A,i,j,k,l,m,n) it the best that can be done. (A[i][j]...[n] would work but would need arrays of pointers to pointer to... and I don't think we want to go there.) On the other hand, syntax like that proposed can be provided in C++. Still, I'd be very surprised if the blitz people haven't already given a great deal of thought to this. They probably can explain why it is slower and why it is difficult to do much better than this. Perry From fperez at colorado.edu Wed Sep 3 16:17:06 2003 From: fperez at colorado.edu (Fernando Perez) Date: Wed, 03 Sep 2003 14:17:06 -0600 Subject: [SciPy-dev] Re: [SciPy-user] Low-level code integrationdiscussion at scipy'03? In-Reply-To: References: Message-ID: <3F564C42.5030907@colorado.edu> Perry Greenfield wrote: > So given that, I (with my admittedly limited imagination :-) can't see > how syntax such as > > A(i,j,k,l,m,n) > > can work in C (I hope someone can set me straight). Perhaps something > like > > Index6d(A,i,j,k,l,m,n) > > it the best that can be done. Agreed. In pure C, I don't see any other option either. > (A[i][j]...[n] would work but would need > arrays of pointers to pointer to... and I don't think we want to go > there.) Yup. I've done that for 2d arrays, writing a simple function which takes a Numpy array and returns the necessary scaffolding of pointers for A[i][j] to work. But it's ugly, wasteful, and doesn't scale cleanly to higher dimensions. > On the other hand, syntax like that proposed can be provided in C++. > Still, I'd be very surprised if the blitz people haven't already given > a great deal of thought to this. They probably can explain why it is > slower and why it is difficult to do much better than this. Well, their stuff works really well for pure indexing. The overhead I mentioned was for the magic U = sum(M(i,j),T(k,l,m,n,j),j); expression, which is setting all U(i,k,l,m,n) for you by building all the nested loops on the fly. If you directly index the arrays, I think blitz performance is indeed very good (at least the benchmarks they provide claim so). I hope Julian Cummings can make it to the discussion, so we can ask him about this directly. Cheers, f. From Chuck.Harris at sdl.usu.edu Wed Sep 3 14:56:54 2003 From: Chuck.Harris at sdl.usu.edu (Chuck Harris) Date: Wed, 3 Sep 2003 12:56:54 -0600 Subject: [SciPy-dev] Re: [SciPy-user] Low-level code integrationdiscussionat scipy'03? Message-ID: <1885D1238A4FFE40B113CEB26F87872B11836C@cobra.usurf.usu.edu> >> >> >> > >> > Are you just talking about C macros to do the indexing >> (including strides? >> >> Well, anything which allows me to say in C/C++: >> >> A(i,j,k,l,m,n) = some_function_of(i,j,k,l,m,n); >> >> square brackets would also work, though >> >> A[i][j][k][l][m][n] >> >> starts to feel ugly real quick. >> >Well, since C is not blessed with convenient handling of multidimensional >array indexing, it would have to be a macro or a function to do this. >A function probably brings in about a factor of 3 penalty for data >access (at least it did a couple years ago when we benchmarked that >approach--hmmm, makes me wonder if blitz resorts to that for large numbers >of dimensions). Mind you, for numarray instances where byteswapped or >nonaligned data are involved, I think a function for indexing is a >necessity. In a temporary moment of madness, I started to write my own version of Numeric. Indexing was a small loop implementing a mixed radix counter that was passed a structure containing the slice info and some function pointers that were called in certain parts of the loop. The overhead I measured was of the order of 10% over a straight loop with embedded function calls on my machine (PIII). So I think it is possible to build pretty efficient functions for this sort of thing. I was most surprised at how low the function call overhead was. Chuck -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3358 bytes Desc: not available URL: From nwagner at mecha.uni-stuttgart.de Wed Sep 10 11:29:02 2003 From: nwagner at mecha.uni-stuttgart.de (Nils Wagner) Date: Wed, 10 Sep 2003 17:29:02 +0200 Subject: [SciPy-dev] Hankel, Toeplitz why not Vandermonde Message-ID: <3F5F433E.D030CEC@mecha.uni-stuttgart.de> Hi all, I found Hankel and Toeplitz matrices in scipy - why not Vandermonde matrices ? Nils From rramachandran at itsc.uah.edu Wed Sep 10 14:40:55 2003 From: rramachandran at itsc.uah.edu (Rahul Ramachandran) Date: Wed, 10 Sep 2003 13:40:55 -0500 Subject: [SciPy-dev] Earth Science Markup Language Message-ID: All: We have implemented a Python wrapper for the ESML data reader library. This allows users to read any kind of unstructured files formats in Binary or ASCII and formats such as HDF-EOS, netCDF, GRIB in Python. I think it will be extremely useful to the SciPy community. Details about ESML can be found at: http://esml.itsc.uah.edu Thnx, -Rahul **************************************** Rahul Ramachandran, Ph.D. Information Technology and Systems Center University of Alabama in Huntsville Huntsville, AL 35899 Tel: (256)-824-5157 Fax: (256)-824-5149 email: rramachandran at itsc.uah.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 2074 bytes Desc: image001.gif URL: From jb.richet at areas.asso.fr Mon Sep 8 10:59:11 2003 From: jb.richet at areas.asso.fr (Jean-Baptiste Richet - AREAS) Date: Mon, 08 Sep 2003 15:59:11 +0100 Subject: [SciPy-dev] problem installing weave under Win ME Message-ID: <3F5C993F.5040307@areas.asso.fr> Hello, weave could be highly usefull to me, but I can't have it work correctly on my PC, and I don't know what to do. Could you help me ? I installed the latest zip package by "python setup.py install", on a python 2.3 installation (OS = Windows ME), and here is a result when trying it : ========================= IDLE 1.0 >>> import weave >>> a = 123 >>> weave.inline('printf("%d\\n",a);',['a']) Exception exceptions.AttributeError: "DbfilenameShelf instance has no attribute 'writeback'" in Traceback (most recent call last): File "", line 1, in -toplevel- weave.inline('printf("%d\\n",a);',['a']) File "C:\DEVELOPPEMENT\PYTHON23\Lib\site-packages\weave\inline_tools.py", line 327, in inline auto_downcast = auto_downcast, File "C:\DEVELOPPEMENT\PYTHON23\Lib\site-packages\weave\inline_tools.py", line 426, in compile_function verbose=verbose, **kw) File "C:\DEVELOPPEMENT\PYTHON23\Lib\site-packages\weave\ext_tools.py", line 349, in compile verbose = verbose, **kw) File "C:\DEVELOPPEMENT\PYTHON23\Lib\site-packages\weave\build_tools.py", line 194, in build_extension setup(name = module_name, ext_modules = [ext],verbose=verb) File "c:\developpement\python23\lib\distutils\core.py", line 166, in setup raise SystemExit, "error: " + str(msg) CompileError: error: command 'g++' failed with exit status 1 >>> =========================== The gcc I use comes from MinGW and is version 3.2. I tried out gcc 2.95, and it was nearly the same (error at the same place, but 'cc' was blaimed). I tried cygwin, and it was not different. I tried python 2.2, and the result was similar. Any idea ? thanks a lot for your help. -- Jean-Baptiste Richet AREAS _________________________________ Association r?gionale pour l'?tude et l'am?lioration des sols 2, avenue Foch 76460 Saint Val?ry en Caux t?l : +33 2 35 97 25 12 fax : +33 2 35 97 25 73 _________________________________ From bhoel at web.de Wed Sep 10 13:25:44 2003 From: bhoel at web.de (Berthold Hoellmann) Date: Wed, 10 Sep 2003 19:25:44 +0200 Subject: [SciPy-dev] Compile failure of CVS version Message-ID: Hello, Trying to compile the latest CVS version of SciPy I get: gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wstrict-prototypes -O3 -fstrict-aliasing -funroll-loops -fschedule-insns2 -ffast-math -fno-math-errno -funsafe-math-optimizations -fno-trapping-math -march=athlon-xp -mfpmath=sse -msse -mmmx -m3dnow -fPIC -I/home/devel/CVS/scipy/Lib_chaco/agg2/include -I/home/devel/CVS/scipy/Lib_chaco/kiva/agg/swig_src/include -I/usr/local/include/python2.3 -c /home/devel/CVS/scipy/Lib_chaco/kiva/agg/swig_src/agg_wrap.cxx -o build/temp.linux-i686-2.3/home/devel/CVS/scipy/Lib_chaco/kiva/agg/swig_src/agg_wrap.o In file included from /home/devel/CVS/scipy/Lib_chaco/kiva/agg/swig_src/agg_wrap.cxx:982: /home/devel/CVS/scipy/Lib_chaco/kiva/agg/swig_src/kiva_graphics_context.h:515: error: invalid type `agg::rgba' for default argument to `agg::rgba&' /home/devel/CVS/scipy/Lib_chaco/kiva/agg/swig_src/agg_wrap.cxx: In function ` PyObject* _wrap__GraphicsContextArray_clear(PyObject*, PyObject*)': /home/devel/CVS/scipy/Lib_chaco/kiva/agg/swig_src/agg_wrap.cxx:5182: error: could not convert `rgba(1.0e+0, 1.0e+0, 1.0e+0, 1.0e+0)' to `agg::rgba&' error: command 'gcc' failed with exit status 1 I am on a SuSE 8.2 but with gcc --version gcc (GCC) 3.3 (SuSE Linux) Copyright (C) 2003 Free Software Foundation, Inc. instead of the original delivered prerelease version and a self compiled Python 2.3. Sorry, but I don't know enough about C++ to provide a patch. Greetings Berthold -- bhoel at web.de / http://starship.python.net/crew/bhoel/ It is unlawful to use this email address for unsolicited ads (USC Title 47 Sec.227). I will assess a US$500 charge for reviewing and deleting each unsolicited ad. From nwagner at mecha.uni-stuttgart.de Tue Sep 16 09:15:45 2003 From: nwagner at mecha.uni-stuttgart.de (Nils Wagner) Date: Tue, 16 Sep 2003 15:15:45 +0200 Subject: [SciPy-dev] powm for scipy Message-ID: <3F670D01.4CB9B306@mecha.uni-stuttgart.de> Dear experts, I wonder, if powm will become a part of scipy in the near future. Nils From nwagner at mecha.uni-stuttgart.de Tue Sep 16 11:03:22 2003 From: nwagner at mecha.uni-stuttgart.de (Nils Wagner) Date: Tue, 16 Sep 2003 17:03:22 +0200 Subject: [SciPy-dev] Contour plots in scipy Message-ID: <3F67263A.212DBC74@mecha.uni-stuttgart.de> Dear experts, I am interested in a plot of equipotential curves. If desired, the regions between contours should be shaded or colored to indicate their magnitude. Is this feature already available in scipy ? Any comments or suggestions ? Nils From bhoel at web.de Tue Sep 16 15:58:49 2003 From: bhoel at web.de (Berthold Hoellmann) Date: Tue, 16 Sep 2003 21:58:49 +0200 Subject: [SciPy-dev] test failures with current CVS Message-ID: Hello, Building and installing the current CVS version of SciPy, import scipy scipy.test() gives 4 errors on SuSE 8.2 with Python 2.3 and gcc3.3: ... ====================================================================== ERROR: check_cdf (test_distributions.test_foldnorm) ---------------------------------------------------------------------- Traceback (most recent call last): File "", line 5, in check_cdf File "/usr/local/lib/python2.3/site-packages/scipy/stats/stats.py", line 1521, in kstest vals = sb.sort(rvs(*args,**kwds)) File "/usr/local/lib/python2.3/site-packages/scipy/stats/distributions.py", line 586, in __call__ return self.rvs(*args, **kwds) File "/usr/local/lib/python2.3/site-packages/scipy/stats/distributions.py", line 338, in rvs vals = reshape(self._rvs(*args),size) File "/usr/local/lib/python2.3/site-packages/scipy/stats/distributions.py", line 1100, in _rvs return abs(norm(loc=c,size=self._size)) TypeError: __call__() got an unexpected keyword argument 'loc' ====================================================================== ERROR: check_cdf (test_distributions.test_nct) ---------------------------------------------------------------------- Traceback (most recent call last): File "", line 5, in check_cdf File "/usr/local/lib/python2.3/site-packages/scipy/stats/stats.py", line 1521, in kstest vals = sb.sort(rvs(*args,**kwds)) File "/usr/local/lib/python2.3/site-packages/scipy/stats/distributions.py", line 586, in __call__ return self.rvs(*args, **kwds) File "/usr/local/lib/python2.3/site-packages/scipy/stats/distributions.py", line 338, in rvs vals = reshape(self._rvs(*args),size) File "/usr/local/lib/python2.3/site-packages/scipy/stats/distributions.py", line 1704, in _rvs return norm(loc=nc,size=self._size)*sqrt(df) / sqrt(chi2(df,size=self._size)) TypeError: __call__() got an unexpected keyword argument 'loc' ====================================================================== ERROR: check_chebyc (test_basic.test_chebyc) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.3/site-packages/scipy/special/tests/test_basic.py", line 214, in check_chebyc chebc = chebyc(1)(.2) File "/usr/local/lib/python2.3/site-packages/scipy/special/orthogonal.py", line 398, in chebyc p = p * 2.0/p(2) File "/usr/local/lib/python2.3/site-packages/scipy_base/polynomial.py", line 445, in __getattr__ raise KeyError KeyError ====================================================================== ERROR: check_chebys (test_basic.test_chebys) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.3/site-packages/scipy/special/tests/test_basic.py", line 220, in check_chebys chebs = chebys(1)(.2) File "/usr/local/lib/python2.3/site-packages/scipy/special/orthogonal.py", line 429, in chebys p = p * (n+1.0)/p(2) File "/usr/local/lib/python2.3/site-packages/scipy_base/polynomial.py", line 445, in __getattr__ raise KeyError KeyError ---------------------------------------------------------------------- Ran 739 tests in 3.045s FAILED (errors=4) >>> Greetings Berthold -- bhoel at web.de / http://starship.python.net/crew/bhoel/ It is unlawful to use this email address for unsolicited ads (USC Title 47 Sec.227). I will assess a US$500 charge for reviewing and deleting each unsolicited ad. From nwagner at mecha.uni-stuttgart.de Thu Sep 18 04:22:51 2003 From: nwagner at mecha.uni-stuttgart.de (Nils Wagner) Date: Thu, 18 Sep 2003 10:22:51 +0200 Subject: [SciPy-dev] Problems with latest cvs Message-ID: <3F696B5B.6831E289@mecha.uni-stuttgart.de> Dear experts, /usr/local/bin/python setup.py install failed with the following message usr/local/lib/python2.1/site-packages/scipy/traits/traits.py:518: SyntaxWarning: local name 'object' in 'sync_trait' shadows use of 'object' as global in nested scope 'lambda' def sync_trait ( self, trait_name, object, alias = None, mutual = True ): /usr/local/lib/python2.1/site-packages/scipy/traits/traits.py:518: SyntaxWarning: local name 'self' in 'sync_trait' shadows use of 'self' as global in nested scope 'lambda' def sync_trait ( self, trait_name, object, alias = None, mutual = True ): /usr/local/lib/python2.1/site-packages/scipy/traits/traits.py:518: SyntaxWarning: local name 'trait_name' in 'sync_trait' shadows use of 'trait_name' as global in nested scope 'lambda' def sync_trait ( self, trait_name, object, alias = None, mutual = True ): /usr/local/lib/python2.1/site-packages/scipy/traits/traits.py:518: SyntaxWarning: local name 'alias' in 'sync_trait' shadows use of 'alias' as global in nested scope 'lambda' def sync_trait ( self, trait_name, object, alias = None, mutual = True ): byte-compiling /usr/local/lib/python2.1/site-packages/scipy/traits/tests/test_timing.py to test_timing.pyc byte-compiling /usr/local/lib/python2.1/site-packages/scipy/traits/tests/test_traits.py to test_traits.pyc Traceback (most recent call last): File "setup.py", line 113, in ? setup_package() File "setup.py", line 103, in setup_package url = "http://www.scipy.org", File "scipy_core/scipy_distutils/core.py", line 42, in setup return old_setup(**new_attr) File "/usr/local/lib/python2.1/distutils/core.py", line 138, in setup dist.run_commands() File "/usr/local/lib/python2.1/distutils/dist.py", line 899, in run_commands self.run_command(cmd) File "/usr/local/lib/python2.1/distutils/dist.py", line 919, in run_command cmd_obj.run() File "/usr/local/lib/python2.1/distutils/command/install.py", line 484, in run self.run_command(cmd_name) File "/usr/local/lib/python2.1/distutils/cmd.py", line 328, in run_command self.distribution.run_command(command) File "/usr/local/lib/python2.1/distutils/dist.py", line 919, in run_command cmd_obj.run() File "/usr/local/lib/python2.1/distutils/command/install_lib.py", line 93, in run self.byte_compile(outfiles) File "/usr/local/lib/python2.1/distutils/command/install_lib.py", line 130, in byte_compile verbose=self.verbose, dry_run=self.dry_run) File "/usr/local/lib/python2.1/distutils/util.py", line 439, in byte_compile compile(file, cfile, dfile) File "/usr/local/lib/python2.1/py_compile.py", line 62, in compile codeobject = __builtin__.compile(codestring, dfile or file, 'exec') OverflowError: ('integer literal too large', ('/usr/local/lib/python2.1/site-packages/scipy/traits/tests/test_traits.py', 839, None, "[ '000-000-000', '999-9999-999', '1234-567-8769', '[123] 456-7890',\n")) Nils From pearu at scipy.org Thu Sep 18 06:56:27 2003 From: pearu at scipy.org (Pearu Peterson) Date: Thu, 18 Sep 2003 13:56:27 +0300 (EEST) Subject: [SciPy-dev] Problems with latest cvs In-Reply-To: <3F696B5B.6831E289@mecha.uni-stuttgart.de> Message-ID: On Thu, 18 Sep 2003, Nils Wagner wrote: > Dear experts, > > /usr/local/bin/python setup.py install failed with the following > message > > OverflowError: ('integer literal too large', > ('/usr/local/lib/python2.1/site-packages/scipy/traits/tests/test_traits.py', > 839, None, "[ '000-000-000', '999-9999-999', '1234-567-8769', '[123] > 456-7890',\n")) This error seems to occur only with Python 2.1. Until it gets fixed, use newer Python or disable building chaco in cvs/scipy/setup.py. Pearu From pearu at scipy.org Thu Sep 18 09:11:00 2003 From: pearu at scipy.org (Pearu Peterson) Date: Thu, 18 Sep 2003 16:11:00 +0300 (EEST) Subject: [SciPy-dev] Re: [Scipy-cvs] world/scipy setup.py,1.10,1.11 In-Reply-To: <20030917202131.B47393EA08@www.scipy.com> Message-ID: On Wed, 17 Sep 2003 martin at scipy.org wrote: > Index: setup.py > =================================================================== > RCS file: /home/cvsroot/world/scipy/setup.py,v > retrieving revision 1.10 > retrieving revision 1.11 > diff -C2 -d -r1.10 -r1.11 > *** setup.py 15 Sep 2003 14:06:42 -0000 1.10 > --- setup.py 17 Sep 2003 20:21:29 -0000 1.11 > *************** > *** 50,54 **** > scipy_packages = standard_packages > scipy_packages += graphics_packages > ! #scipy_packages += chaco_packages > > #--------------- > --- 50,54 ---- > scipy_packages = standard_packages > scipy_packages += graphics_packages > ! scipy_packages += chaco_packages Hmm, why is this uncommented? Now chaco and friends are installed in two locations: both in site-packages/ and site-packages/scipy/. > #--------------- > *************** > *** 60,64 **** > > # enable building chaco here by uncommenting the following line: > ! #separate_packages += chaco_packages > > #------------------------------- > --- 60,64 ---- > > # enable building chaco here by uncommenting the following line: > ! separate_packages += chaco_packages I thought we agreed on disabling chaco build under scipy because of gcc 2.95 bugs showing up on chaco C++ codes. Pearu From kern at ugcs.caltech.edu Sat Sep 20 05:36:04 2003 From: kern at ugcs.caltech.edu (Robert Kern) Date: Sat, 20 Sep 2003 02:36:04 -0700 Subject: [SciPy-dev] Formal Review Process Message-ID: <20030920093604.GA29589@taliesen.caltech.edu> At both SciPy conferences, Eric emphasized the need for a peer-review process for potential contributions to SciPy. I recalled that the Boost community established a formal review process for contributions to their set of libraries, and it seems to work well for them. I got bored of writing a history paper, so I looked at their process and wrote up a proposal for SciPy (liberally using parts of the Boost documents where noted). Please comment on, add to, delete from, and otherwise contribute to the following proposal. You can email me personally as well as respond to the list. As he announced at SciPy '03, Chad Netzer has a wavelets module he would like to submit. If we can hammer out some consensus (enough for a test run at least), then Chad can be our first sacrif^Wclient. I am sure he is as thrilled to learn of this possibility as I am to raise it. The text is included below. HTML and text versions can also be found in the following URLs: http://www.ugcs.caltech.edu/~kern/scipy-review.html http://www.ugcs.caltech.edu/~kern/scipy-review.txt A Proposal for a SciPy Module Submission Review Process ======================================================= :Author: Robert Kern (parts by the Boost_ group and others__) :Version: 0.0 :Date: 2003-09-20 The following proposal is inspired by (and indeed partly outright lifted from) the `Boost Formal Review Process`_ and `Submission Process`_. It is written using ReStructuredText_. .. _Boost: http://www.boost.org __ Acknowledgements_ .. _Boost Formal Review Process: http://www.boost.org/more/formal_review_process.htm .. _Submission Process: http://www.boost.org/more/submission_process.htm .. _ReStructuredText : http://docutils.sourceforge.net Rationale --------- 1. The SciPy package requires a peer-review process to examine new module submissions. The expertise of the CVS maintainer is limited, and the number of fields for which one could write a SciPy module is large. 2. A formal procedure for the review process makes it easier for newcomers to know exactly what they have to do to get their contribution into SciPy. Furthermore, a formal procedure makes it easier for Enthought to parcel out the work to non-Enthought people and even to transition SciPy to a wholly community-driven project if that ever becomes a goal. 3. The Boost project had a similar problem, found a solution that worked for them, and wrote it up all pretty, so why not steal it? [XXX - or something to this effect] Organization ------------ Mailing List ~~~~~~~~~~~~ Discussion of libraries under formal review shall take place on scipy-dev at scipy.org . Actual reviews of a candidate module (see section `Reviews`_) during its formal review period should begin their Subject header with [REVIEW - ] for the convenience of the Review Manager. Reviewers who do not wish to subscribe to scipy-dev at scipy.org may use the GMane_ WWW_ and Usenet_ interfaces to the mailing list. The WWW_ interface does not allow posting. [XXX - to be honest, I don't know if the Usenet_ one allows posting for this list, either.] .. _GMane: http://www.gmane.org .. _WWW: http://news.gmane.org/gmane.comp.python.scientific.devel .. _Usenet: nntp://news.gmane.org/gmane.comp.python.scientific.devel Review Wizard ~~~~~~~~~~~~~ (The Review Wizard/Manager sections are only slightly changed from `Boost Formal Review Process`_.) The Review Wizard: * Maintains a list of review manager volunteers. * When a formal review is requested for a module: - Assigns a review manager and suggests a schedule. - Finalizes the schedule, once the review manager verifies the module is actually ready for review. - Resolves schedule slips or other issues with review managers and submitters. * Maintains a schedule of both past and pending reviews. * Resolves questions from review managers and module submitters, who sometimes want a third opinion on questions such as "Should we extend the review period because ...?" * Monitors the general review process, and makes minor adjustments as needed, or queries the list about possible major adjustments. .. _Boost Formal Review Process: http://www.boost.org/more/formal_review_process.htm Review Manager ~~~~~~~~~~~~~~ The Review Manager: * Checks the submission to make sure it really is complete enough to warrant formal review. If necessary, work with the submitter to verify the code compiles and runs correctly on several platforms. * Finalizes the schedule with the Review Wizard and the submitter. * Posts a notice of the review schedule on the appropriate mailing list(s). - The notice should include a brief description of the module and what it does, to let readers know if the module is one they are interested in reviewing. - If the module is known to fail on certain platforms, please mention them in the review notice so reviewers on those platforms won't waste time diagnosing known problems (or alternatively, will test and try to resolve the failures). * Inspects the SciPy package for modules which may interact with the new submission. These potential interactions should be pointed out in the review announcement, and the author(s) of these modules should be privately notified and urged to participate in the review. * Urges people to do reviews if they aren't forthcoming. * Follows review discussions regarding the modules, moderating or answering questions as needed. * Decides if there is consensus to accept the modules, and if there are any conditions attached. * Posts a message to the appropriate mailing lists informing people of the review results. In other words, it is the Review Manager's responsibility to make sure the review process works smoothly. The Process for a Module Submitter ---------------------------------- Determine Interest ~~~~~~~~~~~~~~~~~~ Post a message to scipy-dev at scipy.org to see if there is interest in your module. Describe what your module does and perhaps include a small snippet of code that shows how one uses it. Preliminary Submission ~~~~~~~~~~~~~~~~~~~~~~ Put your (possibly unfinished) module on a website and post its location to scipy-dev at scipy.org . Contact the maintainers of www.scipy.org for a Zope account if you need web space for your module. Submissions should follow the Code Submission Guidelines [XXX - this should point to something useful. For now, look at the `Code Formatting Guidelines`_ and the `Testing Guidelines`_.] .. _Code Formatting Guidelines: http://www.scipy.org/site_content/tutorials/formatting_guidelines .. _Testing Guidelines: http://www.scipy.org/site_content/tutorials/testing_guidelines Refinement ~~~~~~~~~~ Discuss the module in scipy-dev at scipy.org . Fix bugs; add functionality; change the interface; write docs/unit tests; do whatever comes out of the discussion as necessary or beneficial to including the module in SciPy. Update the preliminary submission until it is ready for formal submission. Submission for Review ~~~~~~~~~~~~~~~~~~~~~ Package your module submission in a ZIP file or gzipped tar file such that it can be unpacked in the scipy CVS directory [XXX - what if you need to patch the setup.py file or scipy_distutils files? Diff file?]. Contact the Review Wizard. He will assign a Review Manager and create a www.scipy.org Zope account for you. Place your submission in [XXX] on the www.scipy.org website. The Review Manager will kick off the reviews on scipy-dev at scipy.org . Reviews ------- (Again, taken from `Boost Formal Review Process`_). Reviews may be submitted to either the list as a whole or the Review Manager privately. Your comments may be brief or lengthy, but basically the Review Manager needs your evaluation of the module. If you identify problems along the way, please note if they are minor, serious, or showstoppers. Here are some questions you might want to answer in your review: * What is your evaluation of the design? * What is your evaluation of the implementation? * What is your evaluation of the documentation? * What is your evaluation of the unit-test coverage? * What is your evaluation of the potential usefulness of the module? * Did you try to use the module? On what platform? Did you have any problems? * How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? * Are you knowledgeable about the problem domain? Are you knowledgeable about the programming techniques used? [E.g. perhaps you know nothing about what the underlying FORTRAN code does, but you are an expert in wrapping code with f2py. Both areas of knowledge are useful for a review.] And finally, every review should answer this question: * Do you think the module should be accepted into the SciPy package? Be sure to say this explicitly so that your other comments don't obscure your overall opinion. The module should be stable during the review period (making exception for serious bug fixes). Modifications suggested during review and accepted by the submitter should be noted on the mailing list so reviewers can make an informed decision. At the end of the review period, the Review Manager will make the final decision to accept or reject the module for inclusion into SciPy. He will post this result to the mailing list along with a rationale and any suggestions or conditions that must be met before the module can be included (or resubmitted for review, as the case may be). Acknowledgements ---------------- I would like to thank the Boost community for their well thought-out review process and the words describing said process. [XXX - Eric, please check if this is OK with them] Eric Jones, [XXX - Your Names Here] provided helpful comments and contributions to this document. [XXX - Kern's Notes] -------------------- The Review Wizard/Manager roles will probably be taken by Enthought people or their designees at least at the start, mostly because they would have to do similar tasks without a formal review process, so who else would volunteer? Note the XXX's. They need content or answers. Boost adds the libraries up for review into a CVS sandbox. We may want to do the same. The process as a whole feels a bit heavyweight to me, but there are some benefits that flow from that. Some nips and tucks couldn't hurt, though. -- Robert Kern kern at caltech.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From pearu at scipy.org Sat Sep 20 06:13:56 2003 From: pearu at scipy.org (Pearu Peterson) Date: Sat, 20 Sep 2003 13:13:56 +0300 (EEST) Subject: [SciPy-dev] Fortran code in scipy_base In-Reply-To: <20030919210853.DA6E23EA08@www.scipy.com> Message-ID: Hi, On Fri, 19 Sep 2003 travo at scipy.org wrote: > Update of /home/cvsroot/world/scipy_core/scipy_base > In directory scipy.org:/tmp/cvs-serv7557/scipy_core/scipy_base > > Modified Files: > function_base.py polynomial.py setup_scipy_base.py > Added Files: > futil.f > Log Message: > Added fortran utility functions to scipy_base as part of f2py demo. Adding Fortran codes to scipy_core contradicts with its design idea. Namely, the modules from scipy_core were factored from the earlier scipy package in order to resolve packaging issues of scipy and f2py2e -- see scipy/PACKAGERS.txt. The solution was that scipy_core f2py2e scipy should be installed strictly in the given order, so, scipy_core may contain only pure Python or C extension modules. So, could futil.f be moved to somewhere under scipy? Pearu From pearu at scipy.org Sat Sep 20 12:45:29 2003 From: pearu at scipy.org (Pearu Peterson) Date: Sat, 20 Sep 2003 19:45:29 +0300 (EEST) Subject: [SciPy-dev] Formal Review Process In-Reply-To: <20030920093604.GA29589@taliesen.caltech.edu> Message-ID: On Sat, 20 Sep 2003, Robert Kern wrote: > A Proposal for a SciPy Module Submission Review Process > ======================================================= > > :Author: Robert Kern (parts by the Boost_ group and others__) > :Version: 0.0 > :Date: 2003-09-20 > > The following proposal is inspired by (and indeed partly outright lifted from) > the `Boost Formal Review Process`_ and `Submission Process`_. It is written > using ReStructuredText_. Did you check out PEP-2? I think there are several things that the Boost document does not cover: maintaining a module, for one instance. > Rationale > --------- > > 1. The SciPy package requires a peer-review process to examine new module > submissions. The expertise of the CVS maintainer is limited, and the > number of fields for which one could write a SciPy module is large. > > 2. A formal procedure for the review process makes it easier for newcomers to > know exactly what they have to do to get their contribution into SciPy. > Furthermore, a formal procedure makes it easier for Enthought to parcel out > the work to non-Enthought people and even to transition SciPy to a wholly > community-driven project if that ever becomes a goal. Agreed. > 3. The Boost project had a similar problem, found a solution that worked for > them, and wrote it up all pretty, so why not steal it? [XXX - or something to > this effect] Hmm, I guess there are several projects with the same problem. Python, for instance. I would leave this item out and mention Boost, say, in acknowledgements (somehow so many references to Boost were annoying to me while reading this document;-). > Organization > ------------ > > Mailing List > ~~~~~~~~~~~~ > > Discussion of libraries under formal review shall take place on > scipy-dev at scipy.org . Actual reviews of a candidate module (see section `Reviews`_) > during its formal review period should begin their Subject header with > [REVIEW - ] for the convenience of the Review Manager. > > Reviewers who do not wish to subscribe to scipy-dev at scipy.org may use the GMane_ > WWW_ and Usenet_ interfaces to the mailing list. The WWW_ interface does not > allow posting. [XXX - to be honest, I don't know if the Usenet_ one allows > posting for this list, either.] AFAIK, only members can post to scipy-dev mailing list. > The Process for a Module Submitter > ---------------------------------- > > Determine Interest > ~~~~~~~~~~~~~~~~~~ > > Post a message to scipy-dev at scipy.org to see if there is interest in your > module. Describe what your module does and perhaps include a small snippet > of code that shows how one uses it. > > Preliminary Submission > ~~~~~~~~~~~~~~~~~~~~~~ > > Put your (possibly unfinished) module on a website and post its location to > scipy-dev at scipy.org . Contact the maintainers of www.scipy.org for a Zope > account if you need web space for your module. > > Submissions should follow the Code Submission Guidelines [XXX - this should > point to something useful. For now, look at the `Code Formatting Guidelines`_ > and the `Testing Guidelines`_.] See also PEP-7 and PEP-8. Along the lines in PEP-2, submission should include a rather complete documentation in order to get a better picture of the module when it will be finished. Such a document may contain also ideas/suggestions on future extensions of the module by proposing, say, function signatures. Such functions are allowed to be implemented even after the module is accepted to Scipy package. Since Scipy should provide a solid base for scientific work, all modules should have a mathematical background documented, most importantly, the formal definitions of used concepts. This would allow verifying the correctness of the results as well as avoids misusing the provided tools by the end users who might know/use slightly different definitions for some mathematical concepts. > Refinement > ~~~~~~~~~~ > > Discuss the module in scipy-dev at scipy.org . Fix bugs; add functionality; > change the interface; write docs/unit tests; do whatever comes out of the > discussion as necessary or beneficial to including the module in SciPy. > Update the preliminary submission until it is ready for formal submission. > > Submission for Review > ~~~~~~~~~~~~~~~~~~~~~ > > Package your module submission in a ZIP file or gzipped tar file such that > it can be unpacked in the scipy CVS directory [XXX - what if you need to > patch the setup.py file or scipy_distutils files? Diff file?]. I think we can work out a setup where no modifications to setup.py files are needed. I am thinking of adding (or using existing pre__init__.py file) a meta file into each module directory in scipy/Lib and scipy/sandbox that contains the following information about the module: - scipy level information - is it standalone or to be installed under scipy, etc. - state of the module (final,beta,...,unstable,ignore) - documentation - platform or software dependencies (setup.py would ignore the module when not running on the required platform or if some software is not installed) - etc. Then setup.py will scan directories and decides whether or not to include a module to a building/installation/distribution process based on the content of pre__init__.py file in these directories. Such a setup should be as simple to use as possible: adding a new module would require two steps: 1) unpack module to the sandbox directory 2) create a pre__init__.py file to the module directory (unless submitter already had done that) Pearu From oliphant at ee.byu.edu Sat Sep 20 14:08:34 2003 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sat, 20 Sep 2003 12:08:34 -0600 Subject: [SciPy-dev] Fortran code in scipy_base In-Reply-To: References: Message-ID: <3F6C97A2.3000608@ee.byu.edu> Pearu Peterson wrote: > Hi, > > On Fri, 19 Sep 2003 travo at scipy.org wrote: > > >>Update of /home/cvsroot/world/scipy_core/scipy_base >>In directory scipy.org:/tmp/cvs-serv7557/scipy_core/scipy_base >> >>Modified Files: >> function_base.py polynomial.py setup_scipy_base.py >>Added Files: >> futil.f >>Log Message: >>Added fortran utility functions to scipy_base as part of f2py demo. > > > Adding Fortran codes to scipy_core contradicts with its design idea. > > Namely, the modules from scipy_core were factored from the earlier scipy > package in order to resolve packaging issues of scipy and f2py2e -- see > scipy/PACKAGERS.txt. The solution was that > scipy_core > f2py2e > scipy > should be installed strictly in the given order, so, scipy_core may > contain only pure Python or C extension modules. > > So, could futil.f be moved to somewhere under scipy? > Yes, It definitely could. I'll do it. -Travis From oliphant at ee.byu.edu Sat Sep 20 14:23:06 2003 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sat, 20 Sep 2003 12:23:06 -0600 Subject: [SciPy-dev] SciPy 0.2 release Message-ID: <3F6C9B0A.6040805@ee.byu.edu> We are going to release SciPy 0.2 next week (Thursday 1:00pm Central Time). Please report any remaining outstanding issues that must be resolved before we tag the CVS tree with the release number. SciPy will still undergo improvements so SciPy 0.3 does not have to be far behind. Thanks for all of your reports so far. -Travis Oliphant From kern at caltech.edu Sat Sep 20 14:56:45 2003 From: kern at caltech.edu (Robert Kern) Date: Sat, 20 Sep 2003 11:56:45 -0700 Subject: [SciPy-dev] Re: Formal Review Process References: <20030920093604.GA29589@taliesen.caltech.edu> Message-ID: Pearu Peterson wrote: > > On Sat, 20 Sep 2003, Robert Kern wrote: > >> A Proposal for a SciPy Module Submission Review Process >> ======================================================= >> >> :Author: Robert Kern (parts by the Boost_ group and others__) >> :Version: 0.0 >> :Date: 2003-09-20 >> >> The following proposal is inspired by (and indeed partly outright lifted >> from) the `Boost Formal Review Process`_ and `Submission Process`_. It is >> written using ReStructuredText_. > > Did you check out PEP-2? I think there are several things that the Boost > document does not cover: maintaining a module, for one instance. Not until now, no. But you're right, it has good ideas to steal. The integrator/maintainer concepts are quite useful. And yes, maintenance needs to be addressed though I think it should be fully covered in a separate SciPy Development Guide or something and briefly summarized here. [snip] >> 3. The Boost project had a similar problem, found a solution that worked >> for >> them, and wrote it up all pretty, so why not steal it? [XXX - or >> something to this effect] > > Hmm, I guess there are several projects with the same problem. Python, for > instance. You're right, and I will look at the PEPs you suggest. Boost was just the most salient project in my head when I started thinking about this. > I would leave this item out and mention Boost, say, in > acknowledgements (somehow so many references to Boost were annoying to me > while reading this document;-). Sorry. When I commit larceny on so comprehensive a scale, I feel guilty and try to give advance warning. :-) >> Organization >> ------------ >> >> Mailing List >> ~~~~~~~~~~~~ >> >> Discussion of libraries under formal review shall take place on >> scipy-dev at scipy.org . Actual reviews of a candidate module (see section >> `Reviews`_) during its formal review period should begin their Subject >> header with >> [REVIEW - ] for the convenience of the Review Manager. >> >> Reviewers who do not wish to subscribe to scipy-dev at scipy.org may use the >> GMane_ WWW_ and Usenet_ interfaces to the mailing list. The WWW_ >> interface does not allow posting. [XXX - to be honest, I don't know if >> the Usenet_ one allows posting for this list, either.] > > AFAIK, only members can post to scipy-dev mailing list. I am posting this via GMane, so it must work. > > >> The Process for a Module Submitter >> ---------------------------------- >> >> Determine Interest >> ~~~~~~~~~~~~~~~~~~ >> >> Post a message to scipy-dev at scipy.org to see if there is interest in your >> module. Describe what your module does and perhaps include a small >> snippet of code that shows how one uses it. >> >> Preliminary Submission >> ~~~~~~~~~~~~~~~~~~~~~~ >> >> Put your (possibly unfinished) module on a website and post its location >> to scipy-dev at scipy.org . Contact the maintainers of www.scipy.org for a >> Zope account if you need web space for your module. >> >> Submissions should follow the Code Submission Guidelines [XXX - this >> should point to something useful. For now, look at the `Code Formatting >> Guidelines`_ and the `Testing Guidelines`_.] > > See also PEP-7 and PEP-8. > > Along the lines in PEP-2, submission should include a rather complete > documentation in order to get a better picture of the module when it will > be finished. Such a document may contain also ideas/suggestions on future > extensions of the module by proposing, say, function signatures. Such > functions are allowed to be implemented even after the module is accepted > to Scipy package. > > Since Scipy should provide a solid base for scientific work, all modules > should have a mathematical background documented, most importantly, > the formal definitions of used concepts. This would allow verifying the > correctness of the results as well as avoids misusing the provided tools > by the end users who might know/use slightly different definitions for > some mathematical concepts. Agreed. As alluded to above, I think all of these requirements should be wrapped up in a separate SciPy Development Guide since the information is useful to maintainers as well as potential contributors and because it is likely to be longer than this document is already. [snip] > Such a setup should be as simple to use as possible: adding a new module > would require two steps: > 1) unpack module to the sandbox directory > 2) create a pre__init__.py file to the module directory (unless > submitter already had done that) Nifty! +1. However, we still may want to deal with the issue of modules needing to modify, say, scipy_core/scipy_distutils/system_info.py . > Pearu -- Robert Kern kern at ugcs.caltech.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From eric at enthought.com Sun Sep 21 04:37:46 2003 From: eric at enthought.com (eric jones) Date: Sun, 21 Sep 2003 02:37:46 -0600 Subject: [SciPy-dev] extract and where Message-ID: <3F6D635A.3070008@enthought.com> extract has the following signature: >>> help(extract) Help on function extract in module __main__: extract(arr, mask) 1D array of those elements of ravel(arr) where ravel(mask) is true. This is backward from most of the Numeric routines where the condition (mask) array comes first (where, compress, and maybe others). Any objections to reversing these? Also, the one argument form of where() included in numarray would be useful to have in SciPy while we are still relying on Numeric. Should we wait until after 0.2, or go ahead and add this? From the numarray docs: Starting with numarray-0.6, where supports a one parameter form that is equivalent to nonzero but reads better: >>> where(arange(10) % 2) (array([1, 3, 5, 7, 9]),) # indices where expression is true < the following would have to use take() instead of indexing while we are still relying on Numeric> Like nonzero, the one parameter form of where can be used to do array indexing: >>> a = arange(10,20) >>> a[ where( a % 2 ) ] array([11, 13, 15, 17, 19]) Note that for array indices which are boolean arrays, using where is not necessary but is still OK: >>> a[(a % 2) != 0] array([11, 13, 15, 17, 19]) >>> a[where((a%2) != 0)] array([11, 13, 15, 17, 19]) eric From prabhu at aero.iitm.ernet.in Tue Sep 23 13:11:25 2003 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Tue, 23 Sep 2003 22:41:25 +0530 Subject: [SciPy-dev] Formal Review Process In-Reply-To: <20030920093604.GA29589@taliesen.caltech.edu> References: <20030920093604.GA29589@taliesen.caltech.edu> Message-ID: <16240.32445.169915.903497@monster.linux.in> Hi Robert, >>>>> "RK" == Robert Kern writes: RK> At both SciPy conferences, Eric emphasized the need for a RK> peer-review process for potential contributions to SciPy. I RK> recalled that the Boost community established a formal review RK> process for contributions to their set of libraries, and it RK> seems to work well for them. I got bored of writing a history RK> paper, so I looked at their process and wrote up a proposal RK> for SciPy (liberally using parts of the Boost documents where RK> noted). Nice! Just one suggestion. You probably need to mention that the code must be released under the BSD licence in order to be considered. I guess the MIT and the Python license are also fine but I don't know what exactly the official stance of the SciPy community is on this. cheers, prabhu From kern at vomit.ugcs.caltech.edu Tue Sep 23 16:03:32 2003 From: kern at vomit.ugcs.caltech.edu (Robert Kern) Date: Tue, 23 Sep 2003 13:03:32 -0700 Subject: [SciPy-dev] Formal Review Process In-Reply-To: <16240.32445.169915.903497@monster.linux.in> References: <20030920093604.GA29589@taliesen.caltech.edu> <16240.32445.169915.903497@monster.linux.in> Message-ID: <20030923200332.GA23532@vomit.ugcs.caltech.edu> On Tue, Sep 23, 2003 at 10:41:25PM +0530, Prabhu Ramachandran wrote: > Hi Robert, > > >>>>> "RK" == Robert Kern writes: > > RK> At both SciPy conferences, Eric emphasized the need for a > RK> peer-review process for potential contributions to SciPy. I > RK> recalled that the Boost community established a formal review > RK> process for contributions to their set of libraries, and it > RK> seems to work well for them. I got bored of writing a history > RK> paper, so I looked at their process and wrote up a proposal > RK> for SciPy (liberally using parts of the Boost documents where > RK> noted). > > Nice! Just one suggestion. You probably need to mention that the > code must be released under the BSD licence in order to be considered. > I guess the MIT and the Python license are also fine but I don't know > what exactly the official stance of the SciPy community is on this. To address this issue and other requirements placed on new modules, I have written up a SciPy Development Guide. Since I made most of it up myself, I'm waiting for Eric to review it before I release it into the wild. The Guide will be included by reference in the review process document. > cheers, > prabhu -- Robert Kern kern at ugcs.caltech.edu "In the fields of Hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From eric at enthought.com Thu Sep 25 03:48:46 2003 From: eric at enthought.com (eric jones) Date: Thu, 25 Sep 2003 01:48:46 -0600 Subject: [SciPy-dev] failure importing real from Numeric in io.array_import Message-ID: <3F729DDE.8010309@enthought.com> Just got the following on one of our internal tests when using the CVS of SciPy: Traceback (most recent call last): File "C:\wrk\proava\trunk\prototype\tests\test_depth_time.py", line 102, in ch eck_single_input data = read_depth_data() File "C:\wrk\proava\trunk\prototype\tests\test_depth_time.py", line 10, in rea d_depth_data data = io.read_array(os.path.join(d,'z2t_input_z_vp.txt'), File "C:\Python23\Lib\site-packages\scipy_base\ppimport.py", line 226, in __ge tattr__ module = self._ppimport_importer() File "C:\Python23\Lib\site-packages\scipy_base\ppimport.py", line 194, in _ppi mport_importer raise PPImportError,\ PPImportError: Traceback (most recent call last): File "C:\Python23\Lib\site-packages\scipy_base\ppimport.py", line 204, in _ppi mport_importer module = __import__(name,None,None,['*']) File "C:\Python23\lib\site-packages\scipy\io\__init__.py", line 11, in ? from array_import import * File "C:\Python23\Lib\site-packages\scipy\io\array_import.py", line 13, in ? from Numeric import array, take, concatenate, Float, asarray, real, imag ImportError: cannot import name real Changing this to from scipy_base import real, imag fixed the problems. I'm using Numeric 23.1. Should I be using the Numeric CVS, or should we change the import statement? thanks, eric From pearu at scipy.org Thu Sep 25 03:03:34 2003 From: pearu at scipy.org (Pearu Peterson) Date: Thu, 25 Sep 2003 02:03:34 -0500 (CDT) Subject: [SciPy-dev] failure importing real from Numeric in io.array_import In-Reply-To: <3F729DDE.8010309@enthought.com> Message-ID: On Thu, 25 Sep 2003, eric jones wrote: > from Numeric import array, take, concatenate, Float, asarray, real, imag > ImportError: cannot import name real > > Changing this to > > from scipy_base import real, imag > > fixed the problems. > > I'm using Numeric 23.1. Should I be using the Numeric CVS, or should we > change the import statement? We should change the import statement if we want to support Numeric starting from 21.0. In fact, transformation Numeric->numarray would be easier when using scipy_base in scipy modules. The only possible drawback that I can see (and live with) is that running tests in situ would require that scipy_base must be installed. Pearu From oliphant at ee.byu.edu Thu Sep 25 14:50:36 2003 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 25 Sep 2003 12:50:36 -0600 Subject: [SciPy-dev] Scheduled release Message-ID: <3F7338FC.2070703@ee.byu.edu> The release of scipy is coming along, but I think we need to postpone the release until Tuesday as I have found some bugs in special that need addressing. Comments? -Travis Oliphant From bhoel at web.de Thu Sep 25 16:06:57 2003 From: bhoel at web.de (Berthold Hoellmann) Date: Thu, 25 Sep 2003 22:06:57 +0200 Subject: [SciPy-dev] Broken .py files in current CVS Message-ID: Hello, Trying to install the latest CVS version of SciPy I get byte-compiling /usr/local/lib/python2.3/site-packages/kiva/agg/graphics_context.swig.py to graphics_context.swig.pyc File "/usr/local/lib/python2.3/site-packages/kiva/agg/graphics_context.swig.py", line 1 %pythoncode { ^ SyntaxError: invalid syntax byte-compiling /usr/local/lib/python2.3/site-packages/kiva/agg/graphics_context.py to graphics_context.pyc File "/usr/local/lib/python2.3/site-packages/kiva/agg/graphics_context.py", line 1 %pythoncode { ^ SyntaxError: invalid syntax byte-compiling /usr/local/lib/python2.3/site-packages/scipy/io/array_import.py to array_import.pyc Is there some cruft left from some kind of code generator? Regards Berthold -- bhoel at web.de / http://starship.python.net/crew/bhoel/ It is unlawful to use this email address for unsolicited ads (USC Title 47 Sec.227). I will assess a US$500 charge for reviewing and deleting each unsolicited ad. From kern at ugcs.caltech.edu Thu Sep 25 17:34:08 2003 From: kern at ugcs.caltech.edu (Robert Kern) Date: Thu, 25 Sep 2003 14:34:08 -0700 Subject: [SciPy-dev] Scheduled release In-Reply-To: <3F7338FC.2070703@ee.byu.edu> References: <3F7338FC.2070703@ee.byu.edu> Message-ID: <20030925213408.GA6115@schilling.ugcs.caltech.edu> On Thu, Sep 25, 2003 at 12:50:36PM -0600, Travis Oliphant wrote: > > The release of scipy is coming along, but I think we need to postpone > the release until Tuesday as I have found some bugs in special that need > addressing. > > Comments? I haven't been tracking CVS for the past week, but the last time I checked, there were some errors in stats/distributions.py . Specifically, triang.stats gives the same mean regardless of the loc parameter (the others may be wrong, too); uniform.ppf(x,...) for x == 0 or 1 gives 0 or 1 respectively despite the loc and shape parameters, and x in between give the right values. These bugs may or may not generalize to other distributions. Prioritize as you see fit. :-) > -Travis Oliphant -- Robert Kern kern at ugcs.caltech.edu "In the fields of Hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From oliphant at ee.byu.edu Thu Sep 25 19:20:45 2003 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 25 Sep 2003 17:20:45 -0600 Subject: [SciPy-dev] Broken .py files in current CVS In-Reply-To: References: Message-ID: <3F73784D.6030708@ee.byu.edu> Berthold Hoellmann wrote: > Hello, > > Trying to install the latest CVS version of SciPy I get > > byte-compiling /usr/local/lib/python2.3/site-packages/kiva/agg/graphics_context.swig.py to graphics_context.swig.pyc > File "/usr/local/lib/python2.3/site-packages/kiva/agg/graphics_context.swig.py", line 1 > %pythoncode { > ^ > SyntaxError: invalid syntax > byte-compiling /usr/local/lib/python2.3/site-packages/kiva/agg/graphics_context.py to graphics_context.pyc > File "/usr/local/lib/python2.3/site-packages/kiva/agg/graphics_context.py", line 1 > %pythoncode { > ^ > SyntaxError: invalid syntax > byte-compiling /usr/local/lib/python2.3/site-packages/scipy/io/array_import.py to array_import.pyc > > Is there some cruft left from some kind of code generator? > I see these errors too with Python 2.3, but the install actually works fine. These errors are not hindering the tests. The byte-compiler is trying to compile this file but probably shouldn't. I don't understand enough about what Eric is trying to do with agg here to comment further. -Travis O. > Regards > Berthold From arnd.baecker at web.de Fri Sep 26 02:17:54 2003 From: arnd.baecker at web.de (Arnd Baecker) Date: Fri, 26 Sep 2003 08:17:54 +0200 (CEST) Subject: [SciPy-dev] Broken .py files in current CVS In-Reply-To: <3F73784D.6030708@ee.byu.edu> References: <3F73784D.6030708@ee.byu.edu> Message-ID: Hi, I think I saw the same errors recently (but not in my last install, if I remember correctly) when I installed a new version over an existing old one. What helped me in this case (I think ;-) was to remove all scipy/kiva/agg stuff from the site-packages directory (I'd guess just kiva might be enough, but I did not play around with this ...). Maybe you can try this and do a `pyton setup.py install`? Best, Arnd From oliphant at ee.byu.edu Fri Sep 26 02:35:16 2003 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 26 Sep 2003 00:35:16 -0600 Subject: [SciPy-dev] Scheduled release In-Reply-To: <20030925213408.GA6115@schilling.ugcs.caltech.edu> References: <3F7338FC.2070703@ee.byu.edu> <20030925213408.GA6115@schilling.ugcs.caltech.edu> Message-ID: <3F73DE24.9050100@ee.byu.edu> Robert Kern wrote: > On Thu, Sep 25, 2003 at 12:50:36PM -0600, Travis Oliphant wrote: > >>The release of scipy is coming along, but I think we need to postpone >>the release until Tuesday as I have found some bugs in special that need >>addressing. >> >>Comments? > > > I haven't been tracking CVS for the past week, but the last time I checked, > there were some errors in stats/distributions.py . Specifically, triang.stats > gives the same mean regardless of the loc parameter (the others may be wrong, > too); uniform.ppf(x,...) for x == 0 or 1 gives 0 or 1 respectively despite the > loc and shape parameters, and x in between give the right values. These bugs may > or may not generalize to other distributions. Thanks Robert. The first bug was probably due to the incorrect formula for the mean of the triangular distribution. That has been fixed (it should change with different loc and scale parameters). The second ppf bug affected all distributions (it was in the base class). Thanks for finding it. It has been corrected. -Travis O. From joe at enthought.com Tue Sep 30 00:21:42 2003 From: joe at enthought.com (Joe Cooper) Date: Mon, 29 Sep 2003 23:21:42 -0500 Subject: [SciPy-dev] Testing scipy-dev Message-ID: <3F7904D6.6010306@enthought.com> Just a test of the mailing list, after a Mailman upgrade. Please ignore. Thanks.