[Numpy-discussion] Proposed Roadmap Overview

[Benjamin Root]

On Sat, Feb 18, 2012 at 2:45 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Sat, Feb 18, 2012 at 1:39 PM, Matthew Brett <matthew.brett at gmail.com>wrote:
>
>> Hi,
>>
>> On Sat, Feb 18, 2012 at 12:35 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> >
>> > On Sat, Feb 18, 2012 at 12:21 PM, Matthew Brett <
>> matthew.brett at gmail.com>
>> > wrote:
>> >>
>> >> Hi.
>> >>
>> >> On Sat, Feb 18, 2012 at 12:18 AM, Christopher Jordan-Squire
>> >> <cjordan1 at uw.edu> wrote:
>> >> > On Fri, Feb 17, 2012 at 11:31 PM, Matthew Brett
>> >> > <matthew.brett at gmail.com> wrote:
>> >> >> Hi,
>> >> >>
>> >> >> On Fri, Feb 17, 2012 at 10:18 PM, Christopher Jordan-Squire
>> >> >> <cjordan1 at uw.edu> wrote:
>> >> >>> On Fri, Feb 17, 2012 at 8:30 PM, Sturla Molden <sturla at molden.no>
>> >> >>> wrote:
>> >> >>>>
>> >> >>>>
>> >> >>>> Den 18. feb. 2012 kl. 05:01 skrev Jason Grout
>> >> >>>> <jason-sage at creativetrax.com>:
>> >> >>>>
>> >> >>>>> On 2/17/12 9:54 PM, Sturla Molden wrote:
>> >> >>>>>> We would have to write a C++ programming tutorial that is based
>> on
>> >> >>>>>> Pyton knowledge instead of C knowledge.
>> >> >>>>>
>> >> >>>>> I personally would love such a thing.  It's been a while since I
>> did
>> >> >>>>> anything nontrivial on my own in C++.
>> >> >>>>>
>> >> >>>>
>> >> >>>> One example: How do we code multiple return values?
>> >> >>>>
>> >> >>>> In Python:
>> >> >>>> - Return a tuple.
>> >> >>>>
>> >> >>>> In C:
>> >> >>>> - Use pointers (evilness)
>> >> >>>>
>> >> >>>> In C++:
>> >> >>>> - Return a std::tuple, as you would in Python.
>> >> >>>> - Use references, as you would in Fortran or Pascal.
>> >> >>>> - Use pointers, as you would in C.
>> >> >>>>
>> >> >>>> C++ textbooks always pick the last...
>> >> >>>>
>> >> >>>> I would show the first and the second method, and perhaps
>> >> >>>> intentionally forget the last.
>> >> >>>>
>> >> >>>> Sturla
>> >> >>>>
>> >> >>
>> >> >>> On the flip side, cython looked pretty...but I didn't get the
>> >> >>> performance gains I wanted, and had to spend a lot of time figuring
>> >> >>> out if it was cython, needing to add types, buggy support for
>> numpy,
>> >> >>> or actually the algorithm.
>> >> >>
>> >> >> At the time, was the numpy support buggy?  I personally haven't had
>> >> >> many problems with Cython and numpy.
>> >> >>
>> >> >
>> >> > It's not that the support WAS buggy, it's that it wasn't clear to me
>> >> > what was going on and where my performance bottleneck was. Even after
>> >> > microbenchmarking with ipython, using timeit and prun, and using the
>> >> > cython code visualization tool. Ultimately I don't think it was
>> >> > cython, so perhaps my comment was a bit unfair. But it was
>> >> > unfortunately difficult to verify that. Of course, as you say,
>> >> > diagnosing and solving such issues would become easier to resolve
>> with
>> >> > more cython experience.
>> >> >
>> >> >>> The C files generated by cython were
>> >> >>> enormous and difficult to read. They really weren't meant for human
>> >> >>> consumption.
>> >> >>
>> >> >> Yes, it takes some practice to get used to what Cython will do, and
>> >> >> how to optimize the output.
>> >> >>
>> >> >>> As Sturla has said, regardless of the quality of the
>> >> >>> current product, it isn't stable.
>> >> >>
>> >> >> I've personally found it more or less rock solid.  Could you say
>> what
>> >> >> you mean by "it isn't stable"?
>> >> >>
>> >> >
>> >> > I just meant what Sturla said, nothing more:
>> >> >
>> >> > "Cython is still 0.16, it is still unfinished. We cannot base NumPy
>> on
>> >> > an unfinished compiler."
>> >>
>> >> Y'all mean, it has a zero at the beginning of the version number and
>> >> it is still adding new features?  Yes, that is correct, but it seems
>> >> more reasonable to me to phrase that as 'active development' rather
>> >> than 'unstable', because they take considerable care to be backwards
>> >> compatible, have a large automated Cython test suite, and a major
>> >> stress-tester in the Sage test suite.
>> >>
>> >
>> > Matthew,
>> >
>> > No one in their right mind would build a large performance library using
>> > Cython, it just isn't the right tool. For what it was designed for -
>> > wrapping existing c code or writing small and simple things close to
>> Python
>> > - it does very well, but it was never designed for making core C/C++
>> > libraries and in that role it just gets in the way.
>>
>> I believe the proposal is to refactor the lowest levels in pure C and
>> move the some or most of the library superstructure to Cython.
>>
>
> Go for it.
>
> Chuck
>
>
>
Just a couple of quick questions:

1.) What is the status of the refactoring that was done for IronPython a
couple of years ago?  The last I heard, the branches diverged too much for
merging the work back into numpy.  Are there lessons that can be learned
from that experience that can be applied to whatever happens next?

2.) My personal preference is an incremental refactor over to C++ using
STL, however, I have to be realistic.  First, the exception issue is
problematic (unsolvable? I don't know). Second, one of Numpy/Scipy's
greatest strengths is the relative ease it has in interfacing with BLAS,
ATLAS, mkl and other optimizations.  Will this still be possible from a C++
(or anything else) core? Third, I am only familiar with STL on gcc. Are
there any subtle differences in implementations of STL in MSVC or any other
compilers.  Pointers are hard to mess up, in cross-platform ways.

3.) Will memory-mapped arrays still be possible after the refactor?  I am
not familiar with the implementation, but I am a big netcdf/hdf user and
mem-mapped arrays are important to me.

4.) Wouldn't depending on Cython create a circular dependency? Can you
build Cython without numpy-devel? (I never tried.  I have only used
packaged Cython). Also, because Cython generates code to compile, is there
a possibility of producing different ABIs depending upon the combinations
of numpy and cython versions (even if unintentional)?  How difficult will
it be for distro maintainers to package numpy and its extensions? How
difficult will it be for users of Macs and Windows who may try combining
different versions? Honest questions because I have never had more than a
cursory exposure to Cython.

Ben Root
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20120218/ddee1e7f/attachment.html>