[Numpy-discussion] Proposed Roadmap Overview

[Matthew Brett]

Hi,

On Sat, Feb 18, 2012 at 2:20 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Sat, Feb 18, 2012 at 22:06, Matthew Brett <matthew.brett at gmail.com> wrote:
>> Hi,
>>
>> On Sat, Feb 18, 2012 at 2:03 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>> On Sat, Feb 18, 2012 at 21:51, Matthew Brett <matthew.brett at gmail.com> wrote:
>>>> On Sat, Feb 18, 2012 at 1:40 PM, Charles R Harris
>>>> <charlesr.harris at gmail.com> wrote:
>>>>>
>>>>>
>>>>> On Sat, Feb 18, 2012 at 2:17 PM, David Cournapeau <cournape at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> On Sat, Feb 18, 2012 at 8:45 PM, Charles R Harris
>>>>>> <charlesr.harris at gmail.com> wrote:
>>>>>> >
>>>>>> >
>>>>>> > On Sat, Feb 18, 2012 at 1:39 PM, Matthew Brett <matthew.brett at gmail.com>
>>>>>> > wrote:
>>>>>> >>
>>>>>> >> Hi,
>>>>>> >>
>>>>>> >> On Sat, Feb 18, 2012 at 12:35 PM, Charles R Harris
>>>>>> >> <charlesr.harris at gmail.com> wrote:
>>>>>> >> >
>>>>>> >> >
>>>>>> >> > On Sat, Feb 18, 2012 at 12:21 PM, Matthew Brett
>>>>>> >> > <matthew.brett at gmail.com>
>>>>>> >> > wrote:
>>>>>> >> >>
>>>>>> >> >> Hi.
>>>>>> >> >>
>>>>>> >> >> On Sat, Feb 18, 2012 at 12:18 AM, Christopher Jordan-Squire
>>>>>> >> >> <cjordan1 at uw.edu> wrote:
>>>>>> >> >> > On Fri, Feb 17, 2012 at 11:31 PM, Matthew Brett
>>>>>> >> >> > <matthew.brett at gmail.com> wrote:
>>>>>> >> >> >> Hi,
>>>>>> >> >> >>
>>>>>> >> >> >> On Fri, Feb 17, 2012 at 10:18 PM, Christopher Jordan-Squire
>>>>>> >> >> >> <cjordan1 at uw.edu> wrote:
>>>>>> >> >> >>> On Fri, Feb 17, 2012 at 8:30 PM, Sturla Molden
>>>>>> >> >> >>> <sturla at molden.no>
>>>>>> >> >> >>> wrote:
>>>>>> >> >> >>>>
>>>>>> >> >> >>>>
>>>>>> >> >> >>>> Den 18. feb. 2012 kl. 05:01 skrev Jason Grout
>>>>>> >> >> >>>> <jason-sage at creativetrax.com>:
>>>>>> >> >> >>>>
>>>>>> >> >> >>>>> On 2/17/12 9:54 PM, Sturla Molden wrote:
>>>>>> >> >> >>>>>> We would have to write a C++ programming tutorial that is
>>>>>> >> >> >>>>>> based
>>>>>> >> >> >>>>>> on
>>>>>> >> >> >>>>>> Pyton knowledge instead of C knowledge.
>>>>>> >> >> >>>>>
>>>>>> >> >> >>>>> I personally would love such a thing.  It's been a while since
>>>>>> >> >> >>>>> I
>>>>>> >> >> >>>>> did
>>>>>> >> >> >>>>> anything nontrivial on my own in C++.
>>>>>> >> >> >>>>>
>>>>>> >> >> >>>>
>>>>>> >> >> >>>> One example: How do we code multiple return values?
>>>>>> >> >> >>>>
>>>>>> >> >> >>>> In Python:
>>>>>> >> >> >>>> - Return a tuple.
>>>>>> >> >> >>>>
>>>>>> >> >> >>>> In C:
>>>>>> >> >> >>>> - Use pointers (evilness)
>>>>>> >> >> >>>>
>>>>>> >> >> >>>> In C++:
>>>>>> >> >> >>>> - Return a std::tuple, as you would in Python.
>>>>>> >> >> >>>> - Use references, as you would in Fortran or Pascal.
>>>>>> >> >> >>>> - Use pointers, as you would in C.
>>>>>> >> >> >>>>
>>>>>> >> >> >>>> C++ textbooks always pick the last...
>>>>>> >> >> >>>>
>>>>>> >> >> >>>> I would show the first and the second method, and perhaps
>>>>>> >> >> >>>> intentionally forget the last.
>>>>>> >> >> >>>>
>>>>>> >> >> >>>> Sturla
>>>>>> >> >> >>>>
>>>>>> >> >> >>
>>>>>> >> >> >>> On the flip side, cython looked pretty...but I didn't get the
>>>>>> >> >> >>> performance gains I wanted, and had to spend a lot of time
>>>>>> >> >> >>> figuring
>>>>>> >> >> >>> out if it was cython, needing to add types, buggy support for
>>>>>> >> >> >>> numpy,
>>>>>> >> >> >>> or actually the algorithm.
>>>>>> >> >> >>
>>>>>> >> >> >> At the time, was the numpy support buggy?  I personally haven't
>>>>>> >> >> >> had
>>>>>> >> >> >> many problems with Cython and numpy.
>>>>>> >> >> >>
>>>>>> >> >> >
>>>>>> >> >> > It's not that the support WAS buggy, it's that it wasn't clear to
>>>>>> >> >> > me
>>>>>> >> >> > what was going on and where my performance bottleneck was. Even
>>>>>> >> >> > after
>>>>>> >> >> > microbenchmarking with ipython, using timeit and prun, and using
>>>>>> >> >> > the
>>>>>> >> >> > cython code visualization tool. Ultimately I don't think it was
>>>>>> >> >> > cython, so perhaps my comment was a bit unfair. But it was
>>>>>> >> >> > unfortunately difficult to verify that. Of course, as you say,
>>>>>> >> >> > diagnosing and solving such issues would become easier to resolve
>>>>>> >> >> > with
>>>>>> >> >> > more cython experience.
>>>>>> >> >> >
>>>>>> >> >> >>> The C files generated by cython were
>>>>>> >> >> >>> enormous and difficult to read. They really weren't meant for
>>>>>> >> >> >>> human
>>>>>> >> >> >>> consumption.
>>>>>> >> >> >>
>>>>>> >> >> >> Yes, it takes some practice to get used to what Cython will do,
>>>>>> >> >> >> and
>>>>>> >> >> >> how to optimize the output.
>>>>>> >> >> >>
>>>>>> >> >> >>> As Sturla has said, regardless of the quality of the
>>>>>> >> >> >>> current product, it isn't stable.
>>>>>> >> >> >>
>>>>>> >> >> >> I've personally found it more or less rock solid.  Could you say
>>>>>> >> >> >> what
>>>>>> >> >> >> you mean by "it isn't stable"?
>>>>>> >> >> >>
>>>>>> >> >> >
>>>>>> >> >> > I just meant what Sturla said, nothing more:
>>>>>> >> >> >
>>>>>> >> >> > "Cython is still 0.16, it is still unfinished. We cannot base
>>>>>> >> >> > NumPy
>>>>>> >> >> > on
>>>>>> >> >> > an unfinished compiler."
>>>>>> >> >>
>>>>>> >> >> Y'all mean, it has a zero at the beginning of the version number and
>>>>>> >> >> it is still adding new features?  Yes, that is correct, but it seems
>>>>>> >> >> more reasonable to me to phrase that as 'active development' rather
>>>>>> >> >> than 'unstable', because they take considerable care to be backwards
>>>>>> >> >> compatible, have a large automated Cython test suite, and a major
>>>>>> >> >> stress-tester in the Sage test suite.
>>>>>> >> >>
>>>>>> >> >
>>>>>> >> > Matthew,
>>>>>> >> >
>>>>>> >> > No one in their right mind would build a large performance library
>>>>>> >> > using
>>>>>> >> > Cython, it just isn't the right tool. For what it was designed for -
>>>>>> >> > wrapping existing c code or writing small and simple things close to
>>>>>> >> > Python
>>>>>> >> > - it does very well, but it was never designed for making core C/C++
>>>>>> >> > libraries and in that role it just gets in the way.
>>>>>> >>
>>>>>> >> I believe the proposal is to refactor the lowest levels in pure C and
>>>>>> >> move the some or most of the library superstructure to Cython.
>>>>>> >
>>>>>> >
>>>>>> > Go for it.
>>>>>>
>>>>>> The proposal of moving to a core C + cython has been discussed by
>>>>>> multiple contributors. It is certainly a valid proposal. *I* have
>>>>>> worked on this (npymath, separate compilation), although certainly not
>>>>>> as much as I would have wanted to. I think much can be done in that
>>>>>> vein. Using the "shut up if you don't do it" is a straw man (and
>>>>>> uncalled for).
>>>>>
>>>>>
>>>>> OK, I was annoyed.
>>>>
>>>> By what?
>>>
>>> Your misunderstanding of what was being discussed. The proposal being
>>> discussed is implementing the core of numpy in C++, wrapped in C to be
>>> usable as a C library that other extensions can use, and then exposed
>>> to Python in an unspecified way. Cython was raised as an alternative
>>> for this core, but as Chuck points out, it doesn't really fit. Your
>>> assertion that what was being discussed was putting the core in C and
>>> using Cython to wrap it was simply a non-sequitur. Discussion of
>>> alternatives is fine. You weren't doing that.
>>
>> You read David's email?  Was he also being annoying?
>
> Not really, because he was responding on-topic to the bizarro-branch
> of the conversation that you spawned about the merits of moving from
> hand-written C extensions to a Cython-wrapped C library. Whatever
> annoyance his email might inspire is your fault, not his. The
> discussion was about whether to use C++ or Cython for the core. Chuck
> argued that Cython was not a suitable implementation language for the
> core. You responded that his objections to Cython didn't apply to what
> you thought was being discussed, using Cython to wrap a pure-C
> library. As Pauli (Wolfgang, not our Pauli) once phrased it, you were
> "not even wrong". It's hard to respond coherently to someone who is
> breaking the fundamental expectations of discourse. Even I had to
> stare at the thread for a few minutes to figure out where things went
> off the rails.

I'm sorry but this seems to me to be aggressive, offensive, and unjust.

The discussion was, from the beginning, mainly about the relative
benefits of rewriting the core with C / Cython, or C++.

I don't think anyone was proposing writing every line of the numpy
core in Cython.  Ergo (sorry to use the debating term), the proposal
to use Cython was always to take some of the higher level code out of
C and leave some of it in C.   It does indeed make the debate
ridiculous to oppose a proposal that no-one has made.

Now I am sure it is obvious to you, that the proposal to refactor the
current C code to into low-level C libraries, and higher level Cython
wrappers, is absurd and off the table.  It isn't obvious to me.  I
don't think I broke a fundamental rule of polite discourse to clarify
that is what I meant,

Best,

Matthew