[Numpy-discussion] Proposed Roadmap Overview

[Robert Kern]

On Sat, Feb 18, 2012 at 22:06, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Sat, Feb 18, 2012 at 2:03 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Sat, Feb 18, 2012 at 21:51, Matthew Brett <matthew.brett at gmail.com> wrote:
>>> On Sat, Feb 18, 2012 at 1:40 PM, Charles R Harris
>>> <charlesr.harris at gmail.com> wrote:
>>>>
>>>>
>>>> On Sat, Feb 18, 2012 at 2:17 PM, David Cournapeau <cournape at gmail.com>
>>>> wrote:
>>>>>
>>>>> On Sat, Feb 18, 2012 at 8:45 PM, Charles R Harris
>>>>> <charlesr.harris at gmail.com> wrote:
>>>>> >
>>>>> >
>>>>> > On Sat, Feb 18, 2012 at 1:39 PM, Matthew Brett <matthew.brett at gmail.com>
>>>>> > wrote:
>>>>> >>
>>>>> >> Hi,
>>>>> >>
>>>>> >> On Sat, Feb 18, 2012 at 12:35 PM, Charles R Harris
>>>>> >> <charlesr.harris at gmail.com> wrote:
>>>>> >> >
>>>>> >> >
>>>>> >> > On Sat, Feb 18, 2012 at 12:21 PM, Matthew Brett
>>>>> >> > <matthew.brett at gmail.com>
>>>>> >> > wrote:
>>>>> >> >>
>>>>> >> >> Hi.
>>>>> >> >>
>>>>> >> >> On Sat, Feb 18, 2012 at 12:18 AM, Christopher Jordan-Squire
>>>>> >> >> <cjordan1 at uw.edu> wrote:
>>>>> >> >> > On Fri, Feb 17, 2012 at 11:31 PM, Matthew Brett
>>>>> >> >> > <matthew.brett at gmail.com> wrote:
>>>>> >> >> >> Hi,
>>>>> >> >> >>
>>>>> >> >> >> On Fri, Feb 17, 2012 at 10:18 PM, Christopher Jordan-Squire
>>>>> >> >> >> <cjordan1 at uw.edu> wrote:
>>>>> >> >> >>> On Fri, Feb 17, 2012 at 8:30 PM, Sturla Molden
>>>>> >> >> >>> <sturla at molden.no>
>>>>> >> >> >>> wrote:
>>>>> >> >> >>>>
>>>>> >> >> >>>>
>>>>> >> >> >>>> Den 18. feb. 2012 kl. 05:01 skrev Jason Grout
>>>>> >> >> >>>> <jason-sage at creativetrax.com>:
>>>>> >> >> >>>>
>>>>> >> >> >>>>> On 2/17/12 9:54 PM, Sturla Molden wrote:
>>>>> >> >> >>>>>> We would have to write a C++ programming tutorial that is
>>>>> >> >> >>>>>> based
>>>>> >> >> >>>>>> on
>>>>> >> >> >>>>>> Pyton knowledge instead of C knowledge.
>>>>> >> >> >>>>>
>>>>> >> >> >>>>> I personally would love such a thing.  It's been a while since
>>>>> >> >> >>>>> I
>>>>> >> >> >>>>> did
>>>>> >> >> >>>>> anything nontrivial on my own in C++.
>>>>> >> >> >>>>>
>>>>> >> >> >>>>
>>>>> >> >> >>>> One example: How do we code multiple return values?
>>>>> >> >> >>>>
>>>>> >> >> >>>> In Python:
>>>>> >> >> >>>> - Return a tuple.
>>>>> >> >> >>>>
>>>>> >> >> >>>> In C:
>>>>> >> >> >>>> - Use pointers (evilness)
>>>>> >> >> >>>>
>>>>> >> >> >>>> In C++:
>>>>> >> >> >>>> - Return a std::tuple, as you would in Python.
>>>>> >> >> >>>> - Use references, as you would in Fortran or Pascal.
>>>>> >> >> >>>> - Use pointers, as you would in C.
>>>>> >> >> >>>>
>>>>> >> >> >>>> C++ textbooks always pick the last...
>>>>> >> >> >>>>
>>>>> >> >> >>>> I would show the first and the second method, and perhaps
>>>>> >> >> >>>> intentionally forget the last.
>>>>> >> >> >>>>
>>>>> >> >> >>>> Sturla
>>>>> >> >> >>>>
>>>>> >> >> >>
>>>>> >> >> >>> On the flip side, cython looked pretty...but I didn't get the
>>>>> >> >> >>> performance gains I wanted, and had to spend a lot of time
>>>>> >> >> >>> figuring
>>>>> >> >> >>> out if it was cython, needing to add types, buggy support for
>>>>> >> >> >>> numpy,
>>>>> >> >> >>> or actually the algorithm.
>>>>> >> >> >>
>>>>> >> >> >> At the time, was the numpy support buggy?  I personally haven't
>>>>> >> >> >> had
>>>>> >> >> >> many problems with Cython and numpy.
>>>>> >> >> >>
>>>>> >> >> >
>>>>> >> >> > It's not that the support WAS buggy, it's that it wasn't clear to
>>>>> >> >> > me
>>>>> >> >> > what was going on and where my performance bottleneck was. Even
>>>>> >> >> > after
>>>>> >> >> > microbenchmarking with ipython, using timeit and prun, and using
>>>>> >> >> > the
>>>>> >> >> > cython code visualization tool. Ultimately I don't think it was
>>>>> >> >> > cython, so perhaps my comment was a bit unfair. But it was
>>>>> >> >> > unfortunately difficult to verify that. Of course, as you say,
>>>>> >> >> > diagnosing and solving such issues would become easier to resolve
>>>>> >> >> > with
>>>>> >> >> > more cython experience.
>>>>> >> >> >
>>>>> >> >> >>> The C files generated by cython were
>>>>> >> >> >>> enormous and difficult to read. They really weren't meant for
>>>>> >> >> >>> human
>>>>> >> >> >>> consumption.
>>>>> >> >> >>
>>>>> >> >> >> Yes, it takes some practice to get used to what Cython will do,
>>>>> >> >> >> and
>>>>> >> >> >> how to optimize the output.
>>>>> >> >> >>
>>>>> >> >> >>> As Sturla has said, regardless of the quality of the
>>>>> >> >> >>> current product, it isn't stable.
>>>>> >> >> >>
>>>>> >> >> >> I've personally found it more or less rock solid.  Could you say
>>>>> >> >> >> what
>>>>> >> >> >> you mean by "it isn't stable"?
>>>>> >> >> >>
>>>>> >> >> >
>>>>> >> >> > I just meant what Sturla said, nothing more:
>>>>> >> >> >
>>>>> >> >> > "Cython is still 0.16, it is still unfinished. We cannot base
>>>>> >> >> > NumPy
>>>>> >> >> > on
>>>>> >> >> > an unfinished compiler."
>>>>> >> >>
>>>>> >> >> Y'all mean, it has a zero at the beginning of the version number and
>>>>> >> >> it is still adding new features?  Yes, that is correct, but it seems
>>>>> >> >> more reasonable to me to phrase that as 'active development' rather
>>>>> >> >> than 'unstable', because they take considerable care to be backwards
>>>>> >> >> compatible, have a large automated Cython test suite, and a major
>>>>> >> >> stress-tester in the Sage test suite.
>>>>> >> >>
>>>>> >> >
>>>>> >> > Matthew,
>>>>> >> >
>>>>> >> > No one in their right mind would build a large performance library
>>>>> >> > using
>>>>> >> > Cython, it just isn't the right tool. For what it was designed for -
>>>>> >> > wrapping existing c code or writing small and simple things close to
>>>>> >> > Python
>>>>> >> > - it does very well, but it was never designed for making core C/C++
>>>>> >> > libraries and in that role it just gets in the way.
>>>>> >>
>>>>> >> I believe the proposal is to refactor the lowest levels in pure C and
>>>>> >> move the some or most of the library superstructure to Cython.
>>>>> >
>>>>> >
>>>>> > Go for it.
>>>>>
>>>>> The proposal of moving to a core C + cython has been discussed by
>>>>> multiple contributors. It is certainly a valid proposal. *I* have
>>>>> worked on this (npymath, separate compilation), although certainly not
>>>>> as much as I would have wanted to. I think much can be done in that
>>>>> vein. Using the "shut up if you don't do it" is a straw man (and
>>>>> uncalled for).
>>>>
>>>>
>>>> OK, I was annoyed.
>>>
>>> By what?
>>
>> Your misunderstanding of what was being discussed. The proposal being
>> discussed is implementing the core of numpy in C++, wrapped in C to be
>> usable as a C library that other extensions can use, and then exposed
>> to Python in an unspecified way. Cython was raised as an alternative
>> for this core, but as Chuck points out, it doesn't really fit. Your
>> assertion that what was being discussed was putting the core in C and
>> using Cython to wrap it was simply a non-sequitur. Discussion of
>> alternatives is fine. You weren't doing that.
>
> You read David's email?  Was he also being annoying?

Not really, because he was responding on-topic to the bizarro-branch
of the conversation that you spawned about the merits of moving from
hand-written C extensions to a Cython-wrapped C library. Whatever
annoyance his email might inspire is your fault, not his. The
discussion was about whether to use C++ or Cython for the core. Chuck
argued that Cython was not a suitable implementation language for the
core. You responded that his objections to Cython didn't apply to what
you thought was being discussed, using Cython to wrap a pure-C
library. As Pauli (Wolfgang, not our Pauli) once phrased it, you were
"not even wrong". It's hard to respond coherently to someone who is
breaking the fundamental expectations of discourse. Even I had to
stare at the thread for a few minutes to figure out where things went
off the rails.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco