On Mon, Aug 22, 2016 at 3:26 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
> To clarify, I'm coming from the case where I
> didn't read the whole tutorial/docs before being faced with pyx in the
> projects I previously mentioned, while on tight turn around time - I was not
> able to grok in that context.

Understood--I am short on time too. Though your self-admitted lack of
familiarity doesn't bolster your argument that Cython's doing it
wrong, we need this new feature.

Bare in mind that was then, and while I'm not Cython developer level I do have a better idea of how Cython works now, possibly better than your average user.  I've thought about this problem and the state of the overall C++ python binding capability alot as I've had to reevaluate it possibly more than most, and I've used them more deeply than playcode.



> For myself and any other ML user who comes across this thread - can you list
> a few libraries that do things the right way?



 
> My point was that multifile multi-level wrapper that I mentioned earlier -
> if you're saying that those projects did Cython extensions wrong, then I'm
> incorrect at faulting Cython and should fault the libraries using it.  I
> didn't say staying close to python caused $blurb.
>
> I don't know in a situation as confusing as to all the binding projects if
> this should be taken as validation of philosophy either - I think it is
> reasonable to consider the attrition of these projects as a function of
> manpower, number of early on project supporters/authors, and if a project
> (like sage) indirectly, through dependency,  kept the project alive.  And
> good old fashioned luck.  I noted most of them don't use distutils and
> something custom but less capable instead which maybe plays a roll in how
> mature/usable/smalltime they where/are.

Certainly the success of a project depends on many external factors,
and even raw luck plays a part. But the approach and philosophy taken
to attack a problem and guard its API (and the users/contributors that
such decisions attract or repel) can't be discounted either,
especially when taken over a long timeframe.

But if anyone wants to believe that Cython's become popular because of
pure luck despite wrongheaded guiding principles or philosophies,
it'll be difficult to persuade them otherwise.

You snidely play with extremes, I never said pure luck was the main factor - I listed several factors of very high significance.  Further, Cython branched off of pyrex - why do you think that project failed (you might actually know)?  Did your philosophy carry separate the two or was it because you needed things to work (for sage), had a larger set of smart developers and improve it faster than pyrex would allow?  I think it's the bustling of life that got Cython where it is, I don't believe it's philosophy in this case - I think it was lack of feature competitiveness and Cython probably worked better (more bugs/cases fixed). 
 

>> I agree that any efforts to trying to parsing C++ without building on
>> an actual compiler are fraught with danger. That's not the case with
>> generating C++ code, which is the direction we're going. In
>> particular, our goal is to understand C++ enough to invoke it, which
>> allows us to be much less pedantic.
>
> I understand and agree with the logic in stating it's a less complicated
> goal but what comparable success stories exist?  I strongly think "devils in
> the details" in correctly making that work and that they will be tough
> solvable problems.  And then you're going and promising on unfamiliar
> territory.  But what the ultimate takeaway for me is that you won't have it
> ready in any near term.  Do you have the skills and resources to implement
> this in under 2 years?  And then the other question is are you and the team
> reasonably confident you will have it working and usable by then.  Otherwise
> you are not being pragmatic.
>
> On the other hand, if it was reasonably simple as many of your other points
> in future emails point out, I'd really like to know why you hadn't addressed
> them earlier.

Other higher priority items for limited resources. And non-type
template args are not necessarily that simple given the way things are
structured now.

There's always limited resources and priorities but hen it sounds like it's going to be a very (ever)? long time until everything is working great unless the need is becoming a clear driver or this email thread has reassigned priorities.  Do you think you can sketch out a time table for the known issues?  Maybe not in this thread but I'd appreciate a back of the envelope calculation - realistic time of usefulness is part of being pragmatic.
 

You're missing the point of (3). The fact that we're generating C
code, and not fortran or directly assembly, should mostly be an
implementation detail. It's not realistic to embed snippets without
caring about the surrounding context in all but the simplest of cases.
And there'd be feature creep here--you're in the middle of a C snippet
and want to report an error, or access a Python object, or ...

I know you took a different philosophy than pyrex, but I don't understand why.  In that sage days presentation, you mentioned pyrex chose to distinguish itself as a way to write c extensions, emphasizing it (I don't know to the degree if it offered what wish cython offered, doubted).  It's pretty clear to me you wanted mostly accelerated python with some c-trampolining capability through externally defined functions.  I *really* think you should remember your root's goals though.

 

>> The escape hatch is to wrap the C++ in an actual C++ file and invoke
>> the wrapping. Typically this is the minority of one's code--if it's
>> the "whole library" then you probably have an API that's only
>> understandable to someone well versed in C++ anyways. You've given a
>> single example (non-type template arguments) that we would like to
>> support that's blocking you.
>
> My lack of examples is due to insufficient time playing with Cython - I hit
> nonstarters so I stop and abandon; as I said, to date, Cython has never been
> able to solve my C++ problems and none of them seem extraordinary.  I think
> you've got alot of more unknown-unknowns here than you give credit to but we
> can't discover that until you at least fix that template bug (properly).

It would be helpful for you to enumerate the (implied many)
nonstarters and problems you've had, to at least get the
known-unknowns out on the table.

Honestly I can't remember the other 2 times I reevaluated its applicability (we're talking a few days each over the last ~10 years), it was too long ago and capability of Cython has grown right?  My answer in each of those times, originally coming from SWIG, was boost.python... and in the mean time it's looking like it might be pybind11 (I don't have the luxury of waiting for what I'm working on, but I'm not forgetting about Cython either).
 

> I'm still not looking forward to forward declaring every
> identifier/function/whatever from C++ land in Cython though

This is largely a separate issue, and I agree a big pain point.

> and I still
> strongly dislike that there's no single source way of doing Cython with
> something like a kernel/ufunc that needs to escape to C/C++.  This makes
> doing something like the mako based templates I mentioned in the OP email
> much more cumbersome/hard and Cython would provide no built in mechanism
> (like inline/inline_module) for making that work.

The issue here that you want to inline C code snippets into your
inlined Cython code snippet?

Multiple files is more a PITA to making the mako-templated strategy work - I didn't say impossible, just more a PITA. I learned the JIT-module compilation technique from OpenCL/PyCUDA and found it makes a metric ton of sense to do, the performance boost can be massive from using -DDEFINES and embedding runtime variables/constants right into C (recall the point of Cython seems to be for speed :-). This is one of the reasons why I'd like to stick with a python driven way of doing C++ bindings (which hopefully can use C++ directly).   It's alot easier for a single file to work and simply pass a string to something like inline which does all the file writing, and module placement.
 
I'm not saying that these projects died because of the string-based
approach they took, rather that if this "embed strings" approach were
so critical, so superior, it should at least kept one of them alive.
Or a new project could have formed around this approach (e.g. letting
all the executable code be C++, with Python syntax for the structure,
could be an interesting point in the design space). It has its pros
and cons. I think for Cython it's the wrong direction. But I'm in
favor of letting many flowers bloom--and we're in luck that these are
all open source to boot.

I think it's because of their as-is-limitations, not of the technique they implemented but as a whole system.  I looked at the source for pyinline the other day and it looked like it only supports primitive types and would choke on C++ for instance and doesn't really offer convenience to extract data from python objects like Cython can so easily (the whole system, including it's custom C (no ++) lexer is ~700 lines).  I tend to think of a few/most of them are just passed the working proof of concept stage and were significantly lacking in overall capability.  Most python projects (on PyPi) are single contributors with short lifespans too which seems to be the case for those projects.  I'm not sure when/how Pyrex started out (1 guy at Uni who set up a full website?) but how did it happen with Cython?


> I watched one of your old talks for Sage Days 29 and at the bottom of a
> slide you have "Cython is a very pragmatic project, driven by user needs".
> I'm calling foul.  Go watch that video again and tell me what's changed
> since 2011.

Embedding code is not a need, it's a means to an end. This is why I've
been asking for concrete features that are the most important to try
to support.

I have another idea/iteration to run by you then.  One of your chief quibbles, although I don't think it's your underlying one, is Cython must understand what's going on.  So how about we support a block of C/C++ code as a proper construct.  Same name but now, I guess braces maybe get involved (since semicolons do too).  Then to transform (mangle) the nested identifiers in accordance with scoping/shadowing rules we pull in the python bindings to llvm/Clang to parse that complicated code, work with the AST (or further down the rabbit-hole) appropriately and spit it back out.  Obviously you could detect and analyze embedded statements to your hearts content to decide to do something smart about that (you mentioned exception handling and return statement as concerning items).

The potential hazards here are that LLVM/Clang is brought in (LLVM is already on ~ everybody's linux boxes as a graphics shader dependency) and it's [python] api changes somewhat often.  I've used it before with success for python driven C++ POD boiler plate (IO, size checks, visitors methods etc) autocoding.

If you can understand the underlying C/C++ and get over that Cython will always run on a C/C++ compiler in our universe,  and make that a feature to embrace, not a detail to hide (who actually benefits in the later case?) - are you still against the significant convenience - and potentially only way to make things work when Cython is not supporting a C++ way of doing things (say things aren't finished/working yet)?


-Jason