[Cython] Cannot cythonize subclasses of setuptools.extension._Extension

[Erik Bray]

On Sat, Apr 16, 2016 at 1:29 PM, Manuel Nuno Melo
<manuel.nuno.melo at gmail.com> wrote:
> Hi Erik,
> Please post your solution; I'm curious to see it.

Will do in a bit. I need to see if I can distill it somewhat from its
original context so that it can be better understood.

> Currently, we're also using setup_requires but are moving away from it
> because of two main issues:
>
> 1- there's no flexibility to hook into it via setuptools.Distribution
> subclassing (unless you rewrite the entire __init__);

I don't really think that's a big issue but maybe you've encountered
some reason that it is.
The worse thing about it is the general chicken-egg problem that you
need to call setup() to get your setup_requires, but you may need some
packages pulled in from setup_requires in order to do anything useful
with setup().

d2to1 [1] solves this problem quite well, but unfortunately
development on it is stalled for the time being.  Another alternative
that can work quite well is some version of Daniel Holth's
"setup-requires" hack [2].

> 2- (and more serious) setuptools installs setup_requires dependencies in the
> local directory, not as a system-wide install.

That's a feature, not a bug.  Build requirements for a package might
be in conflict with the versions of those same requirements you have
already installed.  It also prevents things from being installed that
are *not* required at runtime.  It makes it easier for projects to pin
a little more strictly to their known-working build requirements,
without the constraints imposed by supporting a wider range of runtime
requirements.

> Even worse, it then puts a
> path link to the current directory under the system-wide easy_install.pth.
> This means that if you install from the source directory via 'sudo
> ./setup.py install', you get a Cython egg downloaded into that directory,
> and its path added to sys.path (!!). Needless to say this can break in many
> nasty ways...

Huh??  I'm not sure we're talking about the same thing now.  The local
.eggs cache directory created for setup_requires downloads most
certainly does not get added to easy_install.pth.  Are you sure you're
not thinking of `setup.py develop`?

[1] https://pypi.python.org/pypi/d2to1
[2] https://bitbucket.org/dholth/setup-requires/src

> On Sat, Apr 16, 2016 at 1:10 PM, Erik Bray <erik.m.bray at gmail.com> wrote:
>>
>> On Apr 14, 2016 21:07, "Manuel Nuno Melo" <manuel.nuno.melo at gmail.com>
>> wrote:
>> >
>> > Our need to control cythonization comes from the fact that we implement
>> > cython as a lazy and optional dependency. Lazy in the sense that we delay as
>> > much as possible cythonization so that setuptools or pip have time to
>> > install cython, if needed. Optional because we distribute both .pyx and
>> > cythonized .c files, and decide on which to use based on user flags.
>> >
>> > We, therefore, need an Extension class that only cythonizes if we decide
>> > to.
>> >
>> > Thanks for the feedback,
>> > Manel
>>
>> I have a solution for exactly this in astropy_helpers which makes it
>> possible, for example, to pull in Cython via setup_requires, but only if
>> it's needed.
>>
>> Remind me on Monday and I can point out how it works.
>>
>> > On Apr 14, 2016 8:17 PM, "Matthew Brett" <matthew.brett at gmail.com>
>> > wrote:
>> > >
>> > > On Thu, Apr 14, 2016 at 6:08 AM, Erik Bray <erik.m.bray at gmail.com>
>> > > wrote:
>> > > > On Wed, Apr 13, 2016 at 9:35 PM, Manuel Nuno Melo
>> > > > <manuel.nuno.melo at gmail.com> wrote:
>> > > >> Hello devs,
>> > > >>
>> > > >> I'm developing the setup.py for a scientific package, MDAnalysis
>> > > >> (see PR
>> > > >> #799). We depend on distutils and setuptool. Namely, we use
>> > > >> setuptools.extension.Extension class for our extensions.
>> > > >>
>> > > >> Some older versions of setuptools (<18.0) do filename cythonization
>> > > >> themselves upon initialization of the Extension object.
>> > > >>
>> > > >> Because we want to control name cythonization ourselves I try to
>> > > >> directly
>> > > >> use distutils.extension.Extension, which has none of setuptools'
>> > > >> cythonization. However, this doesn't work because setuptools
>> > > >> patches
>> > > >> distutils, so that distutils.extension.Extension effectively
>> > > >> becomes
>> > > >> setuptools.extension.Extension.
>> > > >
>> > > > I'm wondering what it is specifically you need to do in your
>> > > > subclass--might it still be possible to do with a subclass of the
>> > > > setuptools Extension?  Not saying I disagree with the overall idea,
>> > > > but I also wonder if there isn't a better way.
>> > >
>> > > I know this is a terrible and ugly hack, but the projects I work in
>> > > have a 'fake_pyrex' directory, that fools setuptools into thinking
>> > > that 'pyrex' is installed, and therefore prevents it from doing the
>> > > .pyx -> .c filename conversions in the extension:
>> > >
>> > > https://github.com/regreg/regreg/blob/master/setup.py#L33
>> > >
>> > > Cheers,
>> > >
>> > > Matthew
>> > > _______________________________________________
>> > > cython-devel mailing list
>> > > cython-devel at python.org
>> > > https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> >
>> > _______________________________________________
>> > cython-devel mailing list
>> > cython-devel at python.org
>> > https://mail.python.org/mailman/listinfo/cython-devel
>> >
>>
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>