Phillip J. Eby wrote:

At 11:19 AM 9/3/2007 +0200, Stefan Behnel wrote:

...

In current setuptools, the file setuptools/extension.py checks for Pyrex being installed and otherwise renames all .pyx file entries to .c. This does not work when Cython is installed *instead of* Pyrex, as Cython will never be executed in this case. It just fails saying that the .c file doesn't exist.

I'm currently shipping a fake Pyrex/Distutils/build_ext.py module with lxml to make it work with both, but I don't even see a major advantage in the extension.py module at all. It's plain easy to do that yourself in the setup.py script if (and only if) you want it to build without Pyrex/Cython, but currently, it is enforced in setuptools. This really only works if the distributor wants it and does something to make it work, so always enforcing this behaviour doesn't give you any real advantage.

I'd like to see this module removed from setuptools or at least have Cython supported in the same way.

I guess I'm not clear on the actual problem here. Do you want it to only rename .pyx to .c for .c files that actually exist?

That would be an option. However, I still don't understand why it should be up to setuptools to decide if *my* code distribution can be built without Pyrex (or any other tool that names its files .pyx). The extension module in setuptools could also check if build_ext is the original function or a replacement and only do its magic if it's alone in the field and thinks it can't hurt anyone. That would only be another heuristic, but it would at least have prevented cases like this...

A better way to specify that you can handle .pyx files?

You mean a hook into setuptools so that Cython can fix the extension module? Doesn't sound like a clean solution to me...

Note that removing this feature is not an option that's on the table, as it was one of the original reasons I created setuptools in the first place, years before the idea of eggs even existed.

So you are suggesting that people should choose: it's either eggs or Cython. I don't think eggs are that important to everyone, but I don't think this decision should be enforced either. Maybe you should split setuptools into a part that serves everyone and a part that served you years before the idea of eggs even existed. Don't get me wrong, I really like eggs and the EasyInstall machinery. But it shouldn't get in the way of the projects it aims to serve.

Now, in future versions of setuptools, the ones that eventually replace a lot of distutils framework with new stuff, there will definitely be a better way of handling this. But I have no idea when that will happen.

Well, splitting it up could make it easier to improve things gradually. Stefan

Phillip J. Eby wrote:

At 06:01 PM 9/3/2007 +0200, Stefan Behnel wrote:

...

So you are suggesting that people should choose: it's either eggs or Cython.

No -- I'm suggesting that developers whose projects include C code generated from .pyx files should always include the resulting .c files in their source distributions as a courtesy to their users.

Not only is this the friendly thing for your users, it's also the only sane thing to do from a support standpoint. Otherwise, you have to ensure that your users have exactly the same version of Pyrex (or Cython, or pypy, or whatever) that you do.

Please notice that these points are entirely independent of whether setuptools or eggs exist. I was shipping generated Pyrex-generated .c files to my users for these reasons well before setuptools existed, let alone eggs.

So do I. lxml has been doing that long before we even started using setuptools. The point is that current setuptools break developer builds. You can't change the sources without having (a fake) Pyrex installed, because setuptools will not let you run Cython. Stefan

Phillip J. Eby wrote:

At 06:21 PM 9/3/2007 +0200, Stefan Behnel wrote:

...

The point is that current setuptools break developer builds. You can't change the sources without having (a fake) Pyrex installed, because setuptools will not let you run Cython.

If you want to use the native Extension type, do this:

from distutils.core import Extension from setuptools import setup

Have you tested this lately? I mean, with Python 2.5 distutils? What I get is Traceback (most recent call last): File "setup.py", line 89, in <module> **extra_options File "distutils/core.py", line 151, in setup File "distutils/dist.py", line 974, in run_commands File "distutils/dist.py", line 994, in run_command File "distutils/command/build_ext.py", line 290, in run File "distutils/command/build_ext.py", line 413, in build_extensions File "distutils/command/build_ext.py", line 315, in check_extensions_list TypeError: iteration over non-sequence this is the failing code: if isinstance(ext, Extension): continue # OK! (assume type-checking done # by Extension constructor) ==> (ext_name, build_info) = ext

Alternately, you can use:

from setuptools.extension _import Extension as Extension

I assume you meant from setuptools.extension import _Extension as Extension That's definitely a hack, but it yields the same error as above.

I will add this information to the existing setuptools documentation regarding Pyrex detection.

Meanwhile, if you'd like to suggest a way of detecting Cython's presence, I can probably add that to the setuptools.extension module.

import Cython should do the trick. I still prefer the "only fiddle with extensions if build_ext wasn't replaced" way, though. Stefan

Phillip J. Eby wrote:

At 06:51 PM 9/3/2007 +0200, Stefan Behnel wrote:

...

I still prefer the "only fiddle with extensions if build_ext wasn't replaced" way, though.

If build_ext wasn't replaced where and how? Do you mean in the "cmdclass" argument to setup()?

Exactly. As I said, that would also be only a heuristic, but it would still be better than the current 'feature' enforcement code which basically states: "if Pyrex isn't installed, you can't possibly have meant what you were requesting, so I'll fix your code for you". Problem being that the code isn't broken and the 'fix' breaks it. Stefan

Hi, I noticed you just released a new version of setuptools without mentioning a fix for the problem in the changelog, so I assume it's not fixed? Phillip J. Eby wrote:

At 08:09 AM 9/4/2007 +0200, Stefan Behnel wrote:

...

Phillip J. Eby wrote:

...

At 06:51 PM 9/3/2007 +0200, Stefan Behnel wrote:

...

I still prefer the "only fiddle with extensions if build_ext wasn't replaced" way, though.

If build_ext wasn't replaced where and how? Do you mean in the "cmdclass" argument to setup()?

Exactly.

That would mean that you'd have to write code in *every* setup.py to do the replacement -- *and* you'd have to get it correct so that it does the right thing when Pyrex and Cython are *not* installed.

You already have to write code that imports and replaces build_ext, so having to add a bit of code that replaces .pyx by .c is not asking to much IMHO.

...

but it would still be better than the current 'feature' enforcement code which basically states: "if Pyrex isn't installed, you can't possibly have meant what you were requesting, so I'll fix your code for you".

...and you simply get a different error message than you were going to get anyway, so there's no net increase in problems. You are going to have to fix the real problem (by either including the .c files or installing the compiler), no matter what. So how does setuptools' action here make any difference? All that changes is the error message.

Because there is no easy way around it. That's the problem. Having setuptools do its monkey patching in the background means you have to write code that works around the patching, and, yes, you have to get it right, which is far from obvious, involving a couple of test imports, catching ImportErrors and the like. How is that better than writing a straight forward 1-liner for replacing .pyx by .c?

...

Problem being that the code isn't broken and the 'fix' breaks it.

After thinking this over some more, I disagree.

The key design question I ask for setuptools features is "who gets the pain?", since nearly all design decisions will cause *somebody* pain. Ideally, I want that pain to go to somebody who's in a position to do something that improves things for everybody in the long term.

In this case, the only additional pain is for the developers of new tools that process .pyx files (such as yourself).

Wrong. I cannot fix this as a developer of Cython - at least not without providing a fake build_ext.py in a fake Pyrex package, which would make installing Pyrex and Cython on the same machine near impossible and which would break every tool that requires Pyrex instead on Cython. I can only fix this problem as a developer of lxml which *uses* Cython, by shipping a fake Pyrex with my distribution and adding it to the SVN repository. So you are currently shifting the burden on every developer who wants to support Cython (maybe in addition to Pyrex). Great idea.

[stripped incorrect comments based on a wrong assumption]

In the long run, my plan is to add entry points to the "build" command so that other commands can be added. This would let you have a command whose job is to turn .pyx files to .c, but only runs if it's installed. In the meantime, faking Pyrex-ness is, IMO, the best way to deal with this.

No, it's not. I would rather like to have a way to deal with this from Cython than from Cython-implemented distributions. But the best option IMHO would be to fix setuptools. Stefan

At 09:24 PM 9/4/2007 +0200, Stefan Behnel wrote:

Stefan Behnel wrote:

...

Phillip J. Eby wrote:

...

The key design question I ask for setuptools features is "who gets the pain?", since nearly all design decisions will cause *somebody* pain. Ideally, I want that pain to go to somebody who's in a position to do something that improves things for everybody in the long term.

In this case, the only additional pain is for the developers of new tools that process .pyx files (such as yourself).

Wrong. I cannot fix this as a developer of Cython - at least not without providing a fake build_ext.py in a fake Pyrex package, which would make installing Pyrex and Cython on the same machine near impossible and which would break every tool that requires Pyrex instead on Cython.

Ok, admittedly, I could change sys.path to point it to an internal directory of the installed Cython distribution, so that setuptools would be tricked into succeeding with a fake Pyrex import - but how ugly is that?

Isn't Cython a *replacement* for Pyrex? If not, then why does it use .pyx files? If it is, then why is installing both a problem? (It's not like you can't switch back and forth between them.)

The right way of fixing this is definitely in setuptools, and the quick fix is to extend the test import of Pyrex.Distutils.build_ext.build_ext with an additional test for Cython.Distutils.build_ext.build_ext in the failure case - if you don't feel like taking the cleaner step of checking for a "build_ext" replacement...

Perhaps you'd care to produce a patch to implement that "cleaner step"? It's not at all obvious to me how to do that without introducing instability that would be unsuitable for an 0.6cN release.

At 08:30 AM 9/5/2007 +0200, Stefan Behnel wrote:

...

Perhaps you'd care to produce a patch to implement that "cleaner step"? It's not at all obvious to me how to do that without introducing instability that would be unsuitable for an 0.6cN release.

One way of implementing the above change would be to move the replacement code into build_ext rather than Extension. Something like the (untested) build_ext-patcher.py I attached. Note the type check that tests for build_ext being subclassed.

You're illustrating my point. It's easy to hand-wave about how it should be done, but not so easy to actually *do*. Did you look at where all the .sources attribute gets used? How the build_ext command can get called by other commands? You're also ignoring my larger point: the current mechanism allows you to write setup scripts that *don't* need to subclass build_ext. A setuptools-based setup script just refers to '.pyx' files, and everything else happens automatically. And if you need to be able to distinguish between Cython-specific and Pyrex-specific files, why are you using the same file extension?

At 08:00 PM 9/5/2007 +0200, Stefan Behnel wrote:

Phillip J. Eby wrote:

...

At 08:30 AM 9/5/2007 +0200, Stefan Behnel wrote:

...

...

Perhaps you'd care to produce a patch to implement that "cleaner step"? It's not at all obvious to me how to do that without introducing instability that would be unsuitable for an 0.6cN release.

One way of implementing the above change would be to move the replacement code into build_ext rather than Extension. Something like the (untested) build_ext-patcher.py I attached. Note the type check that tests for build_ext being subclassed.

You're illustrating my point. It's easy to hand-wave about how it should be done, but not so easy to actually *do*. Did you look at where all the .sources attribute gets used? How the build_ext command can get called by other commands?

Sort of. Do you doubt it should work that way?

I know that other commands use the .sources of build_ext indirectly, at a time when run() has not yet been called. That means that your approach may change those other commands' behavior in non-obvious ways. That's why I'm not going to do something like that in an 0.6cN release.

...

You're also ignoring my larger point: the current mechanism allows you to write setup scripts that *don't* need to subclass build_ext. A setuptools-based setup script just refers to '.pyx' files, and everything else happens automatically.

The script I posted doesn't change that. Read it.

Actually, now that I've read it in detail, I see that it's also broken in other ways, such as confusing modules and classes. But it's also broken in the way I said: it *requires* subclassing of build_ext in setup.py, which the current mechanism does not.

...

And if you need to be able to distinguish between Cython-specific and Pyrex-specific files, why are you using the same file extension?

We don't distinguish, it's the same language. But users will have to install both of them

If it's the "same language" then, *why* do they need this? If Pyrex is still required, then clearly it's *not* the "same language".

You're actually forgetting the point you stressed most. If we prevent users from installing Cython and Pyrex at the same time, they will have to start modifying setup.py scripts. So it's pushing the burden on them.

Actually, I think you need to decide whether you are really providing an extended Pyrex compiler. If so, then use .pyx as the extension and include a Pyrex-compatible build_ext module, and then there is no need to have both installed on the same system, and everything works splendidly for everyone. If on the other hand you are *not* providing a Pyrex replacement and must co-exist, then use a different file extension (.cy, perhaps?) and you get exactly the behavior you wanted -- i.e., setuptools doesn't touch your files, and explicit action is required to change the file extensions. And there is still a third solution: provide your own Extension class for users to import, since they will have to import from Cython in order to use your extended build_ext class anyway, nothing stops them from importing the Extension type too. (Just have your Extension.__init__ save the sources and restore them after calling the superclass; it'll then work whether the patched version is present or not.)

At 09:49 PM 9/5/2007 +0200, Stefan Behnel wrote:

get a build_ext replacement, which is required by both Pyrex and Cython.

No, it isn't. A setuptools-based project does not need to import *anything* from Pyrex; it just declares a setuptools Extension() with .pyx sources. Further, if Cython includes a Pyrex-replacing build_ext in the same module location, it's not necessary for Cython either; setuptools will simply call the Cython one. That's why I'm saying that if Cython is actually a Pyrex *replacement*, it should work just fine, with no need for anybody to change anything. Conversely, if Cython is *not* a replacement for Pyrex, then it should use some other file extension and thus make the choice of tool explicit. My initial understanding was that Cython was a Pyrex replacement... which led me to wonder what all the fuss was about. But this assumption about replacing build_ext being necessary is clearly a big part of what's leading us to different conclusions.

At 10:21 PM 9/5/2007 +0200, Stefan Behnel wrote:

Phillip J. Eby wrote:

...

At 09:49 PM 9/5/2007 +0200, Stefan Behnel wrote:

...

get a build_ext replacement, which is required by both Pyrex and Cython.

No, it isn't. A setuptools-based project does not need to import *anything* from Pyrex; it just declares a setuptools Extension() with .pyx sources. [...] But this assumption about replacing build_ext being necessary is clearly a big part of what's leading us to different conclusions.

Ah, I missed that part. So setuptools special cases Pyrex in other places, too, thanks for pointing that out.

From the docs: Distributing Extensions compiled with Pyrex ------------------------------------------- ``setuptools`` includes transparent support for building Pyrex extensions, as long as you define your extensions using ``setuptools.Extension``, *not* ``distutils.Extension``. You must also not import anything from Pyrex in your setup script. If you follow these rules, you can safely list ``.pyx`` files as the source of your ``Extension`` objects in the setup script. ``setuptools`` will detect at build time whether Pyrex is installed or not. If it is, then ``setuptools`` will use it. If not, then ``setuptools`` will silently change the ``Extension`` objects to refer to the ``.c`` counterparts of the ``.pyx`` files, so that the normal distutils C compilation process will occur. Of course, for this to work, your source distributions must include the C code generated by Pyrex, as well as your original ``.pyx`` files. This means that you will probably want to include current ``.c`` files in your revision control system, rebuilding them whenever you check changes in for the ``.pyx`` source files. This will ensure that people tracking your project in CVS or Subversion will be able to build it even if they don't have Pyrex installed, and that your source releases will be similarly usable with or without Pyrex.

Maybe I should really consider adding a Pyrex replacement to Cython... but that needs to be discussed on the cython-dev list.

I still wouldn't mind having a few additional if-Pyrex-import-fails-try-Cython lines in setuptools for now...

Well, does that need to be done for the actual build_ext command, too? (See setuptools.command.build_ext)