Just because there are configuration problems associated with adding a feature like the one I needed is absolutely no reason to abandon it when it can bring value to the tool if used correctly and in some circumstances. I considered some of those exact complications "what if it was already installed, etc" and with my company's project, where I am using this useful tool in a circumstance you may overlook (it is perfectly acceptable to have such a feature, *despite* the list of complications you mention), such a feature would have been very valuable. Since it is useful in my case, I understand it would be valuable for others as well. (I don't appreciate the aggressive tone of your reply, though, nor do I see how my good faith efforts to help others warrant this... how did I possibly offend you with my post??) On the other hand, I appreciate your "correct solution" as a good approach and I'll forward this idea to an author of SQLAlchemy for his consideration. On 5/7/2010 1:33 PM, Glyph Lefkowitz wrote:

On May 7, 2010, at 9:09 AM, Kent wrote:

...

Consider the case where you want your setup to install third-party software, but you want/need to pass an argument to the command line that runs "python setup.py --argument install" or "python setup.py -- argument bdist_egg"

As far as I could research, this feature is unavailable.

And for good reason, I should think. I really hope that nobody ever adds this feature.

If you require SQLAlchemy to be installed "--with-cextensions", then what happens when your package is installed in an environment that already has SQLAlchemy installed *without* that flag? Does it stomp on the existing installation? What if the user installed one already with that flag and some other flags as well? What if it's system-installed and you're doing a user-install?

Basically, compile-time and install-time options are a configuration nightmare. They should represent only how and where a package is installed, not what features it has. The correct solution to your problem would be to get SQLAlchemy to fix its broken deployment setup and split itself into 2 packages, "SQLAlchemyCExtensions" and "SQLAlchemy", and then have your project depend on both, not to try to cram installer options into the dependency language.

For confirmation of this theory, you need look no further than the excruciating user-experience of source-based installation systems with 'variant' support, like gentoo's Portage and *BSD's Ports, versus the relatively non-excruciating experience of packaging systems which express compile-time options as different packages like Yum and Apt.

On May 7, 2010, at 2:08 PM, Michael Bayer wrote:

I'd only mention that Storm has a C extension/non C extension flag as well, and only offers one source distribution on Pypi. You have to modify a variable directly within setup.py. Our setup.py features the same capability (its just our C extension is off by default for 0.6 since it was just written, which is the same case for when Storm first introduced its C extension).

It occurs to me that Twisted has a similar problem (except there's no installation flag: it just builds the C extensions if it possibly can). The problem I see here is that the dependencies list of a particular project should be a complete expression of the features that the source code requires to function properly. If a C extension is present for optimization purposes only, then I don't think it ever needs to be mentioned in a dependency listing. Performance tuning is a build and deployment issue, not a dependency-correctness issue. However, if a C extension wraps features necessary for an application to work correctly, without which it will simply traceback and die, then it should be possible for the application to say "I depend on this functionality". After all, it's kind of bogus if I say "I depend on library X", and then library X gets installed, but half of it is missing for some reason. It's bogus if it's missing for any reason at all, really. The "C extension couldn't be compiled" one is common, but there are other configuration and build issues which could prevent a distribution from being fully functional. Is there already a good way to express a dependency on a portion of a source distribution, or "optional" features? A way to list one source distribution on PyPI so that it will be present under multiple names, one for each optional chunk?

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Lennart Regebro wrote:

On Sat, May 8, 2010 at 07:48, Glyph Lefkowitz wrote:

...

a dependency-correctness issue. However, if a C extension wraps features necessary for an application to work correctly, without which it will simply traceback and die, then it should be possible for the application to say "I depend on this functionality".

I think setuptools can do this, but I've never used it so I'm not sure.

Another option would to package the extensions as a separate package, so you get 'MySQL' and 'MySQL-C-extensions'. Problem solved. :)

That is effectivaly what the "extras" bit in setuptools does[ http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional... Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvoUd8ACgkQ+gerLs4ltQ6zxQCePHJfWk8mvoQrt7k8n8s4TfOk FmAAnAl3XT1HZK1oeP2Yq1HgLzQQJHTf =HZXH -----END PGP SIGNATURE-----

FWIW, it is perfectly possible to package the thing separately as Glyph seem to suggest, even if the feature is enabled through an option. For example, Debian does it: http://packages.debian.org/experimental/python-sqlalchemy-ext On Fri, May 7, 2010 at 19:56, Kent Bower wrote:

Just because there are configuration problems associated with adding a feature like the one I needed is absolutely no reason to abandon it when it can bring value to the tool if used correctly and in some circumstances.  I considered some of those exact complications "what if it was already installed, etc" and with my company's project, where I am using this useful tool in a circumstance you may overlook (it is perfectly acceptable to have such a feature, *despite* the list of complications you mention), such a feature would have been very valuable.

Since it is useful in my case, I understand it would be valuable for others as well.

(I don't appreciate the aggressive tone of your reply, though, nor do I see how my good faith efforts to help others warrant this... how did I possibly offend you with my post??)

On the other hand, I appreciate your "correct solution" as a good approach and I'll forward this idea to an author of SQLAlchemy for his consideration.

On 5/7/2010 1:33 PM, Glyph Lefkowitz wrote:

...

On May 7, 2010, at 9:09 AM, Kent wrote:

...

Consider the case where you want your setup to install third-party software, but you want/need to pass an argument to the command line that runs "python setup.py --argument install" or "python setup.py -- argument bdist_egg"

As far as I could research, this feature is unavailable.

And for good reason, I should think.  I really hope that nobody ever adds this feature.

If you require SQLAlchemy to be installed "--with-cextensions", then what happens when your package is installed in an environment that already has SQLAlchemy installed *without* that flag?  Does it stomp on the existing installation?  What if the user installed one already with that flag and some other flags as well?  What if it's system-installed and you're doing a user-install?

Basically, compile-time and install-time options are a configuration nightmare.  They should represent only how and where a package is installed, not what features it has. The correct solution to your problem would be to get SQLAlchemy to fix its broken deployment setup and split itself into 2 packages, "SQLAlchemyCExtensions" and "SQLAlchemy", and then have your project depend on both, not to try to cram installer options into the dependency language.

For confirmation of this theory, you need look no further than the excruciating user-experience of source-based installation systems with 'variant' support, like gentoo's Portage and *BSD's Ports, versus the relatively non-excruciating experience of packaging systems which express compile-time options as different packages like Yum and Apt.

-- Gaëtan de Menten

On May 7, 2010, at 1:56 PM, Kent Bower wrote:

(I don't appreciate the aggressive tone of your reply, though, nor do I see how my good faith efforts to help others warrant this... how did I possibly offend you with my post??)

Let me start here to set the tone: Please note that I am not attacking _you_. I thought that this was pretty clear, but if anything I said (in either my previous message or this one) could be construed as a personal attack, then I sincerely apologize: that was not and is not my intention. If anything, I'm grateful to you for bringing this up, and giving me a clear example to express my views on this issue, so they are recorded for future askers of similar questions. I'm enthusiastically attacking your *proposed feature*, because I think it's a terrible idea that will cause lots of problems if implemented, but I don't believe you suggested it in bad faith or have otherwise malicious intent.

Just because there are configuration problems associated with adding a feature like the one I needed is absolutely no reason to abandon it when it can bring value to the tool if used correctly and in some circumstances.

Here's a somewhat exaggerated example. It may be convenient for some people, in some circumstances, to attach an exposed lawnmower blade to a child's toy. Maybe they're out playing with their kids and want to mow the lawn at the same time: if the toys are also all lawnmowers, it's super convenient. And there's of course no problem if this feature is used correctly, in the right circumstances: by adults, carefully, to mow lawns. However, most toy manufacturers would agree that it is not worth the risk of taking off a child's hands or head to provide someone with that additional convenience. If you want a lawnmower, buy a lawnmower and use it to mow your lawn. Don't go welding exposed blades to every available household object that may be at hand when you're in your yard.

I considered some of those exact complications "what if it was already installed, etc" and with my company's project, where I am using this useful tool in a circumstance you may overlook (it is perfectly acceptable to have such a feature, *despite* the list of complications you mention), such a feature would have been very valuable.

In your particular case, what you're trying to do is deploy an application, not express code dependencies. While I haven't used Buildout personally, my understanding is that this is the sort of thing that Buildout is intended to do: deploy a specific application with specific dependencies in a carefully-controlled environment. I have no idea if Buildout can do precisely what you want, but if not, there are plenty of other tools which do similar things: makefiles, shell scripts, fabric, capistrano... and each of these tools let you encode that consideration that you've done in a dedicated place, rather than injecting it somewhere it doesn't belong (your application logic, your setup.py). The risk here, which I am keen to avoid, is that if you provide the ability to pass command line arguments via setup.py, people will start uploading packages that do that to PyPI. Gradually, half of your dependencies will start wanting to pass conflicting sets of "helpful" options to their dependencies at installation time, and getting your library packages installed will be an even bigger mess than it is right now. For the most part, distutils needs *fewer* knobs that developers and packagers can turn; this is not one of the few places where it needs more. See all the recent work that's gone into statically expressing all dependencies in a data format, without even having a setup.py. This is a manifestation of a social problem in open source projects; rather than fix a problem correctly, there is a temptation to say "just let me pass through my options so it works for me". If you want to depend on C extensions in SQLAlchemy, there should be a proper way to do that so that it works regardless of what other packages you happen to have installed or how they got there. (See "standing up to user pressure", here: http://ometer.com/free-software-ui.html.)

On the other hand, I appreciate your "correct solution" as a good approach and I'll forward this idea to an author of SQLAlchemy for his consideration.

Thanks. Good luck getting that fix in! :)