Donald Stufft writes:

2. Use older versions of setuptools

That's not really an answer, unless downgrading works. For example, a recently created venv would contain the latest setuptools, and perhaps it would be required by other distributions in the venv. How then would e.g. SQLAlchemy specify that an older setuptools version should be used? Would pip downgrade the venv's setuptools to an older version? Even if it did, might that not break other distributions in the venv? Regards, Vinay Sajip

On Mar 6, 2014, at 1:07 PM, Daniel Holth wrote:

pje said:

The "Feature()" facility was never completely implemented or supported, and even if it were, it should be deprecated now, as it will not be compatible with the coming packaging systems based on PEP 426. If you need separate features, use separate distributions and "extras" instead.

wait, ok this is that thing. “separate distributions” means…. SQLAlchemy-0.9.3-nocext.tar.gz SQLAlchemy-0.9.3-cext.tar.gz ?? I think that’s what it means. OK, not sure if I have to say this but that would be…OK very very unworkable. A Python package distribution is a *source distribution*. That suggests that one need to package up separate source distributions in order to specify an option, is… OK very very unworkable. (note by “very very unworkable” I mean “list of hyperboles I had to backspace out because that’s not why I’m here”). If a package has three flags, now there need to be *eight source distros*? Really? On the plus side, I just used math. how does one even maintain this in source control? Do I have setup.py.nocext, setup.py.cext, maintain a copy of 99% the same code in each, then rename them when I do “python setup.py sdist” (since sdist doesn’t take command line options either!!) ? The suggestion is completely inappropriate. I don’t know the metadata format in pep426 well enough to comment (as I wanted to use it one day and found that it seemed to still be pretty much vapor), but I’ll reiterate: these are source distributions, not binaries or wheels or anything like that. A source distro has to build, and builds need options. A metadata format that purports to represent metadata about a source distribution and does not support flags is a broken metadata format.

Michael Bayer writes:

I don’t know the metadata format in pep426 well enough to comment (as I wanted to use it one day and found that it seemed to still be pretty much vapor), but I’ll reiterate: these are source distributions, not binaries or wheels or anything like that. A source distro has to build, and builds need options. A metadata format that purports to represent metadata about a source distribution and does not support flags is a broken metadata format.

PEP 426 only covers installation metadata - its scope does not cover source distributions (there will be another PEP covering that). Apart from the most recent changes being not yet implemented and some open issues relating to hooks, PEP 426 seems to cover installation metadata reasonably well. However, sdist metadata is a different kettle of fish. Regards, Vinay Sajip

From:Michael Bayer

OK so why does PEP-426 compatibility imply removal of command line switches from setup.py files ?

As far as I know, it doesn't. My distil tools complies with a fairly recent version of PEP 426 and AFAIK has no problem building / installing SQLAlchemy with / without extensions, but it uses an extended set of metadata which is not specified in any PEP - for example, http://www.red-dove.com/pypi/projects/S/SQLAlchemy/package-0.9.3.json This is just FYI - it's experimental metadata [which has some redundancy for historical reasons, to be tidied up in the future] and is extracted from what is passed to setup.py - it includes the PEP 426 metadata as a subset ("index-metadata"), and the extra stuff is needed for building. This is used by the distil tool, but there's no support for any specific command-line flags to exclude the building extensions. (That's not because of PEP compliance - it's just that I haven't needed that level of control in the tool.) Regards, Vinay Sajip

On Mar 6, 2014, at 4:15 PM, Marcus Smith wrote:

...

On Thu, Mar 6, 2014 at 10:37 AM, Michael Bayer wrote:

On Mar 6, 2014, at 1:07 PM, Daniel Holth wrote:

...

pje said:

The "Feature()" facility was never completely implemented or supported, and even if it were, it should be deprecated now, as it will not be compatible with the coming packaging systems based on PEP 426. If you need separate features, use separate distributions and "extras" instead.

wait, ok this is that thing. “separate distributions” means….

SQLAlchemy-0.9.3-nocext.tar.gz SQLAlchemy-0.9.3-cext.tar.gz

I'm new to understanding setuptools "Feature" myself, so don't crank too much, if I get anything wrong.

afaik, it means something like:

SQLAlchemy-X.Y.tar.gz SQLAlchemy-cext-X.Y.tar.gz

i.e. isolating the feature into a separate project.

and then instead of having the handy "--with-cext" option, it has to be "pip install SQLAlchemy[SQLAlchemy-cext]

This is my understanding as well except it inverses the default. SQLAlchemy wants the C exts installed unless they can't be if the user opts out. Extras would cause if to not be installed unless the user opts in. This is probably not a reasonable solution. One note if could just be SQLAlchemy[cext]

PJ, Jason: can you clarify what the alternative is supposed to look like?

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On Mar 6, 2014, at 4:35 PM, Marcus Smith wrote:

This is my understanding as well except it inverses the default. SQLAlchemy wants the C exts installed unless they can't be if the user opts out. Extras would cause if to not be installed unless the user opts in. This is probably not a reasonable solution.

ok, that's odd, so there is no extras replacement for the --without-cext scenario. so even more important to understand from PJ/Jason the rationale of how Feature could be removed

One note if could just be SQLAlchemy[cext]

oh yea, you just pass a label

show me that

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On Mar 8, 2014, at 9:19 AM, Jason R. Coombs wrote:

From: Distutils-SIG [mailto:distutils-sig-bounces+jaraco=jaraco.com@python.org] On Behalf Of Marcus Smith Sent: Thursday, 06 March, 2014 16:16 To: DistUtils mailing list Subject: Re: [Distutils] tourist here, with a dumb RTFM question

On Thu, Mar 6, 2014 at 10:37 AM, Michael Bayer wrote:

On Mar 6, 2014, at 1:07 PM, Daniel Holth wrote:

...

pje said:

The "Feature()" facility was never completely implemented or supported, and even if it were, it should be deprecated now, as it will not be compatible with the coming packaging systems based on PEP 426. If you need separate features, use separate distributions and "extras" instead.

wait, ok this is that thing. “separate distributions” means….

SQLAlchemy-0.9.3-nocext.tar.gz SQLAlchemy-0.9.3-cext.tar.gz

I'm new to understanding setuptools "Feature" myself, so don't crank too much, if I get anything wrong.

afaik, it means something like:

SQLAlchemy-X.Y.tar.gz SQLAlchemy-cext-X.Y.tar.gz

i.e. isolating the feature into a separate project.

and then instead of having the handy "--with-cext" option, it has to be "pip install SQLAlchemy[SQLAlchemy-cext]

PJ, Jason: can you clarify what the alternative is supposed to look like?

I’m new to understanding Features as well, but based on PJ’s post, I would agree with Marcus here. This is how I interpreted the use of extras to supersede Features. I see a lot of advantages to this approach over build-time selection. Because it decouples the main project from the C-extension speedups, it gives the installer control over what behavior is present. It also gives deployment tools visibility over what capability is present (with features, it’s harder to tell if the C-extension speedups are present; with extras, one can query pkg_resources).

If there’s an easy way to do this from setup.py it wouldn’t be so bad: python setup.py —without-cext sdist upload python setup.py sdist upload but it still seems kind of strange to deliver the exact same source files with some flag somewhere different? I’d assume that flag is in setup.cfg, which means, it’s a setup.py flag in any case! not to mention its still needed as part of the “sdist” and other commands to build dists. I think the issue of “you should have individual projects on pypi with different options to make life easier for builders” is orthogonal to “setup.py should or should not support flags”.

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 9 Mar 2014 01:50, "Daniel Holth" wrote:

Some packaging systems do not have the same 1:1 source : distribution relationship that is so prominent in the distutils model. This should probably change long term.

Huh, I hadn't thought about that, and I really should have, since RPM is one such system where you can get multiple binary RPMs from a single source RPM. On the other hand, "installing" an SRPM just drops it into the rpmbuild SRPM directory - it doesn't implicitly build anything. So yes, I agree that once we get to fixing the build side of things, we may need to figure out a way to handle multiple binary artifacts coming from a single source artifact *on the same platform and Python version*. Cheers, Nick.

On Sat, Mar 8, 2014 at 9:53 AM, Michael Bayer

...

On Mar 8, 2014, at 9:19 AM, Jason R. Coombs wrote:

From: Distutils-SIG [mailto:distutils-sig-bounces+jaraco=jaraco.com@python.org] On Behalf Of Marcus Smith Sent: Thursday, 06 March, 2014 16:16 To: DistUtils mailing list Subject: Re: [Distutils] tourist here, with a dumb RTFM question

On Thu, Mar 6, 2014 at 10:37 AM, Michael Bayer

...

wrote:

On Mar 6, 2014, at 1:07 PM, Daniel Holth wrote:

...

pje said:

The "Feature()" facility was never completely implemented or supported, and even if it were, it should be deprecated now, as it will not be compatible with the coming packaging systems based on PEP 426. If you need separate features, use separate distributions and "extras" instead.

wait, ok this is that thing. "separate distributions" means....

SQLAlchemy-0.9.3-nocext.tar.gz SQLAlchemy-0.9.3-cext.tar.gz

I'm new to understanding setuptools "Feature" myself, so don't crank too much, if I get anything wrong.

afaik, it means something like:

SQLAlchemy-X.Y.tar.gz SQLAlchemy-cext-X.Y.tar.gz

i.e. isolating the feature into a separate project.

and then instead of having the handy "--with-cext" option, it has to be "pip install SQLAlchemy[SQLAlchemy-cext]

PJ, Jason: can you clarify what the alternative is supposed to look

wrote: like?

...

I'm new to understanding Features as well, but based on PJ's post, I

would

...

agree with Marcus here. This is how I interpreted the use of extras to supersede Features. I see a lot of advantages to this approach over build-time selection. Because it decouples the main project from the C-extension speedups, it gives the installer control over what behavior is present. It also gives deployment tools visibility over what capability is present (with features, it's harder to tell if the C-extension speedups are present; with extras, one can query pkg_resources).

If there's an easy way to do this from setup.py it wouldn't be so bad:

python setup.py --without-cext sdist upload

python setup.py sdist upload

but it still seems kind of strange to deliver the exact same source files with some flag somewhere different? I'd assume that flag is in setup.cfg, which means, it's a setup.py flag in any case! not to mention its still needed as part of the "sdist" and other commands to build dists.

I think the issue of "you should have individual projects on pypi with different options to make life easier for builders" is orthogonal to "setup.py should or should not support flags".

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Thanks for this response, Jason! Your thoughtfulness is appreciated. - mike On Mar 8, 2014, at 9:10 AM, Jason R. Coombs wrote:

Thanks Mike for the report. I haven't read the rest of this thread yet, and I want to respond to your OP before I do. As a result, there will likely be some duplication.

You're absolutely right to be dismayed.

I had incorrectly assessed the potential impact of this change. I had expected the impact to be minor and isolated. I underestimated the number of systems that automatically upgrade Setuptools as well as the number of packages that could be using Features. I relied too heavily on the DeprecationWarning to have flagged the breakage (to package maintainers and their users). I had expected issues like you raised in this post to have been hashed out or at least raised long before I considered the release.

I hope my decision to backout the release serves as a strong signal that this type of breakage is unacceptable. I hope the tickets filed and discussion will lead to a better mechanism for communicating these types of changes in the future. In the future, I will leverage Twitter to announce in advance any backward-incompatible releases, giving additional visibility and oversight.

I do also want to ask that deployment teams consider the implications of always upgrading to the latest release of Setuptools. I'm encouraged that they do, and I do want to keep that as a viable deployment workflow. However, at the same time, there is some inherent risk with automatically upgrading to known-backward-breaking versions. The purpose of adopting semver for versioning is to allow the project to signal directly and unambiguously when a release is backward-breaking. I haven't yet decided when or where to raise this discussion, and it certainly doesn't excuse the challenges faced Thursday.

It's clear to me from your post that there are use-cases for Features that aren't covered by simply "using extras". Namely, extras doesn't support exclusion, only inclusion. I'll make mention of that in the deprecation ticket (https://bitbucket.org/pypa/setuptools/issue/65/) and follow up after understanding the problem better.

I'm the one that needs to apologize, not you. Thanks for your patience and understanding as I and the rest of the PyPA work toward the best means of communication (that doesn't require every package maintainer to follow distutils-sig).

Best regards, Jason

...

-----Original Message----- From: Distutils-SIG [mailto:distutils-sig- bounces+jaraco=jaraco.com@python.org] On Behalf Of Michael Bayer Sent: Thursday, 06 March, 2014 12:56 To: distutils-sig@python.org Subject: [Distutils] tourist here, with a dumb RTFM question

Hi Distutils !

I don't follow this list and haven't looked at it in a long time. However, I'm learning via twitter that a brand new setuptools release that's just gone out has just removed the "Feature" mechanism.

Now as you're all rolling your eyes and preparing to bang out frustrated replies how this was well announced and deprecated and warned about and wow really didn't you know the term paper was due today, OK first off let me say I am sorry! I am not a distutils/setuptools maintainer. I am just someone that uses it, and as I include setuptools in my setup.py, I am also getting thousands of other people who download my product to use it as well. And I don't read the setuptools changelog! Or the setuptools blog. Or this list. I assume you guys have it under control (and you certainly do!). There seem to be other people like me (people who write very widely used software) who also seem surprised! And that is surprising too (as I am usually the only person to be surprised by these things that should not be surprising). So I hope you can hold back your frustration with my clueless RTFMness long enough to answer these questions:

1. where was this announced? I'm wondering why there weren't loud blaring blog posts and tweets all over the place stating that on March 6, dozens of major packages are going to all break.

2. what is the plan for unmaintained packages and old versions of existing packages on pypi that "import Feature" and can no longer be installed? Is it just the case that a large amount of pypi just won't install anymore?

3. What, if any, is the recommended approach going forward to a Python package that wishes to specify a command-line flag during install. Here's multiple choice:

a. you can use new setuptools API

b. you can roll it yourself in setup.py using <hack X>

c. your setup.py should never accept any kind of flags as that interferes with <up-and-coming use case Q>

d. other

If choice c., then here is another question. My library includes optional C extensions. On certain platforms, these C extensions don't build (like on Pypy, or on Windows if you don't have compilation tools installed). In this regard it gracefully degrades. But also, I want to be able to have a command line option to determine this as well. Because! Maybe I'm installing within a test suite where I need to test the entire library without the C extensions built. Maybe some user has found a bug in the C extensions, and that user needs to temporarily install the tool without the extensions built. Other cases for flags are, maybe your library can build with or without SSL support, something like that.

Keep in mind, I actually *won't* be getting bug reports about this because my setup.py already gracefully degrades to distutils! But still, I'd like to have my -without-cextensions flag somehow.

Thanks for listening and again I apologize for not following the setuptools changelog on a regular basis!

- mike

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 9 Mar 2014 01:20, "Nick Coghlan" wrote:

On 9 Mar 2014 00:26, "Jason R. Coombs" wrote:

...

I had incorrectly assessed the potential impact of this change. I had

expected the impact to be minor and isolated. I underestimated the number of systems that automatically upgrade Setuptools as well as the number of packages that could be using Features. I relied too heavily on the DeprecationWarning to have flagged the breakage (to package maintainers and their users). I had expected issues like you raised in this post to have been hashed out or at least raised long before I considered the release.

Ouch, I just realised this is likely a nasty consequence of the decision

we made in CPython quite some time ago to silence deprecation warnings by default.

The reason we did that was because *end users* were getting the warnings,

not just developers of affected projects.

At the same time, unittest and other test runners were updated to enable

the warnings, under the theory that at least projects with reasonable test coverage would still receive advance notice of the problem, providing a counterbalance to the risks introduced by silencing them by default. Oops, missed a key paragraph here: the reason this is a problem for setuptools deprecations in particular is that most developers are unlikely to run their setup.py file under the control of a test runner that would enable the warning.

Perhaps once wheel files are more widespread, we can look at having pip

always enable deprecation warnings when building from source.

Cheers, Nick.