On Monday, November 19, 2012 at 7:37 PM, PJ Eby wrote:

Can we maybe kill Provides-Dist and its associated baggage first, though?

I would love to kill Provides-Dist. The biggest question there is how do you handle it's functionality? If someone needs setuptools but they have distribute installed they both shouldn't get installed. The need for it for the "2 packages being distributed together" I'm (personally) less concerned about since with proper dependency data we should be able to just depend on things instead of bundling them.

The "I bundled a renamed copy of six" is a totally different case which would not invoke provides-dist. "I merged sqlalchemy with a previously separate but wildly popular declarative / database support / whatever extension" would invoke provides-dist. Daniel Holth On Nov 19, 2012, at 7:41 PM, Donald Stufft wrote:

On Monday, November 19, 2012 at 7:37 PM, PJ Eby wrote:

...

Can we maybe kill Provides-Dist and its associated baggage first, though? I would love to kill Provides-Dist. The biggest question there is how do you handle it's functionality? If someone needs setuptools but they have distribute installed they both shouldn't get installed.

The need for it for the "2 packages being distributed together" I'm (personally) less concerned about since with proper dependency data we should be able to just depend on things instead of bundling them.

The distro repos are centrally managed, too. Try getting setuptools to provide a virtual package just because you want to fork.. and then update the dependent packages? Daniel Holth On Nov 19, 2012, at 7:49 PM, Donald Stufft wrote:

Other languages seem to get along fine without it. Even the OS managers which have it don't allow it to be used to masquerade as another project, only to make generic virtual packages (e.g. "email"). On Monday, November 19, 2012 at 7:43 PM, Daniel Holth wrote:

...

Does it really have baggage? I think it is necessary, although it doesn't do favors to the implementer (and has never been implemented). How else is anyone supposed to fork or merge projects?

Daniel Holth

On Nov 19, 2012, at 7:37 PM, PJ Eby wrote:

...

On Mon, Nov 19, 2012 at 6:53 PM, Daniel Holth wrote:

...

I think this PEP is a significant improvement from its predecessor. It represents features like extras (provides-extra) and build requirements (setup-requires-dist) that are in use in the Python community but cannot be represented in older versions of the format, it finally specifies a UTF-8 encoding, removes RFC 822, provides an extension mechanism, and allows the description to be placed in the document payload.

Can we maybe kill Provides-Dist and its associated baggage first, though?

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com

We are getting along fine too. No tool parses metadata 1.x for package management reasons and provides has existed forever with no implementation. So it is not inconveniencing anyone. I would prefer to leave it alone. Daniel Holth On Nov 19, 2012, at 7:49 PM, Donald Stufft wrote:

Other languages seem to get along fine without it. Even the OS managers which have it don't allow it to be used to masquerade as another project, only to make generic virtual packages (e.g. "email"). On Monday, November 19, 2012 at 7:43 PM, Daniel Holth wrote:

...

Does it really have baggage? I think it is necessary, although it doesn't do favors to the implementer (and has never been implemented). How else is anyone supposed to fork or merge projects?

Daniel Holth

On Nov 19, 2012, at 7:37 PM, PJ Eby wrote:

...

On Mon, Nov 19, 2012 at 6:53 PM, Daniel Holth wrote:

...

I think this PEP is a significant improvement from its predecessor. It represents features like extras (provides-extra) and build requirements (setup-requires-dist) that are in use in the Python community but cannot be represented in older versions of the format, it finally specifies a UTF-8 encoding, removes RFC 822, provides an extension mechanism, and allows the description to be placed in the document payload.

Can we maybe kill Provides-Dist and its associated baggage first, though?

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com

On Mon, Nov 19, 2012 at 07:49:41PM -0500, Donald Stufft wrote:

Other languages seem to get along fine without it. Even the OS managers which have it don't allow it to be used to masquerade as another project, only to make generic virtual packages (e.g. "email").

I'm not sure this assertion about OS package managers is correct. I've only just read: http://www.python.org/dev/peps/pep-0426/#provides-dist-multiple-use but the rough rpm analogue seems to be the Provides: tag. Provides is given a string which is parsed into a name or a name and version like this: Provides: python Provides: python = 3.1.0 rpm has no way at package build time to tell that a particular name given in a provides in one package is the actual name of another package. At installtime, rpm keeps package names and provides names separately but in dependency comparisons either one can be used to satisfy a requirement. What that means is that when asking about information on a package with name "python", you'll get information about the python package with that name and not about anything else that Provides: "python". But if you are installing something that has a requirement on "python" either the package with the name python or any package that Provides: python can satisfy the requirement. Package managers with builtin dep solvers can be built on top of rpm. The one that I am familiar with is yum. Since yum is downloading the packages that are being fed into rpm, yum could choose to prefer the package name instead of things in Provides when it downloads. It doesn't, though. Just like the underlying rpm, it treats package names and names specificed through Provides: as equivalent. -Toshio

On Monday, November 19, 2012 at 7:43 PM, Daniel Holth wrote:

Does it really have baggage? I think it is necessary, although it doesn't do favors to the implementer (and has never been implemented). How else is anyone supposed to fork or merge projects?

Daniel Holth

On Nov 19, 2012, at 7:37 PM, PJ Eby wrote:

On Mon, Nov 19, 2012 at 6:53 PM, Daniel Holth wrote:

I think this PEP is a significant improvement from its predecessor. It represents features like extras (provides-extra) and build requirements (setup-requires-dist) that are in use in the Python community but cannot be represented in older versions of the format, it finally specifies a UTF-8 encoding, removes RFC 822, provides an extension mechanism, and allows the description to be placed in the document payload.

Can we maybe kill Provides-Dist and its associated baggage first, though?

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ donald.stufft%40gmail.com

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/a.badger%40gmail.com

Daniel Holth writes:

Does [Provides-Dist] really have baggage? I think it is necessary, although it doesn't do favors to the implementer (and has never been implemented). How else is anyone supposed to fork or merge projects?

It doesn't bother me personally if this traffic is on python-dev, but this looks rather technical. Shouldn't the distutils-sig at least be CC'd? Steve

On Monday, November 19, 2012 at 9:01 PM, Nick Coghlan wrote:

Look more closely at the docs for "Obsoletes" in RPM, not just those for "Provides". Being able to transparently replace an existing package with a renamed one that installs files with the same names is certainly part of the purpose/capabilities of the RPM dependency machinery (i.e. precisely the distribute vs setuptools situation). We may want to clarify the wording to ensure it is clear that the provision of the dist name (as posted on PyPI) is implied, though. Cheers, Nick. I'm not sure if you're responding to me or not, but we also have Obsoletes-Dist in the metadata. (Which I don't like the name of but that's for another argument).

On Monday, November 19, 2012 at 9:24 PM, Daniel Holth wrote:

Mostly it seems a bit silly to have so much conversations about parts of the pep that remain unchanged from previously accepted versions...

Well, I think the PEP should describe what we expect to be implemented *shrug*. Either we should expect it to be implemented and it should be part of the spec, or we shouldn't expect people to implement it and it should be removed.

Daniel Holth writes:

Mostly it seems a bit silly to have so much conversations about parts of the pep that remain unchanged from previously accepted versions...

I don't agree with the suggestion that we shouldn't discuss it because it was accepted in a previous version. Perhaps it didn't receive the right scrutiny at that time, but since it hasn't been implemented, it's reasonable to discuss it. ISTM that implementing it as suggested in the PEP can lead to certain problems, since it is a multi-valued field. If it is left in, then something should be said in the PEP about the potential difficulties and if/how they can be resolved. The difficulties I am talking about relate to dependency resolution. Given the current definition of Provides-Dist, it is possible for a package A on PyPI to "Provide" all of e.g. "A (1.0)", "B (1.2)" and "C (1.5)", and it is also possible for packages B and C on PyPI to provide the same (or slightly different) versions of logical packages of A, B, and C. This will likely lead to the need for a sophisticated dependency resolver because the dependency graph can get quite convoluted. (Remember, we might need to do this resolution when removing packages as well as when installing them.) I know there are SAT solvers and such, but I'm not sure we need that level of sophistication, or whether its complexity cost is outweighed by any benefit. Remember, we are managing fine without multi-valued Provides-Dist, and while a case has been made for virtual packages and forks (which just require a single-valued field), no compelling case has been made for bundling packages in general (I understand that such requirements might sometimes arise in certain corporate environments, but they don't seem to be a mainstream use case). Hence, no strong case has been made for a multi-valued "Provides" field. If we have a good index and packaging infrastructure, there is no general need for packages to bundle other packages, unless those bundled packages are changed in some way to suit the bundler's needs. In that case, I don't know how you could be sure that a bundled "A (1.0)" hasn't diverged from the equivalent package on PyPI. The "Provides" seems essentially useless in a metadata index, since if, when asked to install D which has a dependency on A, you would download and install A to resolve it rather than B or C, and I can't see when you would want to query the index to say "who provides A?" and then use some heuristic to pick e.g. B or C, rather than A. distlib currently contains support for the multi-valued "Provides", but I'm not confident that will work as expected given pathological cases like the example I suggested, without getting "complicated" in the Zen of Python sense. I'm not convinced that the maintenance burden of a complicated solution is worth the heretofore unnecessary ability to bundle stuff in arbitrary ways. Regards, Vinay Sajip

Daniel Holth writes:

If you don't have Provides-Dist, then distribute must continue to bundle an extra .egg-info directory to emulate the feature. This is more than enough justification for me. Name: is essentially an alias for Provides-Dist: (or vice-versa) so there is no such thing as a single-valued Provides-Dist. Having two names for a package is just as complicated as having twenty.

I'm not so sure. In the case of two names, it could be assumed that one was a fork of the other (as in the specific cases of distribute/setuptools, or PIL/ Pillow). You cannot reasonably make this assumption if you have twenty entries in your Provides-Dist.

You should not implement Provides-Dist by searching for every Provides-Dist: name on PyPI.

I wasn't seriously suggesting that this approach be taken - merely pointing out that Provides-Dist isn't of much use in a metadata index.

should only use it when deciding whether to download setuptools when distribute is already installed and a package depends on setuptools.The bundling term was bad wording on the part of the PEP. No one should ever include non-renamed copies of other dists in their dists "import six" vs. "import django.util.six". I've suggested a new wording in this thread.

So apart from the setuptools/distribute and PIL/Pillow scenarios, what are the scenarios where you would have 3 or more values in Provides-Dist? If they are e.g. a bundled SQLAlchemy, why would that be preferable to an entry in Requires-Dist? Regards, Vinay Sajip

Nick Coghlan writes:

Provides/Requires/Obsoletes are *not* for bundling. Publishing bundled packages on the index is bad, and people shouldn't do it. [detail snipped] It's likely fine if an installer doesn't use sophisticated graph analysis to find the "best" way to satisfy a set of requirements - you can just as easily use it in the simple way Daniel describes of only using these fields to check for existing locally installed packages with the necessary capabilities, before going out to get whatever is missing from the package index based purely on the distribution names.

In which case, it seems sensible to put these constraints into the PEP, so that both PEP implementers and users of those implementations have these guidelines clarified. Regards, Vinay Sajip

On Wed, Nov 21, 2012 at 2:04 AM, Vinay Sajip wrote:

Nick Coghlan writes:

...

Provides/Requires/Obsoletes are *not* for bundling. Publishing bundled packages on the index is bad, and people shouldn't do it. [detail snipped] It's likely fine if an installer doesn't use sophisticated graph analysis to find the "best" way to satisfy a set of requirements - you can just as easily use it in the simple way Daniel describes of only using these fields to check for existing locally installed packages with the necessary capabilities, before going out to get whatever is missing from the package index based purely on the distribution names.

In which case, it seems sensible to put these constraints into the PEP, so that both PEP implementers and users of those implementations have these guidelines clarified.

Yes, I thought Daniel's rewording looked pretty reasonable on that front. However, the details of how an installer uses this information is really up to the installer developers and what their users expect/demand. It certainly isn't *practical* to do a full dependency analysis when PyPI doesn't provide the same kind of precalculated metadata that a yum repo does, but that's not something that should be spelled out in the distribution metadata PEP, any more than it is spelled out in the RPM format spec. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, Nov 20, 2012 at 11:49 AM, Vinay Sajip wrote:

Also: what happens when a requirement is for setuptools (>= X.Y), but the distribute fork hasn't kept pace, and so only supports setuptools at a lower version than X.Y? I take it we're entirely comfortable with installing setuptools X.Y in that case? How would you ensure the right setuptools is always loaded, since presumably both are on sys.path?

Egg-based tools don't have any problem with this, since they set sys.path to include the eggs needed for the running program. Other tools will have to tell the user and let them work it out, e.g. by using a different virtualenv. I personally don't think that forks claiming to "provide" something is really a good thing to encourage; ISTM that saying a package *conflicts* with another is more accurate, e.g. distribute Conflicts-Dist setuptools. I also think distributions should say they are obsoleted, rather than allowing other distributions to obsolete them. That is, centralized packaging systems rely on a central authority to resolve issues of who provides what and obsoletes what; there's an implicit "x obsoletes y [by decree of semi-independent third-party z]". However, in Python package metadata, it's "x obsoletes y [by decree of x]". IMO, this should be reversed to, "Y is obsoleted by x [by decree of y]", and "installing Y will conflict with X [by decree of X]", so that in each case the scope of authority for the statement is clear. That is, in each case (conflict or obsolescence), the project's developers are declaring under what conditions they will not be supporting an installation. In the case of obsolescence, the developer is saying, "this is being phased out, you should use that other thing instead". In the case of forks, the developer is saying, "If you install both versions, something's gonna break." Note that installation conflict is a more conservative claim anyway: a conflict between forked "foobar" packages is permanent, in the sense that it doesn't matter what versions of both packages you're interested in: they both want to install a foobar/__init__.py. (Of course, installers can and should detect that condition automatically, but not until they download the package first.)

On 11/20/2012 12:46 PM, PJ Eby wrote:

On Tue, Nov 20, 2012 at 11:49 AM, Vinay Sajip mailto:vinay_sajip@yahoo.co.uk> wrote:

Also: what happens when a requirement is for setuptools (>= X.Y), but the distribute fork hasn't kept pace, and so only supports setuptools at a lower version than X.Y? I take it we're entirely comfortable with installing setuptools X.Y in that case? How would you ensure the right setuptools is always loaded, since presumably both are on sys.path?

Egg-based tools don't have any problem with this, since they set sys.path to include the eggs needed for the running program. Other tools will have to tell the user and let them work it out, e.g. by using a different virtualenv.

I personally don't think that forks claiming to "provide" something is really a good thing to encourage; ISTM that saying a package *conflicts* with another is more accurate, e.g. distribute Conflicts-Dist setuptools. I also think distributions should say they are obsoleted, rather than allowing other distributions to obsolete them.

Obsolete distributions won't say they are obsoleted, unless they receive further maintenance. However, if the distribution is obsolete because the maintainer has lost interest, they won't receive further maintenance.

That is, centralized packaging systems rely on a central authority to resolve issues of who provides what and obsoletes what; there's an implicit "x obsoletes y [by decree of semi-independent third-party z]".

However, in Python package metadata, it's "x obsoletes y [by decree of x]". IMO, this should be reversed to, "Y is obsoleted by x [by decree of y]", and "installing Y will conflict with X [by decree of X]", so that in each case the scope of authority for the statement is clear.

That is, in each case (conflict or obsolescence), the project's developers are declaring under what conditions they will not be supporting an installation. In the case of obsolescence, the developer is saying, "this is being phased out, you should use that other thing instead". In the case of forks, the developer is saying, "If you install both versions, something's gonna break."

Note that installation conflict is a more conservative claim anyway: a conflict between forked "foobar" packages is permanent, in the sense that it doesn't matter what versions of both packages you're interested in: they both want to install a foobar/__init__.py. (Of course, installers can and should detect that condition automatically, but not until they download the package first.)

On Tue, Nov 20, 2012 at 4:07 PM, Glenn Linderman wrote:

On 11/20/2012 12:46 PM, PJ Eby wrote:

I personally don't think that forks claiming to "provide" something is really a good thing to encourage; ISTM that saying a package *conflicts* with another is more accurate, e.g. distribute Conflicts-Dist setuptools. I also think distributions should say they are obsoleted, rather than allowing other distributions to obsolete them.

Obsolete distributions won't say they are obsoleted, unless they receive further maintenance. However, if the distribution is obsolete because the maintainer has lost interest, they won't receive further maintenance.

(We've been over this before, the last time this discussion came up on the Distutils-SIG for a previous Metadata PEP a year or two back, but here goes....) Obsoleting a package is for handling renames and support transitions. For example, if it actually did anything to do so, I'd mark RuleDispatch as obsoleted-by PEAK, the Pylons folks might mark some version of that as obsoleted-by Pyramid, etc. To put it another way, marking a package obsolete is part of deprecation and replacement, not an unsubstantiated third-party claim about the maintenance status of an unrelated project. If a package is *actually* dead, there's no real point to declaring that something else obsoletes it, and certainly no reason to put it in metadata form. Otherwise, we could have Twisted claiming to obsolete GEvent and vice-versa at the same time. Which one should an installer believe? It makes no sense in a standard where the project's maintainers can say whatever they want about somebody else's project. The scope of authority for automatically-consumed metadata should *only* encompass the project that provided the metadata.

I think the Metadata 1.1 treatment of these concepts is in some ways better. (Metadata 1.2 added the -Dist suffix to the fields in an attempt to make it clear that dependency names are PyPI names and not "import x" names.) http://www.python.org/dev/peps/pep-0314/ says: Provides (multiple use) Each entry contains a string describing a package or module that will be provided by this package once it is installed. These strings should match the ones used in Requirements fields. A version declaration may be supplied (without a comparison operator); the package's version number will be implied if none is specified. Example: Provides: xml Provides: xml.utils Provides: xml.utils.iso8601 Provides: xml.dom Provides: xmltools (1.3) Obsoletes (multiple use) Each entry contains a string describing a package or module that this package renders obsolete, meaning that the two packages should not be installed at the same time. Version declarations can be supplied. The most common use of this field will be in case a package name changes, e.g. Gorgon 2.3 gets subsumed into Torqued Python 1.0. When you install Torqued Python, the Gorgon package should be removed. Example: Obsoletes: Gorgon They mean pretty much what the same words mean in RPM and do not need further bikeshedding.

No. We trust the packages we install, including the way they decide to use the metadata. A bad package could delete all our files or cause dependency resolution to fail. Mostly they won't. Daniel Holth On Nov 20, 2012, at 5:26 PM, Vinay Sajip wrote:

Daniel Holth writes:

...

They mean pretty much what the same words mean in RPM and do not need further bikeshedding.

But isn't it the case that the scenarios are different because in the case of RPMs, we have a presumed authority which can determine e.g. what obsoletes what, whereas with Python distributions, there's no central authority that has this function?

Regards,

Vinay Sajip

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

On 11/20/2012 1:48 PM, PJ Eby wrote:

(We've been over this before, the last time this discussion came up on the Distutils-SIG for a previous Metadata PEP a year or two back, but here goes....)

Thanks. I wasn't over there. Makes it clear that clarifying PEPs to reflect discussions is a good idea.

On Tue, Nov 20, 2012 at 5:01 PM, Daniel Holth wrote:

http://www.python.org/dev/peps/pep-0314/ says:

The most common use of this field will be in case a package name changes, e.g. Gorgon 2.3 gets subsumed into Torqued Python 1.0. When you install Torqued Python, the Gorgon package should be removed.

Example:

Obsoletes: Gorgon

They mean pretty much what the same words mean in RPM and do not need further bikeshedding.

The problem is that the above *makes no sense*. "Torqued Python" and "Gorgon" are veiled pseudonyms for Twisted and Medusa.... and Twisted is not actually a plug-and-play substitute for Medusa, AFAIK. Can anybody suggest an *actual* use case for "Obsoletes", and explain how it is supposed to work in software? The last time this discussion came up, nobody had any use cases that stood up to the "how's that actually going to work and/or help?" test. Here's a post of mine summarizing this and related points in the previous thread: http://mail.python.org/pipermail/catalog-sig/2010-October/003364.html

On Tue, Nov 20, 2012 at 6:43 PM, Daniel Holth wrote:

No. We trust the packages we install, including the way they decide to use the metadata. A bad package could delete all our files or cause dependency resolution to fail. Mostly they won't.

That's sort of beside the point. The *only* use case which Obsoletes provides over Obsoleted-By is that it allows third parties to unilaterally advertise their forked project as a substitute for the original, and maybe block users from switching back to the un-forked project -- regardless of the status of the original project or the consent of the original project's maintainer. This use case, however, benefits nobody besides the forkers. There are many other legitimate channels by which the forkers can advertise themselves as a replacement for their parent project, and no reason for the installing end user to be bothered with the subject, except in case of a conflict. For somebody obsoleting their own package, on the other hand, it's likely well worth the effort to at least update their PyPI metadata to reflect the change in status -- especially if this can be done through the web interface. It's likely they would wish to update their description as well, to notify human beings of the change. But here's the thing that kills "Obsoletes" dead in the first place as a practical tool: unless installers use a PyPI search before installing *every single project*, there is no way for them to realize that the obsoleting package exists! By contrast, if a package is "Obsoleted-By", then installing that package (or declaring a dependency on it) provides an opportunity to inform the user of the need to make a transition. This can't be done with an "Obsoletes" field. Conversely, if you have already installed a package that says it "Obsoletes" another package, this does *not* tell you that the obsolete package shouldn't still be installed! A replacement project doesn't necessarily share the same API, and may exist in a different package namespace altogether. In short, "Obsoletes" is virtually *useless* as a machine-consumed metadata field, because there is nothing you can actually do with it in a practical installer. I'm against adding more fields to the metadata which do not have a specification for how they should be used in practice; the presence of such fields has been a problem with most of the preceding metadata specs, IMO.

I re-named a package once just because I did not like the name. I used "Obsoletes" for that. It is documentation.

Note that "Obsoleted-By" would also serve that use case, and have the additional benefit of being able to notify people who install new copies of the replaced project. (By the way Daniel, I'm sorry I didn't comment on this PEP sooner; I'd forgotten about the previous PEP 345 rehashing in 2010, or rather, I just sort of assumed that the results of that discussion had been incorporated into the newer PEP, and didn't notice the reappearance of the noise fields until your call for approval just now. Sorry!)

On Tue, Nov 20, 2012 at 7:18 PM, Jim Jewett wrote:

On 11/20/12, Daniel Holth wrote:

...

On Tue, Nov 20, 2012 at 3:58 PM, Jim J. Jewett wrote:

...

...

Vinay Sajip reworded the 'Provides-Dist' definition to explicitly say:

...

...

...

The use of multiple names in this field *must not* be used for bundling distributions together. It is intended for use when projects are forked and merged over time ...

...

...

(1) Then how *should* the "bundle-of-several-components" case be represented?

...

The useful way to bundle a bunch of things would be to just include them all in an executable folder or zipfile with __main__.py. PEP 426 and the package database would not get involved. The bundle would be distributed as an application you can download and use, not as an sdist on PyPI.

When I look at, for example, twisted, there are some fairly fine distinctions.

I can imagine some people wanting to handle each little piece differently, since that is the level at which they would be replaced by a more efficient implementation. That doesn't mean that someone using the default should have to manage 47 separate little packages individually.

Also note that ZODB is mentioned as a bundling example in the current (2012-11-14) PEP. What does the PEP recommend that they do? Stop including transaction? Keep including it but stop 'Provides-Dist'-ing it?

The current PEP also specifies that "This field must include the project identified in the Name field, followed by the version : Name (Version)." but the examples do not always include version. Why is the MUST there?

ZODB is a bad example. The word "bundling" will be struck from the PEP entirely. Two sdists should not be combined into one sdist when both packages are still being developed. If A and B are merged into a single PyPI package C, and A and B will no longer be developed, then C may Provides-Dist A and B. http://www.python.org/dev/peps/pep-0426/#requires-dist-multiple-use No MUST on the Requires-Dist version. If no version is there, it should satisfy any version requirement. Is there some way to distinguish between concrete and abstract

provisions? For example, if MyMail (2012.11.10) includes 'Provides-Dist: email', does that really get parsed as 'Provides-Dist: email (2012.11.10)'?

No.

...

(2) How is 'Provides-Dist' different from 'Obsoletes-Dist'? The only difference I can see is that it may be a bit more polite to people who do want to install multiple versions of a (possibly abstract) package.

The intent of Provides and Obsoletes is different. Obsoletes would not

...

satisfy a requirement during dependency resolution.

...

The RPM guide explains a similar system:

As best I can understand, Obsoletes means "Go ahead and uninstall that other package." Saying that *without* providing the same functionality seems like a sneaky spelling of "Please break whatever relies on that other package."

I'm willing to believe that there is a more useful meaning. I'm also willing to believe that they are logically redundant but express different intentions. The current wording doesn't tell me which is true. (Admittedly, that is arguably an upstream bug with other package systems, but you should still either fix it or explicitly delegate the definitions.)

And as long as I'm asking for clarification, can foopkg-3.4 obsolete foopgk3.2? If not, is it a semantics problem, or just not idiomatic? If so, does it have a precise meaning, such as "no longer interoperates with"?

When I used Obsoletes, it meant "I am no longer developing this other package that is identical to this re-named package". The system of requirements/conflicts (as the RPM system) does not appear to be entirely orthogonal. And now that I've looked more carefully ...

Can a "Key: Value" pair be continued onto another line? The syntax description under "Metadata Files" does not say so, but later text suggests that either leading whitespace or a leading tab specifically (from the example code) will work. (And is description a special case?)

Description (now in the payload, please) is the only field that is commonly multi-line. Any field could continue onto the next line as far as the parser is concerned. It probably would not make sense. Is the payload assumed to be utf8 text? Can it be itself a mime message?

The entire file needs to be utf-8. The payload is assumed to be utf-8 text in this version. Wouldn't a mime message also be utf-8 text? (we wouldn't know what to do with it)

Are there any restrictions on 'Name'? e.g., Can the name include spaces? line breaks? Must it be a valid python identifier? A valid python qualname?

setuptools constrains it to alphanumeric characters. Metadata 1.3 doesn't say. 'Version' says that it must be in the format specified in PEP 386.

Unfortunately, it doesn't say which part of 386. Do you mean that it must be acceptable to verlib.NormalizedVersion without first having to call suggest_normalized_version?

It means it is expected to match: http://www.python.org/dev/peps/pep-0386/#the-new-versioning-algorithm expr = r"""^ (?P<version>\d+\.\d+) # minimum 'N.N' (?P<extraversion>(?:\.\d+)*) # any number of extra '.N' segments (?: (?P<prerel>[abc]|rc) # 'a' = alpha, 'b' = beta # 'c' or 'rc' = release candidate (?P<prerelversion>\d+(?:\.\d+)*) )? (?P<postdev>(\.post(?P<post>\d+))?(\.dev(?P<dev>\d+))?)? $""" Please do not enforce this regexp in your Metadata parser.

'Summary' specifies that it must be one line. Is there a character limit, or do you just mean "no line breaks"? Do you want to add a "Should be less than 80 characters" or some such, based on typical tool presentation? Would it be worth repeating the advice that longer descriptions should go in the payload, after all headers? (Otherwise, they have to find 'Description' *and* notice that it is deprecated and figure out what to do instead.)

Under 'Description', it isn't entirely clear whether what terminates the field. "Multiple paragraphs" suggests that there can be multiple lines, but I'm guessing that -- in practice -- they have to be a single logical line, with all but the first starting with whitespace.

Non-blank lines that don't start with whitespace are keys. email.parser.Parser() takes care of this in an e-mail-inspired (but not any literal RFC) way. The distutils documentation has guidelines on the short description / summary.

Under 'Classifier', is PEP 301 really the current authority for classifiers? I would prefer at least a reference to http://pypi.python.org/pypi?%3Aaction=list_classifiers demonstrating which classifiers are currently meaningful.

Under 'Requires-Dist', there is an unclosed parenthesis.

Does the 'Setup-Requires-Dist' set implicitly include the 'Requires-Dist' set, or should a package be listed both ways if it is required at both setup and runtime?

The Summary of Differences from PEP 345 mentions changes to Requires-Dist, but I don't know what they were -- even the unclosed parentheses seemed the same.

The addition of the allowed "extra" variable in the ; condition is the most significant change.

The appendix gives code for generating and parsing continuation lines that suggests the continuation whitespace is exactly one tab -- is other whitespace OK too?

Any whitespace would work.

On Wed, Nov 21, 2012 at 1:20 PM, Nick Coghlan wrote:

On Wed, Nov 21, 2012 at 1:10 PM, PJ Eby wrote:

...

Conversely, if you have already installed a package that says it "Obsoletes" another package, this does *not* tell you that the obsolete package shouldn't still be installed! A replacement project doesn't necessarily share the same API, and may exist in a different package namespace altogether.

Then that's a bug in the metadata of the project misusing "Obsoletes", and should be reported as such. If the new package is not a drop-in replacement, then it has no business claiming to obsolete the other package.

I think one of the big reasons this kind of use is rare in the Python community is that project name changes are almost always accompanied by *package* name changes, and as soon as you change the package name, you're changing the public API, and thus it is no longer appropriate to use Provides or Obsoletes, as the renamed project is no longer a drop-in replacement for the original.

I realised that my comments above are more about the appropriate use of "Provides", rather than "Obsoletes". For a practically useful "Obsoletes", I think I'm inclined to agree with you, as "Obsoleted-By" provides a way for a maintainer to explicitly declare that a project is no longer receiving updates, and users should migrate to the replacement project if they want to continue to receive fixes and improvements. The current version of "Obsoletes" is, as Daniel describes, really only useful as documentation. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Nov 21, 2012 at 12:00 AM, Stephen J. Turnbull wrote:

Daniel Holth writes:

...

When I used Obsoletes, it meant "I am no longer developing this other package that is identical to this re-named package".

But as a user I could care less! The authors may care, but I don't care if Torqued "obsoletes" Gorgon, because in using Torqued I'm DTRT'ing even though I don't know it.

What I care about is when I'm using Gorgon, and there's something "better" (or worse, "correct") to use in my application.

Hence my suggestion for an Obsoleted-By field, in which Gorgon would be able to suggest alternatives.

It might be a good idea to have a just-like-Amazon

While-This-Package-Is-Great-You-Might-Also-Consider:

field.

Yeah, that's basically what Obsoleted-By is for.

How to use Obsoletes: The author of B decides A is obsolete. A releases an empty version of itself that Requires: B B Obsoletes: A The package manager says "These packages are obsolete: A". Would you like to remove them? User says "OK". On Wed, Nov 21, 2012 at 2:54 AM, Stephen J. Turnbull wrote:

PJ Eby writes:

...

On Wed, Nov 21, 2012 at 12:00 AM, Stephen J. Turnbull < stephen@xemacs.org>wrote:

...

...

What I care about is when I'm using Gorgon, and there's something "better" (or worse, "correct") to use in my application.

Hence my suggestion for an Obsoleted-By field, in which Gorgon would be able to suggest alternatives.

My bad, my precise intention was to follow up on your idea (which, credit where credit is due, I had *not* hit upon independently). I should have made that clear.

(I really shouldn't be answering English email at a Japanese-speaking conference, my brain thinks it knows what it's doing but shirazuni 日 本文化が染み込む....)

...

...

It might be a good idea to have a just-like-Amazon

While-This-Package-Is-Great-You-Might-Also-Consider:

field.

Yeah, that's basically what Obsoleted-By is for.

Well, Obsoleted-By is pretty strong language for suggesting possible alternatives. But I suspect that few projects would really want to be suggesting competitors' products *or* their own oldie-but-still-goodie that they'd really like to obsolete ASAP (put an Obsoleted-By line in every Python 2 distribution, anyone? :-) _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

On Mon, Dec 3, 2012 at 2:43 PM, Daniel Holth wrote:

How to use Obsoletes:

The author of B decides A is obsolete.

A releases an empty version of itself that Requires: B

B Obsoletes: A

The package manager says "These packages are obsolete: A". Would you like to remove them?

User says "OK".

Um, no. Even if the the author of A and B are the same person, you can't remove A if there are other things on the user's system using it. The above scenario does not work *at all*, ever, except in the case where B is simply an updated version of A (i.e. identical API) -- in which case, why bother? To change the project name? (Then it should be "Formerly-named" or something like that, not "Obsoletes".) Please, *please* see the previous Catalog-SIG discussion I linked: this is only one of multiple metadata fields that were thoroughly debunked in that discussion as completely useless for automated dependency management.

On Wed, Dec 05, 2012 at 02:46:11AM -0500, Donald Stufft wrote:

On Wednesday, December 5, 2012 at 2:13 AM, PJ Eby wrote:

On Mon, Dec 3, 2012 at 2:43 PM, Daniel Holth wrote:

How to use Obsoletes:

The author of B decides A is obsolete.

A releases an empty version of itself that Requires: B

B Obsoletes: A

The package manager says "These packages are obsolete: A". Would you like to remove them?

User says "OK".

Um, no. Even if the the author of A and B are the same person, you can't remove A if there are other things on the user's system using it. The above scenario does not work *at all*, ever, except in the case where B is simply an updated version of A (i.e. identical API) -- in which case, why bother? To change the project name? (Then it should be "Formerly-named" or something like that, not "Obsoletes".)

You can automatically uninstall A from B in an automatic dependency management system. I *think* RPM does this, at the very least

This is correct.

I believe it refuses to install B if A is already there (and the reverse as well).*

I'd have to test this but I believe you are correct about the first. Not sure about the reverse.

There's nothing preventing an installer from, during it's attempt to install B, see it Obsoletes A, looking at what depends on A and warning the user what is going to happen and prompt it.

In rpm-land, if something depended on A and nothing besides the actual A package provided A, rpm will refuse to install B. But rpm is meant to be used unattended so different package managers could certainly choose to prompt. For package renames, package B would have both an Obsoletes: A <= $OLD_VERSION and a Provides: A = NEW_VERSION -Toshio

On Wed, Dec 5, 2012 at 4:10 PM, PJ Eby wrote:

On Wed, Dec 5, 2012 at 2:46 AM, Donald Stufft wrote:

...

There's nothing preventing an installer from, during it's attempt to install B, see it Obsoletes A, looking at what depends on A and warning the user what is going to happen and prompt it.

Unless the user wrote those things that depend on A, they aren't going to be in a position to do anything about it. (Contrast with a distro, where dependencies are indirect - the other package will depend on an abstraction provided by both A and B, rather than directly depending on A *or* B.)

(Also note that all the user knows at this point is that the author of B *claims* to obsolete A, not that the authority managing the repository as a whole has decreed B to obsolete A.)

...

You can automatically uninstall A from B in an automatic dependency management system

My point is that this can only work if the "obsoleting" is effectively just a rename, in which case the field should be "renames", or better still, "renamed-to" on the originating package.

As I've mentioned repeatedly, Obsoleted-By handles more use cases than Obsoletes, and has at least one practical automated use case (notifying a developer that their project is depending on something that's obsolete).

Also, the example given as a use case in the PEP (Gorgon to Torqued) is not just wrong, it's *actively misleading*. Gorgon and Torqued are transparent renames of Medusa and Twisted, which do not share a common API and thus cannot be used as the subject of any automated processing (in the case of Obsoletes) without doing some kind of PyPI metadata search for every package installed every time a package is installed.

...

I think Obsoletes as is an alright bit of information.

1. It cannot be used to prevent the installation of an obsolete package without a PyPI metadata search, since you must examine every *other* package on PyPI to find out whether some package obsoletes the one you're trying to install.

2. Unlike RPM, where metadata is provided by a trusted third party, Obsoletes can be specified by any random forker (no pun intended), which makes this information a mere advertisement... and an advertisement to the wrong audience at that, because they must have *already* found B in order to discover that it replaces A!

3. Nobody has yet supplied a use case where Obsoletes would not be strictly improved upon by Obsoleted-By. (Note that "the author of package X no longer maintains it" does not equal "package Y is entitled to name itself the successor and enforce this upon all users" -- this can work in RPM only because it is a third party Z who declares Y the successor to X, and there is no such party Z in the Python world.)

...

I don't see this in this thread, could you link it again?

http://mail.python.org/pipermail/catalog-sig/2010-October/003368.html http://mail.python.org/pipermail/catalog-sig/2010-October/003364.html

These posts also address why a "Conflicts" field is *also* unlikely to be particularly useful in practice, in part for reasons that relate to differences between RPM-land and Python-land. (For example, RPMs can conflict over things besides files, due to runtime and configuration issues that are out-of-scope for a Python installer tool.)

While it's certainly desirable to not invent wheels, it's important to

My desire is to invent the useful "wheel" binary package format in a reasonable and limited amount of time by making changes to Metadata 1.2 and implementing the new metadata format and wheel in distribute and pip. Help me out by allowing useless but un-changed fields to remain in this version of the PEP. I am done with the PEP and submit that it is not worse than its predecessor. I can participate in a discussion about any of the following: Summary of Differences From PEP 345http://www.python.org/dev/peps/pep-0345 - Metadata-Version is now 1.3. - Values are now expected to be UTF-8. - A payload (containing the description) may appear after the headers. - Added extra to environment markers. - Most fields are now optional. - Changed fields: - Description - Project-URL - Requires-Dist - Added fields: - Extension - Provides-Extra - Setup-Requires-Dist

On Dec 05, 2012, at 04:10 PM, PJ Eby wrote:

While it's certainly desirable to not invent wheels, it's important to understand that the Python community does not work the same way as a Linux distribution. We are not a single organization shipping a fully-functional and configured machine, we are hundreds of individual authors shipping our own stuff. Conflict resolution and package replacement (and even deciding what it is that things "provide" or "require") are primarily *human* processes, not technical ones. Relationship and support "contracts", IOW, rather than software contracts.

That's why, in the distro world, a package manager can use simple fields to carry out the will of the human organization that made those support and compatibility decisions. For Python, the situation is a bit more complicated, which is why clear thinking is needed. Simply copying fields blindly from other packaging systems just isn't going to cut it.

+1

Now, if the will of the community is to turn PyPI into a distro-style repository, that's fine...

Please no! -1 :) -Barry

On Wednesday, December 5, 2012 at 6:18 PM, Barry Warsaw wrote:

On Dec 05, 2012, at 06:07 PM, Donald Stufft wrote:

...

If you're installing B you've prescribed trust to that author. If you don't trust the author then why are you installing (and then executing) code they wrote.

What you installed Z, but B got installed because it was a dependency three levels down?

Sure, you granted trust to Z, Z granted trust to Y, and Y granted trust to B. Like in SSL certificates there was a chain of trust. If you don't trust Z then don't install their package.

...

Very convenient to declare that one of the major use cases for Obsoletes over Obsoleted-By is not valid because of your own personal opinions. Like I said above, if you're installing a package that someone has uploaded you've implicitly granted them trust. There is far worse things that a bad Python citizen can do during, and after and install that what is allowed by Obsoletes.

Well, basically never installing anything from PyPI except into a virtualenv is probably a good recommendation (maybe even now).

A virtualenv only protects you from well behaved packages. There is no way to prevent a package author from doing very nasty things to you if they wish. Providing more power in the metadata doesn't make this situation better or worse, it just makes more standard paths in the cases where you do need to do it.

...

End systems often times do not have a singular organization controlling every package in their system. The best example is Ubuntu and their PPA's.

Well, PPAs are awesome, but have known and well-publicized trust issues. I wouldn't enable a PPA into my running system without really knowing who the owner is and why I'm using their PPA. Or doing a lot of testing in a chroot first, and probably pinning the package set to just the one(s) from the PPA I care about.

Basically the same thing can be said about packages on PyPI. All the same trust issues exist there. Simply installing a Python package is already granting far more trust than Obsoletes requires since installing a package is executed someone else's python code on your system. Even if you remove setup.py you're still going to be executing their code on your system. If you do not trust the author of the packages you are installing, you do not install their packages.

On Wed, Dec 5, 2012 at 6:07 PM, Donald Stufft wrote:

Arguing over Obsoletes vs Renames is a massive bikeshedding argument.

And is entirely beside the point. The substantive question is whether it's Obsoletes or Obsoleted-By - i.e., which side is it declared on.

So it's a bad example. Hardly an argument against it.

Nobody has actually proposed a better one, outside of package renaming -- and that example featured an author who could just as easily have used an obsoleted-by field.

Will require support from PyPI but this ultimately isn't a big deal.

...and every PyPI clone. And of course the performance issues.

If you're installing B you've prescribed trust to that author. If you don't trust the author then why are you installing (and then executing) code they wrote.

Trusting their code is one thing; trusting whether they understood a PEP (and its interactions with various installation tools) well enough to not accidentally delete *somebody else's code* out of my system is another thing altogether. OTOH, trusting an author to tell me (in an automated fashion), "hey, you should switch to this other thing as soon as you can" is a FAR smaller amount of required trust. Arguing that because I have to trust one thing, means I must trust another, is a "Fallacy of Gray" argument.

Very convenient to declare that one of the major use cases for Obsoletes over Obsoleted-By is not valid because of your own personal opinions.

I didn't say it was invalid, I said: """Note that "the author of package X no longer maintains it" does not equal "package Y is entitled to name itself the successor and enforce this upon all users""" These things are not equal. AFAIK, well-managed Linux distros do not allow random forkers to declare themselves the official successor to a defunct package, so any analogy between this use case in the Python world and the distro world is strained at *best*.

Unless you think there are no cases where two packages can conflict in more than what files are going to be installed

The rationale for that is laid out in the posts I linked.

then there are cases where it would be helpful

Please, present a *real-life instance* where it would have been helpful to you.

and merely having the ability to use it when it is the best tool for the job isn't going to cause any great issue.

One of the posts I linked presents an instance where it would have actually *harmed* things to specify it, and it's quite easy to see how the same problem would arise if used for non-file-related conflicts... And the problem present is *directly* tied to the lack of a third-party Z who decides whether X and Y, as configured for release Q of distro P, "conflict". This is not a problem that is solvable even in *principle* for an automated tool in the absence of party Z, which means that any such field's actual function is limited to a heads-up to a human user.

This is insane. A fairly simple database query is going to "grind the PyPI servers into dust"? You're going to need to back up this FUD or please refrain from spouting it.

I take it you're not familiar with PyPI's history of performance and scaling problems over the last several years, then. The statically cached "/simple" index was developed precisely to stop *today's* class of installation tools from killing the servers... and then mirroring PyPI was still required to scale. Any proposal that calls for encouraging tools to query a metadata field *every time* a package is installed (or even just downloaded) almost certainly needs to be vetted with the PyPI admin team.

On 2012-12-06 02:12, Stephen J. Turnbull wrote:

I understand the PEP author's frustration with continued discussion, but I think this subthread on Obsoletes vs. Obsoleted-By is not mere bikeshedding on names. It matters *which package* presents the information.

Donald Stufft writes:

...

On Wednesday, December 5, 2012 at 6:18 PM, Barry Warsaw wrote:

...

On Dec 05, 2012, at 06:07 PM, Donald Stufft wrote:

...

If you're installing B you've prescribed trust to that author. If you don't trust the author then why are you installing (and then executing) code they wrote.

The author may be a genius when it comes to writing code, and an idiot when it comes to distributing it. Distribution is much harder than it looks, as you know. Trusting the author's *content* and trusting the author's *metadata* are not equivalent!

As far as I can see, the semantics of putting "Obsoletes: A" into B without changing A are the same as the semantics of putting "Provides: A" into B (without changing A).[1] Only if A includes "Obsoleted-By: B" can a user be confident that B is a true successor to A.

Furthermore, as has been pointed out, the presence of "Obsoleted-By" in A has the huge advantage of informing users and developers of dependent packages alike that A is obsolete when they try to update A. If A is not changed, then an attempted update will tell them exactly that, and they may never find out about B. But if A is modified in this trivial way, the package system can automatically inform them. This is also trivial, requiring no database queries.

"Simple is better than complex."

That makes sense. In summary, someone using B won't care that it has replaced A, but someone using A needs to be told that it has been replaced by B.

On Thu, Dec 6, 2012 at 8:30 AM, Daniel Holth wrote:

My desire is to invent the useful "wheel" binary package format in a reasonable and limited amount of time by making changes to Metadata 1.2 and implementing the new metadata format and wheel in distribute and pip. Help me out by allowing useless but un-changed fields to remain in this version of the PEP. I am done with the PEP and submit that it is not worse than its predecessor.

Agreed. PJE's arguments sound reasonable (especially since Obsoletes doesn't get used much in RPM-land either - Provides & Conflicts are both far more common), but they're orthogonal to the current aims of the metadata 1.3 update. If another author wanted to create a subsequent 1.4 update that was focused on replacing Obsoletes with Obsoleted-By, that would be fine (alternatively, a patch to the current PEP draft may be acceptable, but accepting such a change would be up to Daniel as the PEP author).

I can participate in a discussion about any of the following: Summary of Differences From PEP 345http://www.python.org/dev/peps/pep-0345

- Metadata-Version is now 1.3. - Values are now expected to be UTF-8. - A payload (containing the description) may appear after the headers. - Added extra to environment markers. - Most fields are now optional. - Changed fields: - Description - Project-URL - Requires-Dist - Added fields: - Extension - Provides-Extra - Setup-Requires-Dist

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Dec 6, 2012 at 1:12 PM, Daniel Holth wrote:

Makes sense. How about calling it Replacement. 0 or 1?

Hah, you'd think I'd have learned by now to finish reading a thread before replying. It will be nice to get this addressed along with the other changes :) (FWIW, Conflicts and Obsoletes are messy in RPM as well, and especially troublesome as soon as you start enabling multiple upstream repos from different providers. The metadata problem is handled by prebuilding indices when the repo changes, but that's still more work for the server, and more work for clients)

Replacement (optional) ::::::::::::::::::::::

I like verb forms like Obsoleted-By or Replaced-By, as the noun form is ambiguous about the direction of the change. Since the field being replaced is Obsoletes, Obsoleted-By makes sense.

Indicates that this project is no longer being developed. The named project provides a drop-in replacement.

Typically, the new version *won't* be a drop-in replacement (e.g. you'll likely at least have to import from a different top level package). Instead, the field would more often be used as an explicit indicator that the project is no longer receiving updates, as the *development team* has moved on, so users may want to consider either migrating, taking over development (if the former developers are amenable) or forking. If the replacing project *is* a drop-in replacement for the old project, then it should also advertise a Provides-Dist for the original project. Automated tools can then easily detect the two cases: A Obsoleted-By-Dist B and B Provides-Dist A = A is defunct, and B should be a drop-in replacement for A A Obsoleted-By-Dist B (without a Provides-Dist on B) = A is defunct, B is a replacement for A, but some porting will be needed Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Dec 05, 2012 at 07:34:41PM -0500, PJ Eby wrote:

On Wed, Dec 5, 2012 at 6:07 PM, Donald Stufft wrote:

Nobody has actually proposed a better one, outside of package renaming -- and that example featured an author who could just as easily have used an obsoleted-by field.

How about pexpect and pextpect-u as a better example?

...

Very convenient to declare that one of the major use cases for Obsoletes over Obsoleted-By is not valid because of your own personal opinions.

I didn't say it was invalid, I said:

"""Note that "the author of package X no longer maintains it" does not equal "package Y is entitled to name itself the successor and enforce this upon all users"""

These things are not equal. AFAIK, well-managed Linux distros do not allow random forkers to declare themselves the official successor to a defunct package, so any analogy between this use case in the Python world and the distro world is strained at *best*.

Note that although well-managed Linux distros attempt to control random forking internally, the distro package managers don't prevent people from installing from third parties. So Ubuntu PPAs, upstreams that provide their own rpms/debs, and major third party repos (for instance, rpmfusion as an add-on repo to Fedora) all have and sometimes (mis)use the ability to Obsolete packages in the base repository. So Donald isn't stretching the relationship quite as far as you make it out. The ecosystem of packages for a distro carries uncontrolled packages just as much as pypi.

...

and merely having the ability to use it when it is the best tool for the job isn't going to cause any great issue.

One of the posts I linked presents an instance where it would have actually *harmed* things to specify it, and it's quite easy to see how the same problem would arise if used for non-file-related conflicts...

And the problem present is *directly* tied to the lack of a third-party Z who decides whether X and Y, as configured for release Q of distro P, "conflict".

This is not a problem that is solvable even in *principle* for an automated tool in the absence of party Z, which means that any such field's actual function is limited to a heads-up to a human user.

And the same for Provides. (ie: latest foo is 0.6c; bar Provides: foo-0.6d. an automated tool that finds both foo and bar in its dep tree can choose to install bar and not foo.) The ability for this class of fields to cause harm is not, to me, a compelling argument not to include them. It could be an argument to explicitly tell implementers of install tools that they all have caveats when used with pypi and similar unpoliced community package repositories. The install tools can then choose how they wish to deal with those caveats. Some example strategies: choose to prompt the user as to which to install, choose to always treat the fields as human-informational only, mark some repositories as being trusted to contain packages where these fields are active and other repositories where the fields are ignored. -Toshio

On Thursday, December 6, 2012 at 6:28 AM, Vinay Sajip wrote:

Donald Stufft http://gmail.com)> writes:

Never mind the "Obsoletes" information - even the more useful "Requires-Dist" information is not exposed via PyPI, even though it appears to be stored in the database. (Or if it is, please point me to where - I must have missed it.)

Requires-Dist doesn't exist for more than a handful of packages. But PyPI exposes it via the XMLRPC API, possibly the JSON api as well.

Even if this were to be made available, it's presumably obtained from PKG-INFO. As I understand, this data is not considered reliable - for example, pip runs egg_info on downloaded packages to get updated information when determining dependencies to be downloaded. If the Requires-Dist info in PKG-INFO can't be relied on, surely less critical information such as Obsoletes can't be relied on, either?

pip runs egg_info because setuptools does not write out to PKG-INFO what the dependencies are (it does write it out to a different text file though). But IIRC that text file is not guaranteed to exist in the distribution. There's also the history where pip was trying to preserve as much backwards compat with easy_install as it could, and if you used the file that egg_info writes out then you'll only get the requirements for the system that the distribution was packaged on. Any if statements that affect the dependencies won't be in effect.

Regards,

Vinay Sajip

_______________________________________________ Python-Dev mailing list Python-Dev@python.org (mailto:Python-Dev@python.org) http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com

Daniel Holth writes:

The wheel implementation makes sure all the metadata (the .dist-info directory) is at the end of the .zip archive. It's possible to read the metadata with a single HTTP partial request for the end of the archive without downloading the entire archive.

Sounds good, but can you point to any example code which does this? As I understand it, for .zip files you have to read the last part of the file to get a pointer to the directory, then read that to find where each file in the archive is, then seek to a specific position to read the file contents. Regards, Vinay Sajip

On Thu, Dec 6, 2012 at 9:58 AM, Vinay Sajip wrote:

Daniel Holth writes:

...

The wheel implementation makes sure all the metadata (the .dist-info directory) is at the end of the .zip archive. It's possible to read the metadata with a single HTTP partial request for the end of the archive without downloading the entire archive.

Sounds good, but can you point to any example code which does this? As I understand it, for .zip files you have to read the last part of the file to get a pointer to the directory, then read that to find where each file in the archive is, then seek to a specific position to read the file contents.

You have to make a maximum of 3 requests: one for the directory pointer, one for the directory, and one for the file you want. It's not particularly difficult to make an HTTP-backed seekable file object to pass to ZipFile() for this purpose but I don't have an example. Normally the last few k of the file will contain all 3 pieces. 8k or 16k would be a good guess.

On Thu, Dec 6, 2012 at 11:30 AM, Vinay Sajip wrote:

Daniel Holth writes:

...

You have to make a maximum of 3 requests: one for the directory pointer, one for the directory, and one for the file you want. It's not particularly difficult to make an HTTP-backed seekable file object to pass to ZipFile() for this purpose but I don't have an example. Normally the last few k of the file will contain all 3 pieces. 8k or 16k would be a good guess.

I don't need an example for doing it with multiple HTTP requests. I only asked for an example because you said one could read the metadata "with a single HTTP partial request", and I couldn't see how it could always be done with a single request.

PEP 427 is mute on the subject of zip file comments in a .whl, but perhaps it shouldn't be. IIUC, the directory of the zip file *could* be further from the end of the file by more than 16K, due to the possible presence of a pathologically large comment in the end record.

It's just a "usually works" optimization that might be fun when bandwidth is more important than round trip times. The distance between the directory and the end of the file depends on the size of the directory. Django's is an extreme case at nearly half a meg; most are much smaller. On many filesystems it is cheap to create a sparse file the size of the entire archive and write the partial requests into it. The OS doesn't actually store all the 0's. The other reason wheel puts the metadata at the end is so the metadata can be re-written efficiently without re-writing the entire zipfile. The wheel project implements ZipFile.pop() which truncates the last file from a (normal) zip archive. This is especially useful when the last file is the attached digital signature.

On Thu, Dec 6, 2012 at 9:58 AM, Vinay Sajip wrote:

Daniel Holth writes:

...

The wheel implementation makes sure all the metadata (the .dist-info directory) is at the end of the .zip archive. It's possible to read the metadata with a single HTTP partial request for the end of the archive without downloading the entire archive.

Sounds good, but can you point to any example code which does this? As I understand it, for .zip files you have to read the last part of the file to get a pointer to the directory, then read that to find where each file in the archive is, then seek to a specific position to read the file contents.

ISTR that this is especially true for zipimport: I think it depends on a zipfile signature being present at the *end* of the file. Certainly, the standard for .exe and shell wrappers for zipfiles is to place them at the beginning of the file, rather than the end.

On Fri, Dec 07, 2012 at 01:18:40AM -0500, PJ Eby wrote:

On Thu, Dec 6, 2012 at 1:49 AM, Toshio Kuratomi wrote:

...

On Wed, Dec 05, 2012 at 07:34:41PM -0500, PJ Eby wrote:

...

On Wed, Dec 5, 2012 at 6:07 PM, Donald Stufft wrote:

Nobody has actually proposed a better one, outside of package renaming -- and that example featured an author who could just as easily have used an obsoleted-by field.

How about pexpect and pextpect-u as a better example?

Perhaps you could explain? I'm not familiar with those projects.

pexepect was last released in 2008. Upstream went silent with unanswered bugs in its tracker and no mailing list. A fork of pexpect was created that addressed the issue of unicode type in python2, a python3 port, and has slowly evolvd since then. I see that the original upstream has made some commits to their source repository since the fork was created although there has still been no new release.

...

Note that although well-managed Linux distros attempt to control random forking internally, the distro package managers don't prevent people from installing from third parties. So Ubuntu PPAs, upstreams that provide their own rpms/debs, and major third party repos (for instance, rpmfusion as an add-on repo to Fedora) all have and sometimes (mis)use the ability to Obsolete packages in the base repository.

But in each of these cases, the packages are being defined *with reference to* some underlying vision of what the distro (or even "a distro") is. An Ubuntu PPA, if I understand correctly, is still *building an Ubuntu system*. Python packaging as a whole lacks such frames of reference. A forked distro is still a distro, and it's a fork *of something*. Rpmfusion is defining an enhanced Fedora, not slinging random unrelated packages about.

Uhm.... that's both true and false as any complex system is. rpm and deb are just packaging formats. So: *) Not all packages built build on top of that system. There are rpm packages provided by upstreams that users attempt (to greater and lesser degrees of success) to install on SuSE, RHEL, Fedora, Mandriva, etc. There are debs built for Ubuntu that people attempt to install onto Debian. *) PPAs and rpmfusion may both build on top of an existing system but they can change the underlying structure, replacing components that other pieces of the base system depend on. You talk about the setuptools and distribute problem on pypi.... there's absolutley nothing that prevents someone from building a PPA or a package in a third-party rpm repository that packages a setuptools that Obsoletes: distribute or a distribute package that Obsoletes: setuptools.

...

The install tools can then choose how they wish to deal with those caveats. Some example strategies: choose to prompt the user as to which to install, choose to always treat the fields as human-informational only, mark some repositories as being trusted to contain packages where these fields are active and other repositories where the fields are ignored.

A peculiar phenomenon: every defense of these fields seems to refer almost exclusively to how the problems could be fixed or why the problems aren't that bad, rather than *how useful the fields would be* in real-world scenarios. In some cases, the argument for the fields' safety actually runs *counter* to their usefulness, e.g., the fields aren't that bad because we could make them have a limited function or no function at all. Isn't lack of usefulness generally considered an argument for *not* including a feature? ;-)

If you constantly forget why the fields are useful, then I suppose you'll always believe that :-) -Toshio

On Fri, Dec 7, 2012 at 3:47 PM, PJ Eby wrote:

In effect, a "conflicts" field actually *creates* conflicts and maintenance burdens where they did not previously exist, because even after the conflict no longer really existed, an automated tool would have prevented PyDispatch from being installed, or, per your suggestion above, unnecessarily *uninstalled* it after a user installed RuleDispatch.

That's not what a Conflicts field is for. It's to allow a project to say *they don't support* installing in parallel with another package. It doesn't matter why it's unsupported, it's making a conflict perceived by the project explicit in their metadata. Such a field is designed to convey information to users about *supported* configurations, regardless of whether or not they happen to work for a given use case. If a user believes a declared conflict is in error, and having the two installed in parallel is important to them, they can: 1. Use virtual environments to keep the two projects isolated from each other 2. Use an installer that ignores Conflicts information (which will be all of them, since that's the status quo) 3. Make their case to the upstream project that the conflict has been resolved, and installing the two in parallel no longer causes issues Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sat, Dec 8, 2012 at 8:02 AM, PJ Eby wrote:

...

*) Not all packages built build on top of that system. There are rpm packages provided by upstreams that users attempt (to greater and lesser degrees of success) to install on SuSE, RHEL, Fedora, Mandriva, etc. There are debs built for Ubuntu that people attempt to install onto Debian.

Sure. But the reference points still exist, and there is a layer of indirection between "packager" and "developer", even in the case where the packager and developer are the same person or organization. In the Python case, there is usually no such indirection, outside of curated systems like SciPy et al. (Even there, most of what third-party packaging is about in the Python world is taking care of binary builds.)

Again, it's islands in the ocean vs. pools on land.

To strain the analogy, the main value of these fields exists on the beach: at the point where you need to impedance match between the Python community and another packaging community. The ideal is to be able to get a point where you can point an automated tool at a project on PyPI and say "give me that, only packaged as RPM/deb/whatever, with appropriate distro specific metadata". Such a tool is obviously going to be distro specific, since it is going to have to do some remapping based on file names to pick up existing distro packages, but it *also* needs upstream metadata. Even in a distro, a "Conflicts:" field often *does* denote runtime conflicts (e.g. over a particular network port), because, as you say, filesystem level conflicts will usually be picked up automatically. The distro philosophy is to say "OK, we simply won't let you install conflicting projects at the same time, so you won't be surprised later by a conflict that only shows up if you happen to run them both at the same time". It's designed to turn a complex, hard to debug, problem into a simple, explicit error at installation time. People build complex systems (especially web apps) based on the PyPI ecosystem, and the upstream communities *can* assist in flagging potential issues in advance. If people start putting bad metadata in their projects, then that's just a bug to be dealt with like any other. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sat, Dec 8, 2012 at 4:46 PM, PJ Eby wrote:

So if package A includes a "Conflicts: B" declaration, I recommend the following:

* An attempt to install A with B already present refuses to install A without a warning and confirmation * An attempt to install B informs the user of the conflict, and optionally offers to uninstall A

In this way, any collateral damage to B is avoided, while still making the intended "lack of support" declaration clear.

How does that sound?

No, that's not the way it works. A conflict is always symmetric, no matter who declares it. The beneficiary of these notifications is the aggregator attempting to build a systematically coherent system, rather than one with latent incompatibilities waiting to bite them at run time. It doesn't *matter* if "A conflicts with B" or "B conflicts with A", you cannot have a system with both of them installed that will be supported by the developers of both A *and* B. Now, this beneficiary *may* be the packagers for a Linux distribution, but it may also be a larger Python distribution (ActiveState, EPD, etc), a web application developer, a desktop application developer, a system integrator for a large-scale distributed system, or anyone else that combines and deploys an integrated set of packages (even those a developer installs on their personal workstation). It's up to the user to decide who they want to believe. Now, it may be that, for a given use case, the end user doesn't actually care about the potential conflict (e.g. they've done their own research and determined that the conflicting behaviour doesn't affect their system) - that's then a design decision in the installation tools as to whether or not they want to make it easy for users to override the metadata. In the Linux distro case, the installer *and* most of the metadata are largely provided by the same people, so yum/rpm/etc generally *don't* make it easy to install conflicting packages. Python installers are in a different situation though, so forced installs are likely to be an expected feature (in fact, I expect the more likely outcome given the status quo is that the default behaviour will be a warning at installation time with an option to request enforcement of "no conflicts"). Building integrated systems *is hard*. Pretending projects can't conflict just because they're both written in Python isn't sensible, and neither is it sensible to avoid warning users about the the potential for latent defects when particular packages are used in combination. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2012-12-08 20:18, PJ Eby wrote:

On Sat, Dec 8, 2012 at 5:06 AM, Nick Coghlan wrote:

...

On Sat, Dec 8, 2012 at 4:46 PM, PJ Eby wrote:

...

So if package A includes a "Conflicts: B" declaration, I recommend the following:

* An attempt to install A with B already present refuses to install A without a warning and confirmation * An attempt to install B informs the user of the conflict, and optionally offers to uninstall A

In this way, any collateral damage to B is avoided, while still making the intended "lack of support" declaration clear.

How does that sound?

No, that's not the way it works. A conflict is always symmetric, no matter who declares it.

But that *precisely contradicts* what you said in your previous email:

...

It's to allow a project to say *they don't support* installing in parallel with another package.

Just because A doesn't support being installed next to B, doesn't mean B doesn't support being installed next to A. B might work just fine with A installed, and even be explicitly supported by the author of B. Why should the author of A get to decide what happens to B? Just because I trust A about A, doesn't mean I should have to trust them about B.

[snip] If package A says that it conflicts with package B, it may or may not be symmetrical, because it's possible that package B has been updated since the author of package A discovered the conflict, so it's important that the user is told which package is complaining about the conflict, the one that is being installed or the one that is already installed. It may also be helpful if the package that includes the "Conflicts" declaration specifies which version of the other package it was last tested against in case there is a more recent version of the other package that does not cause the conflict, or, indeed, that there's a more recent version of the package that includes the "Conflicts" declaration that does not cause the conflict.

On 09/12/12 08:14, MRAB wrote:

If package A says that it conflicts with package B, it may or may not be symmetrical, because it's possible that package B has been updated since the author of package A discovered the conflict, so it's important that the user is told which package is complaining about the conflict, the one that is being installed or the one that is already installed.

I must admit than in reading this thread, I'm having a bit of trouble understanding why merely *installing* packages should lead to conflicts. Assuming that two software packages Spam and Ham install into directories Spam and Ham, how can merely having them installed side-by-side lead to a conflict? I can see how running or importing Spam and Ham together might lead to problems. And I can see that if package Spam wants to install into directory Ham, that would be bad. But who does that? Have I just demonstrated my naivety when it comes to packaging? Under what circumstances would two well-behaved packages with different names conflict? -- Steven

On 09/12/12 12:32, Chris Angelico wrote:

On Sun, Dec 9, 2012 at 12:15 PM, Steven D'Aprano wrote:

...

Assuming that two software packages Spam and Ham install into directories Spam and Ham, how can merely having them installed side-by-side lead to a conflict?

I can see how running or importing Spam and Ham together might lead to problems. And I can see that if package Spam wants to install into directory Ham, that would be bad. But who does that?

If two packages Spam and Ham both define a module Jam, then the one that gets loaded will depend on the search path. That would be one form of conflict.

import Spam.Jam import Ham.Jam What am I missing? Why would a software package called "Spam" install a top-level module called "Jam" rather than "Spam"? Isn't the whole point of Python packages to solve this namespace problem? -- Steven

On Sun, Dec 9, 2012 at 1:11 PM, Steven D'Aprano wrote:

On 09/12/12 12:32, Chris Angelico wrote:

...

On Sun, Dec 9, 2012 at 12:15 PM, Steven D'Aprano wrote:

...

Assuming that two software packages Spam and Ham install into directories Spam and Ham, how can merely having them installed side-by-side lead to a conflict?

I can see how running or importing Spam and Ham together might lead to problems. And I can see that if package Spam wants to install into directory Ham, that would be bad. But who does that?

If two packages Spam and Ham both define a module Jam, then the one that gets loaded will depend on the search path. That would be one form of conflict.

import Spam.Jam

import Ham.Jam

What am I missing?

Why would a software package called "Spam" install a top-level module called "Jam" rather than "Spam"? Isn't the whole point of Python packages to solve this namespace problem?

That would require/demand that the software package MUST define a module with its own name, and MUST NOT define any other top-level modules, and also that package names MUST be unique. (RFC 2119 keywords.) That would work, as long as those restrictions are acceptable. ChrisA

On Sun, Dec 9, 2012 at 8:41 AM, Tres Seaver wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 12/08/2012 05:06 AM, Nick Coghlan wrote:

...

Building integrated systems *is hard*. Pretending projects can't conflict just because they're both written in Python isn't sensible, and neither is it sensible to avoid warning users about the the potential for latent defects when particular packages are used in combination.

Building such systems is *too hard* to deletgate to the maintainers of every Python distribution registered on the Cheeseshop: there is too much policy involved for the ha'penn'orth of mechanism we are discussing here (decentralized inter-project metadata) to support.

Such metadata *cannot* be useful in the general sense, but only in the context of a "curated" collection of packages, where the *curator* (not the upstream package authors) makes the choices.

The authors of major projects are often in a good position to know when they conflict with other high profile projects and thus can't be used reliably in the same system. Now, *most* of the time, if there's a genuine conflict between two Python packages, it's going to be at install time - two projects attempting to install the same file obviously can't coexist on a single system (distribute and setuptools, for example, conflict at this level - they both want to own the "setuptools" and "easy_install" names). However, Python has plenty of other global state too (the codec registry, the import system, monkeypatching), and there is potential for conflict over underlying OS level resources. So let's look at the case of collections of Python packages that *are* curated. Maybe I'm a Linux distro packager, looking to automate the conversion to distro packages. Maybe I'm a toolsmith for a large corporation trying to build a curated set of packages for internal use (clearly indicating to my internal users which ones don't play nicely with each other and thus shouldn't be used together in the same project). Regardless of the reason, I'm the curator for a collection of Python packages. How shall I express the conflicts I have identified? Shall I go invent my own metadata system? Shall I be forced to choose a particular platform-specific dependency management system? How shall upstream authors communicate to *me* the conflicts that they're already aware of? Or, hey, there's this nice shiny cross-platform dependency management system *right here*. Maybe they'll be nice enough to consider handling *my* use case as well, even if it's a use case *they* don't care about. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Dec 9, 2012 at 6:18 AM, PJ Eby wrote:

On Sat, Dec 8, 2012 at 5:06 AM, Nick Coghlan wrote:

...

On Sat, Dec 8, 2012 at 4:46 PM, PJ Eby wrote:

...

So if package A includes a "Conflicts: B" declaration, I recommend the following:

* An attempt to install A with B already present refuses to install A without a warning and confirmation * An attempt to install B informs the user of the conflict, and optionally offers to uninstall A

In this way, any collateral damage to B is avoided, while still making the intended "lack of support" declaration clear.

How does that sound?

No, that's not the way it works. A conflict is always symmetric, no matter who declares it.

But that *precisely contradicts* what you said in your previous email:

...

It's to allow a project to say *they don't support* installing in parallel with another package.

Just because A doesn't support being installed next to B, doesn't mean B doesn't support being installed next to A. B might work just fine with A installed, and even be explicitly supported by the author of B. Why should the author of A get to decide what happens to B? Just because I trust A about A, doesn't mean I should have to trust them about B.

If I'm installing both A *and* B, I want to know if *either* project doesn't support that configuration. The order in which they get installed should *not* have any impact on my finding out that I am using one of my dependencies in an unsupported way that may cause me unanticipated problems further down the line. The author of A *doesn't* get to decide what happens to B, *I* do. They're merely providing a heads up that they believe there are problems when using their project in conjunction with B. My options will be: - use them both anyway (e.g. perhaps after doing some research, I may find out the conflict relates solely to a feature of B that I'm not using, so I simply update my project documentation to say "do not use feature X from project B, as it conflicts with dependency A") - choose to continue using A, find another solution for B - choose to continue using B, find another solution for A As a concrete example, there are projects out there that are known not to work with gevent's socket monkeypatching, but people don't know that until they try it and it blows up in their face. I now agree that *enforcing* a conflicts field at install time in a Python installer doesn't make any sense, since the nature of Python means it will often be easy to sidestep any such issues once you're aware of their existence (e.g. by avoiding gevent's monkeypatching features and using threads to interact with the uncooperative synchronous library, or by splitting your application into multiple processes, some using gevent and others synchronous sockets). I also believe that *any* Conflicts declaration *should* be backed up with an explicit explanation and rationale for that conflict declaration in the project documentation. Making it impossible to document runtime conflicts in metadata doesn't make those conflicts go away - it just means they will continue to be documented in an ad hoc manner on project web sites (if they get documented at all), making the job of package curation unnecessarily more difficult (since there is no standard way to document runtime conflicts). Adding a metadata field doesn't make sure such known conflicts *will* be documented, but it least makes it possible. So, I still like the idea of including a Conflicts field, but think a few points should be made clear: - the Conflicts field would be for documenting other distributions which have known issues working together in the same process and thus constitute an unsupported configuration - this field would be aimed at package *users*, rather than at installation tools (although it would still be good if they installation tools supported scanning a set of packages for known conflicts) - any use of this field should be backed up with a more detailed explanation in the project documentation Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Dec 9, 2012 at 12:54 AM, Nick Coghlan wrote:

On Sun, Dec 9, 2012 at 6:18 AM, PJ Eby wrote:

...

On Sat, Dec 8, 2012 at 5:06 AM, Nick Coghlan wrote:

...

On Sat, Dec 8, 2012 at 4:46 PM, PJ Eby wrote:

...

So if package A includes a "Conflicts: B" declaration, I recommend the following:

* An attempt to install A with B already present refuses to install A without a warning and confirmation * An attempt to install B informs the user of the conflict, and optionally offers to uninstall A

In this way, any collateral damage to B is avoided, while still making the intended "lack of support" declaration clear.

How does that sound?

No, that's not the way it works. A conflict is always symmetric, no matter who declares it.

But that *precisely contradicts* what you said in your previous email:

...

It's to allow a project to say *they don't support* installing in parallel with another package.

Just because A doesn't support being installed next to B, doesn't mean B doesn't support being installed next to A. B might work just fine with A installed, and even be explicitly supported by the author of B. Why should the author of A get to decide what happens to B? Just because I trust A about A, doesn't mean I should have to trust them about B.

If I'm installing both A *and* B, I want to know if *either* project doesn't support that configuration. The order in which they get installed should *not* have any impact on my finding out that I am using one of my dependencies in an unsupported way that may cause me unanticipated problems further down the line.

This is probably moot now, but I didn't propose that installation order matter -- in both scenarios I described, you end up with a warning and A not installed, regardless of whether A or B were installed first.

The author of A *doesn't* get to decide what happens to B, *I* do.

The reason I said, "(de facto)", is because the default behavior of whatever the next big installation tool is, would be what most users would've gotten by default.

They're merely providing a heads up that they believe there are problems when using their project in conjunction with B. My options will be: - use them both anyway (e.g. perhaps after doing some research, I may find out the conflict relates solely to a feature of B that I'm not using, so I simply update my project documentation to say "do not use feature X from project B, as it conflicts with dependency A") - choose to continue using A, find another solution for B - choose to continue using B, find another solution for A

As a concrete example, there are projects out there that are known not to work with gevent's socket monkeypatching, but people don't know that until they try it and it blows up in their face.

Here's the question, though: who's going to maintain that list? I can see gevent wanting to have a compatibility chart page in their docs, but it seems unlikely they'd want to block installation of non-gevent-compatible projects or vice versa. Similarly, I can't see why any of those other projects would want to block installation of gevent, or vice versa. That being said, I don't object to having the ability for either of them to do so: the utility of the field is *much* enhanced once its connection to installation tools is gone, since a wider variety of issues can be described without inconveniencing users.

I now agree that *enforcing* a conflicts field at install time in a Python installer doesn't make any sense, since the nature of Python means it will often be easy to sidestep any such issues once you're aware of their existence (e.g. by avoiding gevent's monkeypatching features and using threads to interact with the uncooperative synchronous library, or by splitting your application into multiple processes, some using gevent and others synchronous sockets). I also believe that *any* Conflicts declaration *should* be backed up with an explicit explanation and rationale for that conflict declaration in the project documentation.

Beyond that, I think a reference URL should be included *in the field itself*, e.g. to a bug report, support ticket, or other page that documents the incompatibility and will be updated as the situation changes. The actual usefulness of the field to anyone "downstream" seems greatly reduced if they have to go hunting for the information explaining the compatibility issue(s). This is a good example of what I meant about clear thinking on concrete use cases, vs. simply copying fields from distro tools. In the distro world, these kinds of fields reflect the *results* of research and decision-making about compatibility. Whereas, in our "upstream" world, the purpose of the fields is to provide downstream repackagers and integrators with the source materials for such research.

So, I still like the idea of including a Conflicts field, but think a few points should be made clear: - the Conflicts field would be for documenting other distributions which have known issues working together in the same process and thus constitute an unsupported configuration - this field would be aimed at package *users*, rather than at installation tools (although it would still be good if they installation tools supported scanning a set of packages for known conflicts) - any use of this field should be backed up with a more detailed explanation in the project documentation

My concrete recommendation based on your comments, then, is: * The field should be called Known-Incompatibilities (to better clarify its purpose and avoid confusion with similarly-named installation-oriented metadata in other tools) * The field should be of the form (though not necessarily syntax): ProjectName==incompatible_version; info=url That is, each entry lists a project name and a specific version that is known to be incompatible, along with a (required) information URL. The URL should be for: * a page that is updated with any change in the situation * that will remain available indefinitely, and * describes the specific reason that particular project is considered incompatible, along with any available workarounds For minor issues, a bug report or support ticket is acceptable; otherwise, a long-lived documentation link should be used. In-page anchor links are acceptable. A simple link to either project's home page or main documentation page is *not* acceptable: the link must to be a part of the documentation that directly addresses the nature of the incompatibility. I'm not too picky about the version specification approach, though; the simplest thing is to only allow a single version to be named, but it also seems it could be reasonable to list one or more version ranges that appy, as long as they are not open-ended going forward. That is, saying versions 1.1-2.3 are incompatible is ok, but not "1.1 on". (Because the author of A is not in a position to declare on B's behalf that the incompatibility will *never* be fixable.) (I might be overthinking the versions, bit, though, since this is really just about warnings.) I would recommend that tools automatically provide the warning in cases where a project C depends on versions of A and B that are declared incompatible. In this case, while one cannot *prove* the incompatibility to be an issue, it is still a potential issue. (This is more of a package build-time issue, though, as with Replaced-By.) Speaking of Replaced-By, it probably makes sense to require a URL in the field there as well, but that URL can be an unchanging page such as an archived post to a mailing list or blog, announcing the project's renaming or obsolescence, and providing migration help or links thereto. I think it also should be a multi-valued field, just like Known-Incompatibilities. Recently, I came across a Python project "lepl" (a parser combinator library) that just declared its end-of-life, and actually recommended multiple alternatives, each of which would be more appropriate for some uses of a parsing library. (That is, there was no single "does everything" replacement for lepl's full feature set.) Finally, the PEP should document that the audience for both the Replaced-By and Known-Incompatibilities fields is developers and system integrators (such as distro teams). So they are designed to be processed by tools that *build* packages, rather than tools that *install* them. So, if you build a project that depends on something that's replaced, or a pair of things known to be incompatible, that's when you get warnings and such. Tools to check such things on installed projects are also ok, though to avoid unnecessary warnings, it's probably best to only list incompatibilities for co-dependents (and orphaned replaced projects) by default. That is, a checker should probably ignore replacements when there's an installed project depending on the replaced version, and ignore incompatibilities that aren't part of the same requirements subtree (and thus unlikely to be used together). Of course, having options to be more verbose is not an issue, and this isn't really something to legislate anyway -- it's just that listing *every* replaced project or potentially-incompatible pairing in even a moderately-sized installation is likely to be far more noise than signal.

On Sun, Dec 9, 2012 at 8:48 PM, Stephen J. Turnbull wrote:

PJ Eby writes:

...

This is a good example of what I meant about clear thinking on concrete use cases, vs. simply copying fields from distro tools. In the distro world, these kinds of fields reflect the *results* of research and decision-making about compatibility. Whereas, in our "upstream" world, the purpose of the fields is to provide downstream repackagers and integrators with the source materials for such research.

I agree with the meaning of the above paragraph, but would like to dissociate myself from the comparison implied by the expression "clear thinking".

What comparison is that? By "clear", I mean "free of prior assumptions". The assumptions that made the discussion difficult weren't just about the use cases themselves, but about the environments, tools, organizations, concepts, etc. surrounding those use cases. Indeed, even the assumption of what should *qualify* as a "use case" was a stumbling block on occasion. ;-) And by "thinking", I mean, "considering alternatives and consequences", as distinct from debating the merits of a specific position. Put together, the phrase "clear thinking on concrete use cases" means (at least to me), "dropping all preconceptions of the existing design and starting over from square one, to ask how best the problem may be solved, using specific examples as a guide rather than using generalities." Generalities not rooted in concrete examples have a way of leading to non-terminating discussions. ;-) Starting over a discussion in this fashion isn't easy, but the results are usually worth it. I appreciate Nick and Daniel's patience in particular.

PJ Eby writes:

By "clear", I mean "free of prior assumptions".

Ah, well, I guess I've just run into a personal limitation. I can't imagine thinking that is "free of prior assumptions". Not my own<wink/>, and not by others, either. So, unfortunately, I was left with the conventional opposition in thinking: "clear" vs. "muddy". That impression was only strengthened by the phrase "vs. simply copying fields from distro tools."

Put together, the phrase "clear thinking on concrete use cases" means (at least to me), "dropping all preconceptions of the existing design and starting over from square one, to ask how best the problem may be solved, using specific examples as a guide rather than using generalities."

Sure, but ISTM that's the opposite of what you've actually been doing, at least in terms of contributing to my understanding. One obstacle to discussion you have contributed to overcoming in my thinking is the big generality that the packager (ie, the person writing the metadata) is in a position to recommend "good behavior" to the installation tool, vs. being in a position to point out "relevant considerations" for users and tools installing the packager's product. Until that generality is formulated and expressed, it's very difficult to see why the examples and particular solutions to use cases that various proponents have described fail to address some real problems. It was difficult for me to see, at first, what distinction was actually being made. Specifically, I thought that the question about "Obsoletes" vs. "Obsoleted-By" was about which package should be considered authoritative about obsolescence. That is a reasonable distinction for that particular discussion, but there is a deeper, and general, principle behind that. Namely, "metadata is descriptive, not prescriptive." Of course once one understands that principle, the names of the fields don't matter so much, but it is helpful for "naive" users of the metadata if the field names strongly connote description of the package rather than behavior of the tool.

On Sun, Dec 09, 2012 at 12:48:45AM -0500, PJ Eby wrote:

Do any of the distro folks know of a Python project tagged as conflicting with another for their distro, where the conflict does *not* involve any files in conflict?

In Fedora we do work to avoid most types of Conflicts (backporting fixes, etc) but I can give some examples of where Conflivts could have been used in the past: In docutils prior to the latest release, certain portions of docutils was broken if pyxml was installed (since pyxml replaces certain stdlib xml.* functionaltiy). So older docutils versions could have had a Conflicts: PyXML. Nick has since provided a technique for docutils to use that loads from the stdlib first and only goes to PyXML if the functionality is not available there. Various libraries in web stacks have had bugs that prevent the propser functioning of the web framework at the top level. In case of major issues (security, unable to startup), these top level frameworks could use versioned Conflicts to prevent installation. For instance: TurboGears might have a Conflicts: CherryPy < 2.3.1 Note, though, that if parallel installable versions and selection of the proper versions from that work, then this type of Conflict wouldn't be necessary. Instead you'd have versioned Requires: instead. -Toshio

On Mon, Dec 10, 2012 at 3:27 AM, Stephen J. Turnbull wrote:

PJ Eby writes:

...

By "clear", I mean "free of prior assumptions".

Ah, well, I guess I've just run into a personal limitation. I can't imagine thinking that is "free of prior assumptions". Not my own<wink/>, and not by others, either.

I suppose I should have said, "free of *known* prior assumptions", since the trick to suspending assumptions is to find the ones you *have*. The deeper assumptions, alas, can usually only be found by clashing opinions with others... then stepping back and going, "wait... what does he/she believe that's *different* from what I believe, that allows them to have that differing opinion?" And then that's how you find out what it is that *you're* assuming, that you didn't know you were assuming. ;-) (Not to mention what the other person is.)

...

Put together, the phrase "clear thinking on concrete use cases" means (at least to me), "dropping all preconceptions of the existing design and starting over from square one, to ask how best the problem may be solved, using specific examples as a guide rather than using generalities."

Sure, but ISTM that's the opposite of what you've actually been doing, at least in terms of contributing to my understanding. One obstacle to discussion you have contributed to overcoming in my thinking is the big generality that the packager (ie, the person writing the metadata) is in a position to recommend "good behavior" to the installation tool, vs. being in a position to point out "relevant considerations" for users and tools installing the packager's product.

Right, but I started from a concrete scenario I wanted to avoid, which led me to question the assumption that those fields were actually useful. As soon as I began questioning *that* assumption and asking for use cases (2 years ago, in the last PEP 345 revision discussion), it became apparent to me that there was something seriously wrong with the conflicts and obsoletes fields, as they had almost no real utility as they were defined and understood at that point.

Until that generality is formulated and expressed, it's very difficult to see why the examples and particular solutions to use cases that various proponents have described fail to address some real problems.

Unfortunately, it's a chicken-and-egg problem: until you know what assumptions are being made, you can't formulate them. It's an iterative process of exposing assumptions, until you succeed in actually communicating. ;-) Heck, even something as simple as my assumptions about what "clear thinking" meant and what I was trying to say has taken some back and forth to clarify. ;-)

It was difficult for me to see, at first, what distinction was actually being made.

Specifically, I thought that the question about "Obsoletes" vs. "Obsoleted-By" was about which package should be considered authoritative about obsolescence. That is a reasonable distinction for that particular discussion, but there is a deeper, and general, principle behind that. Namely, "metadata is descriptive, not prescriptive."

Actually, the principle I was clinging to for *both* fields was not giving project authors authority over other people's projects. It's fine for metadata to be prescriptive (e.g. requirements), it's just that it should be prescriptive *only* for that project in isolation. (In the broader sense, it also applies to the distro situation: the project author doesn't really have authority over the distro, either, so it can only be a suggestion there, as well.)

On 10 December 2012 16:35, Toshio Kuratomi wrote:

I have no problems with Obsoletes, Conflicts, Requires, and Provides types of fields are marked informational. In fact, there are many cases where packages are overzealous in their use of Requires right now that cause distributions to patch the dependency information in the package metadata.

Given the endless debate on these fields, and the fact that it pretty much all seems to be about what happens when tools enforce them, I'm +1 on this. Particularly as these fields were not the focus of this change to the spec in any case. Paul.