(How) do setuptools/distribute handle circular dependencies?
Working on distlib, I noticed that PyPI contains projects with circular dependencies. Here's an extract from setup.py in the latest Zope2 (2.13.19): install_requires=[ ... 'Products.OFSP >= 2.13.2', ... ] + additional_install_requires, whereas the the setup.py in the latest Products.OFSP (2.13.2) has: install_requires=[ ... 'Zope2 >= 2.13.0a1', ], So, according to the declarations, each package depends on the other. Can setuptools / distribute deal with this sort of situation? If so, how does that work? Regards, Vinay Sajip
Vinay Sajip
So, according to the declarations, each package depends on the other. Can setuptools / distribute deal with this sort of situation? If so, how does that work?
My question also applies to pip, of course. I know it uses setuptools / distribute under the covers, but I'm not sure if it has additional / different dependency resolution logic. Regards, Vinay Sajip
On 2012-11-09 22:56:46 +0000, Vinay Sajip said:
Vinay Sajip
writes: So, according to the declarations, each package depends on the other. Can setuptools / distribute deal with this sort of situation? If so, how does that work?
My question also applies to pip, of course. I know it uses setuptools / distribute under the covers, but I'm not sure if it has additional / different dependency resolution logic.
Just a guess, but I think it works "as expected". If you e.g. pip install Zope2 you get the install_requires Products.OFSP. In order for OFSP installaion to succeed, it's install_requires must be satisfied. It doesn't really matter that the deps are circular, as long as they are sane. If a dep is not sane (e.g. foo 1.0 requires bar which requires foo < 1.0) then installation will fail, regardless of its "circularity" or "non-circularity". Also, I don't think there is any "circular detection" built in or at least if there is, I've never seen any evidence of it in practice.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Alex Clark · https://www.gittip.com/aclark4life/
Hi Vinay, On 11/09/2012 03:56 PM, Vinay Sajip wrote:
Vinay Sajip
writes: So, according to the declarations, each package depends on the other. Can setuptools / distribute deal with this sort of situation? If so, how does that work?
My question also applies to pip, of course. I know it uses setuptools / distribute under the covers, but I'm not sure if it has additional / different dependency resolution logic.
Pip has its own dependency resolution logic, it does not rely on setuptools for that. As Alex says, the "circular" case you mention is not hard; all it requires is that you know what you're already installing and check new requirements you come across against that list to see if they are already satisfied. In pip this happens here: https://github.com/pypa/pip/blob/develop/pip/req.py#L1091 More generally, I wouldn't really recommend pip's dependency resolution logic as a model for new Python code in this area. There are some not uncommon cases that it handles poorly; see https://github.com/pypa/pip/issues/174 and http://bugs.python.org/issue8927. (To be fair to pip, these cases aren't trivial when you have to download and unpack an sdist before you can find out anything about its dependencies, but I'd hope that with the new metadata PEPs Python packaging code could get a bit smarter in this area.) Carl
Carl Meyer
already satisfied. In pip this happens here: https://github.com/pypa/pip/blob/develop/pip/req.py#L1091
More generally, I wouldn't really recommend pip's dependency resolution logic as a model for new Python code in this area. There are some not uncommon cases that it handles poorly; see https://github.com/pypa/pip/issues/174 and http://bugs.python.org/issue8927. (To be fair to pip, these cases aren't trivial when you have to download and unpack an sdist before you can
Thanks for the pointers.
find out anything about its dependencies, but I'd hope that with the new metadata PEPs Python packaging code could get a bit smarter in this area.)
AFAICT, the proposed metadata PEP changes don't offer the same requirement granularity as setuptools / distribute (for example, 'Requires-Dist' as against 'install_requires', 'setup_requires', 'test_requires'). Anyway, I'll take a look at the issue you mentioned and see how the dependency code in distlib stacks up. Currently, it keeps the requirements distinct for 'install', 'setup' and 'test'. The distinctions seem reasonable in theory, though I'm not sure how useful they are in practice. In the case I was quoting, the circular dependency wasn't being treated as any kind of conflict - I just came across cycles when testing topological sorting of dependency graphs, and was curious about them. Regards, Vinay Sajip
The new metadata keeps setup-requires-dist separate, and test and doc
dependencies are well-known extra names. The granularity is preserved.
On Nov 10, 2012 6:52 AM, "Vinay Sajip"
Carl Meyer
writes: already satisfied. In pip this happens here: https://github.com/pypa/pip/blob/develop/pip/req.py#L1091
More generally, I wouldn't really recommend pip's dependency resolution logic as a model for new Python code in this area. There are some not uncommon cases that it handles poorly; see https://github.com/pypa/pip/issues/174 and http://bugs.python.org/issue8927. (To be fair to pip, these cases aren't trivial when you have to download and unpack an sdist before you can
Thanks for the pointers.
find out anything about its dependencies, but I'd hope that with the new metadata PEPs Python packaging code could get a bit smarter in this area.)
AFAICT, the proposed metadata PEP changes don't offer the same requirement granularity as setuptools / distribute (for example, 'Requires-Dist' as against 'install_requires', 'setup_requires', 'test_requires').
Anyway, I'll take a look at the issue you mentioned and see how the dependency code in distlib stacks up. Currently, it keeps the requirements distinct for 'install', 'setup' and 'test'. The distinctions seem reasonable in theory, though I'm not sure how useful they are in practice.
In the case I was quoting, the circular dependency wasn't being treated as any kind of conflict - I just came across cycles when testing topological sorting of dependency graphs, and was curious about them.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
To echo what others have said, pip has no problem with circularity itself,
that I know of. The #174 issue mentioned is the limitation of pip's "first
found, wins" logic for handling duplicate requirements (circular or not).
This logic is intentional for the case when a user is explicitly declaring
requirement versions (in req files or in the cli). Those explicit
declarations are "first found" and win. The user is allowed to be in
charge of the environment. But for everything else, not explicitly
defined, that logic is a shortcoming. I think this could be fixed now in
pip (in our current packaging environment), but not trivial.
On Fri, Nov 9, 2012 at 2:56 PM, Vinay Sajip
Vinay Sajip
writes: So, according to the declarations, each package depends on the other. Can setuptools / distribute deal with this sort of situation? If so, how does that work?
My question also applies to pip, of course. I know it uses setuptools / distribute under the covers, but I'm not sure if it has additional / different dependency resolution logic.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Sat, Nov 10, 2012 at 6:51 AM, Vinay Sajip
Carl Meyer
writes: already satisfied. In pip this happens here: https://github.com/pypa/pip/blob/develop/pip/req.py#L1091
More generally, I wouldn't really recommend pip's dependency resolution logic as a model for new Python code in this area. There are some not uncommon cases that it handles poorly; see https://github.com/pypa/pip/issues/174 and http://bugs.python.org/issue8927. (To be fair to pip, these cases aren't trivial when you have to download and unpack an sdist before you can
Thanks for the pointers.
find out anything about its dependencies, but I'd hope that with the new metadata PEPs Python packaging code could get a bit smarter in this area.)
AFAICT, the proposed metadata PEP changes don't offer the same requirement granularity as setuptools / distribute (for example, 'Requires-Dist' as against 'install_requires', 'setup_requires', 'test_requires').
Anyway, I'll take a look at the issue you mentioned and see how the dependency code in distlib stacks up. Currently, it keeps the requirements distinct for 'install', 'setup' and 'test'. The distinctions seem reasonable in theory, though I'm not sure how useful they are in practice.
Test dependencies allow you to depend on a test framework (e.g. nose) without requiring it to be installed at runtime. Setup dependencies let you depend on tools like Pyrex or Cython in order to compile a binary package, without requiring them to be available at runtime.
In the case I was quoting, the circular dependency wasn't being treated as any kind of conflict - I just came across cycles when testing topological sorting of dependency graphs, and was curious about them.
Pkg_resources uses a "greedy" dependency resolver, so it ignores circularity as a side-effect. That is, as soon as the first requirement for a project is found, a suitable version is grabbed and added to the tentative new path. Thus, if another reference to the same project occurs elsewhere, the project has already been added to the path, so the dependency is already satisfied and does not recurse. Btw, there are other tricky corner cases for dependency resolution; with distribute and sufficiently-old version of setuptools, circular setup-time dependencies can lead to recursion, unless the circular package is available in binary form. This was fixed a few years ago in setuptools by ensuring that nested source build invocations saved and restored pkg_resources state, or something like that, but I don't think distribute ever integrated those changes. In any case, that's more of a practical issue for build tools than a dependency resolver issue per se. Btw, I did consider using a backtracking dependency resolver in pkg_resources -- i.e. one that could handle the sort of conflicts linked above -- but decided against it on the basis that 1. Backtracking resolution could be exponential time in worst-case scenarios, and 2. Dependency information wasn't available from PyPI without actually downloading and building things anyway, so there were limits to how much the backtracking would help.
On Mon, Nov 12, 2012 at 3:39 PM, PJ Eby
On Sat, Nov 10, 2012 at 6:51 AM, Vinay Sajip
wrote: Carl Meyer
writes: already satisfied. In pip this happens here: https://github.com/pypa/pip/blob/develop/pip/req.py#L1091
More generally, I wouldn't really recommend pip's dependency resolution logic as a model for new Python code in this area. There are some not uncommon cases that it handles poorly; see https://github.com/pypa/pip/issues/174 and http://bugs.python.org/issue8927. (To be fair to pip, these cases aren't trivial when you have to download and unpack an sdist before you can
Thanks for the pointers.
find out anything about its dependencies, but I'd hope that with the new metadata PEPs Python packaging code could get a bit smarter in this area.)
AFAICT, the proposed metadata PEP changes don't offer the same requirement granularity as setuptools / distribute (for example, 'Requires-Dist' as against 'install_requires', 'setup_requires', 'test_requires').
Anyway, I'll take a look at the issue you mentioned and see how the dependency code in distlib stacks up. Currently, it keeps the requirements distinct for 'install', 'setup' and 'test'. The distinctions seem reasonable in theory, though I'm not sure how useful they are in practice.
Test dependencies allow you to depend on a test framework (e.g. nose) without requiring it to be installed at runtime. Setup dependencies let you depend on tools like Pyrex or Cython in order to compile a binary package, without requiring them to be available at runtime.
I'm not sure this is well understood, let's make sure everyone gets this. With setup/test requirements as implemented in setuptools nose, Pyrex, Cython, whatever can be on PYTHONPATH during the install but are not available when the install is done. You get to use them without cluttering up your normal runtime environment. This is implemented by installing nose in a uniquely named subdirectory of the sdist that is being installed and adding that directory to PYTHONPATH/sys.path during the install / test command run. I would like to see distutils (1, 2, any number) banished from the stdlib to become a common setup-requires requirement.
PJ Eby
Test dependencies allow you to depend on a test framework (e.g. nose) without requiring it to be installed at runtime. Setup dependencies let you depend on tools like Pyrex or Cython in order to compile a binary package, without requiring them to be available at runtime.
Oh, I understand why the distinctions exist, and they seem reasonable to me. Sorry I wasn't clear - I meant that I wasn't sure how widely these were used. I made the comment because I thought that the distinction wasn't preserved in the new metadata format, but I must have misread - Daniel commented that the distinction *is* preserved.
Pkg_resources uses a "greedy" dependency resolver, so it ignores circularity as a side-effect. That is, as soon as the first requirement for [explanation snipped]
Thanks for the explanation. It seems to me that the new metadata formats make dependency resolution more difficult because they allow for e.g. 'Provides-Dist' as a multi-value field. Perhaps I'm misunderstanding, but this, it seems to me, opens the door for a project named A on PyPI to provide e.g. "A (1.0)", "B (1.5)" and "C (2.0)", and likewise, projects B and C on PyPI could provide slightly different versions of "A", "B" and "C". You can soon get a rat's nest of dependencies in the resolver - and if you get something like the case that Carl linked to, where some element of backtracking might be in order, it doesn't seem computationally straightforward to resolve dependencies, perhaps even with a SAT solver in the mix. Is this a case of practicality losing out to purity? Assuming it's easy to pull any version from an index, I can't see a compelling case for any distribution archive for A to ever provide anything other than e.g. "A (x.y)". Can someone point me to the real need for multi-valued "Provides" fields? Or have I completely misunderstood this aspect of the metadata? Regards, Vinay Sajip
The commonest example is that distribute provides setuptools.
Daniel Holth
On Nov 12, 2012, at 6:10 PM, Vinay Sajip
PJ Eby
writes: Test dependencies allow you to depend on a test framework (e.g. nose) without requiring it to be installed at runtime. Setup dependencies let you depend on tools like Pyrex or Cython in order to compile a binary package, without requiring them to be available at runtime.
Oh, I understand why the distinctions exist, and they seem reasonable to me. Sorry I wasn't clear - I meant that I wasn't sure how widely these were used. I made the comment because I thought that the distinction wasn't preserved in the new metadata format, but I must have misread - Daniel commented that the distinction *is* preserved.
Pkg_resources uses a "greedy" dependency resolver, so it ignores circularity as a side-effect. That is, as soon as the first requirement for [explanation snipped]
Thanks for the explanation. It seems to me that the new metadata formats make dependency resolution more difficult because they allow for e.g. 'Provides-Dist' as a multi-value field. Perhaps I'm misunderstanding, but this, it seems to me, opens the door for a project named A on PyPI to provide e.g. "A (1.0)", "B (1.5)" and "C (2.0)", and likewise, projects B and C on PyPI could provide slightly different versions of "A", "B" and "C". You can soon get a rat's nest of dependencies in the resolver - and if you get something like the case that Carl linked to, where some element of backtracking might be in order, it doesn't seem computationally straightforward to resolve dependencies, perhaps even with a SAT solver in the mix. Is this a case of practicality losing out to purity? Assuming it's easy to pull any version from an index, I can't see a compelling case for any distribution archive for A to ever provide anything other than e.g. "A (x.y)". Can someone point me to the real need for multi-valued "Provides" fields? Or have I completely misunderstood this aspect of the metadata?
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Monday, November 12, 2012 at 6:10 PM, Vinay Sajip wrote:
PJ Eby
http://telecommunity.com)> writes: Thanks for the explanation. It seems to me that the new metadata formats make dependency resolution more difficult because they allow for e.g. 'Provides-Dist' as a multi-value field. Perhaps I'm misunderstanding, but this, it seems to me, opens the door for a project named A on PyPI to provide e.g. "A (1.0)", "B (1.5)" and "C (2.0)", and likewise, projects B and C on PyPI could provide slightly different versions of "A", "B" and "C". You can soon get a rat's nest of dependencies in the resolver - and if you get something like the case that Carl linked to, where some element of backtracking might be in order, it doesn't seem computationally straightforward to resolve dependencies, perhaps even with a SAT solver in the mix. Is this a case of practicality losing out to purity? Assuming it's easy to pull any version from an index, I can't see a compelling case for any distribution archive for A to ever provide anything other than e.g. "A (x.y)". Can someone point me to the real need for multi-valued "Provides" fields? Or have I completely misunderstood this aspect of the metadata?
I think Provides is a misfeature as well. Even RPM, debs, etc which do have a provides feature do not use it as it is used here. They allow you to use it for a virtual package (e.g. email) which any number of packages could provide but it's not there to allow one package to masquerade as another.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org (mailto:Distutils-SIG@python.org) http://mail.python.org/mailman/listinfo/distutils-sig
On Nov 12, 2012, at 6:53 PM, Donald Stufft
PJ Eby
writes: Thanks for the explanation. It seems to me that the new metadata formats make dependency resolution more difficult because they allow for e.g. 'Provides-Dist' as a multi-value field. Perhaps I'm misunderstanding, but this, it seems to me, opens the door for a project named A on PyPI to provide e.g. "A (1.0)", "B (1.5)" and "C (2.0)", and likewise, projects B and C on PyPI could provide slightly different versions of "A", "B" and "C". You can soon get a rat's nest of dependencies in the resolver - and if you get something like the case that Carl linked to, where some element of backtracking might be in order, it doesn't seem computationally straightforward to resolve dependencies, perhaps even with a SAT solver in the mix. Is this a case of practicality losing out to purity? Assuming it's easy to pull any version from an index, I can't see a compelling case for any distribution archive for A to ever provide anything other than e.g. "A (x.y)". Can someone point me to the real need for multi-valued "Provides" fields? Or have I completely misunderstood this aspect of the metadata? I think Provides is a misfeature as well. Even RPM, debs, etc which do have a
On Monday, November 12, 2012 at 6:10 PM, Vinay Sajip wrote: provides feature do not use it as it is used here. They allow you to use it for a virtual package (e.g. email) which any number of packages could provide but it's not there to allow one package to masquerade as another.
I think it's necessary, but you would never hunt for "providing" packages as part of dependency resolution. You would have to install those manually first.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Daniel Holth
The commonest example is that distribute provides setuptools.
I would regard that as a special case - I'm not thinking about forks. Can you point to more substantive cases? Regards, Vinay Sajip
Daniel Holth
I think it's necessary, but you would never hunt for "providing" packages as part of dependency resolution. You would have to install those manually first.
What makes you say that? I agree that, given a requirement "A", it doesn't make sense to search the entire index metadata to see what provides "A". But as for installing anything first - my distro package manager doesn't insist on downloading and installing anything first; it tells me what will need to be downloaded, installed and upgraded and lets me decide whether to go ahead. Why should it be different for Python packages with interdependencies? You haven't answered the question of *why* multi-valued Provides fields are needed as a matter of practicality, nor how one would tackle the practical issues of dependency/conflict resolution I mentioned if multi-valued Provides fields are used completely freely. Regards, Vinay Sajip
On Mon, Nov 12, 2012 at 8:11 PM, Vinay Sajip
Daniel Holth
writes: I think it's necessary, but you would never hunt for "providing"
packages as
part of dependency resolution. You would have to install those manually first.
What makes you say that? I agree that, given a requirement "A", it doesn't make sense to search the entire index metadata to see what provides "A". But as for installing anything first - my distro package manager doesn't insist on downloading and installing anything first; it tells me what will need to be downloaded, installed and upgraded and lets me decide whether to go ahead. Why should it be different for Python packages with interdependencies?
You haven't answered the question of *why* multi-valued Provides fields are needed as a matter of practicality, nor how one would tackle the practical issues of dependency/conflict resolution I mentioned if multi-valued Provides fields are used completely freely.
David, did you mention a paper about advanced dependency resolution algorithms? I don't know how dependency resolution should work. I only claim that the very popular distribute needs to provide setuptools to work; right now it does that with a hack by including a setuptools .egg-info directory. I don't expect provides-dist to be a very widely used feature at all. As for Provides-Dist you should just index that field locally and the remote package index should let you search by provides instead of by the package name (in that index the package name is one of the provides values). You are searching the entire metadata, it's just already indexed so it's efficient. Yum/apt cache all the available package metadata locally too. In Python/pypi, which is mostly libraries and not applications like in Debian, a fork would be the normal use case for provides-dist. If the plugin systems were more widely used then you might have more non-fork provides-dist lines, for example if trac required at least one revision control backend.
Daniel Holth
David, did you mention a paper about advanced dependency resolution algorithms?I don't know how dependency resolution should work. I only claim that the very popular distribute needs to provide setuptools to work; right now it does that with a hack by including a setuptools .egg-info directory.
Do you mean pip? distribute is an incarnation of setuptools, after all.
I don't expect provides-dist to be a very widely used feature at all.As for Provides-Dist you should just index that field locally and the remote package index should let you search by provides instead of by the package name (in that index the package name is one of the provides values). You are searching the entire metadata, it's just already indexed so it's efficient.
So if several different distributions on PyPI say that they "provide" "Foo (1.0)", which one are you supposed to pick? The question of efficiency isn't the main concern here - it's "what do you do with the (no doubt efficiently-returned) results?"
In Python/pypi, which is mostly libraries and not applications like in Debian, a fork would be the normal use case for provides-dist. If the plugin systems were more widely used then you might have more non-fork provides-dist lines, for example if trac required at least one revision control backend.
Linux distros, Debian included, ship lots of libraries too. I get most of the commonly-used libraries (like setuptools, PIL) through the distro package manager. Forks don't need a *multi-valued* Provides field. And ISTM in the comment about Trac, you are referring to what pip calls bundles - which, IIUC, is a deprecated feature of pip. With a good repository infrastructure and packaging tools, I'm not sure why bundles would be needed. Often, packages which bundle other packages don't advertise those other packages as being provided (and rightly so, in my view). For example, Django includes simplejson, six etc. but that doesn't need to be exposed at the PyPI level. It can be considered as implementation detail. The bundled packages could even diverge from their non-bundled counterparts to better serve the needs of the "main" package. Regards, Vinay Sajip
On 2012-11-13 00:57:19 +0000, Vinay Sajip said:
Daniel Holth
writes: The commonest example is that distribute provides setuptools.
I would regard that as a special case - I'm not thinking about forks. Can you point to more substantive cases?
Just jumping in here, but I don't think that the distribute distribution providing the setuptools module is indicative of a fork. It's an example of a distribution that does not contain a module name you'd expect it to, based on the distribution name. PIL is another example: from PIL import Image. And in the case of PIL at least, the fork distribution name does not change anything: pip install Pillow; from PIL import Image.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Alex Clark · https://www.gittip.com/aclark4life/
Alex Clark
Just jumping in here, but I don't think that the distribute distribution providing the setuptools module is indicative of a fork. It's an example of a distribution that does not contain a module name you'd expect it to, based on the distribution name. PIL is another example: from PIL import Image. And in the case of PIL at least, the fork distribution name does not change anything: pip install Pillow; from PIL import Image.
Sure, but my main point was about having "Provides" be a multi-valued field, which opens the door to a lot of complexity for perhaps little or no benefit. I don't have any problem with the use of "Provides" as a canonical name for a distribution. Even before Pillow came along, there used to be confusion, IIRC, between the names "Imaging" and "PIL". Regards, Vinay Sajip
On Tue, Nov 13, 2012 at 4:21 AM, Vinay Sajip
Daniel Holth
writes: David, did you mention a paper about advanced dependency resolution algorithms?I don't know how dependency resolution should work. I only claim that the very popular distribute needs to provide setuptools to work; right now it does that with a hack by including a setuptools .egg-info directory.
Do you mean pip? distribute is an incarnation of setuptools, after all.
I meant distribute. If distribute did not "Provides-Dist" setuptools (although it does it in its own way, because no one has actually implemented Provides-Dist), then all the packages that depend on setuptools would try to install setuptools in addition to distribute.
I don't expect provides-dist to be a very widely used feature at all.As for Provides-Dist you should just index that field locally and the remote package index should let you search by provides instead of by the package name (in that index the package name is one of the provides values). You are searching the entire metadata, it's just already indexed so it's efficient.
So if several different distributions on PyPI say that they "provide" "Foo (1.0)", which one are you supposed to pick? The question of efficiency isn't the main concern here - it's "what do you do with the (no doubt efficiently-returned) results?"
You would ask the user or prefer the package that is actually named Foo by default.
In Python/pypi, which is mostly libraries and not applications like in Debian, a fork would be the normal use case for provides-dist. If the plugin systems were more widely used then you might have more non-fork provides-dist lines, for example if trac required at least one revision control backend.
Linux distros, Debian included, ship lots of libraries too. I get most of the commonly-used libraries (like setuptools, PIL) through the distro package manager.
Forks don't need a *multi-valued* Provides field. And ISTM in the comment about Trac, you are referring to what pip calls bundles - which, IIUC, is a deprecated feature of pip. With a good repository infrastructure and packaging tools, I'm not sure why bundles would be needed.
Often, packages which bundle other packages don't advertise those other packages as being provided (and rightly so, in my view). For example, Django includes simplejson, six etc. but that doesn't need to be exposed at the PyPI level. It can be considered as implementation detail. The bundled packages could even diverge from their non-bundled counterparts to better serve the needs of the "main" package.
That's usually called vendorizing. The vendorized packages would have a different name "from django.utils import six". The package index and installer don't need to know. With trac, a Python app with a good plugin system, I meant it might need at least one package that provides a virtual package "trac-revision-control-backend". Of course trac doesn't actually do this, it supports SVN out of the box and supports other RCS with plugins. Pillow is a great example. It might provide both PIL and Imaging and of course it provides its own name Pillow. If you allow Provides-Dist then the package already has two names. Once you have 2 all the complexity is already there, so you might as well allow n based on the 0, 1 or infinity rule. http://www.catb.org/jargon/html/Z/Zero-One-Infinity-Rule.html a cavallo says: Maybe a sat solver is what you're looking for... libzypp (suse) does implement that for this purpose. A python modules is under the rox project.
Daniel Holth
You would ask the user or prefer the package that is actually named Foo by default.
Asking the user could get old fast for the user, depending on the scenario. If you're going to prefer the package named Foo, there's no point in searching the index to see who provides Foo - just go straight for Foo. This is the sensible thing to do, of course - but it makes Provides-Dist indexing a bit pointless.
Pillow is a great example.
Presumably Pillow manages now without the benefit of Provides-Dist, since no one has actually implemented Provides-Dist yet.
Maybe a sat solver is what you're looking for...
Perhaps, but I'm not sure any existing projects are ready to be used, whether for technical or licensing reasons (e.g. rox is GPL). It seems a bit of a rabbit-hole at this point. Regards, Vinay Sajip
On 2012-11-13 12:16:43 +0000, Vinay Sajip said:
Alex Clark
writes: Just jumping in here, but I don't think that the distribute distribution providing the setuptools module is indicative of a fork. It's an example of a distribution that does not contain a module name you'd expect it to, based on the distribution name. PIL is another example: from PIL import Image. And in the case of PIL at least, the fork distribution name does not change anything: pip install Pillow; from PIL import Image.
Sure, but my main point was about having "Provides" be a multi-valued field, which opens the door to a lot of complexity for perhaps little or no benefit. I don't have any problem with the use of "Provides" as a canonical name for a distribution. Even before Pillow came along, there used to be confusion, IIRC, between the names "Imaging" and "PIL".
Ah, in that case I think that makes me +1 for a mult-valued Provides (whatever that is, is this a PEP discussion?) because there are times when you want to provide multiple packages in a single distribution. E.g. packages=[ 'foo', 'bar', ], package_dir={ 'foo': 'foo', 'bar': 'bar', }, where the distribution looks like this: foo-bar/foo/foo/__init__.py foo-bar/bar/bar/__init__.py And this reminds me of a question I had that I'll now ask in a separate thread :-)
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Alex Clark · https://www.gittip.com/aclark4life/
On Tue, Nov 13, 2012 at 2:16 PM, Alex Clark
On 2012-11-13 12:16:43 +0000, Vinay Sajip said:
Alex Clark
writes: Just jumping in here, but I don't think that the distribute distribution providing the setuptools module is indicative of a fork. It's an example of a distribution that does not contain a module name you'd expect it to, based on the distribution name. PIL is another example: from PIL import Image. And in the case of PIL at least, the fork distribution name does not change anything: pip install Pillow; from PIL import Image.
Sure, but my main point was about having "Provides" be a multi-valued field, which opens the door to a lot of complexity for perhaps little or no benefit. I don't have any problem with the use of "Provides" as a canonical name for a distribution. Even before Pillow came along, there used to be confusion, IIRC, between the names "Imaging" and "PIL".
Ah, in that case I think that makes me +1 for a mult-valued Provides (whatever that is, is this a PEP discussion?) because there are times when you want to provide multiple packages in a single distribution. E.g.
packages=[ 'foo', 'bar', ], package_dir={ 'foo': 'foo', 'bar': 'bar', },
where the distribution looks like this:
foo-bar/foo/foo/__init__.py foo-bar/bar/bar/__init__.py
And this reminds me of a question I had that I'll now ask in a separate thread :-)
Packages often provide a single PyPI name that is the same as the "import x" name, but Provides-Dist: is the distribution name. It is not necessarily the same as the package name(s) "import x". Or are you merging packages now :-)
On Tuesday, November 13, 2012 at 2:16 PM, Alex Clark wrote:
Ah, in that case I think that makes me +1 for a mult-valued Provides (whatever that is, is this a PEP discussion?) because there are times when you want to provide multiple packages in a single distribution. E.g.
Except provides-dist is supposed to say what distributions this provides, not what python packages (as in import foo) it provides. I'm against making it sane to bundle an upstream dep alongside your thing, and have it install as if you installed that dist… why not just requires-dist and be done with it.
packages=[ 'foo', 'bar', ], package_dir={ 'foo': 'foo', 'bar': 'bar', },
where the distribution looks like this:
foo-bar/foo/foo/__init__.py foo-bar/bar/bar/__init__.py
And this reminds me of a question I had that I'll now ask in a separate thread :-)
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org (mailto:Distutils-SIG@python.org) http://mail.python.org/mailman/listinfo/distutils-sig
-- Alex Clark · https://www.gittip.com/aclark4life/
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org (mailto:Distutils-SIG@python.org) http://mail.python.org/mailman/listinfo/distutils-sig
Recall that multi-valued Provides-Dist is from the 8-year-old Metadata 1.2 and is unchanged in the new PEP. When the PEP says "bundle" it means not a pip bundle, but two packages that have been merged together. The example given is ZODB and transaction, which used to be a single PyPI package and now are two. If it had gone the other way, and the two packages became one, then the merged package would Provides-Dist the obsolete name(s). It may not be pretty, but it is not always possible to come up with a perfect distribution factorization the first time around. Provides-Dist (multiple use) http://www.python.org/dev/peps/pep-0345/#id26 Each entry contains a string naming a Distutils project which is contained within this distribution. This field *must* include the project identified in the Name field, followed by the version : Name (Version). A distribution may provide additional names, e.g. to indicate that multiple projects have been bundled together. For instance, source distributions of the ZODB project have historically included the transaction project, which is now available as a separate distribution. Installing such a source distribution satisfies requirements for both ZODB and transaction. A distribution may also provide a "virtual" project name, which does not correspond to any separately-distributed project: such a name might be used to indicate an abstract capability which could be supplied by one of multiple projects. E.g., multiple projects might supply RDBMS bindings for use by a given ORM: each project might declare that it provides ORM-bindings, allowing other projects to depend only on having at most one of them installed. A version declaration may be supplied and must follow the rules described in Version Specifiershttp://www.python.org/dev/peps/pep-0345/#version-specifiers. The distribution's version number will be implied if none is specified. Examples: Provides-Dist: OtherProject Provides-Dist: AnotherProject (3.4) Provides-Dist: virtual_package
participants (7)
-
Alex Clark
-
Carl Meyer
-
Daniel Holth
-
Donald Stufft
-
Marcus Smith
-
PJ Eby
-
Vinay Sajip