dependencies, pip and non-PyPI-hosted packages
Hi I have a couple of packages that I don't want to upload to PyPI just yet but they are hosted some other place, and one (project2) depends on the other (project1). I've packaged them using the standard distutils setup. For project1, setup.py contains: setup(name='project1', ..., requires=[], provides=['project1'], ...) and for project2, setup.py contains: setup(name='project2', ..., requires=['project1'], provides=['project2'], ...) I put them in some host https://my.host.org/repository, but when I run pip install -f https://my.host.org/repository project2 installs project2 correctly, but it does not install project1. So my first question is: is the 'requires' keyword in setup.py actually used when installing packages? or it only works for projects hosted on PyPI? Thanks -- Ernesto Posse Modelling and Analysis in Software Engineering School of Computing Queen's University - Kingston, Ontario, Canada
Hi Ernesto, On 04/19/2011 12:42 PM, Ernesto Posse wrote:
Hi I have a couple of packages that I don't want to upload to PyPI just yet but they are hosted some other place, and one (project2) depends on the other (project1). I've packaged them using the standard distutils setup.
For project1, setup.py contains:
setup(name='project1', ..., requires=[], provides=['project1'], ...)
and for project2, setup.py contains:
setup(name='project2', ..., requires=['project1'], provides=['project2'], ...)
I put them in some host https://my.host.org/repository, but when I run
pip install -f https://my.host.org/repository project2
installs project2 correctly, but it does not install project1.
So my first question is: is the 'requires' keyword in setup.py actually used when installing packages? or it only works for projects hosted on PyPI?
The "standard" requires and provides keywords defined in PEP 314 and accepted by distutils are, AFAIK, not actually used by any existing tools. They are also deprecated by the newer PEP 345 metadata standard. (The reason for this, as I understand it, is that those keywords were defined by PEP 314 to refer to importable module/package names rather than project names. This makes them almost impossible for an installer to use, given that we have no global mapping available for translating an importable module/package name to an installable project name.) In order to have pip (or any other tool I know of) respect dependencies specified in setup.py, you need to use setuptools (or the "distribute" fork) in your setup.py and use its install_requires keyword argument instead. See http://peak.telecommunity.com/DevCenter/setuptools#new-and-changed-setup-key... for details. Carl
Thanks. I have tried it with distribute, and it's quite nice, but I have a couple of questions: On Tue, Apr 19, 2011 at 2:26 PM, Carl Meyer <carl@oddbird.net> wrote:
Hi Ernesto,
On 04/19/2011 12:42 PM, Ernesto Posse wrote:
Hi I have a couple of packages that I don't want to upload to PyPI just yet but they are hosted some other place, and one (project2) depends on the other (project1). I've packaged them using the standard distutils setup.
For project1, setup.py contains:
setup(name='project1', ..., requires=[], provides=['project1'], ...)
and for project2, setup.py contains:
setup(name='project2', ..., requires=['project1'], provides=['project2'], ...)
I put them in some host https://my.host.org/repository, but when I run
pip install -f https://my.host.org/repository project2
installs project2 correctly, but it does not install project1.
So my first question is: is the 'requires' keyword in setup.py actually used when installing packages? or it only works for projects hosted on PyPI?
The "standard" requires and provides keywords defined in PEP 314 and accepted by distutils are, AFAIK, not actually used by any existing tools. They are also deprecated by the newer PEP 345 metadata standard.
(The reason for this, as I understand it, is that those keywords were defined by PEP 314 to refer to importable module/package names rather than project names. This makes them almost impossible for an installer to use, given that we have no global mapping available for translating an importable module/package name to an installable project name.)
In order to have pip (or any other tool I know of) respect dependencies specified in setup.py, you need to use setuptools (or the "distribute" fork) in your setup.py and use its install_requires keyword argument instead. See http://peak.telecommunity.com/DevCenter/setuptools#new-and-changed-setup-key... for details.
1) using distribute and setup(..., install_requires=['project1'],..., dependency_links=['http://my.host.org/repository/'],...) installs as expected both project1 and project2, but pip uninstall project2 does not uninstall project1. This is quite disappointing, as a user may be unaware of dependencies automatically installed, and thus, the uninstall leaves behind something that was installed with the bundle. I imagine that the idea is that the user may install some other package that depends on 'project1' and pip takes the conservative approach (is that the case?) but I would have expected for pip or distribute or setuptools or distutils to keep some dependency reference counter. Does any of these tools have something like that? or is it going to be addressed in distutils2? 2) if install_requires is missing a dependency (project1), the package gets installed without that dependency, but if I add a dependency and the user attempts an install (project2) with the updated setup.py (listing the new dependency) pip will say that the package (project2) is already installed and won't attempt to install the dependencies. Is this correct? If so, is there a way to tell pip to install project2's dependencies? Thanks -- Ernesto Posse Modelling and Analysis in Software Engineering School of Computing Queen's University - Kingston, Ontario, Canada
On 04/19/2011 04:07 PM, Ernesto Posse wrote:
1) using distribute and
setup(..., install_requires=['project1'],..., dependency_links=['http://my.host.org/repository/'],...)
installs as expected both project1 and project2, but
pip uninstall project2
does not uninstall project1. This is quite disappointing, as a user may be unaware of dependencies automatically installed, and thus, the uninstall leaves behind something that was installed with the bundle. I imagine that the idea is that the user may install some other package that depends on 'project1' and pip takes the conservative approach (is that the case?) but I would have expected for pip or distribute or setuptools or distutils to keep some dependency reference counter. Does any of these tools have something like that? or is it going to be addressed in distutils2?
No, there's no current Python packaging tool I know of that keeps a dependency reference counter and automatically uninstalls orphaned dependencies. The new standard installation format (defined in PEP 376 and implemented in distutils2) does record whether a package was installed by explicit user request or as a dependency; combined with checking dependencies of all other installed packages, this will make it possible for an uninstaller to implement automatic (or prompted) dependency uninstalls.
2) if install_requires is missing a dependency (project1), the package gets installed without that dependency, but if I add a dependency and the user attempts an install (project2) with the updated setup.py (listing the new dependency) pip will say that the package (project2) is already installed and won't attempt to install the dependencies. Is this correct? If so, is there a way to tell pip to install project2's dependencies?
With a real package release, the setup.py metadata should never change without a version number change. In which case, if you specify the new version explicitly or use --upgrade, the previous version will be uninstalled and the new one installed in its place, with dependencies. (Actually, if you specify --upgrade and you already have the most recent version installed, it will still uninstall and reinstall it; but this is actually considered a bug.) Carl
On 19/04/2011 22:18, Carl Meyer wrote:
does not uninstall project1. This is quite disappointing, as a user may be unaware of dependencies automatically installed, and thus, the uninstall leaves behind something that was installed with the bundle. I imagine that the idea is that the user may install some other package that depends on 'project1' and pip takes the conservative approach (is that the case?) but I would have expected for pip or distribute or setuptools or distutils to keep some dependency reference counter. Does any of these tools have something like that? or is it going to be addressed in distutils2?
No, there's no current Python packaging tool I know of that keeps a dependency reference counter and automatically uninstalls orphaned dependencies.
I'm pretty sure zc.buildout does this by way of recreating the script when necessary.
2) if install_requires is missing a dependency (project1), the package gets installed without that dependency, but if I add a dependency and the user attempts an install (project2) with the updated setup.py (listing the new dependency) pip will say that the package (project2) is already installed and won't attempt to install the dependencies. Is this correct? If so, is there a way to tell pip to install project2's dependencies?
With a real package release, the setup.py metadata should never change without a version number change.
I should point out that zc.buildout deals gracefully with this case too, by way of regenerating scripts when needed. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Thanks. I have another question. Now that I'm using distribute, pip handles the requirements correctly, but if I install with: python setup.py install the dependencies are not installed. This is a bit perplexing. I am trying the scenario where the user has a plain Python installation with no pip, distribute or setuptools (hence installing the old fashioned way). My setup.py looks like this: # Begin of setup.py from distribute_setup import use_setuptools use_setuptools() from setuptools import setup, find_packages setup(..., install_requires=['project1'],...,dependency_links=['http://my.host.org/repository/'],...) # End of setup.py and I took care of including distribute_setup.py in MANIFEST.in, so it is part of the .tar.gz. When attempting to install, I get this $ sudo python setup.py install Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.15.tar.gz Extracting in /tmp/tmp4VDb5n Now working in /tmp/tmp4VDb5n/distribute-0.6.15 Building a Distribute egg in /home/eposse/Downloads/project2-0.1dev /home/eposse/Downloads/proyecto3-0.1dev/distribute-0.6.15-py2.6.egg /usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'install_requires' warnings.warn(msg) ... And the dependencies are ignored. I don't understand: I explicitly called use_setuptools in my setup, and as can be seen, distribute is automatically downloaded (albeit installed in a temporary directory), and imported setup from setuptools, but it looks like it is calling the plain distutils setup !? Why is it not calling the distribute/setuptools setup? Is there any way around this? Thanks PS: since 'requires' and 'provides' are not used in any tool, why are they in the official documentation? On Tue, Apr 19, 2011 at 5:18 PM, Carl Meyer <carl@oddbird.net> wrote:
On 04/19/2011 04:07 PM, Ernesto Posse wrote:
1) using distribute and
setup(..., install_requires=['project1'],..., dependency_links=['http://my.host.org/repository/'],...)
installs as expected both project1 and project2, but
pip uninstall project2
does not uninstall project1. This is quite disappointing, as a user may be unaware of dependencies automatically installed, and thus, the uninstall leaves behind something that was installed with the bundle. I imagine that the idea is that the user may install some other package that depends on 'project1' and pip takes the conservative approach (is that the case?) but I would have expected for pip or distribute or setuptools or distutils to keep some dependency reference counter. Does any of these tools have something like that? or is it going to be addressed in distutils2?
No, there's no current Python packaging tool I know of that keeps a dependency reference counter and automatically uninstalls orphaned dependencies. The new standard installation format (defined in PEP 376 and implemented in distutils2) does record whether a package was installed by explicit user request or as a dependency; combined with checking dependencies of all other installed packages, this will make it possible for an uninstaller to implement automatic (or prompted) dependency uninstalls.
2) if install_requires is missing a dependency (project1), the package gets installed without that dependency, but if I add a dependency and the user attempts an install (project2) with the updated setup.py (listing the new dependency) pip will say that the package (project2) is already installed and won't attempt to install the dependencies. Is this correct? If so, is there a way to tell pip to install project2's dependencies?
With a real package release, the setup.py metadata should never change without a version number change. In which case, if you specify the new version explicitly or use --upgrade, the previous version will be uninstalled and the new one installed in its place, with dependencies. (Actually, if you specify --upgrade and you already have the most recent version installed, it will still uninstall and reinstall it; but this is actually considered a bug.)
Carl _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Ernesto Posse Modelling and Analysis in Software Engineering School of Computing Queen's University - Kingston, Ontario, Canada
On Fri, Apr 22, 2011 at 12:34 PM, Ernesto Posse <eposse@gmail.com> wrote:
PS: since 'requires' and 'provides' are not used in any tool, why are they in the official documentation?
We answered the chicken-and-egg problem incorrectly. Ignore those. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> "Give me the luxuries of life and I will willingly do without the necessities." --Frank Lloyd Wright
Hi Ernesto, On 04/22/2011 11:34 AM, Ernesto Posse wrote:
Thanks. I have another question. Now that I'm using distribute, pip handles the requirements correctly, but if I install with:
python setup.py install
the dependencies are not installed. This is a bit perplexing.
This is just how it is. "python setup.py install" installs the package in question. Installer tools like pip or easy_install handle dependencies. If you want dependencies handled automatically, your users will need to use pip or easy_install.
I don't understand: I explicitly called use_setuptools in my setup, and as can be seen, distribute is automatically downloaded (albeit installed in a temporary directory), and imported setup from setuptools, but it looks like it is calling the plain distutils setup !?
Why is it not calling the distribute/setuptools setup?
No, it is calling the distribute setup. If you look at how your package is installed, you'll find it in an egg - that's a sure sign of setuptools/distribute. It's just that "python setup.py install" does not handle dependencies, even with setuptools/distribute.
Is there any way around this?
Not that I know of. Carl
On Fri, Apr 22, 2011 at 5:54 PM, Carl Meyer <carl@oddbird.net> wrote:
Hi Ernesto,
On 04/22/2011 11:34 AM, Ernesto Posse wrote:
Thanks. I have another question. Now that I'm using distribute, pip handles the requirements correctly, but if I install with:
python setup.py install
the dependencies are not installed. This is a bit perplexing.
This is just how it is. "python setup.py install" installs the package in question. Installer tools like pip or easy_install handle dependencies. If you want dependencies handled automatically, your users will need to use pip or easy_install.
Or zc.buildout. :) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton
On Fri, Apr 22, 2011 at 5:54 PM, Carl Meyer <carl@oddbird.net> wrote:
I don't understand: I explicitly called use_setuptools in my setup, and as can be seen, distribute is automatically downloaded (albeit installed in a temporary directory), and imported setup from setuptools, but it looks like it is calling the plain distutils setup !?
Why is it not calling the distribute/setuptools setup?
No, it is calling the distribute setup. If you look at how your package is installed, you'll find it in an egg - that's a sure sign of setuptools/distribute. It's just that "python setup.py install" does not handle dependencies, even with setuptools/distribute.
Hmm... OK, I guess I assumed that distribute was extending the standard distutils install command to install dependencies, rather than pip being responsible. So essentially to take advantage of distribute one must use pip? Anyway, is automatic dependency installation going to be a standard feature of distutils2? -- Ernesto Posse Modelling and Analysis in Software Engineering School of Computing Queen's University - Kingston, Ontario, Canada
On 04/22/2011 06:09 PM, Ernesto Posse wrote:
Hmm... OK, I guess I assumed that distribute was extending the standard distutils install command to install dependencies, rather than pip being responsible. So essentially to take advantage of distribute one must use pip?
It extends distutils to do a number of different things (like install actual metadata, for instance), and it provides the metadata related to dependencies - but yes, it's easy_install or pip or zc.buildout (thanks Jim!) that actually makes use of the dependency info to automatically install things.
Anyway, is automatic dependency installation going to be a standard feature of distutils2?
Yes, d2 will come with a "pysetup" command similar to pip/easy_install that will handle dependencies. Carl
At 04:54 PM 4/22/2011 -0500, Carl Meyer wrote:
No, it is calling the distribute setup. If you look at how your package is installed, you'll find it in an egg - that's a sure sign of setuptools/distribute. It's just that "python setup.py install" does not handle dependencies, even with setuptools/distribute.
Uh, yes it does, actually. (At least with setuptools, it does; don't know about distribute.) Ernesto, does it work with the official setuptools? (i.e., if you completely uninstall distribute and install setuptools instead) If it doesn't work, you can file a bug at http://bugs.python.org/setuptools and I'll see what we can do to get your project fixed up. (Just make sure you've *fully* uninstalled Distribute, and use ez_setup.py instead of distribute_setup.py, as Distribute does some patching that keeps setuptools from installing correctly.)
On 04/23/2011 02:00 PM, P.J. Eby wrote:
At 04:54 PM 4/22/2011 -0500, Carl Meyer wrote:
No, it is calling the distribute setup. If you look at how your package is installed, you'll find it in an egg - that's a sure sign of setuptools/distribute. It's just that "python setup.py install" does not handle dependencies, even with setuptools/distribute.
Uh, yes it does, actually. (At least with setuptools, it does; don't know about distribute.)
Erp. Yeah, you're right; just verified that it works fine with both setuptools and distribute. Dunno where I got that idea. Carl
On Sat, Apr 23, 2011 at 3:12 PM, Carl Meyer <carl@oddbird.net> wrote:
On 04/23/2011 02:00 PM, P.J. Eby wrote:
At 04:54 PM 4/22/2011 -0500, Carl Meyer wrote:
No, it is calling the distribute setup. If you look at how your package is installed, you'll find it in an egg - that's a sure sign of setuptools/distribute. It's just that "python setup.py install" does not handle dependencies, even with setuptools/distribute.
Uh, yes it does, actually. (At least with setuptools, it does; don't know about distribute.)
Erp. Yeah, you're right; just verified that it works fine with both setuptools and distribute. Dunno where I got that idea.
Carl
I'm confused now. I have tried both with setuptools alone (using ez_setup.py instead of distribute_setup.py) and distribute, and it doesn't work with distribute alone for me: it doesn't install the dependencies and gives me those warnings about 'install_requires' not being recognized. Any ideas about what could be wrong? -- Ernesto Posse Modelling and Analysis in Software Engineering School of Computing Queen's University - Kingston, Ontario, Canada
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/26/2011 12:02 PM, Ernesto Posse wrote:
On Sat, Apr 23, 2011 at 3:12 PM, Carl Meyer <carl@oddbird.net> wrote:
On 04/23/2011 02:00 PM, P.J. Eby wrote:
At 04:54 PM 4/22/2011 -0500, Carl Meyer wrote:
No, it is calling the distribute setup. If you look at how your package is installed, you'll find it in an egg - that's a sure sign of setuptools/distribute. It's just that "python setup.py install" does not handle dependencies, even with setuptools/distribute.
Uh, yes it does, actually. (At least with setuptools, it does; don't know about distribute.)
Erp. Yeah, you're right; just verified that it works fine with both setuptools and distribute. Dunno where I got that idea.
Carl
I'm confused now. I have tried both with setuptools alone (using ez_setup.py instead of distribute_setup.py) and distribute, and it doesn't work with distribute alone for me: it doesn't install the dependencies and gives me those warnings about 'install_requires' not being recognized.
Any ideas about what could be wrong?
How are you invoking setup? Should be something like: from setuptools import setup setup(name='your.package', ... install_requires=['other.packaage'], ) Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk23VXsACgkQ+gerLs4ltQ5O6gCfd8WzYUb6+bWlGvPC/ZUsfcHu NtIAn1pFAvWtvTV8Slap6Iopl2sT/5UD =Hi4F -----END PGP SIGNATURE-----
On Tue, Apr 26, 2011 at 7:30 PM, Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04/26/2011 12:02 PM, Ernesto Posse wrote:
On Sat, Apr 23, 2011 at 3:12 PM, Carl Meyer <carl@oddbird.net> wrote:
On 04/23/2011 02:00 PM, P.J. Eby wrote:
At 04:54 PM 4/22/2011 -0500, Carl Meyer wrote:
No, it is calling the distribute setup. If you look at how your package is installed, you'll find it in an egg - that's a sure sign of setuptools/distribute. It's just that "python setup.py install" does not handle dependencies, even with setuptools/distribute.
Uh, yes it does, actually. (At least with setuptools, it does; don't know about distribute.)
Erp. Yeah, you're right; just verified that it works fine with both setuptools and distribute. Dunno where I got that idea.
Carl
I'm confused now. I have tried both with setuptools alone (using ez_setup.py instead of distribute_setup.py) and distribute, and it doesn't work with distribute alone for me: it doesn't install the dependencies and gives me those warnings about 'install_requires' not being recognized.
Any ideas about what could be wrong?
How are you invoking setup? Should be something like:
from setuptools import setup
setup(name='your.package', ... install_requires=['other.packaage'], )
The version using distribute (which doesn't get the dependencies and looks as if it was calling the distutils setup) is this: # Begin of setup.py with distribute from distribute_setup import use_setuptools use_setuptools() from setuptools import setup, find_packages setup(name="project3",..., install_requires=['project1','project2'],...,dependency_links=['http://my.host.org/repository/'],...) # End of setup.py with distribute The version that uses only setuptools and not distribute (and does pick the dependencies) is as follows: # Begin of setup.py with setuptools from ez_setup import use_setuptools use_setuptools() from setuptools import setup, find_packages setup(name="project3",..., install_requires=['project1','project2'],...,dependency_links=['http://my.host.org/repository/'],...) # End of setup.py with setuptools The only difference is using ez_setup instead of distribute_setup (the MANIFEST.in file is updated accordingly). I have tried both in "clean" enviroments, removing both setuptools and distribute. Of course, if I do use pip instead of "python setup install" it works just fine.
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk23VXsACgkQ+gerLs4ltQ5O6gCfd8WzYUb6+bWlGvPC/ZUsfcHu NtIAn1pFAvWtvTV8Slap6Iopl2sT/5UD =Hi4F -----END PGP SIGNATURE-----
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Ernesto Posse Modelling and Analysis in Software Engineering School of Computing Queen's University - Kingston, Ontario, Canada
On 27/04/2011 00:45, Ernesto Posse wrote:
# Begin of setup.py with distribute
from distribute_setup import use_setuptools use_setuptools() from setuptools import setup, find_packages
setup(name="project3",..., install_requires=['project1','project2'],...,dependency_links=['http://my.host.org/repository/'],...)
Won't solve your problem, but please don't use dependency_links, it's just plain evil... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
participants (7)
-
Carl Meyer -
Chris Withers -
Ernesto Posse -
Fred Drake -
Jim Fulton -
P.J. Eby -
Tres Seaver