question about future pip capabilities
Hello, I apologize for cross-posting. I am working on improving how I distribute a few python packages, and in the process I have run into some questions that involve easy_install, setuptools/distribute/, pip, and the relationships between them. I was originally planning on using easy_install to take advantage of extras (easy_install foo[bar]) and automatically install dependencies, but I just discovered that easy_install does not support post-installation routines, which I need in order to add entries to the windows start menu. Pip will, I think, support post_installation routines if I implement a custom install command and pass it to setup(). But pip does not yet support extras or installation from windows binary installers, and does not support installation from eggs. The pip documentation states: * It cannot install from eggs. It only installs from source. (In the future it would be good if it could install binaries from Windows .exe or .msi -- binary install on other platforms is not a priority.) * It doesn't understand Setuptools extras (like package[test]). This should be added eventually. * It is incompatible with some packages that customize distutils or setuptools in their setup.py files. * Maybe it doesn't work on Windows. At least, the author doesn't test on Windows often. Questions concerning pip: * Is there a roadmap or timeline concerning installing from windows exe or msi files? I didn't see anything at bitbucket. * Is there a roadmap or timeline concerning extras? I didn't see anything at bitbucket. * Could anyone please expand on the comment that pip is incompatible with some packages that customize distutils or setuptools in their setup.py files, and that it maybe doesn't work on windows? Questions concerning Distribute: * The documentation states that easy_install will be deprecated, and that we should use pip. Isn't this premature, given the current discrepancy between the capabilities and platform support of pip and easy_install? Questions for pip and Distribute: * How closely coupled are these projects? * I have a bit of time after hours, how can I help? Darren
10.12.2009 19:18, Darren Dale kirjoitti:
Hello,
I apologize for cross-posting. I am working on improving how I distribute a few python packages, and in the process I have run into some questions that involve easy_install, setuptools/distribute/, pip, and the relationships between them.
I was originally planning on using easy_install to take advantage of extras (easy_install foo[bar]) and automatically install dependencies, but I just discovered that easy_install does not support post-installation routines, which I need in order to add entries to the windows start menu. Pip will, I think, support post_installation routines if I implement a custom install command and pass it to setup(). But pip does not yet support extras or installation from windows binary installers, and does not support installation from eggs. The pip documentation states:
* It cannot install from eggs. It only installs from source. (In the future it would be good if it could install binaries from Windows .exe or .msi -- binary install on other platforms is not a priority.) * It doesn't understand Setuptools extras (like package[test]). This should be added eventually. * It is incompatible with some packages that customize distutils or setuptools in their setup.py files. * Maybe it doesn't work on Windows. At least, the author doesn't test on Windows often.
Questions concerning pip:
* Is there a roadmap or timeline concerning installing from windows exe or msi files? I didn't see anything at bitbucket. * Is there a roadmap or timeline concerning extras? I didn't see anything at bitbucket. * Could anyone please expand on the comment that pip is incompatible with some packages that customize distutils or setuptools in their setup.py files, and that it maybe doesn't work on windows?
Questions concerning Distribute:
* The documentation states that easy_install will be deprecated, and that we should use pip. Isn't this premature, given the current discrepancy between the capabilities and platform support of pip and easy_install?
The next major distribute release (the one that nixes easy_install) is relatively far in the future, and we (well, I at least) expect pip to have matured enough by then.
Questions for pip and Distribute:
* How closely coupled are these projects? * I have a bit of time after hours, how can I help?
Darren _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Thu, Dec 10, 2009 at 11:18 AM, Darren Dale <dsdale24@gmail.com> wrote:
Questions concerning pip:
* Is there a roadmap or timeline concerning installing from windows exe or msi files? I didn't see anything at bitbucket.
There isn't a timeline, as we don't have anyone actively contributing with respect to Windows.
* Is there a roadmap or timeline concerning extras? I didn't see anything at bitbucket.
There isn't really; extras would be nice, but there hasn't been much demand for them. pip requirement files serve a similar purpose though in a somewhat different way.
* Could anyone please expand on the comment that pip is incompatible with some packages that customize distutils or setuptools in their setup.py files, and that it maybe doesn't work on windows?
Maybe the Windows statement is exaggerated. Unfortunately we don't have a good Windows testing setup, and I don't even know what that would look like. Some buildbot (or buildbot-like-thing, e.g., Hudson) running on a Windows machine would be really helpful. pip itself relies only on the command line and some expected behavior of setup.py files, it doesn't really care how this is implemented. To install a package it does: python -c 'import setuptools; __file__="setup.py"; execfile(__file__)' install --single-version-externally-managed --record=/tmp/XXX The first part installs the setuptools monkeypatching for packages that just use distutils. So you at least have to make something compatible with that monkeypatching (i.e., it can't rely on setuptools not being imported). Then it uses those two command-line switches (and I'm pretty sure that's it; you can see a complete record of what pip does with pip install -vv). Those are provided by Setuptools (or Distribute) and not by Distutils, but pip doesn't care how they are implemented so long as you accept those options. Also to be compatible with pip you have to create a proper Package.egg-info/ directory (to record that the package is installed) and use that --record option to write a list of all the files that are written to disk, so they can be removed when the package is removed.
Questions for pip and Distribute:
* How closely coupled are these projects?
They are not closely coupled at all.
* I have a bit of time after hours, how can I help?
If you are a Windows developer, Windows help for pip would be great. Installing Windows binaries would be the biggest help; I suspect this means unzipping the .exe or .msi and moving the files into place manually (since there is no setup.py), and maybe even generating that .egg-info/ file. I suspect that if you install a Windows installer directly that pip (and Setuptools/Distribute/pkg_resources) might not be able to see that the package is installed. But I've never tried. It would be nice if these installation techniques were compatible with each other, which means the installer creating .egg-info files (or shipping with EGG-INFO, similar to how Windows binary eggs work). In theory Windows binary eggs could be installed similarly, but I don't think there's as much enthusiasm for that among Windows users, and .exe or .msi packages should be mostly equivalent (except that lack of metadata). But I don't consider myself a good judge of Window users' preferences, so maybe I'm wrong. A shorter time commitment is to download pip and run the tests on Windows and report problems, as it's unfortunately quite possible we've introduced Windows regressions. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ian Bicking wrote:
A shorter time commitment is to download pip and run the tests on Windows and report problems, as it's unfortunately quite possible we've introduced Windows regressions.
I did a bit of work on getting pip tests running under Windows last spring, but didn't get very far. There were many test failures, most of them seemingly superficial: posix-specific assumptions about file locations and path separators in the tests themselves, not necessarily broken functionality in pip (though it's hard to say for sure with broken tests). Fixing those issues isn't hard, just tedious. I don't know what deeper issues or regressions might be uncovered once the tests run. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFLIU67FRcxmeyPUXIRAlt5AJoCVlEMF8ZucNPPwypKWETDhe98ZQCfeT5P fs8t8Mr0jaj2gsEwNN3kyvo= =fgc1 -----END PGP SIGNATURE-----
On Thu, Dec 10, 2009 at 1:40 PM, Carl Meyer <carl@meyerloewen.net> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Ian Bicking wrote:
A shorter time commitment is to download pip and run the tests on Windows and report problems, as it's unfortunately quite possible we've introduced Windows regressions.
I did a bit of work on getting pip tests running under Windows last spring, but didn't get very far. There were many test failures, most of them seemingly superficial: posix-specific assumptions about file locations and path separators in the tests themselves, not necessarily broken functionality in pip (though it's hard to say for sure with broken tests). Fixing those issues isn't hard, just tedious. I don't know what deeper issues or regressions might be uncovered once the tests run.
Probably some of those issues could also be fixed in ScriptTest, instead of making the pip tests tediously platform abstracted (e.g., normalizing \ vs /). -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker
On Thu, Dec 10, 2009 at 2:30 PM, Ian Bicking <ianb@colorstudy.com> wrote:
On Thu, Dec 10, 2009 at 11:18 AM, Darren Dale <dsdale24@gmail.com> wrote:
Questions concerning pip:
* Is there a roadmap or timeline concerning installing from windows exe or msi files? I didn't see anything at bitbucket.
There isn't a timeline, as we don't have anyone actively contributing with respect to Windows.
* Is there a roadmap or timeline concerning extras? I didn't see anything at bitbucket.
There isn't really; extras would be nice, but there hasn't been much demand for them. pip requirement files serve a similar purpose though in a somewhat different way.
3.2.0 in my requirements file, because pip is not designed to recursively check the (arbitrarily-named) requirements files of the dependencies I declare in my own requirements file. I have to keep
I brought this up on the virtualenv mailing list once before. Requirements files serve a very different use-case. If I want to define a project extra like foo[traits_qt4] that depends on a project with an extra like traits[qt4] > 3.2.0, I can't just list traits[qt4] track of not only my projects dependencies, but my dependencies dependencies, and on and on.
* I have a bit of time after hours, how can I help?
If you are a Windows developer, Windows help for pip would be great.
I distribute windows installers for a couple projects, but I'm not a windows developer...
A shorter time commitment is to download pip and run the tests on Windows and report problems, as it's unfortunately quite possible we've introduced Windows regressions.
This sounds like a good place to start. Regards, Darren
2009/12/10 Alex Grönholm <alex.gronholm@nextday.fi>:
10.12.2009 19:18, Darren Dale kirjoitti:
* The documentation states that easy_install will be deprecated, and that we should use pip. Isn't this premature, given the current discrepancy between the capabilities and platform support of pip and easy_install?
The next major distribute release (the one that nixes easy_install) is relatively far in the future, and we (well, I at least) expect pip to have matured enough by then.
What gives you that confidence? Ian Bicking just responded that there is currently not a timeline for full windows support, installing from binaries on windows, or project extras. Darren
2009/12/10 Darren Dale <dsdale24@gmail.com>: [..]
Questions for pip and Distribute:
* How closely coupled are these projects?
As Ian said, they are not closely coupled, but some people are contributing to both projects, and in Distribute we want to stop developing easy_install because a package installer should be project on its own. And pip development is very active these days. IOW if Pip needs apis Distutils doesn't (yet) provide, Distribute will try to provide them,
* I have a bit of time after hours, how can I help?
If you want to help on Distribute, there are many things you could do, from bug fixes to new development in the 0.7 versions. Contact me online if you want to discuss this (#distutils on freenode) Tarek
2009/12/10 Darren Dale <dsdale24@gmail.com>:
2009/12/10 Alex Grönholm <alex.gronholm@nextday.fi>:
10.12.2009 19:18, Darren Dale kirjoitti:
* The documentation states that easy_install will be deprecated, and that we should use pip. Isn't this premature, given the current discrepancy between the capabilities and platform support of pip and easy_install?
The next major distribute release (the one that nixes easy_install) is relatively far in the future, and we (well, I at least) expect pip to have matured enough by then.
What gives you that confidence? Ian Bicking just responded that there is currently not a timeline for full windows support, installing from binaries on windows, or project extras.
He said that because no one worked on it yet. By the time we will want to release Distribute 0.7 (without easy_install), if pip doesn't have what is required to support win32, I (or some other Distribute dev) will try to contribute to pip to help making it happen (if no one did before). IOW, the release of Distribute 0.7.x will be done once pip has what is required todrop easy_install for good Tarek
participants (5)
-
Alex Grönholm -
Carl Meyer -
Darren Dale -
Ian Bicking -
Tarek Ziadé