At least one package management tool for 2.7
I wonder if there are many people here who don't use some kind of "easy_install" package for package management in their Python / virtualenv installations? I propose to include at least one such package that is capable to auto-update itself in Python 2.7 C:\~env\Python27>python.exe -m easy_install C:\~env\Python27\python.exe: No module named easy_install C:\~env\Python27>python.exe -m pip C:\~env\Python27\python.exe: No module named pip It bugs me when I have to troubleshoot things on yet another machine that doesn't have some kind of `setuptools` installed. Or when I have to test some bug in my package on different Python version with a clean install and need some dependencies. -- anatoly t.
On Wed, Mar 24, 2010 at 10:59 AM, anatoly techtonik <techtonik@gmail.com> wrote:
I wonder if there are many people here who don't use some kind of "easy_install" package for package management in their Python / virtualenv installations? I propose to include at least one such package that is capable to auto-update itself in Python 2.7
C:\~env\Python27>python.exe -m easy_install C:\~env\Python27\python.exe: No module named easy_install
C:\~env\Python27>python.exe -m pip C:\~env\Python27\python.exe: No module named pip
It bugs me when I have to troubleshoot things on yet another machine that doesn't have some kind of `setuptools` installed. Or when I have to test some bug in my package on different Python version with a clean install and need some dependencies.
We are working on distutils2 right now to improve the situation, and Ian has proposed to work on the possible inclusion of virtualenv in the stldib as well. I'll talk for distutils2 : The plan is to provide a distutils2 standalone version that can be installed from 2.4 to 3.x, and that will provide a basic installer/uninstaller via -m. Distutils2 is planned to be reintegrated in the stdlib in Python 3.3, and my goal is to release it when Python 2.7 final is released. The open question is: do we want to include a full installer that takes care of installing / removing dependencies as well ? I think not. Pip already provides this feature on the top of distutils (and distutils2 later I guess) and is not hard to install on the top of Python. But the "auto-update" story seems interesting, can you expand on this ? Tarek -- Tarek Ziadé | http://ziade.org
On Wed, Mar 24, 2010 at 12:26 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Distutils2 is planned to be reintegrated in the stdlib in Python 3.3, and my goal is to release it when Python 2.7 final is released.
Does that means "after" Python 2.7, because I meant it to be "before" or at least "with"?
The open question is: do we want to include a full installer that takes care of installing / removing dependencies as well ?
If there is a risk to get nothing at all in 2.7 distribution, because it just won't be ready/accepted by that time, then I it is certainly optional.
But the "auto-update" story seems interesting, can you expand on this ?
Sure. Package management tool should have an ability to update itself when required regardless of Python release. For example:: python.exe -m easy_install setuptools This will get you new version of `setuptools` and `easy_install` module from it automagically. You do not need to install new version of `setuptools` manually or copy files from SVN if you want to see fixes before next Python release. The stuff you would likely need to do with distutils bugs, which I personally find awkward. Package management is orthogonal to Python releases, and it is more oriented at Python users who don't like to wait or follow PEPs. That's why package management tool such as 'easy_install' has shorter development cycle, and it should faster react to user feedback. What can be one of the reasons that no package management tool is included with Python. In various README you may often see "requires setuptools > 0.6c9" or similar. I can't see why package management tool can not detect this dependency and propose to update itself. If it is impossible to ship the whole package management system then at least Python distribution may carry small bootstrap script for it. When user tries to execute package management tools, it warns him that these are not installed and gives a hint where to get them::
python -m easy_install bla-bla-bla Error: easy_install module is not shipped with this Python release. Please execute the following command to install the latest version.
python -m easy_bootstrap -- anatoly t.
On Wed, Mar 24, 2010 at 7:23 AM, anatoly techtonik <techtonik@gmail.com> wrote:
Sure. Package management tool should have an ability to update itself when required regardless of Python release. For example::
python.exe -m easy_install setuptools
This should be:
python -m easy_install -U setuptools
JFTR More precisely, what I use for CI is {{{ #!sh $ easy_install -U setuptools==dev }}} but the `python -m` part should work as well ;o) PS: e.g. that allows to check out code from SVN in order to use setuptools 0.7.x `test_runner` switch like this {{{ python -W ignore::DeprecationWarning setup.py test -r ciutils:junitrunner }}} ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: TracRpc: API v2: Test cases for XML-RPC ... PASS - http://bitbucket.org/osimons/trac-rpc-mq/changeset/228ef43726b0/
On Wed, Mar 24, 2010 at 12:20 PM, anatoly techtonik <techtonik@gmail.com> wrote:
On Wed, Mar 24, 2010 at 12:26 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Distutils2 is planned to be reintegrated in the stdlib in Python 3.3, and my goal is to release it when Python 2.7 final is released.
Does that means "after" Python 2.7, because I meant it to be "before" or at least "with"?
The goal is to provide a first version by the time 2.7 is out.
The open question is: do we want to include a full installer that takes care of installing / removing dependencies as well ?
If there is a risk to get nothing at all in 2.7 distribution, because it just won't be ready/accepted by that time, then I it is certainly optional.
Understand that the distutils2 project is happening outside the stdlib at this time, so you will have to install it on the top of Python in any case. Last, its release cycle will be shorter until it gets back in the stdlib, that will make it easy to add features.
But the "auto-update" story seems interesting, can you expand on this ?
Sure. Package management tool should have an ability to update itself when required regardless of Python release. For example::
python.exe -m easy_install setuptools
This will get you new version of `setuptools` and `easy_install` module from it automagically. You do not need to install new version of `setuptools` manually or copy files from SVN if you want to see fixes before next Python release. The stuff you would likely need to do with distutils bugs, which I personally find awkward.
Package management is orthogonal to Python releases, and it is more oriented at Python users who don't like to wait or follow PEPs. That's why package management tool such as 'easy_install' has shorter development cycle, and it should faster react to user feedback. What can be one of the reasons that no package management tool is included with Python.
In various README you may often see "requires setuptools > 0.6c9" or similar. I can't see why package management tool can not detect this dependency and propose to update itself.
If it is impossible to ship the whole package management system then at least Python distribution may carry small bootstrap script for it. When user tries to execute package management tools, it warns him that these are not installed and gives a hint where to get them::
python -m easy_install bla-bla-bla Error: easy_install module is not shipped with this Python release. Please execute the following command to install the latest version.
python -m easy_bootstrap
Interesting, can you start a new thread on distutils-SIG for this part, so we can discuss it there ? Regards Tarek -- Tarek Ziadé | http://ziade.org
anatoly techtonik wrote:
If it is impossible to ship the whole package management system then at least Python distribution may carry small bootstrap script for it. When user tries to execute package management tools, it warns him that these are not installed and gives a hint where to get them::
python -m easy_install bla-bla-bla Error: easy_install module is not shipped with this Python release. Please execute the following command to install the latest version.
python -m easy_bootstrap
Note that this idea has come up before and is *much* more likely to get traction than including a full package management system. The nature of package management is such that tying an installation system to the release cycle of the core interpreter is unlikely to end well. A bootstrapping tool that only knows how to download a single specific package would be much easier to keep stable. Even such a scaled back idea is still far from a certainty to be accepted though, given the number of people who vehemently object to duplicating any significant part of the functionality of the system package management tools (i.e. RPM, apt et al). At this point, the packaging story is in the hands of distutils-sig and they're pressing forward with several initiatives that will permit the construction of more robust and reliable Python-specific package management systems (such as supporting listing and uninstallation of installed packages). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Wed, Mar 24, 2010 at 6:26 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
The open question is: do we want to include a full installer that takes care of installing / removing dependencies as well ?
I think not. Pip already provides this feature on the top of distutils (and distutils2 later I guess) and is not hard to install on the top of Python.
Is pip able to determine and install dependencies recursively, like easy_install does? Or is it up to the requested package to it specify its dependencies (and its dependencies dependencies) in a pip requirements file that is distributed separately? Darren
On Wed, Mar 24, 2010 at 12:50 PM, Darren Dale <dsdale24@gmail.com> wrote:
On Wed, Mar 24, 2010 at 6:26 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
The open question is: do we want to include a full installer that takes care of installing / removing dependencies as well ?
I think not. Pip already provides this feature on the top of distutils (and distutils2 later I guess) and is not hard to install on the top of Python.
Is pip able to determine and install dependencies recursively, like easy_install does? Or is it up to the requested package to it specify its dependencies (and its dependencies dependencies) in a pip requirements file that is distributed separately?
Pip uses the same standard: the install_requires option in setuptools, that is put in a requires.txt file when the egg-info is computed. Notice that we have defined this field in the new version of the metadata (PEP 345) for future interoperability. Let's remove python-dev for packaging talks in the next answers Regards Tarek
Darren
-- Tarek Ziadé | http://ziade.org
On Wed, Mar 24, 2010 at 7:50 AM, Darren Dale <dsdale24@gmail.com> wrote:
On Wed, Mar 24, 2010 at 6:26 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
The open question is: do we want to include a full installer that takes care of installing / removing dependencies as well ?
I think not. Pip already provides this feature on the top of distutils (and distutils2 later I guess) and is not hard to install on the top of Python.
Is pip able to determine and install dependencies recursively, like easy_install does? Or is it up to the requested package to it specify its dependencies (and its dependencies dependencies) in a pip requirements file that is distributed separately?
My experience is that only `install_requires` is needed (unless you want to create app bundles AFAICR) , but in practice I've noticed that *some* easy_installable packages are not pip-able (though I had no time to figure out why :-/ ) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: On adding Hessian (RPC) support for Trac - http://feedproxy.google.com/~r/simelo-en/~3/Vit6dRudChU/on-adding-hessian-rp...
On Wed, Mar 24, 2010 at 7:27 AM, Olemis Lang <olemis@gmail.com> wrote:
My experience is that only `install_requires` is needed (unless you want to create app bundles AFAICR) , but in practice I've noticed that *some* easy_installable packages are not pip-able (though I had no time to figure out why :-/ )
Usually this is because Setuptools is poking at objects to do its work, while pip tries to work mostly with subprocesses. Though to complicate things a bit, pip makes sure the Setuptools monkeypatches to distutils are applied, so that it's always as though the setup.py says "from setuptools import setup". easy_install *also* does this. But then easy_install starts calling methods and whatnot, while pip just does: setup.py install --single-version-externally-managed --no-deps --record some_tmp_file The --no-deps keeps Setuptools from resolving dependencies (because it does so using easy_install), and --single-version-externally-managed keeps Setuptools doing egg directories. And --record keeps track of installed files, which are later moved around to facilitate uninstall. But some distributions pay extra attention to those options, or do other tricky things in their setup.py, and as a result this causes failures. Because easy_install is just calling internal methods, it kind of sidelines those tricks (also people don't tend to give these options to easy_install, and some don't even exist, so some code paths just aren't exercised with typical easy_install usage). Oh, the other reason is the link searching mechanism is slightly different between the two installers. Not particularly intentionally, just incidentally. Also because pip doesn't install anything but source packages, some packages are installable via easy_install but not pip -- usually this is an oversight on the part of the person doing the packaging, but they just never noticed. -- Ian Bicking | http://blog.ianbicking.org | http://twitter.com/ianbicking
On Wed, Mar 24, 2010 at 1:19 PM, Ian Bicking <ianb@colorstudy.com> wrote:
On Wed, Mar 24, 2010 at 7:27 AM, Olemis Lang <olemis@gmail.com> wrote:
My experience is that only `install_requires` is needed (unless you want to create app bundles AFAICR) , but in practice I've noticed that *some* easy_installable packages are not pip-able (though I had no time to figure out why :-/ )
Usually this is because Setuptools is poking at objects to do its work, while pip tries to work mostly with subprocesses. Though to complicate things a bit, pip makes sure the Setuptools monkeypatches to distutils are applied, so that it's always as though the setup.py says "from setuptools import setup". easy_install *also* does this.
But then easy_install starts calling methods and whatnot, while pip just does:
setup.py install --single-version-externally-managed --no-deps --record some_tmp_file
The --no-deps keeps Setuptools from resolving dependencies
Seeking clarification: how can pip recursively install dependencies *and* keep Setuptools from resolving dependencies? Darren
On Wed, Mar 24, 2010 at 12:53 PM, Darren Dale <dsdale24@gmail.com> wrote:
On Wed, Mar 24, 2010 at 1:19 PM, Ian Bicking <ianb@colorstudy.com> wrote:
On Wed, Mar 24, 2010 at 7:27 AM, Olemis Lang <olemis@gmail.com> wrote:
My experience is that only `install_requires` is needed (unless you want to create app bundles AFAICR) , but in practice I've noticed that *some* easy_installable packages are not pip-able (though I had no time to figure out why :-/ )
Usually this is because Setuptools is poking at objects to do its work, while pip tries to work mostly with subprocesses. Though to complicate things a bit, pip makes sure the Setuptools monkeypatches to distutils are applied, so that it's always as though the setup.py says "from setuptools import setup". easy_install *also* does this.
But then easy_install starts calling methods and whatnot, while pip just does:
setup.py install --single-version-externally-managed --no-deps --record some_tmp_file
The --no-deps keeps Setuptools from resolving dependencies
Seeking clarification: how can pip recursively install dependencies *and* keep Setuptools from resolving dependencies?
Using the --no-deps option to setup.py
anatoly techtonik <techtonik <at> gmail.com> writes:
I wonder if there are many people here who don't use some kind of "easy_install" package for package management in their Python / virtualenv installations? I propose to include at least one such package that is capable to auto-update itself in Python 2.7
C:\~env\Python27>python.exe -m easy_install C:\~env\Python27\python.exe: No module named easy_install
C:\~env\Python27>python.exe -m pip C:\~env\Python27\python.exe: No module named pip
It will not happen in 2.7, which is almost in beta phase. Based on previous discussions, it will most likely not happen in 3.2 either. The consensus is that setuptools, which both pip and easy_install depend on, should not be included into the core. However, Tarek's work on "distutils2" (a mostly forward-compatible replacement for distutils) will include features such as dependencies, and make it much easier to create and perhaps bundle a pip-like utility. So perhaps in 3.3 you have a chance ;) Regards Antoine.
participants (8)
-
anatoly techtonik -
Antoine Pitrou -
Darren Dale -
Ian Bicking -
Jason Baker -
Nick Coghlan -
Olemis Lang -
Tarek Ziadé