Re: [Distutils] pip vs easy_install vs distutils2
At 10:19 AM 5/29/2010 +0200, Tarek Ziadé wrote:
I was not thinking about this proposal. If this what Guido proposed at the summit, then I misunderstood.
I don't know what he proposed at the summit - I'm referring to the bootstrap script he wrote that actually does this. It was a couple years ago on Python-Dev, IIRC.
The problem with the feature you describe (bootstrap embed in the archive itself) is that we imply that the packager chooses himself an installer and forces the end-user to use it when he installs the software. This means that the end user might end up with several installers installed for the same Python.
So? It's not like they're going to accidentally run 'easy_install' when they meant to type 'pip', or vice versa. ;-)
It also implies that the developer provides a smart setup.py script that embeds that bootstrap, and runs some code.
Only if they need something beyond what the distutils provide.
It will also allow distributions to be "dumb" envelopes with static metadata that are the same all the time, no matter which tool created them, and eventually remove setup.py in favor of statically described metadata using PEP 345.
For packages with complex build requirements or distutils extensions (e.g. numpy), this is unlikely to happen any time soon. Conversely, for packages where this *is* the case, the current distutils is adequate, and having a bootstrapper that can install them would be a win.
Today, for instance, if an installer wants to install a distribution based on setuptools, it has to run the "egg_info" command to extract the metadata, on the target system.
It's a good thing Guido loaned me the time machine so I could go back to 2005 and fix that one, then: http://svn.python.org/view/sandbox/trunk/setuptools/setuptools/command/sdist.py?view=diff&r1=41093&r2=41094 Five years should be long enough ago that by now all the setuptools-based source distributions on PyPI today will already have the metadata you need... so when you go back and try again to build your new installer today, it will already be there for you to use. Just download any random setuptools-based sdist from PyPI and run "tar -tzf" or "unzip -v" on it to see! For that matter, it's long enough ago that you should also have the fix already retroactively included in Distribute as well, so you should probably even be able to look at one of Distribute's own source distributions and see that it now contains the metadata you require. ;-)
I'm not sure I understand this thread, mainly because I though distutils2 was to provide an installer. :) So with the risk of me not understanding it, here goes: I think both proposals, both the older bootstrap script and the summit proposal, is to make it easy to install an installer, which in practice means easy_install or pip. This is only necessary if we don't have a full installer in stdlib. I can see a couple of options: 0. We have a pyinstall script that runs pip *or* easy_install, depending on which one you choose when you first run it. I believe this is a bad idea, and will just create great amounts of confusion. 1. We include easy_install or pip in stdlib. However, I think we shouldn't include any installer in stdlib, until it has evolved into a proper package handling utility which also can uninstall, etc. 2. We include a pyinstall script that downloads from PyPI and runs setup.py install. This will cover 90% of install needs, for more advanced installers, you need to pyinstall pip or similar. This means that we end up with one more way to install, which isn't a disaster, but isn't really helping the situation. Also, the objection to including installers but not uninstallers are still relevant even though this is a super simplistic script that could be argues isn't really an installer at all. 3. We include a pypiget script, that will download a file from PyPI but *not* install it. That wouldn't create a one step install process, but it would speed up and simplify the install process without sacrificing flexibility. It also isn't an installer, so nobody can expect of require an uninstaller. But on the other hand, it's not exactly a massive helper. :-) 4. Write a proper installer/uninstaller and include that in stdlib. I suggest that it's call "pypi" and have commands like download/install/uninstall/upgrade. I'm not sure how much it needs to be based on/integrated with distutils2, as I don't know what the plans are for uninstaller support. Other than that it would pretty much only need to know how to download and run setup.py on packages, so I'm not sure it needs to be powered by distutils2 in any deeper sense. :-) Sorry if I'm answering something that wasn't a question. :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
Lennart Regebro wrote:
1. We include easy_install or pip in stdlib. However, I think we shouldn't include any installer in stdlib, until it has evolved into a proper package handling utility which also can uninstall, etc.
You are aware that pip uninstalls? It also queries PyPI, lists locally installed packages, etc. It at least has a start on most of the things one would want a "proper package handling utility" to do.
4. Write a proper installer/uninstaller and include that in stdlib. I suggest that it's call "pypi" and have commands like download/install/uninstall/upgrade. I'm not sure how much it needs to be based on/integrated with distutils2, as I don't know what the plans are for uninstaller support. Other than that it would pretty much only need to know how to download and run setup.py on packages, so I'm not sure it needs to be powered by distutils2 in any deeper sense. :-)
This sounds like a pip clone. Pip's installation "interface" is nothing more than "setup.py install" in a subshell, so distutils2 support for installation will likely be quite simple. Replacing pip's use of pkg_resources with distutils2's pkgutil for querying local package info will be a bit more work, but the APIs are pretty similar. The question about a hypothetical "distutils2 installer" is backwards compatibility. If it is not backwards-compatible with existing distutils/setuptools packages, it will be useless to most people (there will be legacy packages hanging around, ones that people depend on, for many years to come). If it is backwards-compatible with existing packages, then it is a reimplementation of pip. I sympathize with the ease-of-use reasons for having something in the stdlib. And I'm not sure putting pip in the stdlib would be good for pip's development. So I don't have a solution to propose. But I have a hard time believing that reimplementing pip is even under serious consideration. It smells of seriously underestimating the amount of work involved. Carl
On Sun, May 30, 2010 at 7:13 PM, Carl Meyer <carl@oddbird.net> wrote: [..]
I sympathize with the ease-of-use reasons for having something in the stdlib. And I'm not sure putting pip in the stdlib would be good for pip's development. So I don't have a solution to propose. But I have a hard time believing that reimplementing pip is even under serious consideration. It smells of seriously underestimating the amount of work involved.
I won't speak for others, but on my side, I don't underestimate the amount of work involved in an installer (heck, if I was scared by the amount of work in packaging, I'd quit 2 years ago ;)), but rather trying to figure out what is the path for a better packaging ecosystem. (it's a bit of a wicked problem) If you put aside the backward compatibility issues for a moment, I see several problems right now: A - If distutils2 doesn't include an installer, a fresh Python install will not be fully functional. I don't think it's the best option. Python should provide an installer/uninstaller that is compatible with the latest standards distutils2 provides. I don't want to release a software that will have to rely on a third party installer which don't exists yet : there's no clear plan to support the new standards yet in pip or in any other project afaik. B - adding within distutils2 an installer like pip (with the support of new standards) at a given version doesn't mean you can't develop the next version and release it as a third party package, so people can upgrade their installer and get the bleeding edge thing. This is doable as long as the installer is configurable. C - easy_install for instance, is already installed in Mac OS X by default. That makes sense from a end-user pov. So why wouldn't we have an installer in distutils2 in the stdlib ? D - letting *each* distribution decide what installer will be used when "python setup.py install" is called, without letting the end-user decides, (that's the ez_setup embed thing) means that we bury our goal to get rid of this file and to move toward statically defined metadata+ content. IOW have a clean separation between a Distribution and the code that is used to install it (eg the installer). Now about the backward compatibility issues: - as discussed with Ian, we are going to add backward compatibility in pkgutil, so we can read old egg-info formats. IOW, it deprecates the usage of pkg_resources in favor of an unified API. - if the installer part of distutils2 includes backward compatibility, it's ok I guess, as long as it's isolated to that part. So, I am still proposing to merge both projects and to have two kinds of releases: - distutils2 + pip included (maybe a renamed script) - pip, alone. And with the new metadata we have "provides-dist", so we can avoid conflicts here. Regards, Tarek -- Tarek Ziadé | http://ziade.org
On Sun, May 30, 2010 at 22:57, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
So, I am still proposing to merge both projects and to have two kinds of releases:
- distutils2 + pip included (maybe a renamed script)
I don't think this is necessary. Sure, distutils without an installer is kinda crippled, but the people who want to install distutils2 are capable of installing pip as well. :-) And an install script, like the one for distribute, could download and install both anyway. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
Tarek Ziadé wrote:
I won't speak for others, but on my side, I don't underestimate the amount of work involved in an installer (heck, if I was scared by the amount of work in packaging, I'd quit 2 years ago ;)), but rather trying to figure out what is the path for a better packaging ecosystem. (it's a bit of a wicked problem)
Indeed! (If anyone's put in enough time in packaging in the last couple years to make writing a new installer from scratch seem possible; it would be you. I hope we can find a solution that allows that energy to be put into something more productive, however!)
A - If distutils2 doesn't include an installer, a fresh Python install will not be fully functional. I don't think it's the best option. Python should provide an installer/uninstaller that is compatible with the latest standards distutils2 provides. I don't want to release a software that will have to rely on a third party installer which don't exists yet : there's no clear plan to support the new standards yet in pip or in any other project afaik.
I agree that it makes sense for there to be an installer packaged with distutils2 that supports all the features of the metadata standard, including dependencies. As far as "clear plan," what sort of plan are you looking for? The direction of pip mainline is up to Ian, of course, but I personally plan to do what I can towards a branch of pip that works with the new standards by the time distutils2 is released with a Python release. Given that pip will largely sit "atop" distutils2, I don't think it makes sense to start on the work in pip until distutils2 has stabilized.
B - adding within distutils2 an installer like pip (with the support of new standards) at a given version doesn't mean you can't develop the next version and release it as a third party package, so people can upgrade their installer and get the bleeding edge thing. This is doable as long as the installer is configurable.
Yep.
Now about the backward compatibility issues: - as discussed with Ian, we are going to add backward compatibility in pkgutil, so we can read old egg-info formats. IOW, it deprecates the usage of pkg_resources in favor of an unified API.
Great! I must have missed the end of that discussion in IRC.
- if the installer part of distutils2 includes backward compatibility, it's ok I guess, as long as it's isolated to that part.
If the pkgutil API is integrated and backwards-compatible, the remaining backwards-compatibility considerations would be: 1) Package installation. This shouldn't be too difficult ("setup.py install"), though if we want to be able to install old packages using the new PEP 376 installation format, that would involve some work. 2) Package-finding. I don't know if there's been discussion yet about whether the new distutils2 world will continue the same liberal link-searching algorithms that easy_install pioneered, or whether there will be a move towards something more structured.
So, I am still proposing to merge both projects and to have two kinds of releases:
- distutils2 + pip included (maybe a renamed script) - pip, alone.
This sounds good to me. I don't really care what the included one is called, though to me renaming sounds mostly like asking for confusion. Carl
On Sun, May 30, 2010 at 19:13, Carl Meyer <carl@oddbird.net> wrote:
Lennart Regebro wrote:
1. We include easy_install or pip in stdlib. However, I think we shouldn't include any installer in stdlib, until it has evolved into a proper package handling utility which also can uninstall, etc.
You are aware that pip uninstalls?
Not reliably, as it doesn't keep track of files, so it can't remove anything installed outside of the package.
It also queries PyPI, lists locally installed packages, etc. It at least has a start on most of the things one would want a "proper package handling utility" to do.
Absolutely.
This sounds like a pip clone.
Indeed. I should have clarified that the name "pypi" I suggested was to satisfy up Perl people claiming that Python don't have CPAN. ;-) I decided against saying that to not make anyone upset, seems I failed. I also did not suggest the inclusion of pip not not make easy_install fans angry. Seriously, all packaging people: You need to be less tetchy. Not every single debate needs to result in a flame war. :-) I'm serious. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
Hi Lennart, Lennart Regebro wrote:
You are aware that pip uninstalls?
Not reliably, as it doesn't keep track of files, so it can't remove anything installed outside of the package.
Not true, inasmuch as it depends on pip. When pip installs distributions it uses the --record option to keep a list of all installed files. When pip uninstalls distributions it installed, it reliably uninstalls everything (though it doesn't have the PEP 376 hash cleverness available yet to detect modified files). When pip is asked to uninstall packages installed via other methods that do not provide the full installed-files record, it does so on a best-effort basis. In the case of easy_installed packages, that means it uninstalls installed packages and console scripts; in most cases, this is everything that was installed.
Indeed. I should have clarified that the name "pypi" I suggested was to satisfy up Perl people claiming that Python don't have CPAN. ;-) I decided against saying that to not make anyone upset, seems I failed. I also did not suggest the inclusion of pip not not make easy_install fans angry.
Seriously, all packaging people: You need to be less tetchy. Not every single debate needs to result in a flame war. :-) I'm serious.
Sorry! I wouldn't say I was upset, just concerned that unnecessary duplication of work be avoided if possible. I am not concerned in some territorial way about the name of the installer that's included; "pypi" is just fine with me. I am concerned that it not be needlessly reimplemented from scratch; we have enough work to do without that! Carl
On Mon, May 31, 2010 at 16:55, Carl Meyer <carl@oddbird.net> wrote:
Not true, inasmuch as it depends on pip. When pip installs distributions it uses the --record option to keep a list of all installed files. When pip uninstalls distributions it installed, it reliably uninstalls everything (though it doesn't have the PEP 376 hash cleverness available yet to detect modified files).
Ah, OK, cool. That must be reasonably new, last time I saw a discussion about this it wasn't the case. That *could* be some time ago, though. :)
Lennart Regebro wrote:
On Mon, May 31, 2010 at 16:55, Carl Meyer <carl@oddbird.net> wrote:
Not true, inasmuch as it depends on pip. When pip installs distributions it uses the --record option to keep a list of all installed files. When pip uninstalls distributions it installed, it reliably uninstalls everything (though it doesn't have the PEP 376 hash cleverness available yet to detect modified files).
Ah, OK, cool. That must be reasonably new, last time I saw a discussion about this it wasn't the case. That *could* be some time ago, though. :)
Nope, pip's used --record on installation for years, and the above has been true since the moment uninstall landed in pip. There are enough different ways things can get installed that it's not surprising that some discussions may have been confused ;-) Carl
On Mon, May 31, 2010 at 19:02, Carl Meyer <carl@oddbird.net> wrote:
Nope, pip's used --record on installation for years, and the above has been true since the moment uninstall landed in pip. There are enough different ways things can get installed that it's not surprising that some discussions may have been confused ;-)
That may be it. Forcing --record in Python 3.2 would be a step forward then? :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
Lennart Regebro wrote:
On Mon, May 31, 2010 at 19:02, Carl Meyer <carl@oddbird.net> wrote:
Nope, pip's used --record on installation for years, and the above has been true since the moment uninstall landed in pip. There are enough different ways things can get installed that it's not surprising that some discussions may have been confused ;-)
That may be it. Forcing --record in Python 3.2 would be a step forward then? :-)
Would be, except that it's a setuptools-only option. Pip is only able to use it for all packages because it forces the import of setuptools, and thus setuptools' command overrides. Fortunately we have the right answer in PEP 376, now we just need to get it implemented. Carl
On Mon, May 31, 2010 at 7:10 PM, Carl Meyer <carl@oddbird.net> wrote:
Lennart Regebro wrote:
On Mon, May 31, 2010 at 19:02, Carl Meyer <carl@oddbird.net> wrote:
Nope, pip's used --record on installation for years, and the above has been true since the moment uninstall landed in pip. There are enough different ways things can get installed that it's not surprising that some discussions may have been confused ;-)
That may be it. Forcing --record in Python 3.2 would be a step forward then? :-)
Would be, except that it's a setuptools-only option.
No, that's a distutils option. But setuptools has its own implementation, but they seem like clone as far as I can tell (I didn't digg into that yet) Tarek
On Mon, May 31, 2010 at 7:04 PM, Lennart Regebro <regebro@gmail.com> wrote:
On Mon, May 31, 2010 at 19:02, Carl Meyer <carl@oddbird.net> wrote:
Nope, pip's used --record on installation for years, and the above has been true since the moment uninstall landed in pip. There are enough different ways things can get installed that it's not surprising that some discussions may have been confused ;-)
That may be it. Forcing --record in Python 3.2 would be a step forward then? :-)
You mean in the current distutils ? Because distutils2 will have the PEP 376 implementation, where we create a RECORD file for each installed project in its dist-info/
-- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Tarek Ziadé | http://ziade.org
On Mon, May 31, 2010 at 19:10, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
You mean in the current distutils ? Because distutils2 will have the PEP 376 implementation, where we create a RECORD file for each installed project in its dist-info/
Good. I suffer from PEP overload, and have long since given up on remembering what has been decided. :-) -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64
On 31 May, 2010, at 19:10, Tarek Ziadé wrote:
On Mon, May 31, 2010 at 7:04 PM, Lennart Regebro <regebro@gmail.com> wrote:
On Mon, May 31, 2010 at 19:02, Carl Meyer <carl@oddbird.net> wrote:
Nope, pip's used --record on installation for years, and the above has been true since the moment uninstall landed in pip. There are enough different ways things can get installed that it's not surprising that some discussions may have been confused ;-)
That may be it. Forcing --record in Python 3.2 would be a step forward then? :-)
You mean in the current distutils ? Because distutils2 will have the PEP 376 implementation, where we create a RECORD file for each installed project in its dist-info/
I haven't carefully read the entire discussion, but do you mean that distutils won't follow PEP 376 during installation? Ronald
On Thu, Jun 3, 2010 at 9:15 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 31 May, 2010, at 19:10, Tarek Ziadé wrote:
On Mon, May 31, 2010 at 7:04 PM, Lennart Regebro <regebro@gmail.com> wrote:
On Mon, May 31, 2010 at 19:02, Carl Meyer <carl@oddbird.net> wrote:
Nope, pip's used --record on installation for years, and the above has been true since the moment uninstall landed in pip. There are enough different ways things can get installed that it's not surprising that some discussions may have been confused ;-)
That may be it. Forcing --record in Python 3.2 would be a step forward then? :-)
You mean in the current distutils ? Because distutils2 will have the PEP 376 implementation, where we create a RECORD file for each installed project in its dist-info/
I haven't carefully read the entire discussion, but do you mean that distutils won't follow PEP 376 during installation?
If people use the old Distutils, (what I've called the current distutils), and trigger an installation via 'python setup.py install'. It'll use the existing code, so install it the 'old' way.
Ronald
-- Tarek Ziadé | http://ziade.org
On 3 Jun, 2010, at 9:50, Tarek Ziadé wrote:
On Thu, Jun 3, 2010 at 9:15 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 31 May, 2010, at 19:10, Tarek Ziadé wrote:
On Mon, May 31, 2010 at 7:04 PM, Lennart Regebro <regebro@gmail.com> wrote:
On Mon, May 31, 2010 at 19:02, Carl Meyer <carl@oddbird.net> wrote:
Nope, pip's used --record on installation for years, and the above has been true since the moment uninstall landed in pip. There are enough different ways things can get installed that it's not surprising that some discussions may have been confused ;-)
That may be it. Forcing --record in Python 3.2 would be a step forward then? :-)
You mean in the current distutils ? Because distutils2 will have the PEP 376 implementation, where we create a RECORD file for each installed project in its dist-info/
I haven't carefully read the entire discussion, but do you mean that distutils won't follow PEP 376 during installation?
If people use the old Distutils, (what I've called the current distutils), and trigger an installation via 'python setup.py install'. It'll use the existing code, so install it the 'old' way.
I know that distutils in python 2. 6 and 3.1 won't support PEP 376, but why won't distutils conform to PEP 376 in python 2.7 and 3.2? The RECORD file from PEP 376 would allow manually installed package (e.g. "python setup.py install") to be further managed by a PEP 376 compliant install tool. This should be pretty easy to add if it isn't in already, although 2.7rc1 is awfully close. Ronald
On Thu, Jun 3, 2010 at 10:02 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 3 Jun, 2010, at 9:50, Tarek Ziadé wrote:
On Thu, Jun 3, 2010 at 9:15 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 31 May, 2010, at 19:10, Tarek Ziadé wrote:
On Mon, May 31, 2010 at 7:04 PM, Lennart Regebro <regebro@gmail.com> wrote:
On Mon, May 31, 2010 at 19:02, Carl Meyer <carl@oddbird.net> wrote:
Nope, pip's used --record on installation for years, and the above has been true since the moment uninstall landed in pip. There are enough different ways things can get installed that it's not surprising that some discussions may have been confused ;-)
That may be it. Forcing --record in Python 3.2 would be a step forward then? :-)
You mean in the current distutils ? Because distutils2 will have the PEP 376 implementation, where we create a RECORD file for each installed project in its dist-info/
I haven't carefully read the entire discussion, but do you mean that distutils won't follow PEP 376 during installation?
If people use the old Distutils, (what I've called the current distutils), and trigger an installation via 'python setup.py install'. It'll use the existing code, so install it the 'old' way.
I know that distutils in python 2. 6 and 3.1 won't support PEP 376, but why won't distutils conform to PEP 376 in python 2.7 and 3.2? The RECORD file from PEP 376 would allow manually installed package (e.g. "python setup.py install") to be further managed by a PEP 376 compliant install tool. This should be pretty easy to add if it isn't in already, although 2.7rc1 is awfully close.
Are you thinking about a full implicit switch to PEP 376, or an optional behavior ? -- Tarek Ziadé | http://ziade.org
On 3 Jun, 2010, at 10:10, Tarek Ziadé wrote:
On Thu, Jun 3, 2010 at 10:02 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 3 Jun, 2010, at 9:50, Tarek Ziadé wrote:
On Thu, Jun 3, 2010 at 9:15 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 31 May, 2010, at 19:10, Tarek Ziadé wrote:
On Mon, May 31, 2010 at 7:04 PM, Lennart Regebro <regebro@gmail.com> wrote:
On Mon, May 31, 2010 at 19:02, Carl Meyer <carl@oddbird.net> wrote: > Nope, pip's used --record on installation for years, and the above has > been true since the moment uninstall landed in pip. There are enough > different ways things can get installed that it's not surprising that > some discussions may have been confused ;-)
That may be it. Forcing --record in Python 3.2 would be a step forward then? :-)
You mean in the current distutils ? Because distutils2 will have the PEP 376 implementation, where we create a RECORD file for each installed project in its dist-info/
I haven't carefully read the entire discussion, but do you mean that distutils won't follow PEP 376 during installation?
If people use the old Distutils, (what I've called the current distutils), and trigger an installation via 'python setup.py install'. It'll use the existing code, so install it the 'old' way.
I know that distutils in python 2. 6 and 3.1 won't support PEP 376, but why won't distutils conform to PEP 376 in python 2.7 and 3.2? The RECORD file from PEP 376 would allow manually installed package (e.g. "python setup.py install") to be further managed by a PEP 376 compliant install tool. This should be pretty easy to add if it isn't in already, although 2.7rc1 is awfully close.
Are you thinking about a full implicit switch to PEP 376, or an optional behavior ?
A full switch to PEP 376. An option to enable/disable would just be confusing, any option increases the mental load of developers (and anyone that just wants to install a python package to use its functionality) and in this case I don't think adding an option would be worth that. This would break current releases of tools that use the current "egg-info" information due to the different filesystem name (currently .egg-info, .dist-info in PEP 376), but those tools will have to be updated sometime anyway. Ronald
On Thu, Jun 3, 2010 at 10:24 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote: [..]
A full switch to PEP 376. An option to enable/disable would just be confusing, any option increases the mental load of developers (and anyone that just wants to install a python package to use its functionality) and in this case I don't think adding an option would be worth that.
This would break current releases of tools that use the current "egg-info" information due to the different filesystem name (currently .egg-info, .dist-info in PEP 376), but those tools will have to be updated sometime anyway.
I don't see what would be the advantage, since we are adding in pkgutil APIs (PEP 376 implementation) the ability to read old formats as well, to make it easier to query what's installed. Plus, distutils was marked "frozen" to avoid instability (any change in distutils potentially breaks tools out there, even internal changes). Regards Tarek
On 3 Jun, 2010, at 10:38, Tarek Ziadé wrote:
On Thu, Jun 3, 2010 at 10:24 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote: [..]
A full switch to PEP 376. An option to enable/disable would just be confusing, any option increases the mental load of developers (and anyone that just wants to install a python package to use its functionality) and in this case I don't think adding an option would be worth that.
This would break current releases of tools that use the current "egg-info" information due to the different filesystem name (currently .egg-info, .dist-info in PEP 376), but those tools will have to be updated sometime anyway.
I don't see what would be the advantage, since we are adding in pkgutil APIs (PEP 376 implementation) the ability to read old formats as well, to make it easier to query what's installed. Plus, distutils was marked "frozen" to avoid instability (any change in distutils potentially breaks tools out there, even internal changes).
The RECORD file in PEP 376 enables scripted removal of a distribution, which cannot be done without PEP 376 unless the user that installed the distribution was careful enough to use either the --record option or a tool like pip. I agree that the fragility of distutils is a problem, which would making changing it this far along the 2.7 release process rather dangerous. Ronald
On Thu, Jun 3, 2010 at 10:02, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
I know that distutils in python 2. 6 and 3.1 won't support PEP 376, but why won't distutils conform to PEP 376 in python 2.7 and 3.2?
Correct me if I'm wrong here, but there is a lot of PEPs regarding distutils, and it was simply decided to move all that work out from trunk into distutils2, as I understand it. Makes sense to me anyway. :) -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python3porting.com/ +33 661 58 14 64
On Sun, May 30, 2010 at 12:13 PM, Carl Meyer <carl@oddbird.net> wrote:
I sympathize with the ease-of-use reasons for having something in the stdlib. And I'm not sure putting pip in the stdlib would be good for pip's development.
This is my primary concern. I have no problem with merging pip with distutils2, so that pip can share utility code, etc. While it would still have to be backward compatible with old packages, I don't think that would have to change just because it merged with distutils2. But... the standard library is where I get stuck. And release cycles. pip has its own release cycle right now. It's not particularly formal, but it's a fairly regular release schedule. In terms of scope I don't think pip is finished. For instance, at least pip should have querying abilities (similar to what yolk does). Also the standard library's backward compatibility guarantee's don't seem as relevant to an installer -- an installer is something you run at a certain point in time, it's not part of the runtime of a system itself, and so changes are not as much of a problem. At the same time it has to be compatible with packages written a long time ago... but it's a different sort of compatibility. Well, I could list the problems, but basically I strongly dislike the idea that, say, Python 3.2 will ship distutils2 with a certain version of pip, and that people would have to upgrade Python or wait for a Python release to get upgrades of pip. Or, if that's not the case, then we need to figure what it means to have distutils2 "in" the standard library while still making regular releases. If Python shipped a "install distutils2" script then that I could work with, because it would imply distutils2 was blessed (and whatever the version at the time of release was, it could possibly be installed) but without attaching release cycles. -- Ian Bicking | http://blog.ianbicking.org
On Mon, May 31, 2010 at 4:55 PM, Ian Bicking <ianb@colorstudy.com> wrote: [..]
Well, I could list the problems, but basically I strongly dislike the idea that, say, Python 3.2 will ship distutils2 with a certain version of pip, and that people would have to upgrade Python or wait for a Python release to get upgrades of pip.
.. or allow people to configure in distutils2 an alternative installer. That is: a newer version of Pip. So they don't suffer from the Python release cycle by being able to upgrade the installer. I don't think being able to upgrade the installer from within the stdlib breaks the stdlib stability promise, since an installer is not something people use in their code as a library. It doesn't really suffer from API deprecation: it's a high-level end-user tool that people use to install stuff. It's interface (that is, the command line options I guess) can be quite stable. If people are using some packaging API in their code, I'd expect them to be in the Distutils2 'toolbox' part, which can benefit from the stdlib stability. Then Pip-the-installer could rely on it, and maybe improve those API by having its own version for a while until the next Python version is released. Python would ship with a version of the installer that can be upgraded by people that know what they do. Tarek -- Tarek Ziadé | http://ziade.org
On Mon, May 31, 2010 at 10:11 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
On Mon, May 31, 2010 at 4:55 PM, Ian Bicking <ianb@colorstudy.com> wrote: [..]
Well, I could list the problems, but basically I strongly dislike the idea that, say, Python 3.2 will ship distutils2 with a certain version of pip, and that people would have to upgrade Python or wait for a Python release to get upgrades of pip.
.. or allow people to configure in distutils2 an alternative installer. That is: a newer version of Pip. So they don't suffer from the Python release cycle by being able to upgrade the installer.
The biggest advantage I see of directly merging pip and distutils2 is that they could share code... but if you can upgrade pip without distutils2 then pip will have to work around any bugs in distutils2 to maintain compatibility with old versions of distutils2 (assuming it supports upgrading *only* pip). At which point they aren't "merged" at all, and pip is exactly where it is now. If distutils2 *and* pip were upgrading together consistently (i.e., they were actually merged), and if releases were essentially a superset of both projects' releases, then I'd have no problem with that. I think we can offer *some* backward compatibility guarantees for pip, like command-line interface (but not for any internal interfaces at this time, though maybe some in the future). In terms of release cycles I'm also somewhat uncomfortable with the extraction of the VCS support... it feels more like a political concession than a good technical decision. I don't want the brokenness of the stdlib process to drive choices in pip. I also don't think it should drive choices in distutils2, and I don't think it's healthy for distutils2, but it's not my project so I'll only offer that as advice. I strongly support the addition of a new category of library that is somewhere between the standard library and just-another-library, a library with some authority and support in the community, but with a separate release cycle entirely. There's a lot of libraries that should be like this, but it's not too painful because most of them are pretty much "done" -- they've had very few changes or additions (maybe for good reasons, maybe not). I think distutils/setuptools/distutils2/pip reveal the need for this category more than many packages because they are both important and need community approval, but are very much Not Done and I think will probably never be done (since they interact with so many moving parts some of which aren't part of Python at all). -- Ian Bicking | http://blog.ianbicking.org
On Tue, Jun 1, 2010 at 5:12 AM, Ian Bicking <ianb@colorstudy.com> wrote: [..]
In terms of release cycles I'm also somewhat uncomfortable with the extraction of the VCS support... it feels more like a political concession than a good technical decision. I don't want the brokenness of the stdlib process to drive choices in pip. I also don't think it should drive choices in distutils2, and I don't think it's healthy for distutils2, but it's not my project so I'll only offer that as advice.
Sounds like the stdlib is hell to you ;) More seriously, how do you define a package that is a good candidate for the stdlib in that case? If there's no good definition, doesn't it mean that there's a higher level problem ? Like, the length of the release cycle of the sdtlib itself ?
I strongly support the addition of a new category of library that is somewhere between the standard library and just-another-library, a library with some authority and support in the community, but with a separate release cycle entirely. There's a lot of libraries that should be like this, but it's not too painful because most of them are pretty much "done" -- they've had very few changes or additions (maybe for good reasons, maybe not). I think distutils/setuptools/distutils2/pip reveal the need for this category more than many packages because they are both important and need community approval, but are very much Not Done and I think will probably never be done (since they interact with so many moving parts some of which aren't part of Python at all).
I am not sure what this would mean for the end-user (a fat release of Python ?) and maybe Pip belongs to that category. For Distutils2 though, most part of it will not change a lot within a Python release, (beside bugfix of course). For example, modules like "version", or "metadata" won't move too much. I guess we will work on a lighter script on our side then. Regards Tarek -- Tarek Ziadé | http://ziade.org
On Sat, May 29, 2010 at 6:32 PM, P.J. Eby <pje@telecommunity.com> wrote: [..]
So? It's not like they're going to accidentally run 'easy_install' when they meant to type 'pip', or vice versa. ;-)
They will accidentally run easy_install through a simple "python setup.py install" call if the given project bootstraps ez_setup and uses setuptools' easy_install command. The end-user is not asked in the process you describe, what installer he wants to use. It's implicitly done. That's the point I am trying to express: it's "implicit, embed, installers in project's setup.py" vs "an installer globally installed, knowing how to install projects that follows a given standard" Some project with ez_setup.py embed will get installed using setuptools own standard, and will also install setuptools of course in the end-user Python system. PEP 345 adds what setuptools had in its own standard on the top of distutils metadata, and one of the goal now is to be able to rely on this new standard without having to embed your own installer and hook it in your setup.py. I mean, why would we have different metadata standards ? with should we have different installers that are not interoperable ? why should people need to add some code in their setup.py to make sure their stuff is correctly installed ?
It also implies that the developer provides a smart setup.py script that embeds that bootstrap, and runs some code.
Only if they need something beyond what the distutils provide.
So what will happen when people uses the new metadata 1.2 ? (PEP 345) I know I don't want to ask the developer to add a special bootstrap in their project just because they used PEP 345 fields. That's the PEP 345-compatible installers job.
It will also allow distributions to be "dumb" envelopes with static metadata that are the same all the time, no matter which tool created them, and eventually remove setup.py in favor of statically described metadata using PEP 345.
For packages with complex build requirements or distutils extensions (e.g. numpy), this is unlikely to happen any time soon.
Conversely, for packages where this *is* the case, the current distutils is adequate, and having a bootstrapper that can install them would be a win.
I don't understand where the win is in this case. For me there are two distinct things: - python projects, following some standards and conventions - installers, that know how to install projects that follow some standards and conventions
Today, for instance, if an installer wants to install a distribution based on setuptools, it has to run the "egg_info" command to extract the metadata, on the target system.
It's a good thing Guido loaned me the time machine so I could go back to 2005 and fix that one, then:
Five years should be long enough ago that by now all the setuptools-based source distributions on PyPI today will already have the metadata you need... so when you go back and try again to build your new installer today, it will already be there for you to use. Just download any random setuptools-based sdist from PyPI and run "tar -tzf" or "unzip -v" on it to see!
This egg-info is useless AFAIK. The egg_info command *has* to be run again because it can change depending on the running platform and the code in setup.py. The one in the source distribution is not guaranteed to be accurate. Moreover, you can't get them at PyPI unless you download the tarball. PEP 345 is different in that aspect because environment-specific metadata fields can be described. And, you can query those metadata at PyPI. As a matter of fact it's already the case: PyPI is now Metadata 1.2 compatible. You can do XML-RPC queries at PyPI to get the metadata 1.2 fields (so the dependencies) for the Distutils2 project. Regards Tarek -- Tarek Ziadé | http://ziade.org
On May 29, 2010, at 09:12 PM, Tarek Ziadé wrote:
That's the point I am trying to express: it's "implicit, embed, installers in project's setup.py" vs "an installer globally installed, knowing how to install projects that follows a given standard"
I feel pretty firmly that the choice of installer should /not/ be left up to the project's setup.py. That's within the domain of the user, or by proxy, their operating system. -Barry
Barry Warsaw <barry@python.org> writes:
On May 29, 2010, at 09:12 PM, Tarek Ziadé wrote:
That's the point I am trying to express: it's "implicit, embed, installers in project's setup.py" vs "an installer globally installed, knowing how to install projects that follows a given standard"
I feel pretty firmly that the choice of installer should /not/ be left up to the project's setup.py. That's within the domain of the user, or by proxy, their operating system.
That is the ideal, yes. We need to get to the point where the packaging data stops being an executable program, and is instead purely declarative metadata with a well-defined format and semantic speification. -- \ “Software patents provide one more means of controlling access | `\ to information. They are the tool of choice for the internet | _o__) highwayman.” —Anthony Taylor | Ben Finney
On Jun 04, 2010, at 01:10 AM, Ben Finney wrote:
Barry Warsaw <barry@python.org> writes:
On May 29, 2010, at 09:12 PM, Tarek Ziadé wrote:
That's the point I am trying to express: it's "implicit, embed, installers in project's setup.py" vs "an installer globally installed, knowing how to install projects that follows a given standard"
I feel pretty firmly that the choice of installer should /not/ be left up to the project's setup.py. That's within the domain of the user, or by proxy, their operating system.
That is the ideal, yes. We need to get to the point where the packaging data stops being an executable program, and is instead purely declarative metadata with a well-defined format and semantic speification.
As Tarek knows I'd say: +1 :) -Barry
Hi [Tarek]
I was not thinking about this proposal. If this what Guido proposed at the summit, then I misunderstood. [PJE] I don't know what he proposed at the summit - I'm referring to the bootstrap script he wrote that actually does this. It was a couple years ago on Python-Dev, IIRC.
Could be this: http://mail.python.org/pipermail/python-dev/2008-March/077837.html Cheers
participants (9)
-
Barry Warsaw -
Ben Finney -
Carl Meyer -
Ian Bicking -
Lennart Regebro -
P.J. Eby -
Ronald Oussoren -
Tarek Ziadé -
Éric Araujo