From g2p.code at gmail.com Mon Jul 1 00:39:33 2013 From: g2p.code at gmail.com (Gabriel de Perthuis) Date: Sun, 30 Jun 2013 22:39:33 +0000 (UTC) Subject: [Distutils] Upcoming changes to PEP 426/440 References: Message-ID: On Sun, 30 Jun 2013 21:21:54 +1000, Nick Coghlan wrote: > On 30 June 2013 18:53, Vinay Sajip wrote: >> Nick Coghlan gmail.com> writes: >> >>> No, because the semantic dependencies form a Cartesian product with >>> the extras. You can define :meta:, :run:, :test:, :build: and :dev: >>> dependencies for any extra. So if, for example, you have a "localweb" >>> extra, then you can define extra test dependencies for that. >>> >>> The semantic specifiers determine *which sets of dependencies* you're >>> interested in, while the explicit extras define optional subsets of >>> those. >> >> Isn't that the same as having an additional field in the dependency mapping? >> It seems like that's how one would organise it at the RDBMS level, anyway. >> >> { >> "install": "localweb-test-util [win] (>= 1.0)", >> "extra": "localweb", >> "environment": "sys_platform == 'win32'", >> "kind": ":test:" >> } > > You certainly *could* define it like that, but no existing dependency > system I'm aware of does it that way. If they allow for anything other > than runtime dependencies in the first place, they define a different > top level field: > > * setuptools has requires and install_requires > * PEP 346 has Requires-Dist and Setup-Requires-Dist > * RPM has Requires and BuildRequires > * npm has dependencies and devDependencies At least for Debian, and probably RPM, source dependencies have a different field name because they are carried by a source package rather than a binary one. The nature of the dependencies isn't different, the required packages are binary in both cases. The cartesian product might be overkill. If someone elects to install development dependencies I don't see a point in picking and choosing. There's enough support noise when people fail to build from source, and while an author is knowledgeable and might conceive more than one way to set things up, publishing them would cause more trouble than it's worth. So it would prefer that dev and test be extras with well known names, so that dev, test, and any other extras define dependencies with a minimum of ambiguity and without the need for a second level of qualifiers. From donald at stufft.io Mon Jul 1 00:46:51 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Jun 2013 18:46:51 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: Message-ID: <3EB91E50-ECCD-4397-A25A-2D4300BE7F93@stufft.io> On Jun 30, 2013, at 6:39 PM, Gabriel de Perthuis wrote: > So it would prefer that dev and test be extras with well known names, > so that dev, test, and any other extras define dependencies with a > minimum of ambiguity and without the need for a second level of > qualifiers. "Well known names" is way more ambiguous than a top level field. It's easy to have minor variances across various packages, "test" vs "tests", "docs", "doc", "documentation". Both top level and "kind" share the fact that there is a limited number of allowed names, which makes it simple to validate that the same name is being used everywhere (because anything outside of those limited numbers are rejected). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From g2p.code at gmail.com Mon Jul 1 00:51:24 2013 From: g2p.code at gmail.com (Gabriel de Perthuis) Date: Sun, 30 Jun 2013 22:51:24 +0000 (UTC) Subject: [Distutils] Upcoming changes to PEP 426/440 References: <3EB91E50-ECCD-4397-A25A-2D4300BE7F93@stufft.io> Message-ID: On Sun, 30 Jun 2013 18:46:51 -0400, Donald Stufft wrote: > On Jun 30, 2013, at 6:39 PM, Gabriel de Perthuis wrote: > >> So it would prefer that dev and test be extras with well known names, >> so that dev, test, and any other extras define dependencies with a >> minimum of ambiguity and without the need for a second level of >> qualifiers. > > "Well known names" is way more ambiguous than a top level field. > It's easy to have minor variances across various packages, "test" vs > "tests", "docs", "doc", "documentation". Both top level and "kind" > share the fact that there is a limited number of allowed names, > which makes it simple to validate that the same name is being used > everywhere (because anything outside of those limited numbers are > rejected). These well-known names would also have some tool support. Something like `pip install-dev` would be sufficient. From donald at stufft.io Mon Jul 1 00:52:46 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Jun 2013 18:52:46 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: <3EB91E50-ECCD-4397-A25A-2D4300BE7F93@stufft.io> Message-ID: On Jun 30, 2013, at 6:51 PM, Gabriel de Perthuis wrote: > On Sun, 30 Jun 2013 18:46:51 -0400, Donald Stufft wrote: >> On Jun 30, 2013, at 6:39 PM, Gabriel de Perthuis wrote: >> >>> So it would prefer that dev and test be extras with well known names, >>> so that dev, test, and any other extras define dependencies with a >>> minimum of ambiguity and without the need for a second level of >>> qualifiers. >> >> "Well known names" is way more ambiguous than a top level field. >> It's easy to have minor variances across various packages, "test" vs >> "tests", "docs", "doc", "documentation". Both top level and "kind" >> share the fact that there is a limited number of allowed names, >> which makes it simple to validate that the same name is being used >> everywhere (because anything outside of those limited numbers are >> rejected). > > These well-known names would also have some tool support. > Something like `pip install-dev` would be sufficient. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig But when defining them, it's very easy to accidentally use "tests" instead of "test". ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From g2p.code at gmail.com Mon Jul 1 00:58:15 2013 From: g2p.code at gmail.com (Gabriel de Perthuis) Date: Sun, 30 Jun 2013 22:58:15 +0000 (UTC) Subject: [Distutils] Upcoming changes to PEP 426/440 References: <3EB91E50-ECCD-4397-A25A-2D4300BE7F93@stufft.io> Message-ID: On Sun, 30 Jun 2013 18:52:46 -0400, Donald Stufft wrote: > On Jun 30, 2013, at 6:51 PM, Gabriel de Perthuis wrote: >> On Sun, 30 Jun 2013 18:46:51 -0400, Donald Stufft wrote: >>> On Jun 30, 2013, at 6:39 PM, Gabriel de Perthuis wrote: >>> >>>> So it would prefer that dev and test be extras with well known names, >>>> so that dev, test, and any other extras define dependencies with a >>>> minimum of ambiguity and without the need for a second level of >>>> qualifiers. >>> >>> "Well known names" is way more ambiguous than a top level field. >>> It's easy to have minor variances across various packages, "test" vs >>> "tests", "docs", "doc", "documentation". Both top level and "kind" >>> share the fact that there is a limited number of allowed names, >>> which makes it simple to validate that the same name is being used >>> everywhere (because anything outside of those limited numbers are >>> rejected). >> >> These well-known names would also have some tool support. >> Something like `pip install-dev` would be sufficient. >> > But when defining them, it's very easy to accidentally use "tests" > instead of "test". A lint tool can warn about these names, and a PyPI server could even block them for new-style packages. From donald at stufft.io Mon Jul 1 01:00:31 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Jun 2013 19:00:31 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: <3EB91E50-ECCD-4397-A25A-2D4300BE7F93@stufft.io> Message-ID: <531E0B11-62E9-4806-ADFB-6F84BDA1DC99@stufft.io> On Jun 30, 2013, at 6:58 PM, Gabriel de Perthuis wrote: > On Sun, 30 Jun 2013 18:52:46 -0400, Donald Stufft wrote: >> On Jun 30, 2013, at 6:51 PM, Gabriel de Perthuis wrote: >>> On Sun, 30 Jun 2013 18:46:51 -0400, Donald Stufft wrote: >>>> On Jun 30, 2013, at 6:39 PM, Gabriel de Perthuis wrote: >>>> >>>>> So it would prefer that dev and test be extras with well known names, >>>>> so that dev, test, and any other extras define dependencies with a >>>>> minimum of ambiguity and without the need for a second level of >>>>> qualifiers. >>>> >>>> "Well known names" is way more ambiguous than a top level field. >>>> It's easy to have minor variances across various packages, "test" vs >>>> "tests", "docs", "doc", "documentation". Both top level and "kind" >>>> share the fact that there is a limited number of allowed names, >>>> which makes it simple to validate that the same name is being used >>>> everywhere (because anything outside of those limited numbers are >>>> rejected). >>> >>> These well-known names would also have some tool support. >>> Something like `pip install-dev` would be sufficient. >>> >> But when defining them, it's very easy to accidentally use "tests" >> instead of "test". > > A lint tool can warn about these names, and a PyPI server could even > block them for new-style packages. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Or use a separate field (either the name, or the aforementioned "kind" field) and remove all ambiguity from the concept and remove the need to have a lint tool or guess what the person might mean. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Mon Jul 1 01:04:58 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 1 Jul 2013 09:04:58 +1000 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: Message-ID: On 1 Jul 2013 08:40, "Gabriel de Perthuis" wrote: > > On Sun, 30 Jun 2013 21:21:54 +1000, Nick Coghlan wrote: > > On 30 June 2013 18:53, Vinay Sajip wrote: > >> Nick Coghlan gmail.com> writes: > >> > >>> No, because the semantic dependencies form a Cartesian product with > >>> the extras. You can define :meta:, :run:, :test:, :build: and :dev: > >>> dependencies for any extra. So if, for example, you have a "localweb" > >>> extra, then you can define extra test dependencies for that. > >>> > >>> The semantic specifiers determine *which sets of dependencies* you're > >>> interested in, while the explicit extras define optional subsets of > >>> those. > >> > >> Isn't that the same as having an additional field in the dependency mapping? > >> It seems like that's how one would organise it at the RDBMS level, anyway. > >> > >> { > >> "install": "localweb-test-util [win] (>= 1.0)", > >> "extra": "localweb", > >> "environment": "sys_platform == 'win32'", > >> "kind": ":test:" > >> } > > > > You certainly *could* define it like that, but no existing dependency > > system I'm aware of does it that way. If they allow for anything other > > than runtime dependencies in the first place, they define a different > > top level field: > > > > * setuptools has requires and install_requires > > * PEP 346 has Requires-Dist and Setup-Requires-Dist > > * RPM has Requires and BuildRequires > > * npm has dependencies and devDependencies > > At least for Debian, and probably RPM, source dependencies have a > different field name because they are carried by a source package > rather than a binary one. The nature of the dependencies isn't > different, the required packages are binary in both cases. > > The cartesian product might be overkill. If someone elects to install > development dependencies I don't see a point in picking and choosing. > There's enough support noise when people fail to build from source, > and while an author is knowledgeable and might conceive more than one > way to set things up, publishing them would cause more trouble than > it's worth. I've had to port stuff to build on s390s - it would have made my life much easier if the dependencies that were only needed for optional x86_64 specific C accelerators had been clearly marked, rather than my having to weed them out through trial and error. What you're talking about is a rationale for sensible defaults and helper commands in tools (and the PEP does go into that), but it's not a good reason to limit the expressiveness of the format itself. > > So it would prefer that dev and test be extras with well known names, > so that dev, test, an any other extras define dependencies with a > minimum of ambiguity and without the need for a second level of > qualifiers. How would you express an optional dependency on Cython for optional C accelerators in such a system? The PEP is as it is because I think the payoff in expressiveness is worth the increase in complexity. Saying "You shouldn't want to describe such situations clearly and succinctly" is not a compelling argument. Cheers, Nick. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Jul 1 15:11:47 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 1 Jul 2013 09:11:47 -0400 Subject: [Distutils] PyPI Hosting Mode Migration Message-ID: <787B20BB-1369-4CCE-974D-AE7715FC5282@stufft.io> Hello! I've just migrated PyPI to mostly no external url hosting modes as per PEP438. This should result in a huge speedup when installing a wide number of packages. Some stats: 18,753 packages migrated to pypi-explicit (First option) 1,022 packages migrated to pypi-scrape (Second option) 25,759 total packages using pypi-explicit 1,046 total packages using pypi-scrape 5,384 total packages using pypi-scrape-crawl The scripts that discovered what version a project could be are located at https://github.com/dstufft/pypi.linkcheck/tree/migration. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon Jul 1 15:14:57 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 1 Jul 2013 09:14:57 -0400 Subject: [Distutils] PyPI Hosting Mode Migration In-Reply-To: <787B20BB-1369-4CCE-974D-AE7715FC5282@stufft.io> References: <787B20BB-1369-4CCE-974D-AE7715FC5282@stufft.io> Message-ID: On Jul 1, 2013, at 9:11 AM, Donald Stufft wrote: > Hello! > > I've just migrated PyPI to mostly no external url hosting modes as per PEP438. This should result in a huge speedup when installing a wide number of packages. > > Some stats: > > 18,753 packages migrated to pypi-explicit (First option) > 1,022 packages migrated to pypi-scrape (Second option) > > 25,759 total packages using pypi-explicit > 1,046 total packages using pypi-scrape > 5,384 total packages using pypi-scrape-crawl > > The scripts that discovered what version a project could be are located at https://github.com/dstufft/pypi.linkcheck/tree/migration. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig s/version a project/mode a project/ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ethan at stoneleaf.us Mon Jul 1 20:19:01 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 01 Jul 2013 11:19:01 -0700 Subject: [Distutils] PyPI upload: zip okay, tar fails with "invalid distribution file" Message-ID: <51D1C815.6030303@stoneleaf.us> I migrated my split py2/py3 code base into one as I couldn't get setup to do what I wanted, and now I am having this problem. Here's my setup.py: ======================================================================== import sys, os from distutils.core import setup long_desc = open('enum/doc/enum.rst').read() setup( name='enum34', version='0.9.1', url='https://pypi.python.org/pypi/enum34', packages=['enum'], package_data={ 'enum' : [ 'LICENSE', 'README', 'doc/enum.rst', 'doc/enum.pdf', 'test/test_enum.py', ] }, license='BSD License', description='Python 3.4 Enum backported to 3.3, 3.2, 3.1, 2.7, 2.6, 2.5, and 2.4', long_description=long_desc, provides=['enum'], author='Ethan Furman', author_email='ethan at stoneleaf.us', classifiers=[ 'Development Status :: 5 - Production/Stable', 'Intended Audience :: Developers', 'License :: OSI Approved :: BSD License', 'Programming Language :: Python', 'Topic :: Software Development' ], ) ======================================================================== and an example run: ======================================================================== ethan at media:~/source/enum$ python setup.py sdist --formats tar upload running sdist running check reading manifest template 'MANIFEST.in' writing manifest file 'MANIFEST' creating enum34-0.9.1 creating enum34-0.9.1/enum creating enum34-0.9.1/enum/doc creating enum34-0.9.1/enum/test making hard links in enum34-0.9.1... hard linking LICENSE -> enum34-0.9.1 hard linking README -> enum34-0.9.1 hard linking setup.py -> enum34-0.9.1 hard linking enum/__init__.py -> enum34-0.9.1/enum hard linking enum/doc/enum.pdf -> enum34-0.9.1/enum/doc hard linking enum/doc/enum.rst -> enum34-0.9.1/enum/doc hard linking enum/test/test_enum.py -> enum34-0.9.1/enum/test Creating tar archive removing 'enum34-0.9.1' (and everything under it) running upload Submitting dist/enum34-0.9.1.tar to http://pypi.python.org/pypi Upload failed (400): invalid distribution file ======================================================================== Any ideas? -- ~Ethan~ From p.f.moore at gmail.com Mon Jul 1 21:11:26 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 1 Jul 2013 20:11:26 +0100 Subject: [Distutils] PyPI upload: zip okay, tar fails with "invalid distribution file" In-Reply-To: <51D1C815.6030303@stoneleaf.us> References: <51D1C815.6030303@stoneleaf.us> Message-ID: On 1 July 2013 19:19, Ethan Furman wrote: > ethan at media:~/source/enum$ python setup.py sdist --formats tar upload > running sdist > running check > reading manifest template 'MANIFEST.in' > writing manifest file 'MANIFEST' > creating enum34-0.9.1 > creating enum34-0.9.1/enum > creating enum34-0.9.1/enum/doc > creating enum34-0.9.1/enum/test > making hard links in enum34-0.9.1... > hard linking LICENSE -> enum34-0.9.1 > hard linking README -> enum34-0.9.1 > hard linking setup.py -> enum34-0.9.1 > hard linking enum/__init__.py -> enum34-0.9.1/enum > hard linking enum/doc/enum.pdf -> enum34-0.9.1/enum/doc > hard linking enum/doc/enum.rst -> enum34-0.9.1/enum/doc > hard linking enum/test/test_enum.py -> enum34-0.9.1/enum/test > Creating tar archive > removing 'enum34-0.9.1' (and everything under it) > running upload > Submitting dist/enum34-0.9.1.tar to http://pypi.python.org/pypi > Upload failed (400): invalid distribution file > ==============================**==============================** > ============ > > Any ideas? > You probably want format gztar rather than tar. I don't think I've ever seen an uncompressed tar on PyPI - they probably aren't allowed... Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Jul 1 21:12:51 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 01 Jul 2013 12:12:51 -0700 Subject: [Distutils] PyPI upload: zip okay, tar fails with "invalid distribution file" In-Reply-To: References: <51D1C815.6030303@stoneleaf.us> Message-ID: <51D1D4B3.2090901@stoneleaf.us> On 07/01/2013 12:11 PM, Paul Moore wrote: > On 1 July 2013 19:19, Ethan Furman > wrote: > > ethan at media:~/source/enum$ python setup.py sdist --formats tar upload > running sdist > running check > reading manifest template 'MANIFEST.in' > writing manifest file 'MANIFEST' > creating enum34-0.9.1 > creating enum34-0.9.1/enum > creating enum34-0.9.1/enum/doc > creating enum34-0.9.1/enum/test > making hard links in enum34-0.9.1... > hard linking LICENSE -> enum34-0.9.1 > hard linking README -> enum34-0.9.1 > hard linking setup.py -> enum34-0.9.1 > hard linking enum/__init__.py -> enum34-0.9.1/enum > hard linking enum/doc/enum.pdf -> enum34-0.9.1/enum/doc > hard linking enum/doc/enum.rst -> enum34-0.9.1/enum/doc > hard linking enum/test/test_enum.py -> enum34-0.9.1/enum/test > Creating tar archive > removing 'enum34-0.9.1' (and everything under it) > running upload > Submitting dist/enum34-0.9.1.tar to http://pypi.python.org/pypi > Upload failed (400): invalid distribution file > ==============================__==============================__============ > > Any ideas? > > > You probably want format gztar rather than tar. I don't think I've ever seen an uncompressed tar on PyPI - they probably > aren't allowed... I could've sworn I've used tar before, but at any rate going with gztar did the trick. Thanks! -- ~Ethan~ From iwan at reahl.org Tue Jul 2 12:59:52 2013 From: iwan at reahl.org (Iwan Vosloo) Date: Tue, 02 Jul 2013 12:59:52 +0200 Subject: [Distutils] Finding modules in an egg / distribution Message-ID: <51D2B2A8.5060404@reahl.org> Hi there, We have been struggling to find a nice way to list all the modules (or packages) that are part of a particular Distribution (or egg). Nice should also mean that it works when the egg is installed. We have a need to do some introspection on the code shipped as an egg. Any ideas? Regards - Iwan From m.van.rees at zestsoftware.nl Tue Jul 2 13:32:58 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Tue, 02 Jul 2013 13:32:58 +0200 Subject: [Distutils] PyPI upload: zip okay, tar fails with "invalid distribution file" In-Reply-To: <51D1D4B3.2090901@stoneleaf.us> References: <51D1C815.6030303@stoneleaf.us> <51D1D4B3.2090901@stoneleaf.us> Message-ID: Op 01-07-13 21:12, Ethan Furman schreef: > On 07/01/2013 12:11 PM, Paul Moore wrote: >> You probably want format gztar rather than tar. I don't think I've >> ever seen an uncompressed tar on PyPI - they probably >> aren't allowed... > > I could've sworn I've used tar before, but at any rate going with gztar > did the trick. I always use --formats=zip, because in some corner cases Python2.4 has problems with the gztar format, not while creating a distribution file but when installing it. -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From davbo at davbo.org Tue Jul 2 11:33:47 2013 From: davbo at davbo.org (David King) Date: Tue, 02 Jul 2013 10:33:47 +0100 Subject: [Distutils] PyPI mirrors Message-ID: <1372757627.28766.140661250871401.45F5CCD5@webmail.messagingengine.com> Hi all, Has the relationship between PyPI mirrors changed since PyPI has started being served behind a CDN? I know people have been recommending against using "--use-mirrors" with pip since it doesn't take advantage of the CDN. I've been considering trying to get a public PyPI mirror setup and wanted to know how/if they're still being used. Thanks, Dave, From wking at tremily.us Tue Jul 2 13:47:32 2013 From: wking at tremily.us (W. Trevor King) Date: Tue, 02 Jul 2013 07:47:32 -0400 Subject: [Distutils] PyPI upload failed (302): Moved Temporarily Message-ID: <20130702114732.GC8649@odin.tremily.us> Hello list! I'm having a bit of trouble getting a new package setup on PyPI. I've done this a few times in the past, but maybe not since the wiki hacking that inspired the HTTPS transition back in February. I did change my password back then, so I don't think that's the problem. Anyhow, with the old URL (still mentioned in the distutils docs [1]): http://www.python.org/pypi in my ~/.pypirc, I get: $ python setup.py register -r pypi running register running check Registering pycalendar to http://www.python.org/pypi Server response (403): You are not allowed to store 'pycalendar' package information With https://pypi.python.org/ in my ~/.pypirc, I get: $ python setup.py register -r pypi running register running check Registering pycalendar to https://pypi.python.org/ Server response (200): OK Ok, how about uploading a tarball? With the https:// URL: $ python setup.py sdist upload -r pypi ? running upload Submitting dist/pycalendar-0.1.tar.gz to https://pypi.python.org/ Upload failed (302): Moved Temporarily With the http:// URL: $ python setup.py sdist upload -r pypi ? running upload Submitting dist/pycalendar-0.1.tar.gz to http://www.python.org/pypi Upload failed (403): You are not allowed to edit 'pycalendar' package information What am I doing wrong? Thanks, Trevor [1]: http://docs.python.org/3/distutils/packageindex.html#the-pypirc-file -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From pje at telecommunity.com Tue Jul 2 17:08:13 2013 From: pje at telecommunity.com (PJ Eby) Date: Tue, 2 Jul 2013 11:08:13 -0400 Subject: [Distutils] Finding modules in an egg / distribution In-Reply-To: <51D2B2A8.5060404@reahl.org> References: <51D2B2A8.5060404@reahl.org> Message-ID: On Tue, Jul 2, 2013 at 6:59 AM, Iwan Vosloo wrote: > Hi there, > > We have been struggling to find a nice way to list all the modules (or > packages) that are part of a particular Distribution (or egg). Nice should > also mean that it works when the egg is installed. We have a need to do some > introspection on the code shipped as an egg. > > Any ideas? If you are targeting at least Python 2.5, see: http://docs.python.org/2/library/pkgutil.html#pkgutil.walk_packages From noah at coderanger.net Tue Jul 2 19:14:42 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Tue, 2 Jul 2013 10:14:42 -0700 Subject: [Distutils] PyPI mirrors In-Reply-To: <1372757627.28766.140661250871401.45F5CCD5@webmail.messagingengine.com> References: <1372757627.28766.140661250871401.45F5CCD5@webmail.messagingengine.com> Message-ID: On Jul 2, 2013, at 2:33 AM, David King wrote: > Hi all, > > Has the relationship between PyPI mirrors changed since PyPI has started > being served behind a CDN? > > I know people have been recommending against using "--use-mirrors" with > pip since it doesn't take advantage of the CDN. I've been considering > trying to get a public PyPI mirror setup and wanted to know how/if > they're still being used. Yes, the use of public mirrors is no longer recommended as a best practice. The idea is that mirrors will continue to be an important part of the ecosystem for things like deploy caching, internal company mirrors, etc, but the federated, public mirror network concept is being retired. Several of the public mirrors have already shut down and just point back at PyPI, but others are still available if you want to use them. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From qwcode at gmail.com Tue Jul 2 19:25:41 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 2 Jul 2013 10:25:41 -0700 Subject: [Distutils] "Python Packaging User Guide" Message-ID: Everyone: Soon after pycon, the "The Hitchhiker?s Guide to Packaging" (HHGTP) was forked into the "Python Packaging User Guide". src: https://bitbucket.org/pypa/python-packaging-user-guide built: https://python-packaging-user-guide.readthedocs.org/en/latest/ Here's the original discussion on distutils-sig: http://mail.python.org/pipermail/distutils-sig/2013-March/020215.html It's a PyPA project, and the goal is for it to become the de facto place for tutorials and where to find the current state of PEPs and development efforts. The idea is that once it's worthy, the python core docs would re-link from the HHGTP to the new guide, but it **needs work.** The TOC is sound and some of the sections are OK, but the two main tutorials are sorely lacking. I intend to keep working on it when I can, but help is appreciated. The project is open for issue logging and pull requests. Thanks, Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From iwan at reahl.org Tue Jul 2 19:30:35 2013 From: iwan at reahl.org (Iwan Vosloo) Date: Tue, 02 Jul 2013 19:30:35 +0200 Subject: [Distutils] Finding modules in an egg / distribution In-Reply-To: References: <51D2B2A8.5060404@reahl.org> Message-ID: <51D30E3B.4060608@reahl.org> On 02/07/2013 17:08, PJ Eby wrote: > If you are targeting at least Python 2.5, see: > http://docs.python.org/2/library/pkgutil.html#pkgutil.walk_packages We're targeting Python 2.7. Trouble is that pkgutil.walk_packages needs a path to search from. Distribution.location is always your site-packages directory once a Distribution is installed, so walking that just gives ALL installed packages. If your Distribution contains some sort of main package that contains everything in it, you can use that package's .__path__, but then you'd need to discover what that package is. Distributions could also contain more than one package next to each other, and top-level modules. (The __path__ of a top-level module is also simply the site-packages directory.) There is a metadata file top_level.txt which one could use to get the names of top-level packages/modules in the Distribution. This can however contain a namespace package too - and you don't want all the packages inside the namespace package - just the bits inside the chosen Distribution... Regards - Iwan From merwok at netwok.org Tue Jul 2 19:37:07 2013 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 02 Jul 2013 13:37:07 -0400 Subject: [Distutils] Finding modules in an egg / distribution In-Reply-To: <51D30E3B.4060608@reahl.org> References: <51D2B2A8.5060404@reahl.org> <51D30E3B.4060608@reahl.org> Message-ID: <51D30FC3.7010702@netwok.org> Hello Iwan, This project can answer your questions: https://pypi.python.org/pypi/pkginfo FTR we planned to include something like it when the distutils2 project was active; distlib or the variant of distlib that will end up in the 3.4 standard library could include similar features. Regards From donald at stufft.io Tue Jul 2 19:52:15 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 2 Jul 2013 13:52:15 -0400 Subject: [Distutils] PyPI mirrors In-Reply-To: References: <1372757627.28766.140661250871401.45F5CCD5@webmail.messagingengine.com> Message-ID: <60EA19B6-3F4C-4CB9-B0AB-85B2F3853B53@stufft.io> Of course as I look it appears all if the official mirrors are out if date again. On Jul 2, 2013, at 1:14 PM, Noah Kantrowitz wrote: > > On Jul 2, 2013, at 2:33 AM, David King wrote: > >> Hi all, >> >> Has the relationship between PyPI mirrors changed since PyPI has started >> being served behind a CDN? >> >> I know people have been recommending against using "--use-mirrors" with >> pip since it doesn't take advantage of the CDN. I've been considering >> trying to get a public PyPI mirror setup and wanted to know how/if >> they're still being used. > > Yes, the use of public mirrors is no longer recommended as a best practice. The idea is that mirrors will continue to be an important part of the ecosystem for things like deploy caching, internal company mirrors, etc, but the federated, public mirror network concept is being retired. Several of the public mirrors have already shut down and just point back at PyPI, but others are still available if you want to use them. > > --Noah > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From pje at telecommunity.com Tue Jul 2 20:27:43 2013 From: pje at telecommunity.com (PJ Eby) Date: Tue, 2 Jul 2013 14:27:43 -0400 Subject: [Distutils] Finding modules in an egg / distribution In-Reply-To: <51D30E3B.4060608@reahl.org> References: <51D2B2A8.5060404@reahl.org> <51D30E3B.4060608@reahl.org> Message-ID: On Tue, Jul 2, 2013 at 1:30 PM, Iwan Vosloo wrote: > On 02/07/2013 17:08, PJ Eby wrote: >> >> If you are targeting at least Python 2.5, see: >> http://docs.python.org/2/library/pkgutil.html#pkgutil.walk_packages > > > We're targeting Python 2.7. > > Trouble is that pkgutil.walk_packages needs a path to search from. > Distribution.location is always your site-packages directory once a > Distribution is installed, so walking that just gives ALL installed > packages. If your Distribution contains some sort of main package that > contains everything in it, you can use that package's .__path__, but then > you'd need to discover what that package is. > > Distributions could also contain more than one package next to each other, > and top-level modules. (The __path__ of a top-level module is also simply > the site-packages directory.) > > There is a metadata file top_level.txt which one could use to get the names > of top-level packages/modules in the Distribution. This can however contain > a namespace package too - and you don't want all the packages inside the > namespace package - just the bits inside the chosen Distribution... Ah, well in that case you'll have to inspect either .egg-info/SOURCES.txt or the PEP 376 installation manifest. I don't know of any reliable way to do what you want for system-installed packages at the moment. From alexjeffburke at gmail.com Wed Jul 3 01:18:16 2013 From: alexjeffburke at gmail.com (Alex Burke) Date: Wed, 3 Jul 2013 01:18:16 +0200 Subject: [Distutils] "Python Packaging User Guide" Message-ID: Hi Marcus, I would very much like to get involved and help with this effort - in fact I was looking to package up some of my own work recently and stumbled on The Hitchhiker?s Guide to Packaging but could tell some of the advice was out of date having followed the discussions for a while now. What I was thinking is it could be useful my documenting exactly what I'm having to do, things to remember etc and it'd help me learn things much more deeply on the way too. My only concern however would be where to go to find the right 'new world' answers. Aside from reading the PEPs etc, are there any other points of contact? Is it ok to ask such questions on this list? Thanks, Alex J Burke. From dholth at gmail.com Wed Jul 3 02:57:20 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 2 Jul 2013 20:57:20 -0400 Subject: [Distutils] "Python Packaging User Guide" In-Reply-To: References: Message-ID: On Tue, Jul 2, 2013 at 7:18 PM, Alex Burke wrote: > Hi Marcus, > > I would very much like to get involved and help with this effort - in fact I was looking to package up some of my own work recently and stumbled on The Hitchhiker?s Guide to Packaging but could tell some of the advice was out of date having followed the discussions for a while now. > > What I was thinking is it could be useful my documenting exactly what I'm having to do, things to remember etc and it'd help me learn things much more deeply on the way too. My only concern however would be where to go to find the right 'new world' answers. Aside from reading the PEPs etc, are there any other points of contact? Is it ok to ask such questions on this list? Sounds like a great idea, assemble your FAQ, and we will remember what's confusing about packaging when you haven't been embroiled in it for years. From qwcode at gmail.com Wed Jul 3 07:28:37 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 2 Jul 2013 22:28:37 -0700 Subject: [Distutils] pip and virtualenv release candidates Message-ID: pip-1.4rc2 and virtualenv-1.10rc3 are available for testing from github. A few highlights: - pip added support for installing and building wheel archives. ( http://www.pip-installer.org/en/latest/cookbook.html#building-and-installing-wheels ) - virtualenv is now using the new merged setuptools, and no longer supports distribute. - pip now only installs stable versions by default, and offers a new --pre option to also find pre-releases. - Dropped support for Python 2.5. Changelogs: pip: http://www.pip-installer.org/en/release-1.4/news.html virtualenv: http://www.virtualenv.org/en/release-1.10/news.html Download Links: pip: gz: https://github.com/pypa/pip/archive/1.4rc2.tar.gz md5=0426430fc8a261c83bcd083fb03fb7d6 zip: https://github.com/pypa/pip/archive/1.4rc2.zip md5=c86dc0d94ed787eadba6dceb06f1676f virtualenv: gz: https://github.com/pypa/virtualenv/archive/1.10rc3.tar.gz md5=b24cdf59b561acf26ae3f639098d5a34 zip: https://github.com/pypa/virtualenv/archive/1.10rc3.zip md5=a6ee1a1570a751aa50f95833d9898649 Installation: The easiest way to try them both and *not* affect your current system, is like so: e.g. on Linux: $ curl -L -O https://github.com/pypa/virtualenv/archive/1.10rc3.tar.gz $ echo "b24cdf59b561acf26ae3f639098d5a34 1.10rc3.tar.gz" | md5sum -c 1.10rc3.tar.gz: OK $ tar zxf 1.10rc3.tar.gz $ python virtualenv-1.10rc3/virtualenv.py myVE $ myVE/bin/pip --version pip 1.4rc2 *Note*: If instead, you choose to upgrade an existing pip (and setuptools), know this: 1) pip's wheel support requires setuptools>=0.8b2 (this will become final before pip is released final) 2) setuptools-0.8bx is not on pypi and can be found here: https://bitbucket.org/pypa/setuptools/downloads 3) Older pip's can not currently upgrade distribute to setuptools (until distribute-0.7.3 is released on ~July-7th) (for more upgrade details: http://www.pip-installer.org/en/latest/installing.html#requirements) Offering Feedback: You can respond to this email, or log issues in our tracker: https://github.com/pypa/pip/issues?state=open -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Jul 3 10:03:21 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 3 Jul 2013 18:03:21 +1000 Subject: [Distutils] "Python Packaging User Guide" In-Reply-To: References: Message-ID: On 3 July 2013 09:18, Alex Burke wrote: > Hi Marcus, > > I would very much like to get involved and help with this effort - in fact > I was looking to package up some of my own work recently and stumbled on > The Hitchhiker?s Guide to Packaging but could tell some of the advice was > out of date having followed the discussions for a while now. > > What I was thinking is it could be useful my documenting exactly what I'm > having to do, things to remember etc and it'd help me learn things much > more deeply on the way too. My only concern however would be where to go to > find the right 'new world' answers. Aside from reading the PEPs etc, are > there any other points of contact? Is it ok to ask such questions on this > list? > Asking here, or filing tracker issues on https://bitbucket.org/pypa/python-packaging-user-guide would be good. Unless things have changed in the past couple of weeks, the aim is for people starting from scratch now to grab setuptools 0.7 (or later) and pip 1.4 (or later & once it is released) I'm not sure on the specific plan for wheel uploads to PyPI myself - I *believe* we're going to recommend using pip to retrieve http://wheel.readthedocs.org/en/latest/ to obtain the bdist_wheel command. There's also the problem of hosting for the setuptools & pip bootstrap scripts. I believe the plan is to host it at a canonical URL on pypi.python.org, but I'm not sure of the current status of that. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alexander.Schneider at uni-duesseldorf.de Wed Jul 3 11:17:51 2013 From: Alexander.Schneider at uni-duesseldorf.de (Alexander Schneider) Date: Wed, 03 Jul 2013 11:17:51 +0200 Subject: [Distutils] Metadataformat PEP 426 on PyPI? In-Reply-To: References: Message-ID: <51D3EC3F.8040405@uni-duesseldorf.de> Hello and sorry if I am on the wrong mailing list. I'm working on a dependency resolution resolver and wanted to implement support for the new Metadata format. (As of now i'm parsing the setup.py for dependency information and am dependent on a self build metadata database of all PyPI packages) Will there be build-in Metadata in the new PEP 426 format online on PyPI for all packages? If yes, are there already specifications on how they will be retreavable? Thanks, Alexander Schneider From ncoghlan at gmail.com Wed Jul 3 13:58:21 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 3 Jul 2013 21:58:21 +1000 Subject: [Distutils] Metadataformat PEP 426 on PyPI? In-Reply-To: <51D3EC3F.8040405@uni-duesseldorf.de> References: <51D3EC3F.8040405@uni-duesseldorf.de> Message-ID: On 3 July 2013 19:17, Alexander Schneider < Alexander.Schneider at uni-duesseldorf.de> wrote: > Hello and sorry if I am on the wrong mailing list. > > I'm working on a dependency resolution resolver and wanted to implement > support for the new Metadata format. (As of now i'm parsing the setup.py > for dependency information and am dependent on a self build metadata > database of all PyPI packages) > > Will there be build-in Metadata in the new PEP 426 format online on PyPI > for all packages? If yes, are there already specifications on how they > will be retreavable? > Yes there will, but actually figuring out the details of those APIs is some time away. Note that the version currently referenced from the PEP is a little out of date ( http://mail.python.org/pipermail/distutils-sig/2013-June/021357.html). I will hopefully get it updated at the PyCon AU sprints next week. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Jul 3 16:51:02 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 3 Jul 2013 14:51:02 +0000 (UTC) Subject: [Distutils] Metadataformat PEP 426 on PyPI? References: <51D3EC3F.8040405@uni-duesseldorf.de> Message-ID: Alexander Schneider uni-duesseldorf.de> writes: > Will there be build-in Metadata in the new PEP 426 format online on PyPI > for all packages? If yes, are there already specifications on how they > will be retreavable? I have experimental support for PEP 426 metadata available, which is up-to-date with the spec apart from the changes Nick linked to. For any given package, you can access some JSON at an URL based on the project name. For example, setuptools 0.7.5 metadata is available at http://www.red-dove.com/pypi/projects/S/setuptools/package-0.7.5.json If you deserialize the JSON at an URL like the above into a dict, the PEP 426 metadata is available in the subdict at key "index-metadata" in the top-level dict. Example from setuptools 0.7.5: "index-metadata": { "description": "omitted for brevity", "license": "PSF or ZPL", "contacts": [ { "role": "author", "name": "The fellowship of the packaging", "email": "distutils-sig at python.org" } ], "summary": "Easily download, build, install, upgrade, and uninstall Python packages", "project_urls": { "Home": "https://pypi.python.org/pypi/setuptools" }, "keywords": [ "CPAN", "PyPI", "distutils", "eggs", "package", "management" ], "metadata_version": "2.0", "extras": [ "certs", "ssl" ], "version": "0.7.5", "run_may_require": [ { "environment": "sys_platform=='win32'", "dependencies": [ "wincertstore (== 0.1)" ], "extra": "ssl" }, { "environment": "sys_platform=='win32' and python_version=='2.4'", "dependencies": [ "ctypes (== 1.0.2)" ], "extra": "ssl" }, { "dependencies": [ "certifi (== 0.0.8)" ], "extra": "certs" }, { "environment": "python_version in '2.4, 2.5'", "dependencies": [ "ssl (== 1.16)" ], "extra": "ssl" } ], "classifiers": [ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: Python Software Foundation License", "License :: OSI Approved :: Zope Public License", "Operating System :: OS Independent", "Programming Language :: Python :: 2.4", "Programming Language :: Python :: 2.5", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.1", "Programming Language :: Python :: 3.2", "Programming Language :: Python :: 3.3", "Topic :: Software Development :: Libraries :: Python Modules", "Topic :: System :: Archiving :: Packaging", "Topic :: System :: Systems Administration", "Topic :: Utilities" ], "name": "setuptools" }, I expect this metadata to track the PEP as changes to it are published. Currently, the top-level dict contains some legacy representations of the metadata which will be removed in due course. I would hope that once the dust settles on the PEP, this metadata (the PEP 426 part) can be migrated to PyPI. Regards, Vinay Sajip From alanwilter at gmail.com Wed Jul 3 15:55:17 2013 From: alanwilter at gmail.com (Alan) Date: Wed, 3 Jul 2013 14:55:17 +0100 Subject: [Distutils] $MACOSX_DEPLOYMENT_TARGET mismatch Message-ID: Hi there, I am trying to install readline and cx_Oracle on my Mac OSX 10.8.4. I've just installed python 2.7.3 standard in my $HOME dir and then I did: export PATH=$HOME/bin:$PATH ~/setuptools-0.7.7 python setup.py install --prefix=$HOME which worked. I tried to installed ipython 0.13.2 which seems to have worked but it fails to start because of readline. easy_install readline Searching for readline Reading https://pypi.python.org/simple/readline/ Best match: readline 6.2.4.1 Downloading https://pypi.python.org/packages/source/r/readline/readline-6.2.4.1.tar.gz#md5=578237939c81fdbc2c8334d168b17907 Processing readline-6.2.4.1.tar.gz Writing /var/folders/0c/n_0by3qj74bg5m68fxrw2kw40000gp/T/easy_install-3WQA6K/readline-6.2.4.1/setup.cfg Running readline-6.2.4.1/setup.py -q bdist_egg --dist-dir /var/folders/0c/n_0by3qj74bg5m68fxrw2kw40000gp/T/easy_install-3WQA6K/readline-6.2.4.1/egg-dist-tmp-eyGcUE error: Setup script exited with error: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.3" but "10.4" during configure How can solve this problem please? Thanks in advance, Alan -- Alan Wilter SOUSA da SILVA, DSc Bioinformatician, UniProt - PANDA, EMBL-EBI CB10 1SD, Hinxton, Cambridge, UK +44 1223 49 4588 -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Wed Jul 3 18:26:19 2013 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 3 Jul 2013 09:26:19 -0700 Subject: [Distutils] [venv] pip and virtualenv release candidates In-Reply-To: <51D44736.7050502@gmail.com> References: <51D44736.7050502@gmail.com> Message-ID: > FWIW, the following works to update a distribute-baesd virtualenv to the > new setuptools / vr: > > $ bin/easy_install \ > --find-links https://bitbucket.org/pypa/setuptools/downloads/ \ > -U distribute > > good pt. if you use the unofficial release location, you can do it with pip as well. pip install -U --find-links=https://bitbucket.org/pypa/setuptools/downloads/distribute P.S. if you already have pip-1.4, add "--pre" so it can find the setuptools-0.8 betas (that pip also needs to make this work) Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From tres.seaver at gmail.com Wed Jul 3 17:45:58 2013 From: tres.seaver at gmail.com (Tres Seaver) Date: Wed, 03 Jul 2013 11:45:58 -0400 Subject: [Distutils] [venv] pip and virtualenv release candidates In-Reply-To: References: Message-ID: <51D44736.7050502@gmail.com> On 07/03/2013 01:28 AM, Marcus Smith wrote: > 3) Older pip's can not currently upgrade distribute to setuptools > (until distribute-0.7.3 is released on ~July-7th) > (for more upgrade details: > http://www.pip-installer.org/en/latest/installing.html#requirements) FWIW, the following works to update a distribute-baesd virtualenv to the new setuptools / vr: $ bin/easy_install \ --find-links https://bitbucket.org/pypa/setuptools/downloads/ \ -U distribute No-looking-back'ly, Tres. -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com From pje at telecommunity.com Wed Jul 3 20:19:16 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 3 Jul 2013 14:19:16 -0400 Subject: [Distutils] Metadataformat PEP 426 on PyPI? In-Reply-To: References: <51D3EC3F.8040405@uni-duesseldorf.de> Message-ID: On Wed, Jul 3, 2013 at 10:51 AM, Vinay Sajip wrote: > If you deserialize the JSON at an URL like the above into a dict, the PEP > 426 metadata is available in the subdict at key "index-metadata" in the > top-level dict. Example from setuptools 0.7.5: > > "index-metadata": { > .... > "name": "setuptools" > }, > > I expect this metadata to track the PEP as changes to it are published. > Currently, the top-level dict contains some legacy representations of the > metadata which will be removed in due course. Just an FYI, not sure if this is an issue with your converter or with the new spec, but the metadata shown for setuptools is missing something important: 0.7.x pins specific distributions of its dependencies using dependency_links URLs with #md5 hashes, so that SSL support can be installed in a reasonably secure manner, as long as you're starting from a trusted copy of the distribution. The converted metadata you show lacks this pinning. Granted, the pinning is somewhat kludged, and the specific need is perhaps a rare use case outside of installer tools themselves. But I thought it worth pointing out as a limitation of either the converter or with the spec itself in relation to version support. From donald at stufft.io Wed Jul 3 20:34:55 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 3 Jul 2013 14:34:55 -0400 Subject: [Distutils] Metadataformat PEP 426 on PyPI? In-Reply-To: References: <51D3EC3F.8040405@uni-duesseldorf.de> Message-ID: On Jul 3, 2013, at 2:19 PM, PJ Eby wrote: > On Wed, Jul 3, 2013 at 10:51 AM, Vinay Sajip wrote: >> If you deserialize the JSON at an URL like the above into a dict, the PEP >> 426 metadata is available in the subdict at key "index-metadata" in the >> top-level dict. Example from setuptools 0.7.5: >> >> "index-metadata": { >> .... >> "name": "setuptools" >> }, >> >> I expect this metadata to track the PEP as changes to it are published. >> Currently, the top-level dict contains some legacy representations of the >> metadata which will be removed in due course. > > Just an FYI, not sure if this is an issue with your converter or with > the new spec, but the metadata shown for setuptools is missing > something important: 0.7.x pins specific distributions of its > dependencies using dependency_links URLs with #md5 hashes, so that SSL > support can be installed in a reasonably secure manner, as long as > you're starting from a trusted copy of the distribution. The > converted metadata you show lacks this pinning. > > Granted, the pinning is somewhat kludged, and the specific need is > perhaps a rare use case outside of installer tools themselves. But I > thought it worth pointing out as a limitation of either the converter > or with the spec itself in relation to version support. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig PEP426 does not support dependency_links. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From chris.barker at noaa.gov Wed Jul 3 20:50:28 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 3 Jul 2013 11:50:28 -0700 Subject: [Distutils] $MACOSX_DEPLOYMENT_TARGET mismatch In-Reply-To: References: Message-ID: This is really a build question, rather than a distributuion question -- I"d try the pythonmac list: http://mail.python.org/mailman/listinfo/pythonmac-sig I recall that readline is a bit of a pain on the Mac, but don't recall the solution (nor am I running 10.8 yet). Good luck, -Chris On Wed, Jul 3, 2013 at 6:55 AM, Alan wrote: > Hi there, > > I am trying to install readline and cx_Oracle on my Mac OSX 10.8.4. I've > just installed python 2.7.3 standard in my $HOME dir and then I did: > > export PATH=$HOME/bin:$PATH > ~/setuptools-0.7.7 > python setup.py install --prefix=$HOME > > which worked. > > I tried to installed ipython 0.13.2 which seems to have worked but it fails > to start because of readline. > > easy_install readline > Searching for readline > Reading https://pypi.python.org/simple/readline/ > Best match: readline 6.2.4.1 > Downloading > https://pypi.python.org/packages/source/r/readline/readline-6.2.4.1.tar.gz#md5=578237939c81fdbc2c8334d168b17907 > Processing readline-6.2.4.1.tar.gz > Writing > /var/folders/0c/n_0by3qj74bg5m68fxrw2kw40000gp/T/easy_install-3WQA6K/readline-6.2.4.1/setup.cfg > Running readline-6.2.4.1/setup.py -q bdist_egg --dist-dir > /var/folders/0c/n_0by3qj74bg5m68fxrw2kw40000gp/T/easy_install-3WQA6K/readline-6.2.4.1/egg-dist-tmp-eyGcUE > error: Setup script exited with error: $MACOSX_DEPLOYMENT_TARGET mismatch: > now "10.3" but "10.4" during configure > > How can solve this problem please? > > Thanks in advance, > > Alan > > -- > Alan Wilter SOUSA da SILVA, DSc > Bioinformatician, UniProt - PANDA, EMBL-EBI > CB10 1SD, Hinxton, Cambridge, UK > +44 1223 49 4588 > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From m.van.rees at zestsoftware.nl Wed Jul 3 21:39:25 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Wed, 03 Jul 2013 21:39:25 +0200 Subject: [Distutils] PyPI upload failed (302): Moved Temporarily In-Reply-To: <20130702114732.GC8649@odin.tremily.us> References: <20130702114732.GC8649@odin.tremily.us> Message-ID: Op 02-07-13 13:47, W. Trevor King schreef: > Hello list! > > I'm having a bit of trouble getting a new package setup on PyPI. I've > done this a few times in the past, but maybe not since the wiki > hacking that inspired the HTTPS transition back in February. I did > change my password back then, so I don't think that's the problem. > Anyhow, with the old URL (still mentioned in the distutils docs [1]): > > http://www.python.org/pypi > > in my ~/.pypirc, I get: > > $ python setup.py register -r pypi > running register > running check > Registering pycalendar to http://www.python.org/pypi > Server response (403): You are not allowed to store 'pycalendar' package information > > With https://pypi.python.org/ in my ~/.pypirc, I get: > > $ python setup.py register -r pypi > running register > running check > Registering pycalendar to https://pypi.python.org/ > Server response (200): OK > > Ok, how about uploading a tarball? With the https:// URL: > > $ python setup.py sdist upload -r pypi > ? > running upload > Submitting dist/pycalendar-0.1.tar.gz to https://pypi.python.org/ > Upload failed (302): Moved Temporarily > > With the http:// URL: > > $ python setup.py sdist upload -r pypi > ? > running upload > Submitting dist/pycalendar-0.1.tar.gz to http://www.python.org/pypi > Upload failed (403): You are not allowed to edit 'pycalendar' package information > > What am I doing wrong? > > Thanks, > Trevor > > [1]: http://docs.python.org/3/distutils/packageindex.html#the-pypirc-file I'll post my ~/.pypirc below for comparison. I have a second index-server, which you should not need. Note that I did not enter a repository url for pypi, so apparently that is not necessary. [distutils] index-servers = pypi plone [pypi] username:maurits password:secret [plone] repository:http://plone.org/products username:maurits password:secret If you have basically the same, then triple check that you have the correct username and password, that you have probably already done that. Maybe change the password again just to be sure. BTW, I am using Python 2.6 or 2.7. I guess it works the same with Python 3 though. -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From alanwilter at gmail.com Wed Jul 3 21:12:57 2013 From: alanwilter at gmail.com (Alan) Date: Wed, 3 Jul 2013 20:12:57 +0100 Subject: [Distutils] $MACOSX_DEPLOYMENT_TARGET mismatch In-Reply-To: References: Message-ID: Well, I found out that if before compiling my python I set export MACOSX_DEPLOYMENT_TARGET=10.3 and then do all the rest, then I get easy_install to work. Besides, I had the same error trying to install cx_Oracle. So, somehow my python (or setuptools) need to build for "10.3". For me it's a bug in setuptools since the function that do this check and raise the error are from setuptools package. Alan On 3 July 2013 19:50, Chris Barker - NOAA Federal wrote: > This is really a build question, rather than a distributuion question > -- I"d try the pythonmac list: > > http://mail.python.org/mailman/listinfo/pythonmac-sig > > I recall that readline is a bit of a pain on the Mac, but don't recall > the solution (nor am I running 10.8 yet). > > Good luck, > -Chris > > On Wed, Jul 3, 2013 at 6:55 AM, Alan wrote: > > Hi there, > > > > I am trying to install readline and cx_Oracle on my Mac OSX 10.8.4. I've > > just installed python 2.7.3 standard in my $HOME dir and then I did: > > > > export PATH=$HOME/bin:$PATH > > ~/setuptools-0.7.7 > > python setup.py install --prefix=$HOME > > > > which worked. > > > > I tried to installed ipython 0.13.2 which seems to have worked but it > fails > > to start because of readline. > > > > easy_install readline > > Searching for readline > > Reading https://pypi.python.org/simple/readline/ > > Best match: readline 6.2.4.1 > > Downloading > > > https://pypi.python.org/packages/source/r/readline/readline-6.2.4.1.tar.gz#md5=578237939c81fdbc2c8334d168b17907 > > Processing readline-6.2.4.1.tar.gz > > Writing > > > /var/folders/0c/n_0by3qj74bg5m68fxrw2kw40000gp/T/easy_install-3WQA6K/readline-6.2.4.1/setup.cfg > > Running readline-6.2.4.1/setup.py -q bdist_egg --dist-dir > > > /var/folders/0c/n_0by3qj74bg5m68fxrw2kw40000gp/T/easy_install-3WQA6K/readline-6.2.4.1/egg-dist-tmp-eyGcUE > > error: Setup script exited with error: $MACOSX_DEPLOYMENT_TARGET > mismatch: > > now "10.3" but "10.4" during configure > > > > How can solve this problem please? > > > > Thanks in advance, > > > > Alan > > > > -- > > Alan Wilter SOUSA da SILVA, DSc > > Bioinformatician, UniProt - PANDA, EMBL-EBI > > CB10 1SD, Hinxton, Cambridge, UK > > +44 1223 49 4588 > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -- Alan Wilter SOUSA da SILVA, DSc Bioinformatician, UniProt - PANDA, EMBL-EBI CB10 1SD, Hinxton, Cambridge, UK +44 1223 49 4588 -------------- next part -------------- An HTML attachment was scrubbed... URL: From js at hipro.co.in Wed Jul 3 22:03:43 2013 From: js at hipro.co.in (Joe Steeve) Date: Thu, 04 Jul 2013 01:33:43 +0530 Subject: [Distutils] "python" option in buildout 2.1.1 Message-ID: <1372881823.4672.3.camel@localhost> From the buildout 2.1.1 docs on pypi: python The name of a section containing information about the default Python interpreter. Recipes that need a installation typically have options to tell them which Python installation to use. By convention, if a section-specific option isn't used, the option is looked for in the buildout section. The option must point to a section with an executable option giving the path to a Python executable. By default, the buildout section defines the default Python as the Python used to run the buildout. Is the above still relevant? I am trying to build a custom python using zc.recipe.cmmi and use the interpreter in other parts, like in: http://goo.gl/ufWwT But, that simply does not work. Regards, Joe -- Joe Steeve HiPro IT Solutions Private Limited http://hipro.co.in/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From jim at zope.com Wed Jul 3 22:32:10 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 3 Jul 2013 16:32:10 -0400 Subject: [Distutils] "python" option in buildout 2.1.1 In-Reply-To: <1372881823.4672.3.camel@localhost> References: <1372881823.4672.3.camel@localhost> Message-ID: On Wed, Jul 3, 2013 at 4:03 PM, Joe Steeve wrote: > From the buildout 2.1.1 docs on pypi: > > python > The name of a section containing information about the That's a documentation bug. :( I'll fix that. > Is the above still relevant? No. The support for multiple Python interpreters in a single buildout was dropped from buildout 2. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From chris.barker at noaa.gov Wed Jul 3 22:36:38 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 3 Jul 2013 13:36:38 -0700 Subject: [Distutils] $MACOSX_DEPLOYMENT_TARGET mismatch In-Reply-To: References: Message-ID: On Wed, Jul 3, 2013 at 12:12 PM, Alan wrote: > Well, I found out that if before compiling my python I set > > export MACOSX_DEPLOYMENT_TARGET=10.3 > > and then do all the rest, then I get easy_install to work. cool - well done. > So, somehow my python (or setuptools) need to build for "10.3". > > For me it's a bug in setuptools since the function that do this check and > raise the error are from setuptools package. >>> I've just installed python 2.7.3 standard How did you install it? not sure what "standard" is. The build is supposed to setup distutils to do the right thing, and I'm pretty sure it does with the binaries from python.org. I will note that unless you want to re-package (i.e. with py2app) and distribute to folks with old machines, you probably don't need to support way back to 10.3 -- the "intel build" on python.org is set up for 10.5 and greater, and may avoid this issue. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From vinay_sajip at yahoo.co.uk Wed Jul 3 23:41:52 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 3 Jul 2013 21:41:52 +0000 (UTC) Subject: [Distutils] Metadataformat PEP 426 on PyPI? References: <51D3EC3F.8040405@uni-duesseldorf.de> Message-ID: PJ Eby telecommunity.com> writes: > Just an FYI, not sure if this is an issue with your converter or with > the new spec, but the metadata shown for setuptools is missing > something important: 0.7.x pins specific distributions of its > dependencies using dependency_links URLs with #md5 hashes, so that SSL > support can be installed in a reasonably secure manner, as long as > you're starting from a trusted copy of the distribution. The > converted metadata you show lacks this pinning. True, although I do capture the dependency links under the 'dependency-urls' key of the top level dict of the JSON I linked to. While dependency_links is not directly supported by PEP 426, the intent is there via "direct references". When installing using distlib/distil, SSL host verification and hash verification are done, even when direct references are not specified, since the versions of dependencies are pinned. For example, if I install setuptools into a fresh venv: $ pyvenv-3.3 /tmp/venv $ distil -e /tmp/venv install "setuptools [ssl,certs]" Checking requirements for setuptools (0.7.7) ... done. The following new packages will be downloaded and installed: certifi (0.0.8) [for setuptools] setuptools (0.7.7) Proceed? (y/n) y Downloading certifi-0.0.8.tar.gz to /tmp/tmpccek0f [for setuptools] 115KB @ 667 KB/s 100 % Done: 00:00:00 Unpacking ... done. Downloading setuptools-0.7.7.tar.gz to /tmp/tmpchxc1x 736KB @ 393 KB/s 100 % Done: 00:00:01 Unpacking ... done. [installation feedback snipped] Below is an extract from distil.log for the above installation, showing the downloading and verification operations: Downloading certifi-0.0.8.tar.gz to /tmp/tmpccek0f [for setuptools] Digest specified: dc5f5e7f0b5fc08d27654b17daa6ecec Host verified: pypi.python.org Digest verified: dc5f5e7f0b5fc08d27654b17daa6ecec Library location: venv site-packages Downloading setuptools-0.7.7.tar.gz to /tmp/tmpchxc1x Digest specified: 0d7bc0e1a34b70a97e706ef74aa7f37f Host verified: pypi.python.org Digest verified: 0d7bc0e1a34b70a97e706ef74aa7f37f Library location: venv site-packages Distil includes the Mozilla certs and thus is able to do SSL host validation. The hash support is currently limited to MD5 because PyPI has not supported other formats, but I expect that will be rectified in due course. Regards, Vinay Sajip From js at hipro.co.in Wed Jul 3 22:57:51 2013 From: js at hipro.co.in (Joe Steeve) Date: Thu, 04 Jul 2013 02:27:51 +0530 Subject: [Distutils] zc.recipe.egg skips script-generation Message-ID: <1372885071.28665.8.camel@localhost> Hello all, If this is not the right place to talk about 'zc.recipe.egg', please be kind and point me to the right place :) I am trying to create a buildout to work with some eggs using ipython. I have a buildout.cfg like this: [buildout] develop = . parts = app ipython_part [app] recipe = zc.recipe.egg:scripts eggs = stool [ipython_part] recipe = zc.recipe.egg:scripts dependent-scripts = true eggs = ${app:eggs} ipython scripts = ipython I have an ipython (0.13.1) installed in my system python's site-packages. When I run buildout with the above, buildout simply says "Installing ipython_part" and does not do anything else. If I remove my system ipython (apt-get purge ipython), then buildout installs a script locally. I tried creating a virtualenv, and bootstrapping with the virtualenv's python. I have the same effect. How do I get an 'ipython' script with my desired eggs in its path? Regards, Joe -- Joe Steeve HiPro IT Solutions Private Limited http://hipro.co.in/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From js at hipro.co.in Wed Jul 3 22:39:27 2013 From: js at hipro.co.in (Joe Steeve) Date: Thu, 04 Jul 2013 02:09:27 +0530 Subject: [Distutils] "python" option in buildout 2.1.1 In-Reply-To: References: <1372881823.4672.3.camel@localhost> Message-ID: <1372883967.28665.0.camel@localhost> On Wed, 2013-07-03 at 16:32 -0400, Jim Fulton wrote: > That's a documentation bug. :( I'll fix that. Thank you. -- Joe Steeve HiPro IT Solutions Private Limited http://hipro.co.in/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From donald at stufft.io Thu Jul 4 01:04:22 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 3 Jul 2013 19:04:22 -0400 Subject: [Distutils] Weekly Download Counters Enabled Message-ID: <7760F9D0-A7A7-4BD1-A964-3DDB41D39B5C@stufft.io> Just a quick follow up to last weeks email the download counters for weekly numbers has now been enabled on PyPI. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From pje at telecommunity.com Thu Jul 4 01:38:24 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 3 Jul 2013 19:38:24 -0400 Subject: [Distutils] Metadataformat PEP 426 on PyPI? In-Reply-To: References: <51D3EC3F.8040405@uni-duesseldorf.de> Message-ID: On Wed, Jul 3, 2013 at 2:34 PM, Donald Stufft wrote: > PEP426 does not support dependency_links. Right - I would've expected direct references in this scenario, assuming the PEP still has them. From donald at stufft.io Thu Jul 4 01:40:52 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 3 Jul 2013 19:40:52 -0400 Subject: [Distutils] Metadataformat PEP 426 on PyPI? In-Reply-To: References: <51D3EC3F.8040405@uni-duesseldorf.de> Message-ID: <059B6EC3-B5CD-48D2-A972-480FDD6D5BA5@stufft.io> On Jul 3, 2013, at 7:38 PM, PJ Eby wrote: > On Wed, Jul 3, 2013 at 2:34 PM, Donald Stufft wrote: >> PEP426 does not support dependency_links. > > Right - I would've expected direct references in this scenario, > assuming the PEP still has them. Yea PEP440 direct references can be used as an approximate dependency_links replacement with the caveat you can't publish them on PyPI. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From js at hipro.co.in Thu Jul 4 00:55:05 2013 From: js at hipro.co.in (Joe Steeve) Date: Thu, 04 Jul 2013 04:25:05 +0530 Subject: [Distutils] zc.recipe.egg skips script-generation In-Reply-To: <1372887902.31748.0.camel@localhost> References: <1372887902.31748.0.camel@localhost> Message-ID: <1372892105.31748.5.camel@localhost> On Thu, 2013-07-04 at 03:15 +0530, Joe Steeve wrote: > I have an ipython (0.13.1) installed in my system python's > site-packages. When I run buildout with the above, buildout simply > says "Installing ipython_part" and does not do anything else. If I > remove my system ipython (apt-get purge ipython), then buildout > installs a script locally. > > I tried creating a virtualenv, and bootstrapping with the virtualenv's > python. I have the same effect. Sorry, false alarm. Retried with a virtualenv after clearing out all the buildout directories. Works fine now. Thanks, Joe -- Joe Steeve HiPro IT Solutions Private Limited http://hipro.co.in/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From donald at stufft.io Thu Jul 4 07:38:19 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 4 Jul 2013 01:38:19 -0400 Subject: [Distutils] PyPI CDN Updates For Greater Availability Message-ID: <3C448D97-1210-4227-AC1B-147EECC7F79B@stufft.io> Several changes were just deployed to PyPI's CDN. The general theme behind the changes is making it so that PyPI appears as functional as possible through a failure of the server hosting it. This should increase the availability of PyPI and enable things such as installation and browsing the site to continue to work through a catastrophic host failure on the PSF infrastructure. The details of what changes are: - Anonymous users will find that /pypi* pages are now cached for a short amount of time (currently 60s). - Objects will be stored in the cache for some time past their expiration date. They will not be used except in two circumstances: - A request is taking longer than 15s to complete, a "stale" object will be returned to prevent a pile up from occurring. - The backend[1] has been deemed unhealthy, in which case stale objects will be served in order to allow some level of functionality until the backend has been restored. - In the advent of an unhealthy backend all requests will be forced to be anonymous, making them eligible for the stale objects that have been cached. - The /mirrors and /security pages will be cached for a week, allowing them to likely be available through a backend failure making it easy to locate mirrors[2] or report a security issue. - Miscellaneous changes to normalize various things so that a single item in the cache will be able to be used for more requests, making it more likely that any particular request will be served from the Cache. [1] Backend in this context means the server hosting PyPI itself, what the CDN itself connects too. [2] Using the mirrors is done so at your own risk. None of the tools currently verify the downloads and they are downloaded over HTTP. This makes it trivial for an attacker to execute arbitrary code on your machine via a MITM. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Thu Jul 4 10:51:13 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 4 Jul 2013 08:51:13 +0000 (UTC) Subject: [Distutils] Upcoming changes to PEP 426/440 References: Message-ID: Nick Coghlan gmail.com> writes: > * "install": the installation specifier for the dependency > * "extra": as per the current PEP (for conditional dependencies) > * "environment": as per the current PEP (for conditional dependencies) > > 4. The "install" subfield is compulsory, the other two are optional > (as now, using either of the latter creates a "conditional > dependency", while dependency declarations with only the "install" > subfield are unconditional) > > 5. An installation specifier is what PEP 426 currently calls a > dependency specifier: the "name [extras] (constraints)" format. They > will get their own top level section (similar to the existing Extras > and Environment markers sections) Is there a particular benefit of the install subfield being a single installation specifier, as opposed to a list of such specifiers? It's perhaps neither here nor there for machine-processed metadata, but I expect this metadata would have human readers too. Not using a list would lead to more verbose metadata. Regards, Vinay Sajip From ncoghlan at gmail.com Thu Jul 4 13:00:19 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 4 Jul 2013 21:00:19 +1000 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: Message-ID: On 4 Jul 2013 18:52, "Vinay Sajip" wrote: > > Nick Coghlan gmail.com> writes: > > > * "install": the installation specifier for the dependency > > * "extra": as per the current PEP (for conditional dependencies) > > * "environment": as per the current PEP (for conditional dependencies) > > > > 4. The "install" subfield is compulsory, the other two are optional > > (as now, using either of the latter creates a "conditional > > dependency", while dependency declarations with only the "install" > > subfield are unconditional) > > > > 5. An installation specifier is what PEP 426 currently calls a > > dependency specifier: the "name [extras] (constraints)" format. They > > will get their own top level section (similar to the existing Extras > > and Environment markers sections) > > Is there a particular benefit of the install subfield being a single > installation specifier, as opposed to a list of such specifiers? It's > perhaps neither here nor there for machine-processed metadata, but I expect > this metadata would have human readers too. Not using a list would lead to > more verbose metadata. Hmm, I guess as long as it's consistent, the only difference when processing is list.append vs list.extend. There's a little extra work when serialising to group like entries together, but I'm OK with that (and that would be a SHOULD rather than a MUST anyway). If I don't hear a good argument against it, I'll make that field a list. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jul 4 13:35:17 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 4 Jul 2013 07:35:17 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: Message-ID: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> On Jul 4, 2013, at 7:00 AM, Nick Coghlan wrote: > > On 4 Jul 2013 18:52, "Vinay Sajip" wrote: > > > > Nick Coghlan gmail.com> writes: > > > > > * "install": the installation specifier for the dependency > > > * "extra": as per the current PEP (for conditional dependencies) > > > * "environment": as per the current PEP (for conditional dependencies) > > > > > > 4. The "install" subfield is compulsory, the other two are optional > > > (as now, using either of the latter creates a "conditional > > > dependency", while dependency declarations with only the "install" > > > subfield are unconditional) > > > > > > 5. An installation specifier is what PEP 426 currently calls a > > > dependency specifier: the "name [extras] (constraints)" format. They > > > will get their own top level section (similar to the existing Extras > > > and Environment markers sections) > > > > Is there a particular benefit of the install subfield being a single > > installation specifier, as opposed to a list of such specifiers? It's > > perhaps neither here nor there for machine-processed metadata, but I expect > > this metadata would have human readers too. Not using a list would lead to > > more verbose metadata. > > Hmm, I guess as long as it's consistent, the only difference when processing is list.append vs list.extend. > > There's a little extra work when serialising to group like entries together, but I'm OK with that (and that would be a SHOULD rather than a MUST anyway). > > If I don't hear a good argument against it, I'll make that field a list. > > > > > Regards, > > > > Vinay Sajip > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig I would prefer a single entry. It makes the serialization format map to the modeling simpler, and I think it's simpler for humans too. I don't see much benefit to making it a list except arbitrarily adding another level of nesting. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Thu Jul 4 14:26:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 4 Jul 2013 22:26:51 +1000 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> Message-ID: On 4 Jul 2013 21:35, "Donald Stufft" wrote: > > > On Jul 4, 2013, at 7:00 AM, Nick Coghlan wrote: > >> >> On 4 Jul 2013 18:52, "Vinay Sajip" wrote: >> > >> > Nick Coghlan gmail.com> writes: >> > >> > > * "install": the installation specifier for the dependency >> > > * "extra": as per the current PEP (for conditional dependencies) >> > > * "environment": as per the current PEP (for conditional dependencies) >> > > >> > > 4. The "install" subfield is compulsory, the other two are optional >> > > (as now, using either of the latter creates a "conditional >> > > dependency", while dependency declarations with only the "install" >> > > subfield are unconditional) >> > > >> > > 5. An installation specifier is what PEP 426 currently calls a >> > > dependency specifier: the "name [extras] (constraints)" format. They >> > > will get their own top level section (similar to the existing Extras >> > > and Environment markers sections) >> > >> > Is there a particular benefit of the install subfield being a single >> > installation specifier, as opposed to a list of such specifiers? It's >> > perhaps neither here nor there for machine-processed metadata, but I expect >> > this metadata would have human readers too. Not using a list would lead to >> > more verbose metadata. >> >> Hmm, I guess as long as it's consistent, the only difference when processing is list.append vs list.extend. >> >> There's a little extra work when serialising to group like entries together, but I'm OK with that (and that would be a SHOULD rather than a MUST anyway). >> >> If I don't hear a good argument against it, I'll make that field a list. >> >> > >> > Regards, >> > >> > Vinay Sajip >> > >> > _______________________________________________ >> > Distutils-SIG maillist - Distutils-SIG at python.org >> > http://mail.python.org/mailman/listinfo/distutils-sig >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > > I would prefer a single entry. It makes the serialization format map to the modeling simpler, and I think it's simpler for humans too. I don't see much benefit to making it a list except arbitrarily adding another level of nesting. The main benefit is that all the dependencies for an extra will typically be in one place. However, I briefly forgot the "machine readable" part again, and for that TOOWTDI is to have one entry per dependency. Merging common criteria would then be a UI thing with multiple ways to do it (e.g. whether to group by extra or environment first for conditional dependencies). If you allow a list instead, then you have the problem of offering two ways to say the same thing (all in one entry or split across multiple entries). So the install subfield will remain a single string in the data interchange format, even if tools choose to structure it differently in their UI. Note repeating the key names as well some subfield values doesn't bother me - that's what streaming compression is for. This is what happens when I don't write my rationale down, though - I forget why I did things a certain way :) Cheers, Nick. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Thu Jul 4 14:31:45 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 4 Jul 2013 12:31:45 +0000 (UTC) Subject: [Distutils] Upcoming changes to PEP 426/440 References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > I would prefer a single entry. It makes the serialization format map to the > modeling simpler, and I think it's simpler for humans too. I don't see much > benefit to making it a list except arbitrarily adding another level of > nesting. It's a question of { "install": ["a", "b", "c"] } versus { "install": "a" }, { "install": "b" }, { "install": "c" } and I can't see why you think the latter is in any way better. IMO implementation details (such as "it's easier for the Django ORM to map it") should not take precedence over other considerations of readability/simplicity. In any case, I can't see why there would be any particular modelling problem with the scheme I've suggested. Is the modelling work you're doing public? I had a quick look at your warehouse repo (github.com/dstufft/warehouse) and I don't see any models beyond User and Email. Is that the correct location? I'd be happy to take a closer look to get a better understanding of what modelling problem you're seeing/foreseeing. FYI the metadata that I'm maintaining on red-dove.com is stored in a SQL database. While my SQL schema is not yet fully aligned with the PEP (as it's WIP), I don't see any modelling problem between an RDBMS backend and any of the JSON formats which have been published in the various revisions of the PEP. Some more detail would help :-) Regards, Vinay Sajip From donald at stufft.io Thu Jul 4 14:37:04 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 4 Jul 2013 08:37:04 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> Message-ID: <2A394821-338D-47D6-B4EB-24C83B0F103A@stufft.io> On Jul 4, 2013, at 8:31 AM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > >> I would prefer a single entry. It makes the serialization format map to the >> modeling simpler, and I think it's simpler for humans too. I don't see much >> benefit to making it a list except arbitrarily adding another level of >> nesting. > > It's a question of > > { > "install": ["a", "b", "c"] > } > > versus > > { > "install": "a" > }, > { > "install": "b" > }, > { > "install": "c" > } > > and I can't see why you think the latter is in any way better. IMO > implementation details (such as "it's easier for the Django ORM to map it") > should not take precedence over other considerations of > readability/simplicity. In any case, I can't see why there would be any > particular modelling problem with the scheme I've suggested. It's not that it's easier for the Django ORM to map it, it's just a simpler structure all together. It goes from a single relation to what's essentially a M2M with extra data on the intermediate table. I think it's better because there's less moving parts and this is designed for machines. Human readability is a nice to have but not hardly a requirement. Simpler, not impossible vs impossible ;) > > Is the modelling work you're doing public? I had a quick look at your > warehouse repo (github.com/dstufft/warehouse) and I don't see any models > beyond User and Email. Is that the correct location? I'd be happy to take a > closer look to get a better understanding of what modelling problem you're > seeing/foreseeing. And yea it's in that repo. Still in a branch though as I haven't finished it. > > FYI the metadata that I'm maintaining on red-dove.com is stored in a SQL > database. While my SQL schema is not yet fully aligned with the PEP (as it's > WIP), I don't see any modelling problem between an RDBMS backend and any of > the JSON formats which have been published in the various revisions of the > PEP. Some more detail would help :-) > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Thu Jul 4 15:23:44 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 4 Jul 2013 09:23:44 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: <2A394821-338D-47D6-B4EB-24C83B0F103A@stufft.io> References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> <2A394821-338D-47D6-B4EB-24C83B0F103A@stufft.io> Message-ID: I also prefer the list install : [] Have you played with Postgresql's JSON support :-) On Thu, Jul 4, 2013 at 8:37 AM, Donald Stufft wrote: > > On Jul 4, 2013, at 8:31 AM, Vinay Sajip wrote: > >> Donald Stufft stufft.io> writes: >> >>> I would prefer a single entry. It makes the serialization format map to the >>> modeling simpler, and I think it's simpler for humans too. I don't see much >>> benefit to making it a list except arbitrarily adding another level of >>> nesting. >> >> It's a question of >> >> { >> "install": ["a", "b", "c"] >> } >> >> versus >> >> { >> "install": "a" >> }, >> { >> "install": "b" >> }, >> { >> "install": "c" >> } >> >> and I can't see why you think the latter is in any way better. IMO >> implementation details (such as "it's easier for the Django ORM to map it") >> should not take precedence over other considerations of >> readability/simplicity. In any case, I can't see why there would be any >> particular modelling problem with the scheme I've suggested. > > It's not that it's easier for the Django ORM to map it, it's just a simpler structure > all together. It goes from a single relation to what's essentially a M2M with > extra data on the intermediate table. I think it's better because there's less > moving parts and this is designed for machines. Human readability is a > nice to have but not hardly a requirement. > > Simpler, not impossible vs impossible ;) > >> >> Is the modelling work you're doing public? I had a quick look at your >> warehouse repo (github.com/dstufft/warehouse) and I don't see any models >> beyond User and Email. Is that the correct location? I'd be happy to take a >> closer look to get a better understanding of what modelling problem you're >> seeing/foreseeing. > > And yea it's in that repo. Still in a branch though as I haven't finished it. > >> >> FYI the metadata that I'm maintaining on red-dove.com is stored in a SQL >> database. While my SQL schema is not yet fully aligned with the PEP (as it's >> WIP), I don't see any modelling problem between an RDBMS backend and any of >> the JSON formats which have been published in the various revisions of the >> PEP. Some more detail would help :-) >> >> Regards, >> >> Vinay Sajip >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From donald at stufft.io Thu Jul 4 15:29:05 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 4 Jul 2013 09:29:05 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> <2A394821-338D-47D6-B4EB-24C83B0F103A@stufft.io> Message-ID: <6C675963-D954-4392-ABBB-E31662E07B01@stufft.io> Yea. It's slow and requires invoking plv8 to do much of anything useful ;) On Jul 4, 2013, at 9:23 AM, Daniel Holth wrote: > Have you played with Postgresql's JSON support :-) From dholth at gmail.com Thu Jul 4 15:34:07 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 4 Jul 2013 09:34:07 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: <6C675963-D954-4392-ABBB-E31662E07B01@stufft.io> References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> <2A394821-338D-47D6-B4EB-24C83B0F103A@stufft.io> <6C675963-D954-4392-ABBB-E31662E07B01@stufft.io> Message-ID: If you don't waste your time enforcing the uniqueness of (condition, extra) in the list of requirements then you can pretend install: is a single item if you want to... Wheel converts the flat Metadata 1.3 format to 2.0 draft easily with a defaultdict: https://bitbucket.org/dholth/wheel/src/fb7a900808f31f440049b89a656089b5f55027ce/wheel/metadata.py?at=default#cl-49 On Thu, Jul 4, 2013 at 9:29 AM, Donald Stufft wrote: > Yea. It's slow and requires invoking plv8 to do much of anything useful ;) > > On Jul 4, 2013, at 9:23 AM, Daniel Holth wrote: > >> Have you played with Postgresql's JSON support :-) From donald at stufft.io Thu Jul 4 15:36:59 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 4 Jul 2013 09:36:59 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> <2A394821-338D-47D6-B4EB-24C83B0F103A@stufft.io> <6C675963-D954-4392-ABBB-E31662E07B01@stufft.io> Message-ID: Yea I just spent significant effort cleaning up the database from a lack if enforced constraints. I will pass on not using them. On Jul 4, 2013, at 9:34 AM, Daniel Holth wrote: > If you don't waste your time enforcing the uniqueness of (condition, > extra) in the list of requirements then you can pretend install: is a > single item if you want to... From dholth at gmail.com Thu Jul 4 15:45:35 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 4 Jul 2013 09:45:35 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> <2A394821-338D-47D6-B4EB-24C83B0F103A@stufft.io> <6C675963-D954-4392-ABBB-E31662E07B01@stufft.io> Message-ID: On Thu, Jul 4, 2013 at 9:36 AM, Donald Stufft wrote: > Yea I just spent significant effort cleaning up the database from a lack if enforced constraints. I will pass on not using them. > > On Jul 4, 2013, at 9:34 AM, Daniel Holth wrote: > >> If you don't waste your time enforcing the uniqueness of (condition, >> extra) in the list of requirements then you can pretend install: is a >> single item if you want to... [ { 'extra':'foo', install:['thing1']}, {'extra:'foo', install:['thing2']} ] From vinay_sajip at yahoo.co.uk Thu Jul 4 21:38:25 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 4 Jul 2013 19:38:25 +0000 (UTC) Subject: [Distutils] Upcoming changes to PEP 426/440 References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> Message-ID: Nick Coghlan gmail.com> writes: > The main benefit is that all the dependencies for an extra will typically > be in one place. > However, I briefly forgot the "machine readable" part again, and for that > TOOWTDI is to have one entry per dependency. One record per dependency is indeed the case at the RDBMS level, but there's no reason why that scheme needs to be slavishly copied over to the JSON. Let me try to illustrate this. I couldn't find any modelling code in Donald's public repo - I checked both branches and couldn't find it (Donald, please point me to it if it's on GitHub rather than just your local clone). So I knocked up a simple model (using SQLAlchemy, but the model is so simple that just about any ORM should do). The entities are Project, Release and Dependency. I've created a simple script, depmodel.py, along with two JSON files which have the relevant subset of the PEP 426 metadata for setuptools 0.7.7 and Pyramid 1.4.2. These are available at https://gist.github.com/vsajip/5929707 This code/data uses the older schema (run_requires / run_may_require, etc. and using 'dependencies' rather than 'install' as a key). This is the JSON which is supposed to be problematic, so I wanted to see what the problems might be. I couldn't find any, so I'm linking to the code here so that Donald/Nick can point out any misunderstanding on my part. The script allows importing the dependencies from JSON to RDBMS (34 lines for the import function) and also exporting from RDBMS to JSON (43 lines for the export function). I've used SQLite for the database. python depmodel.py -i setuptools-0.7.7.json will read the dependencies into SQLite, and python depmodel.py -e setuptools/0.7.7 will print the SQLite records as JSON. I understand that people might have particular preferences, but I can't see any technical reason why we couldn't have lists in the JSON. The import and export code looks pretty simple to me. What have I missed? Regards, Vinay Sajip From donald at stufft.io Fri Jul 5 01:52:06 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 4 Jul 2013 19:52:06 -0400 Subject: [Distutils] g.pypi.python.org out of date Message-ID: <6A6F245F-2E27-4540-997E-0DB20EE1A050@stufft.io> It appears the "g" mirror is ~10 days out of date. Do we know who owns it? It needs kicked to start back up or we need to remove it from the pool if we can't get it going again. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Fri Jul 5 02:09:23 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 4 Jul 2013 20:09:23 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> Message-ID: On the plus side if we're arguing about something as banal as this, maybe we are almost done! On Thu, Jul 4, 2013 at 3:38 PM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> The main benefit is that all the dependencies for an extra will typically >> be in one place. >> However, I briefly forgot the "machine readable" part again, and for that >> TOOWTDI is to have one entry per dependency. > > One record per dependency is indeed the case at the RDBMS level, but there's > no reason why that scheme needs to be slavishly copied over to the JSON. Let > me try to illustrate this. > > I couldn't find any modelling code in Donald's public repo - I checked both > branches and couldn't find it (Donald, please point me to it if it's on > GitHub rather than just your local clone). So I knocked up a simple model > (using SQLAlchemy, but the model is so simple that just about any ORM should > do). The entities are Project, Release and Dependency. I've created a simple > script, depmodel.py, along with two JSON files which have the relevant > subset of the PEP 426 metadata for setuptools 0.7.7 and Pyramid 1.4.2. These > are available at > > https://gist.github.com/vsajip/5929707 > > This code/data uses the older schema (run_requires / run_may_require, etc. > and using 'dependencies' rather than 'install' as a key). This is the JSON > which is supposed to be problematic, so I wanted to see what the problems > might be. I couldn't find any, so I'm linking to the code here so that > Donald/Nick can point out any misunderstanding on my part. > > The script allows importing the dependencies from JSON to RDBMS (34 lines > for the import function) and also exporting from RDBMS to JSON (43 lines for > the export function). I've used SQLite for the database. > > python depmodel.py -i setuptools-0.7.7.json > > will read the dependencies into SQLite, and > > python depmodel.py -e setuptools/0.7.7 > > will print the SQLite records as JSON. > > I understand that people might have particular preferences, but I can't see > any technical reason why we couldn't have lists in the JSON. The import and > export code looks pretty simple to me. What have I missed? > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From vinay_sajip at yahoo.co.uk Fri Jul 5 02:24:37 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 5 Jul 2013 00:24:37 +0000 (UTC) Subject: [Distutils] Upcoming changes to PEP 426/440 References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> Message-ID: Daniel Holth gmail.com> writes: > On the plus side if we're arguing about something as banal as this, > maybe we are almost done! I don't exactly see it as an argument - it's just a discussion (although of course we "argue" for our point of view). I don't think we're done by a long chalk, but as I see it, we might as well polish the various pieces of the puzzle as best we can as we go along. Code is more malleable than the spec, so we should try to get that to be as good as we reasonably can. Regards, Vinay Sajip From dholth at gmail.com Fri Jul 5 02:29:47 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 4 Jul 2013 20:29:47 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> Message-ID: On Thu, Jul 4, 2013 at 8:24 PM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > >> On the plus side if we're arguing about something as banal as this, >> maybe we are almost done! > > I don't exactly see it as an argument - it's just a discussion (although of > course we "argue" for our point of view). > > I don't think we're done by a long chalk, but as I see it, we might as well > polish the various pieces of the puzzle as best we can as we go along. Code > is more malleable than the spec, so we should try to get that to be as good > as we reasonably can. ... certainly not with the whole project, just with this particular piece of metadata ... From ncoghlan at gmail.com Fri Jul 5 03:45:10 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 5 Jul 2013 11:45:10 +1000 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> Message-ID: On 4 Jul 2013 22:32, "Vinay Sajip" wrote: > > Donald Stufft stufft.io> writes: > > > I would prefer a single entry. It makes the serialization format map to the > > modeling simpler, and I think it's simpler for humans too. I don't see much > > benefit to making it a list except arbitrarily adding another level of > > nesting. > > It's a question of > > { > "install": ["a", "b", "c"] > } > > versus > > { > "install": "a" > }, > { > "install": "b" > }, > { > "install": "c" > } > > and I can't see why you think the latter is in any way better. The basic problem with the list form is that allowing two representations for the same metadata makes for extra complexity we don't really want. It means we have to decide if the decomposed version (3 separate entries with one item in each install list) is still legal. What I will do is draft PEP text for the list version that explicitly declares the decomposed form non-compliant with the spec. If I think the extra complexity looks tolerable, I'll switch it over. Cheers, Nick. >IMO > implementation details (such as "it's easier for the Django ORM to map it") > should not take precedence over other considerations of > readability/simplicity. In any case, I can't see why there would be any > particular modelling problem with the scheme I've suggested. > > Is the modelling work you're doing public? I had a quick look at your > warehouse repo (github.com/dstufft/warehouse) and I don't see any models > beyond User and Email. Is that the correct location? I'd be happy to take a > closer look to get a better understanding of what modelling problem you're > seeing/foreseeing. > > FYI the metadata that I'm maintaining on red-dove.com is stored in a SQL > database. While my SQL schema is not yet fully aligned with the PEP (as it's > WIP), I don't see any modelling problem between an RDBMS backend and any of > the JSON formats which have been published in the various revisions of the > PEP. Some more detail would help :-) > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Jul 5 03:47:09 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 4 Jul 2013 21:47:09 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> Message-ID: On Jul 4, 2013, at 9:45 PM, Nick Coghlan wrote: > > What I will do is draft PEP text for the list version that explicitly declares the decomposed form non-compliant with the spec. If I think the extra complexity looks tolerable, I'll switch it over. > > What is the "decomposed form" ? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Fri Jul 5 03:50:46 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 4 Jul 2013 21:50:46 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> Message-ID: I don't think you can get around the complexity. Consider: {extra:'foo', condition:'platform == win32', install=[]} {extra:'foo', condition:'platform == linux', install=[]} They have to be flattened into a single list of all the 'foo' extras that are installable in the current environment anyway. It's exactly the same work you might try to avoid by worrying about whether install is a list. On Thu, Jul 4, 2013 at 9:45 PM, Nick Coghlan wrote: > > On 4 Jul 2013 22:32, "Vinay Sajip" wrote: >> >> Donald Stufft stufft.io> writes: >> >> > I would prefer a single entry. It makes the serialization format map to >> > the >> > modeling simpler, and I think it's simpler for humans too. I don't see >> > much >> > benefit to making it a list except arbitrarily adding another level of >> > nesting. >> >> It's a question of >> >> { >> "install": ["a", "b", "c"] >> } >> >> versus >> >> { >> "install": "a" >> }, >> { >> "install": "b" >> }, >> { >> "install": "c" >> } >> >> and I can't see why you think the latter is in any way better. > > The basic problem with the list form is that allowing two representations > for the same metadata makes for extra complexity we don't really want. It > means we have to decide if the decomposed version (3 separate entries with > one item in each install list) is still legal. > > What I will do is draft PEP text for the list version that explicitly > declares the decomposed form non-compliant with the spec. If I think the > extra complexity looks tolerable, I'll switch it over. > > Cheers, > Nick. > >>IMO >> implementation details (such as "it's easier for the Django ORM to map >> it") >> should not take precedence over other considerations of >> readability/simplicity. In any case, I can't see why there would be any >> particular modelling problem with the scheme I've suggested. >> >> Is the modelling work you're doing public? I had a quick look at your >> warehouse repo (github.com/dstufft/warehouse) and I don't see any models >> beyond User and Email. Is that the correct location? I'd be happy to take >> a >> closer look to get a better understanding of what modelling problem you're >> seeing/foreseeing. >> >> FYI the metadata that I'm maintaining on red-dove.com is stored in a SQL >> database. While my SQL schema is not yet fully aligned with the PEP (as >> it's >> WIP), I don't see any modelling problem between an RDBMS backend and any >> of >> the JSON formats which have been published in the various revisions of the >> PEP. Some more detail would help :-) >> >> Regards, >> >> Vinay Sajip >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From vinay_sajip at yahoo.co.uk Fri Jul 5 10:25:16 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 5 Jul 2013 08:25:16 +0000 (UTC) Subject: [Distutils] Upcoming changes to PEP 426/440 References: <6DB07C69-0A0B-48C7-A9A6-80BB1B76D2B5@stufft.io> Message-ID: Nick Coghlan gmail.com> writes: > The basic problem with the list form is that allowing two representations > for the same metadata makes for extra complexity we don't really want. It > means we have to decide if the decomposed version (3 separate entries > with one item in each install list) is still legal. I'm not sure how prescriptive we need to be. For example, posit metadata like: { "install": ["a", "b", "c"], "extra": "foo" }, { "install": ["d", "e", "f"], "extra": "foo" }, { "install": ["g"], "extra": "foo" } Even though there's no particular rationale for structuring it like this, the intention is clear: "a" .. "g" are dependencies when extra "foo" is specified. As long as the method by which these entries are processed is clear in the PEP, then it's not clear what's to be gained by being overly constraining. There are numerous ways in which dependency information can be represented which are not worth the effort to canonicalise. For example, the order in which extras or version constraints are declared in a dependency specifier: dist-name [foo,bar] (>= 1.0, < 2.0) and dist-name [bar,foo] (< 2.0, >= 1.0) are equivalent, but in any simplistic handling this would slip past e.g. database uniqueness constraints. More sophisticated handling (by modelling below the Dependency level) is possible, but whether it's worth it is debatable. Regards, Vinay Sajip From jaraco at jaraco.com Fri Jul 5 20:05:19 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Fri, 5 Jul 2013 18:05:19 +0000 Subject: [Distutils] Setuptools 0.8 and Distribute 0.7.3 (legacy wrapper) now released Message-ID: <216687c5a1bb4f288c29324a7466008d@BLUPR06MB003.namprd06.prod.outlook.com> The PyPA is excited to announce the public release of Setuptools 0.8. This release of setuptools provides no additional functionality over Setuptools 0.7.x except that it no longer requires 2to3 to build/install on Python 3. What this means for packaging is that tools like pip and virtualenv can now invoke setuptools directly on all supported Python versions (currently 2.4+). This build enables more natural upgrades and helps address many of the bugs that the 2to3 conversion process triggered. Additionally, Distribute 0.7.3 has also been released to PyPI. Distribute 0.7 was designed to ease the upgrade process from Distribute 0.6.x to Setuptools 0.7. This new version, 0.7.3, is a re-release of the legacy wrapper 0.7, but additionally bundles the Setuptools 0.8 code for the purposes of bootstrapping the upgrade. This version specifically eases upgrades on systems running older systems. Now, one can readily upgrade any environment with Distribute 0.6 by simply upgrading (using pip or easy_install) to Distribute 0.7.3, which will replace the 'distribute' package with an empty shell leaving setuptools >= 0.7 (probably 0.8) installed. Enjoy, and please report any issues with either of these packages at the Setuptools project page (https://bitbucket.org/pypa/setuptools). -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From jim at zope.com Fri Jul 5 20:30:49 2013 From: jim at zope.com (Jim Fulton) Date: Fri, 5 Jul 2013 14:30:49 -0400 Subject: [Distutils] Buildout 2.2.0 released Message-ID: zc.buildout 2.2.0 has been released to PyPI. The main feature of this release is support for setuptools 0.7 and later. See: https://pypi.python.org/pypi/zc.buildout/2.2.0#id3 for a full list of changes. To get this release, you need to download and run a new bootstrap.py file from: http://downloads.buildout.org/2/bootstrap.py Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From sdouche at gmail.com Sat Jul 6 03:08:00 2013 From: sdouche at gmail.com (Sebastien Douche) Date: Sat, 6 Jul 2013 03:08:00 +0200 Subject: [Distutils] Buildout 2.2.0 released In-Reply-To: References: Message-ID: On Fri, Jul 5, 2013 at 8:30 PM, Jim Fulton wrote: > zc.buildout 2.2.0 has been released to PyPI. Yeah \o/. Thanks Jim and all contributors. And big up Tres :). -- Sebastien Douche Twitter: @sdouche / G+: +sdouche From ncoghlan at gmail.com Sat Jul 6 06:07:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 6 Jul 2013 14:07:41 +1000 Subject: [Distutils] Setuptools 0.8 and Distribute 0.7.3 (legacy wrapper) now released In-Reply-To: <216687c5a1bb4f288c29324a7466008d@BLUPR06MB003.namprd06.prod.outlook.com> References: <216687c5a1bb4f288c29324a7466008d@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: On 6 Jul 2013 04:08, "Jason R. Coombs" wrote: > > The PyPA is excited to announce the public release of Setuptools 0.8. This release of setuptools provides no additional functionality over Setuptools 0.7.x except that it no longer requires 2to3 to build/install on Python 3. What this means for packaging is that tools like pip and virtualenv can now invoke setuptools directly on all supported Python versions (currently 2.4+). This build enables more natural upgrades and helps address many of the bugs that the 2to3 conversion process triggered. Great news! Thanks for working to get these out quickly to improve the upgrade path from older versions :) Cheers, Nick. > > > > Additionally, Distribute 0.7.3 has also been released to PyPI. Distribute 0.7 was designed to ease the upgrade process from Distribute 0.6.x to Setuptools 0.7. This new version, 0.7.3, is a re-release of the legacy wrapper 0.7, but additionally bundles the Setuptools 0.8 code for the purposes of bootstrapping the upgrade. This version specifically eases upgrades on systems running older systems. Now, one can readily upgrade any environment with Distribute 0.6 by simply upgrading (using pip or easy_install) to Distribute 0.7.3, which will replace the ?distribute? package with an empty shell leaving setuptools >= 0.7 (probably 0.8) installed. > > > > Enjoy, and please report any issues with either of these packages at the Setuptools project page (https://bitbucket.org/pypa/setuptools). > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From setuptools at bugs.python.org Sat Jul 6 08:52:05 2013 From: setuptools at bugs.python.org (mbogosian) Date: Sat, 06 Jul 2013 06:52:05 +0000 Subject: [Distutils] [issue152] setuptools breaks with from __future__ import unicode_literals in setup.py Message-ID: <1373093525.02.0.347910542596.issue152@psf.upfronthosting.co.za> New submission from mbogosian: unicode_literals break a bunch of stuff in setuptools. Considering they may become the default at some point, this should be fixed...? I do not know if this is related to issue 78. To reproduce, run the attached setup.py (output below). Comment out the unicode_literals line in setup.py and try it again (everything should work). % DISTUTILS_DEBUG=t python -c 'import setuptools ; print setuptools.__version__' 0.8 % unzip -d foo_test.zip ; cd foo_test ... % DISTUTILS_DEBUG=t python setup.py build options (after parsing config files): options (after parsing command line): option dict for 'aliases' command: {} option dict for 'build' command: {} option dict for 'nosetests' command: {'all_modules': ('setup.cfg', '1'), 'cover_package': ('setup.cfg', 'foo'), 'detailed_errors': ('setup.cfg', '1'), 'verbosity': ('setup.cfg', '2'), 'with_coverage': ('setup.cfg', '1'), 'with_doctest': ('setup.cfg', '1')} running build Distribution.get_command_obj(): creating 'build' command object running build_py Distribution.get_command_obj(): creating 'build_py' command object Traceback (most recent call last): File "setup.py", line 58, in setuptools.setup(**_SETUP_ARGS) File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build.py", line 127, in run self.run_command(cmd_name) File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/command/build_py.py", line 89, in run self.build_packages() File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py", line 372, in build_packages self.build_module(module, module_file, package) File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/command/build_py.py", line 106, in build_module outfile, copied = _build_py.build_module(self, module, module_file, package) File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py", line 333, in build_module "'package' must be a string (dot-separated), list, or tuple") TypeError: 'package' must be a string (dot-separated), list, or tuple % DISTUTILS_DEBUG=t python setup.py nosetests options (after parsing config files): options (after parsing command line): option dict for 'aliases' command: {} option dict for 'nosetests' command: {'all_modules': ('setup.cfg', '1'), 'cover_package': ('setup.cfg', 'foo'), 'detailed_errors': ('setup.cfg', '1'), 'verbosity': ('setup.cfg', '2'), 'with_coverage': ('setup.cfg', '1'), 'with_doctest': ('setup.cfg', '1')} running nosetests Distribution.get_command_obj(): creating 'nosetests' command object setting options for 'nosetests' command: with_coverage = 1 (from setup.cfg) verbosity = 2 (from setup.cfg) cover_package = foo (from setup.cfg) all_modules = 1 (from setup.cfg) with_doctest = 1 (from setup.cfg) detailed_errors = 1 (from setup.cfg) running egg_info Distribution.get_command_obj(): creating 'egg_info' command object Traceback (most recent call last): File "setup.py", line 58, in setuptools.setup(**_SETUP_ARGS) File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/commands.py", line 132, in run self.run_command('egg_info') File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 971, in run_command cmd_obj.ensure_finalized() File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized self.finalize_options() File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 103, in finalize_options self.ensure_dirname('egg_base') File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 269, in ensure_dirname "'%s' does not exist or is not a directory") File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 255, in _ensure_tested_string val = self._ensure_stringlike(option, what, default) File ".../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 216, in _ensure_stringlike "'%s' must be a %s (got `%s`)" % (option, what, val) distutils.errors.DistutilsOptionError: 'egg_base' must be a directory name (got `src`) % ... # Comment out unicode_literals line in setup.py % python setup.py nosetests running nosetests running egg_info writing src/foo.egg-info/PKG-INFO writing namespace_packages to src/foo.egg-info/namespace_packages.txt writing top-level names to src/foo.egg-info/top_level.txt writing dependency_links to src/foo.egg-info/dependency_links.txt reading manifest file 'src/foo.egg-info/SOURCES.txt' writing manifest file 'src/foo.egg-info/SOURCES.txt' running build_ext Doctest: foo.base.pity.Foo ... ok testFoo (test.base.pity.TestFoo) ... ok Name Stmts Miss Cover Missing --------------------------------------------- foo 3 0 100% foo.base 0 0 100% foo.base.pity 9 0 100% --------------------------------------------- TOTAL 12 0 100% ---------------------------------------------------------------------- Ran 2 tests in 0.034s OK ---------- files: foo_test.zip messages: 724 nosy: mbogosian priority: bug status: unread title: setuptools breaks with from __future__ import unicode_literals in setup.py Added file: http://bugs.python.org/setuptools/file92/foo_test.zip _______________________________________________ Setuptools tracker _______________________________________________ -------------- next part -------------- A non-text attachment was scrubbed... Name: foo_test.zip Type: application/zip Size: 4252 bytes Desc: not available URL: From a.badger at gmail.com Sat Jul 6 10:11:50 2013 From: a.badger at gmail.com (Toshio Kuratomi) Date: Sat, 6 Jul 2013 01:11:50 -0700 Subject: [Distutils] [issue152] setuptools breaks with from __future__ import unicode_literals in setup.py In-Reply-To: <1373093525.02.0.347910542596.issue152@psf.upfronthosting.co.za> References: <1373093525.02.0.347910542596.issue152@psf.upfronthosting.co.za> Message-ID: <20130706081150.GA30804@unaka.lan> On Sat, Jul 06, 2013 at 06:52:05AM +0000, mbogosian wrote: > > New submission from mbogosian: > > unicode_literals break a bunch of stuff in setuptools. Considering they may become the default at some point, this should be fixed...? I do not know if this is related to issue 78. > > To reproduce, run the attached setup.py (output below). Comment out the unicode_literals line in setup.py and try it again (everything should work). > > % DISTUTILS_DEBUG=t python -c 'import setuptools ; print setuptools.__version__' > 0.8 > % unzip -d foo_test.zip ; cd foo_test > ... > % DISTUTILS_DEBUG=t python setup.py build [snip output] > % DISTUTILS_DEBUG=t python setup.py nosetests [snip output] Not sure what the unicode model is in setuptools but one way to look at this is that in python2, the setuptools API takes byte str and in python3, the API takes unicode str. So this is a case of the setup.py being invalid. If you have: from __future__ import unicode_literals That doesn't change what the api takes as input; it only changes how you express it. So a package author who does from __future__ import unicode_literals would also need to do this to make things work: 'package_dir' : { '': b'src' }, 'packages' : setuptools.find_packages(b'src', exclude = ( b'foo', b'test', b'test.*' )), Someone else will have to speak to whether that's the intended model, though. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From vinay_sajip at yahoo.co.uk Sat Jul 6 12:19:07 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 6 Jul 2013 10:19:07 +0000 (UTC) Subject: [Distutils] =?utf-8?q?=5Bissue152=5D_setuptools_breaks_with_from_?= =?utf-8?q?=5F=5Ffuture=5F=5F=09import=09unicode=5Fliterals_in_setu?= =?utf-8?q?p=2Epy?= References: <1373093525.02.0.347910542596.issue152@psf.upfronthosting.co.za> Message-ID: mbogosian bugs.python.org> writes: > unicode_literals break a bunch of stuff in setuptools. Considering they may become the default at some > point, this should be fixed...? I do not know if this is related to issue 78. > These appear to be because distutils (not setuptools) is unduly restrictive, refusing to accept unicode when it could do so. > ".../lib/python2.7/distutils/command/build_py.py", > line 333, in build_module > "'package' must be a string (dot-separated), list, or tuple") > TypeError: 'package' must be a string (dot-separated), list, or tuple distutils code is testing for str (as opposed to list/tuple), but it could as well test for basestring here. > File ".../lib/python2.7/distutils/cmd.py", > line 216, in _ensure_stringlike > "'%s' must be a %s (got `%s`)" % (option, what, val) > distutils.errors.DistutilsOptionError: 'egg_base' must be a directory name (got `src`) Again, this is in distutils. It seems as if testing for basestring rather than str would be needed. There could well be other places in distutils where it's testing for str where basestring would do, but I'm not sure whether these would be regarded as bugs for 2.7. Regards, Vinay Sajip From qwcode at gmail.com Sat Jul 6 21:14:51 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 6 Jul 2013 12:14:51 -0700 Subject: [Distutils] pip and virtualenv release candidates In-Reply-To: References: Message-ID: pip-1.4rc3 and virtualenv-1.10rc4 are now available the changes from the previous RCs: - a bug fix for a new use of urlparse ( https://github.com/pypa/pip/pull/1032) - "pip list" doesn't ignore showing setuptools and pip anymore (although "pip freeze" still does) - the wheel setuptools requirement was changed to ">=0.8" - updated installation instructions related to the release of setuptools-0.8 (http://www.pip-installer.org/en/latest/installing.html#requirements) here's the gist of the RC install instructions again: $ curl -L -O https://github.com/pypa/virtualenv/archive/1.10rc4.tar.gz $ tar zxf 1.10rc4.tar.gz $ python virtualenv-1.10rc4/virtualenv.py myVE $ myVE/bin/pip --version pip 1.4rc3 On Tue, Jul 2, 2013 at 10:28 PM, Marcus Smith wrote: > pip-1.4rc2 and virtualenv-1.10rc3 are available for testing from github. > > A few highlights: > - pip added support for installing and building wheel archives. > ( > http://www.pip-installer.org/en/latest/cookbook.html#building-and-installing-wheels > ) > - virtualenv is now using the new merged setuptools, and no longer > supports distribute. > - pip now only installs stable versions by default, and offers a new > --pre option to also find pre-releases. > - Dropped support for Python 2.5. > > Changelogs: > pip: http://www.pip-installer.org/en/release-1.4/news.html > virtualenv: http://www.virtualenv.org/en/release-1.10/news.html > > Download Links: > pip: > gz: https://github.com/pypa/pip/archive/1.4rc2.tar.gz > md5=0426430fc8a261c83bcd083fb03fb7d6 > zip: https://github.com/pypa/pip/archive/1.4rc2.zip > md5=c86dc0d94ed787eadba6dceb06f1676f > virtualenv: > gz: https://github.com/pypa/virtualenv/archive/1.10rc3.tar.gz > md5=b24cdf59b561acf26ae3f639098d5a34 > zip: https://github.com/pypa/virtualenv/archive/1.10rc3.zip > md5=a6ee1a1570a751aa50f95833d9898649 > > Installation: > The easiest way to try them both and *not* affect your current system, is > like so: > e.g. on Linux: > $ curl -L -O https://github.com/pypa/virtualenv/archive/1.10rc3.tar.gz > $ echo "b24cdf59b561acf26ae3f639098d5a34 1.10rc3.tar.gz" | md5sum -c > 1.10rc3.tar.gz: OK > $ tar zxf 1.10rc3.tar.gz > $ python virtualenv-1.10rc3/virtualenv.py myVE > $ myVE/bin/pip --version > pip 1.4rc2 > > *Note*: If instead, you choose to upgrade an existing pip (and > setuptools), know this: > 1) pip's wheel support requires setuptools>=0.8b2 (this will become > final before pip is released final) > 2) setuptools-0.8bx is not on pypi and can be found here: > https://bitbucket.org/pypa/setuptools/downloads > 3) Older pip's can not currently upgrade distribute to setuptools > (until distribute-0.7.3 is released on ~July-7th) > (for more upgrade details: > http://www.pip-installer.org/en/latest/installing.html#requirements) > > Offering Feedback: > You can respond to this email, or log issues in our tracker: > https://github.com/pypa/pip/issues?state=open > -------------- next part -------------- An HTML attachment was scrubbed... URL: From setuptools at bugs.python.org Sun Jul 7 04:29:32 2013 From: setuptools at bugs.python.org (Tom Brennan) Date: Sun, 07 Jul 2013 02:29:32 +0000 Subject: [Distutils] [issue153] Jython - easy_install -- ssl_support.py: module has no attribute 'CERT_REQUIRED' Message-ID: <1373164172.5.0.684661114137.issue153@psf.upfronthosting.co.za> New submission from Tom Brennan: Can't install anything with easy_install or pip. Happens in at least v.0.8 and 0.7.8. (stack trace attached is from 0.7.8) ---------- files: easy_install_stack_trace.txt messages: 729 nosy: tjb1982 priority: bug status: unread title: Jython - easy_install -- ssl_support.py: module has no attribute 'CERT_REQUIRED' Added file: http://bugs.python.org/setuptools/file93/easy_install_stack_trace.txt _______________________________________________ Setuptools tracker _______________________________________________ -------------- next part -------------- Searching for pip Reading https://pypi.python.org/simple/pip/ Traceback (most recent call last): File "/home/tjb1982/jython2.7/bin/easy_install", line 8, in sys.exit( File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/command/easy_install.py", line 1978, in main File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/command/easy_install.py", line 1965, in with_ei_usage File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/command/easy_install.py", line 1978, in File "/home/tjb1982/jython2.7/Lib/distutils/core.py", line 152, in setup dist.run_commands() File "/home/tjb1982/jython2.7/Lib/distutils/core.py", line 152, in setup dist.run_commands() File "/home/tjb1982/jython2.7/Lib/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/home/tjb1982/jython2.7/Lib/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/command/easy_install.py", line 373, in run File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/command/easy_install.py", line 602, in easy_install File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/package_index$py.class", line 523, in fetch_distribution File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/package_index$py.class", line 359, in find_packages File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/package_index$py.class", line 698, in scan_url File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/package_index$py.class", line 236, in process_url File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/package_index$py.class", line 639, in open_url File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/package_index$py.class", line 639, in open_url File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/package_index$py.class", line 873, in _socket_timeout File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/package_index$py.class", line 920, in open_with_auth File "/home/tjb1982/jython2.7/Lib/urllib2.py", line 400, in open response = self._open(req, data) File "/home/tjb1982/jython2.7/Lib/urllib2.py", line 417, in _open result = self._call_chain(self.handle_open, protocol, protocol + File "/home/tjb1982/jython2.7/Lib/urllib2.py", line 378, in _call_chain result = func(*args) File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/ssl_support$py.class", line 175, in https_open File "/home/tjb1982/jython2.7/Lib/urllib2.py", line 1174, in do_open h.request(req.get_method(), req.get_selector(), req.data, headers) File "/home/tjb1982/jython2.7/Lib/urllib2.py", line 1174, in do_open h.request(req.get_method(), req.get_selector(), req.data, headers) File "/home/tjb1982/jython2.7/Lib/httplib.py", line 958, in request self._send_request(method, url, body, headers) File "/home/tjb1982/jython2.7/Lib/httplib.py", line 992, in _send_request self.endheaders(body) File "/home/tjb1982/jython2.7/Lib/httplib.py", line 954, in endheaders self._send_output(message_body) File "/home/tjb1982/jython2.7/Lib/httplib.py", line 814, in _send_output self.send(msg) File "/home/tjb1982/jython2.7/Lib/httplib.py", line 776, in send self.connect() File "/home/tjb1982/jython2.7/Lib/site-packages/setuptools-0.7.8-py2.7.egg/setuptools/ssl_support$py.class", line 189, in connect AttributeError: 'module' object has no attribute 'CERT_REQUIRED' From chris at simplistix.co.uk Sun Jul 7 09:08:37 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Sun, 07 Jul 2013 08:08:37 +0100 Subject: [Distutils] buildout/setuptools/distribute unhelpful error message (0.7.x issue?) Message-ID: <51D913F5.3080804@simplistix.co.uk> Hi All, What is this exception trying to tell me? Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-0.7.2.tar.gz Extracting in /tmp/tmpJNVsOY Now working in /tmp/tmpJNVsOY/setuptools-0.7.2 Building a Setuptools egg in /tmp/tmpBLZGeg /tmp/tmpBLZGeg/setuptools-0.7.2-py2.6.egg Traceback (most recent call last): File "bootstrap.py", line 91, in pkg_resources.working_set.add_entry(path) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 451, in add_entry self.add(dist, entry, False) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 542, in add self._added_new(dist) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 705, in _added_new callback(dist) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2727, in add_activation_listener(lambda dist: dist.activate()) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2227, in activate self.insert_on(path) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2328, in insert_on "with distribute. Found one at %s" % str(self.location)) ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /tmp/tmpBLZGeg/setuptools-0.7.2-py2.6.egg I don't see any distribute in there, and I don't know where it might be... $ python2.6 Python 2.6.8 (unknown, Jan 29 2013, 10:05:44) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import setuptools Traceback (most recent call last): File "", line 1, in ImportError: No module named setuptools cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From tseaver at palladion.com Sun Jul 7 15:32:08 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 07 Jul 2013 09:32:08 -0400 Subject: [Distutils] buildout/setuptools/distribute unhelpful error message (0.7.x issue?) In-Reply-To: <51D913F5.3080804@simplistix.co.uk> References: <51D913F5.3080804@simplistix.co.uk> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/07/2013 03:08 AM, Chris Withers wrote: > > I don't see any distribute in there, and I don't know where it might > be... 'pkg_resources' (/usr/lib/python2.6/dist-packages/pkg_resources.py) comes from either setuptools or distribute -- in your case, distribute (probably via a 'python-distribute' Debian package, given that path). Either you need to get that system pacakge updated, or else removed, or else insulate yourself from it (e.g., with a 'virtualenv --no-setuptools' or a self-built Python). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHZbdIACgkQ+gerLs4ltQ7y1ACfSuEQBXY/THSNDsoLysgORTQv HeQAoI5TH6WOqNG3TR9Fuu4VKURpeil2 =kkLS -----END PGP SIGNATURE----- From pje at telecommunity.com Sun Jul 7 21:32:42 2013 From: pje at telecommunity.com (PJ Eby) Date: Sun, 7 Jul 2013 15:32:42 -0400 Subject: [Distutils] [issue153] Jython - easy_install -- ssl_support.py: module has no attribute 'CERT_REQUIRED' In-Reply-To: <1373164172.5.0.684661114137.issue153@psf.upfronthosting.co.za> References: <1373164172.5.0.684661114137.issue153@psf.upfronthosting.co.za> Message-ID: On Sat, Jul 6, 2013 at 10:29 PM, Tom Brennan wrote: > > New submission from Tom Brennan: > > Can't install anything with easy_install or pip. Happens in at least v.0.8 and 0.7.8. (stack trace attached is from 0.7.8) > > ---------- > files: easy_install_stack_trace.txt > messages: 729 > nosy: tjb1982 > priority: bug > status: unread > title: Jython - easy_install -- ssl_support.py: module has no attribute 'CERT_REQUIRED' > Added file: http://bugs.python.org/setuptools/file93/easy_install_stack_trace.txt It appears that Jython's ssl module is not 100% compatible with Python's. I'm not sure what to do about that. ( By the way, bugs for setuptools 0.7 and higher should be reported to the issue tracker at https://bitbucket.org/pypa/setuptools ) From ct at gocept.com Mon Jul 8 09:55:32 2013 From: ct at gocept.com (Christian Theune) Date: Mon, 8 Jul 2013 09:55:32 +0200 Subject: [Distutils] bandersnatch 1.0.2: CDN compatibility for mirrors Message-ID: Hi, over the last weeks I worked with Donald to get mirrors working consistently again. I updated the bandersnatch mirror client and the 1.0.2 release has been running on my mirror (f.pypi.python.org) for a few days correctly. If you're running a mirror, please get it and report any further errors you should encounter to the bitbucket issue page: https://bitbucket.org/ctheune/bandersnatch/issues Cheers, Christian Small caveat, nothing big: the statistics that are generated by pypi-mirrors.org show a reduced number of packages available on my mirror, however, that is due to the fact that the master reports a number that includes all packages even without releases. The master serves 404s for their simple pages. Bandersnatch, thus, does not keep their directories. To support the CDN I have to generate the big simple index page myself. The only reliable way to do this is to list all simple pages I actually have. As packages without releases don't have simple pages -> back to square one. From reinout at vanrees.org Mon Jul 8 12:29:40 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Mon, 08 Jul 2013 12:29:40 +0200 Subject: [Distutils] Setuptools 0.8 and Distribute 0.7.3 (legacy wrapper) now released In-Reply-To: <216687c5a1bb4f288c29324a7466008d@BLUPR06MB003.namprd06.prod.outlook.com> References: <216687c5a1bb4f288c29324a7466008d@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: On 05-07-13 20:05, Jason R. Coombs wrote: > > Additionally, Distribute 0.7.3 has also been released to PyPI. > Distribute 0.7 was designed to ease the upgrade process from Distribute > 0.6.x to Setuptools 0.7. This new version, 0.7.3, is a re-release of the > legacy wrapper 0.7, but additionally bundles the Setuptools 0.8 code for > the purposes of bootstrapping the upgrade. This version specifically > eases upgrades on systems running older systems. Now, one can readily > upgrade any environment with Distribute 0.6 by simply upgrading (using > pip or easy_install) to Distribute 0.7.3, which will replace the > ?distribute? package with an empty shell leaving setuptools >= 0.7 > (probably 0.8) installed. I tried this on OSX and ran into a problem: $ sudo pip install -U distribute Password: Downloading/unpacking distribute from https://pypi.python.org/packages/source/d/distribute/distribute-0.7.3.zip#md5=c6c59594a7b180af57af8a0cc0cf5b4a Downloading distribute-0.7.3.zip (145Kb): 145Kb downloaded Running setup.py egg_info for package distribute Downloading/unpacking setuptools>=0.7 (from distribute) Downloading setuptools-0.8.tar.gz (756Kb): 756Kb downloaded Running setup.py egg_info for package setuptools Installing collected packages: distribute, setuptools Found existing installation: distribute 0.6.28 Uninstalling distribute: Successfully uninstalled distribute Running setup.py install for distribute Found existing installation: distribute 0.6.28 Exception: Traceback (most recent call last): File "/Library/Python/2.7/site-packages/pip-1.1-py2.7.egg/pip/basecommand.py", line 104, in main status = self.run(options, args) File "/Library/Python/2.7/site-packages/pip-1.1-py2.7.egg/pip/commands/install.py", line 250, in run requirement_set.install(install_options, global_options) File "/Library/Python/2.7/site-packages/pip-1.1-py2.7.egg/pip/req.py", line 1129, in install requirement.uninstall(auto_confirm=True) File "/Library/Python/2.7/site-packages/pip-1.1-py2.7.egg/pip/req.py", line 477, in uninstall config.readfp(FakeFile(dist.get_metadata_lines('entry_points.txt'))) File "build/bdist.macosx-10.8-intel/egg/pkg_resources.py", line 1213, in get_metadata_lines File "build/bdist.macosx-10.8-intel/egg/pkg_resources.py", line 1205, in get_metadata File "build/bdist.macosx-10.8-intel/egg/pkg_resources.py", line 1270, in _get IOError: zipimport: can not open file /Library/Python/2.7/site-packages/distribute-0.6.28-py2.7.egg So it downloaded 0.7.3 just fine, but barfed on a non-zipfile distribute 0.6.28. The directory didn't exist afterwards, so I assume the upgrade process somehow removed it (or it got confused because it was or wasn't a zipfile). Afterwards nothing pip-related worked as it missed setuptools. The `/usr/bin/easy_install setuptools` also didn't work. So I had to set it up anew, which worked: $ wget https://bitbucket.org/pypa/setuptools/raw/0.8/ez_setup.py $ sudo /usr/bin/python ez_setup.py Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From lgautier at gmail.com Mon Jul 8 13:01:25 2013 From: lgautier at gmail.com (Laurent Gautier) Date: Mon, 08 Jul 2013 13:01:25 +0200 Subject: [Distutils] Python packaging + conditional use of C library shipped with the package Message-ID: <51DA9C05.5050802@gmail.com> Hi, I would like to make a package that is able to use a system's given C-library if found, or compile its own version shipped with the package. This is bringing up two questions: - Is there a distutils/distribute facility to help test for the presence (and version) of a C library, or do I have to roll my own system ? - When having the source for a C library shipping with a package, is there a way to get distutils/distribute compile it, and get it seen by Python at runtime (so I can just use ctypes, or cffi, and even C extensions in other Python package see the headers and compiled libraries) ? Best, Laurent PS: I have noticed that pyzmq seems to be doing a lot of that, but they also appear to have a relatively big set of custom code do so (setup.py is already a little longer than most install scripts, to which a complete package 'buildutils' should be added)... this makes me anticipate that the answer to my two questions will be "no" ( :/ ) but I'd like a confirmation from the experts. From g2p.code at gmail.com Mon Jul 8 13:32:31 2013 From: g2p.code at gmail.com (Gabriel de Perthuis) Date: Mon, 8 Jul 2013 11:32:31 +0000 (UTC) Subject: [Distutils] Python packaging + conditional use of C library shipped with the package References: <51DA9C05.5050802@gmail.com> Message-ID: On Mon, 08 Jul 2013 13:01:25 +0200, Laurent Gautier wrote: > Hi, > > I would like to make a package that is able to use a system's given > C-library if found, or compile its own version shipped with the package. > > This is bringing up two questions: > > - Is there a distutils/distribute facility to help test for the presence > (and version) of a C library, or do I have to roll my own system ? No. Here is a rather clean script that encapsulates pkg-config: http://git.enlightenment.org/bindings/python/python-efl.git/tree/setup.py > - When having the source for a C library shipping with a package, is > there a way to get distutils/distribute compile it, and get it seen by > Python at runtime (so I can just use ctypes, or cffi, and even C > extensions in other Python package see the headers and compiled libraries) ? I don't think you should do any kind of bundling. That will easily decuple the complexity of your setup scripts, for a subpar result. Point to the C project and let people install it as they will. From ct at gocept.com Mon Jul 8 15:09:21 2013 From: ct at gocept.com (Christian Theune) Date: Mon, 8 Jul 2013 15:09:21 +0200 Subject: [Distutils] bandersnatch 1.0.2: CDN compatibility for mirrors References: Message-ID: On 2013-07-08 07:55:32 +0000, Christian Theune said: > Hi, > > over the last weeks I worked with Donald to get mirrors working > consistently again. > > I updated the bandersnatch mirror client and the 1.0.2 release has been > running on my mirror (f.pypi.python.org) for a few days correctly. > > If you're running a mirror, please get it and report any further errors > you should encounter to the bitbucket issue page: > https://bitbucket.org/ctheune/bandersnatch/issues Someone pointed out that I managed to create a brownbag release. The 1.0.2 code is fine, but I managed to screw up releasing an updated requirements.txt and updating the "stable" tag. 1.0.3 fixes this. Christian From tk47 at students.poly.edu Mon Jul 8 14:47:50 2013 From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy) Date: Mon, 8 Jul 2013 20:47:50 +0800 Subject: [Distutils] bandersnatch 1.0.2: CDN compatibility for mirrors In-Reply-To: References: Message-ID: <51DAB4F6.5090502@students.poly.edu> Christian, I don't think I see 1.0.2 yet, it seems to be still 1.0.1 for me. I use this command to upgrade --- is it obsolete? pip install --upgrade -r https://bitbucket.org/ctheune/bandersnatch/raw/stable/requirements.txt On 07/08/2013 03:55 PM, Christian Theune wrote: > Hi, > > over the last weeks I worked with Donald to get mirrors working > consistently again. > > I updated the bandersnatch mirror client and the 1.0.2 release has > been running on my mirror (f.pypi.python.org) for a few days correctly. > > If you're running a mirror, please get it and report any further > errors you should encounter to the bitbucket issue page: > https://bitbucket.org/ctheune/bandersnatch/issues > > Cheers, > Christian > > Small caveat, nothing big: the statistics that are generated by > pypi-mirrors.org show a reduced number of packages available on my > mirror, however, that is due to the fact that the master reports a > number that includes all packages even without releases. The master > serves 404s for their simple pages. Bandersnatch, thus, does not keep > their directories. To support the CDN I have to generate the big > simple index page myself. The only reliable way to do this is to list > all simple pages I actually have. As packages without releases don't > have simple pages -> back to square one. > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From zooko at zooko.com Mon Jul 8 17:33:05 2013 From: zooko at zooko.com (zooko) Date: Mon, 8 Jul 2013 19:33:05 +0400 Subject: [Distutils] Python packaging + conditional use of C library shipped with the package In-Reply-To: <51DA9C05.5050802@gmail.com> References: <51DA9C05.5050802@gmail.com> Message-ID: <20130708153304.GA18441@zooko.com> On Mon, Jul 08, 2013 at 01:01:25PM +0200, Laurent Gautier wrote: > Hi, > > I would like to make a package that is able to use a system's given > C-library if found, or compile its own version shipped with the > package. We do something similar in pycryptopp, but instead of automatically testing for the locally-available C library, we just ask the human to manually pass "--disable-embedded-cryptopp" if they want it to attempt to link to a library external to its own bundled one: https://tahoe-lafs.org/trac/pycryptopp/browser/git/setup.py?annotate=blame&rev=f789ed951b49b33e7cc49d16fdc8b398f7ec7223 > - Is there a distutils/distribute facility to help test for the > presence (and version) of a C library, or do I have to roll my own > system ? If you succeed at this, I'd like to know how you did it! Maybe we could do something similar for pycryptopp. > - When having the source for a C library shipping with a package, is > there a way to get distutils/distribute compile it, and get it seen > by Python at runtime (so I can just use ctypes, or cffi, and even C > extensions in other Python package see the headers and compiled > libraries) ? I don't understand the question. This sounds like the normal thing that distutils has always done for modules made up of compiled C code. By the way, if I were starting pycryptopp today I would use cffi. (And I would name it "crpyto".) Regards, Zooko From lgautier at gmail.com Mon Jul 8 18:26:45 2013 From: lgautier at gmail.com (Laurent Gautier) Date: Mon, 08 Jul 2013 18:26:45 +0200 Subject: [Distutils] Python packaging + conditional use of C library shipped with the package In-Reply-To: <20130708153304.GA18441@zooko.com> References: <51DA9C05.5050802@gmail.com> <20130708153304.GA18441@zooko.com> Message-ID: <51DAE845.9090501@gmail.com> On 07/08/2013 05:33 PM, zooko wrote: > On Mon, Jul 08, 2013 at 01:01:25PM +0200, Laurent Gautier wrote: >> Hi, >> >> I would like to make a package that is able to use a system's given >> C-library if found, or compile its own version shipped with the >> package. > We do something similar in pycryptopp, but instead of automatically testing for > the locally-available C library, we just ask the human to manually pass > "--disable-embedded-cryptopp" if they want it to attempt to link to a library > external to its own bundled one: > > https://tahoe-lafs.org/trac/pycryptopp/browser/git/setup.py?annotate=blame&rev=f789ed951b49b33e7cc49d16fdc8b398f7ec7223 Thanks for this, I'll look at it. >> - Is there a distutils/distribute facility to help test for the >> presence (and version) of a C library, or do I have to roll my own >> system ? > If you succeed at this, I'd like to know how you did it! Maybe we could do > something similar for pycryptopp. pyzmq seems to be doing this sort of thing (from a quick look, it seems like they are implementing something similar to what autoconf can do), but that's a lot of work if each project must reimplement its own. > >> - When having the source for a C library shipping with a package, is >> there a way to get distutils/distribute compile it, and get it seen >> by Python at runtime (so I can just use ctypes, or cffi, and even C >> extensions in other Python package see the headers and compiled >> libraries) ? > I don't understand the question. This sounds like the normal thing that > distutils has always done for modules made up of compiled C code. May be I was not clear enough. What I mean is the C library is just a C library, not a C-extension to Python. For example: mypackage/ clib/ myClibrary.c myClibrary.h src/ mypackage.py (calling myClibrary.so) myotherpackage/ clib/ myotherClibrary.c (requiring myClibrary.h, and obviously link to myClibrary.so) I'll look at the code example you are giving and see it this is automagically taken care of by the package installation system. L. > > By the way, if I were starting pycryptopp today I would use cffi. (And I would > name it "crpyto".) > > Regards, > > Zooko From r1chardj0n3s at gmail.com Wed Jul 10 05:16:47 2013 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Wed, 10 Jul 2013 13:16:47 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated Message-ID: [firstly, my apologies for posting the announcement yesterday of the pip bootstrap implementation and PEP updates to the pypa-dev list instead of distutils-sig... I blame PyCon AU exhaustion :-)] Firstly, I've just made some additional changes to PEP 439 to include: - installing virtualenv as well (so now pip, setuptools and virtualenv are installed) - mention the possibility of inclusion in a future Python 2.7 release - clarify the SSL certificate situation The bootstrap code has also been updated to: - not run the full pip command if it's "pip3 install setuptools" or either of the other two packages it has just installed (thus preventing a possibly confusing message to the user) - also install virtualenv The intention is that the pip, setuptools and actually all Python projects will promote a single bootstrap process: "pip3 install setuptools" or "pip3 install Django" And then there's instructions for getting "pip" if it's not installed. Exact wording etc. to be determined :-) The original message I sent to pypa-dev yesterday is below: The bootstrap that I wrote at the PyCon AU sprints to implement PEP 439 has been added to pypa on bitbucket: https://bitbucket.org/pypa/bootstrap I've also updated the PEP with the following changes: - mention current plans for HTTPS cert verification in Python 3.4+ (sans PEP reference for now) - remove setuptools note; setuptools will now be installed - mention bootstrapping target (user vs. system) and command-line options - mention python 2.6+ bootstrap possibility - remove consideration of support for unnecessary installation options (beyond -i/--index-url) - mention having pip default to --user when itself installed in ~/.local What the last item alludes to is the idea that it'd be nice if pip installed in ~/.local would default to installing packages also in ~/.local as though the --user switch had been provided. Otherwise the user needs to remember it every time they install a package. Note that the bootstrapping uses two different flags to control where the pip implementation is installed: --bootstrap and --bootstrap-system (these were chosen to encourage user installs). It would be ideal if pip could support those flags, as the pip3 command currently must remove them before invoking pip main. Once we're happy with the shape of pip3 we can fork it to Python 2 and use it as the canonical bootstrap script for installing pip and setuptools. I think we should also consider installing virtualenv in Python 2... Happy to clarify where needed and code review is welcome. It's been a looong four days here :-) Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Jul 10 05:20:38 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 9 Jul 2013 23:20:38 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: Message-ID: On Jul 9, 2013, at 11:16 PM, Richard Jones wrote: > [firstly, my apologies for posting the announcement yesterday of the pip bootstrap implementation and PEP updates to the pypa-dev list instead of distutils-sig... I blame PyCon AU exhaustion :-)] > > Firstly, I've just made some additional changes to PEP 439 to include: > > - installing virtualenv as well (so now pip, setuptools and virtualenv are installed) doesn't "PyEnv" which is bundled with Python 3.3+ replace virtualenv? What's the purpose of including virtualenv in the bootstrap? http://www.python.org/dev/peps/pep-0405/ > - mention the possibility of inclusion in a future Python 2.7 release > - clarify the SSL certificate situation > > The bootstrap code has also been updated to: > > - not run the full pip command if it's "pip3 install setuptools" or either of the other two packages it has just installed (thus preventing a possibly confusing message to the user) > - also install virtualenv > > The intention is that the pip, setuptools and actually all Python projects will promote a single bootstrap process: > > "pip3 install setuptools" or "pip3 install Django" > > And then there's instructions for getting "pip" if it's not installed. Exact wording etc. to be determined :-) > > The original message I sent to pypa-dev yesterday is below: > > The bootstrap that I wrote at the PyCon AU sprints to implement PEP 439 has been added to pypa on bitbucket: > > https://bitbucket.org/pypa/bootstrap > > I've also updated the PEP with the following changes: > > - mention current plans for HTTPS cert verification in Python 3.4+ (sans PEP reference for now) > - remove setuptools note; setuptools will now be installed > - mention bootstrapping target (user vs. system) and command-line options > - mention python 2.6+ bootstrap possibility > - remove consideration of support for unnecessary installation options (beyond -i/--index-url) > - mention having pip default to --user when itself installed in ~/.local > > What the last item alludes to is the idea that it'd be nice if pip installed in ~/.local would default to installing packages also in ~/.local as though the --user switch had been provided. Otherwise the user needs to remember it every time they install a package. > > Note that the bootstrapping uses two different flags to control where the pip implementation is installed: --bootstrap and --bootstrap-system (these were chosen to encourage user installs). It would be ideal if pip could support those flags, as the pip3 command currently must remove them before invoking pip main. > > Once we're happy with the shape of pip3 we can fork it to Python 2 and use it as the canonical bootstrap script for installing pip and setuptools. I think we should also consider installing virtualenv in Python 2... > > Happy to clarify where needed and code review is welcome. It's been a looong four days here :-) > > > Richard > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From r1chardj0n3s at gmail.com Wed Jul 10 05:47:44 2013 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Wed, 10 Jul 2013 13:47:44 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: Message-ID: On 10 July 2013 13:20, Donald Stufft wrote: > On Jul 9, 2013, at 11:16 PM, Richard Jones wrote: > Firstly, I've just made some additional changes to PEP 439 to include: > > - installing virtualenv as well (so now pip, setuptools and virtualenv are installed) > > > doesn't "PyEnv" which is bundled with Python 3.3+ replace virtualenv? What's the purpose of including virtualenv in the bootstrap? http://www.python.org/dev/peps/pep-0405/ It's my understanding that people still install virtualenv in py3k. Richard From donald at stufft.io Wed Jul 10 06:18:53 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 10 Jul 2013 00:18:53 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: Message-ID: <5C868F28-F2E9-4960-9E9C-2C766F7F15D2@stufft.io> On Jul 9, 2013, at 11:47 PM, Richard Jones wrote: > On 10 July 2013 13:20, Donald Stufft wrote: >> On Jul 9, 2013, at 11:16 PM, Richard Jones wrote: >> Firstly, I've just made some additional changes to PEP 439 to include: >> >> - installing virtualenv as well (so now pip, setuptools and virtualenv are installed) >> >> >> doesn't "PyEnv" which is bundled with Python 3.3+ replace virtualenv? What's the purpose of including virtualenv in the bootstrap? http://www.python.org/dev/peps/pep-0405/ > > It's my understanding that people still install virtualenv in py3k. > > > Richard I just talked to Carl. He basically said that for 3.3+ pyenv itself should probably used and that "hopefully virtualenv will die in favor of of pyenv". Another reason I think that the bootstrap script shouldn't install virtualenv is that of scope. The point of bootstrapping was to make it so pip could be "included" with Python without actually including it. As far as i'm personally concerned it should concern itself with installing pip and setuptools (assuming we can't make setuptools optional in pip or bundled?). We don't need virtualenv to enable ``pip3 install foo`` so it shouldn't be installing it. Otoh it would be nicer if PyEnv was taken to integrate with pip (although this is possibly a different pip) in that when creating a new environment if pip has already been installed in the "parent" environment it would be copied over into the pyenv created environment. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From r1chardj0n3s at gmail.com Wed Jul 10 06:37:47 2013 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Wed, 10 Jul 2013 14:37:47 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: <5C868F28-F2E9-4960-9E9C-2C766F7F15D2@stufft.io> References: <5C868F28-F2E9-4960-9E9C-2C766F7F15D2@stufft.io> Message-ID: On 10 July 2013 14:18, Donald Stufft wrote: > On Jul 9, 2013, at 11:47 PM, Richard Jones wrote: >> On 10 July 2013 13:20, Donald Stufft wrote: >>> On Jul 9, 2013, at 11:16 PM, Richard Jones wrote: >>> Firstly, I've just made some additional changes to PEP 439 to include: >>> >>> - installing virtualenv as well (so now pip, setuptools and virtualenv are installed) >>> >>> >>> doesn't "PyEnv" which is bundled with Python 3.3+ replace virtualenv? What's the purpose of including virtualenv in the bootstrap? http://www.python.org/dev/peps/pep-0405/ >> >> It's my understanding that people still install virtualenv in py3k. > > > I just talked to Carl. He basically said that for 3.3+ pyenv itself should probably used and that "hopefully virtualenv will die in favor of of pyenv". OK, thanks. I wonder whether virtualenv.org could mention pyvenv for Py3k users? > Another reason I think that the bootstrap script shouldn't install virtualenv is that of scope. The point of bootstrapping was to make it so pip could be "included" with Python without actually including it. As far as i'm personally concerned it should concern itself with installing pip and setuptools (assuming we can't make setuptools optional in pip or bundled?). We don't need virtualenv to enable ``pip3 install foo`` so it shouldn't be installing it. pip without virtualenv in python 2 contexts is pretty rare (or at least *should* be ) so I think I'll retain it in that bootstrap code. > Otoh it would be nicer if PyEnv was taken to integrate with pip (although this is possibly a different pip) in that when creating a new environment if pip has already been installed in the "parent" environment it would be copied over into the pyenv created environment. There's also the idea I mentioned yesterday: if pip is installed to the user local site-packages then it would be really good if pip's installs could also default to that rather than the system site-packages. In fact I consider it a bug that it does not, and I hope the pip devs will come to think that too :-) Richard From donald at stufft.io Wed Jul 10 06:40:51 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 10 Jul 2013 00:40:51 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <5C868F28-F2E9-4960-9E9C-2C766F7F15D2@stufft.io> Message-ID: On Jul 10, 2013, at 12:37 AM, Richard Jones wrote: > On 10 July 2013 14:18, Donald Stufft wrote: >> On Jul 9, 2013, at 11:47 PM, Richard Jones wrote: >>> On 10 July 2013 13:20, Donald Stufft wrote: >>>> On Jul 9, 2013, at 11:16 PM, Richard Jones wrote: >>>> Firstly, I've just made some additional changes to PEP 439 to include: >>>> >>>> - installing virtualenv as well (so now pip, setuptools and virtualenv are installed) >>>> >>>> >>>> doesn't "PyEnv" which is bundled with Python 3.3+ replace virtualenv? What's the purpose of including virtualenv in the bootstrap? http://www.python.org/dev/peps/pep-0405/ >>> >>> It's my understanding that people still install virtualenv in py3k. >> >> >> I just talked to Carl. He basically said that for 3.3+ pyenv itself should probably used and that "hopefully virtualenv will die in favor of of pyenv". > > OK, thanks. I wonder whether virtualenv.org could mention pyvenv for Py3k users? Probably! > > >> Another reason I think that the bootstrap script shouldn't install virtualenv is that of scope. The point of bootstrapping was to make it so pip could be "included" with Python without actually including it. As far as i'm personally concerned it should concern itself with installing pip and setuptools (assuming we can't make setuptools optional in pip or bundled?). We don't need virtualenv to enable ``pip3 install foo`` so it shouldn't be installing it. > > pip without virtualenv in python 2 contexts is pretty rare (or at > least *should* be ) so I think I'll retain it in that bootstrap > code. Ok, I don't really care enough about that minor scope creep to object too heavily :) > > >> Otoh it would be nicer if PyEnv was taken to integrate with pip (although this is possibly a different pip) in that when creating a new environment if pip has already been installed in the "parent" environment it would be copied over into the pyenv created environment. > > There's also the idea I mentioned yesterday: if pip is installed to > the user local site-packages then it would be really good if pip's > installs could also default to that rather than the system > site-packages. In fact I consider it a bug that it does not, and I > hope the pip devs will come to think that too :-) I don't have an opinion on this as I can't think of a single time I (personally) want to use the user local site-packages so that'd be something to convince the other pip devs of :D > > > Richard ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From carl at oddbird.net Wed Jul 10 06:19:01 2013 From: carl at oddbird.net (Carl Meyer) Date: Tue, 09 Jul 2013 22:19:01 -0600 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: Message-ID: <51DCE0B5.6030506@oddbird.net> Hi Richard, On 07/09/2013 09:47 PM, Richard Jones wrote: > On 10 July 2013 13:20, Donald Stufft wrote: >> On Jul 9, 2013, at 11:16 PM, Richard Jones wrote: >> Firstly, I've just made some additional changes to PEP 439 to include: >> >> - installing virtualenv as well (so now pip, setuptools and virtualenv are installed) >> >> >> doesn't "PyEnv" which is bundled with Python 3.3+ replace virtualenv? What's the purpose of including virtualenv in the bootstrap? http://www.python.org/dev/peps/pep-0405/ > > It's my understanding that people still install virtualenv in py3k. They certainly do today, but that's primarily because pyvenv isn't very useful yet, since the stdlib has no installer and thus a newly-created pyvenv has no way to install anything in it. The bootstrap should fix this very problem (i.e. make an installer available in every newly-created pyvenv) and thus encourage use of pyvenv (which is simpler, more reliable, and built-in) in place of virtualenv. I don't think it makes sense for the stdlib bootstrapper to install an inferior third-party tool instead of using a tool that is now built-in to the standard library on 3.3+. Certainly if the bootstrap is ever ported to 2.7 or 3.2, it would make sense for it to install virtualenv there (or, probably even better, for pyvenv to be backported along with the bootstrap). Carl From richard at python.org Wed Jul 10 06:54:18 2013 From: richard at python.org (Richard Jones) Date: Wed, 10 Jul 2013 14:54:18 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: <51DCE0B5.6030506@oddbird.net> References: <51DCE0B5.6030506@oddbird.net> Message-ID: On 10 July 2013 14:19, Carl Meyer wrote: > They certainly do today, but that's primarily because pyvenv isn't very > useful yet, since the stdlib has no installer and thus a newly-created > pyvenv has no way to install anything in it. Ah, thanks for clarifying that. > Certainly if the bootstrap is ever ported to 2.7 or 3.2, it would make > sense for it to install virtualenv there (or, probably even better, for > pyvenv to be backported along with the bootstrap). I intend to create two forks; one for consideration in a 2.7.5 release as "pip" and the other for users of 2.6+ called "get-pip.py". Richard From p.f.moore at gmail.com Wed Jul 10 08:28:21 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 10 Jul 2013 07:28:21 +0100 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: <51DCE0B5.6030506@oddbird.net> References: <51DCE0B5.6030506@oddbird.net> Message-ID: On 10 July 2013 05:19, Carl Meyer wrote: > > It's my understanding that people still install virtualenv in py3k. > > They certainly do today, but that's primarily because pyvenv isn't very > useful yet, since the stdlib has no installer and thus a newly-created > pyvenv has no way to install anything in it. > One other problem I have, personally, with pyvenv, is that the activate code for powershell is significantly less user-friendly than that in virtualenv. Add to that the fact that the Python release cycle is significantly slower than that of virtualenv, and using dev versions of Python is far less practical for day to day use, and that's why I stick to virtualenv at the moment (that and the pip point mentioned already). I really ought to post a patch for Python to upgrade the activate script to use the one from virtualenv. Are there any licensing/ownership issues that might make this problematic? For example, the script is signed and I don't know if that signature is attributable to someone specific... Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Wed Jul 10 10:28:22 2013 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 10 Jul 2013 04:28:22 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/09/2013 11:20 PM, Donald Stufft wrote: > doesn't "PyEnv" which is bundled with Python 3.3+ replace virtualenv? > What's the purpose of including virtualenv in the bootstrap? > http://www.python.org/dev/peps/pep-0405/ Environments generated by pyvenv lack setuptools, which makes them un-useful compared to those generated by virtualenv. Virtualenv is also useful across the important set of Python versions (2.6, 2.7, 3.2, 3.3), which pyvenv (or any shipped-in-core varieant) can never be. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHdGyYACgkQ+gerLs4ltQ5gMQCfZuHj7XyIWv+Wru0rA5VTk//1 JxkAoILDxz0Yn8zOLWP0jOGCc/gDikY8 =15US -----END PGP SIGNATURE----- From vinay_sajip at yahoo.co.uk Wed Jul 10 11:08:37 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 10 Jul 2013 09:08:37 +0000 (UTC) Subject: [Distutils] PEP 439 and pip bootstrap updated References: <5C868F28-F2E9-4960-9E9C-2C766F7F15D2@stufft.io> Message-ID: Richard Jones gmail.com> writes: > pip without virtualenv in python 2 contexts is pretty rare (or at > least *should* be ) so I think I'll retain it in that bootstrap > code. Perhaps I misunderstand, but what's the relevance of Python 2 contexts here? Aren't we talking about Python 3.4 and later? I agree with Donald's suggestion that virtualenv *not* be included, or are you saying that you want to include it for those users who have 3.4 *and* 2.x installed? If you include virtualenv, it makes it possible for people to mistakenly use it even though the recommended approach is to use the built-in venv support in Python. Exactly what benefit does including virtualenv provide in a 3.4+ installation? Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Wed Jul 10 11:16:47 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 10 Jul 2013 09:16:47 +0000 (UTC) Subject: [Distutils] PEP 439 and pip bootstrap updated References: <51DCE0B5.6030506@oddbird.net> Message-ID: Carl Meyer oddbird.net> writes: > They certainly do today, but that's primarily because pyvenv isn't very > useful yet, since the stdlib has no installer and thus a newly-created > pyvenv has no way to install anything in it. True, though I've provided a script to do that very thing: https://gist.github.com/vsajip/4673395 Of course, that'll now need to be changed to install setuptools rather than distribute :-) Regards, Vinay Sajip From richard at python.org Wed Jul 10 11:38:31 2013 From: richard at python.org (Richard Jones) Date: Wed, 10 Jul 2013 19:38:31 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: Message-ID: On 10 July 2013 18:28, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 07/09/2013 11:20 PM, Donald Stufft wrote: > >> doesn't "PyEnv" which is bundled with Python 3.3+ replace virtualenv? >> What's the purpose of including virtualenv in the bootstrap? >> http://www.python.org/dev/peps/pep-0405/ > > Environments generated by pyvenv lack setuptools, which makes them > un-useful compared to those generated by virtualenv. Yes, but Python 3.4 will have the pip bootstrap which automatically installs setuptools. Unless you mean that pyvenv itself (sans pip) would be more useful with setuptools? > Virtualenv is also > useful across the important set of Python versions (2.6, 2.7, 3.2, 3.3), > which pyvenv (or any shipped-in-core varieant) can never be. Yes, that's why I suggested the Python 2 version will install virtualenv :-) There's currently no plan to release a Python 3.3 version of the bootstrap, and certainly not one for a Py3k version lower than that. Hm. We can think about it though. Richard From richard at python.org Wed Jul 10 11:40:26 2013 From: richard at python.org (Richard Jones) Date: Wed, 10 Jul 2013 19:40:26 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <5C868F28-F2E9-4960-9E9C-2C766F7F15D2@stufft.io> Message-ID: On 10 July 2013 19:08, Vinay Sajip wrote: > Richard Jones gmail.com> writes: > >> pip without virtualenv in python 2 contexts is pretty rare (or at >> least *should* be ) so I think I'll retain it in that bootstrap >> code. > > Perhaps I misunderstand, but what's the relevance of Python 2 contexts here? > Aren't we talking about Python 3.4 and later? It makes sense to me (and Nick) to simplify the packaging overhead for users of Python 2. Currently the story is a bit of a mess (multiple sites with different approaches). > Exactly what benefit does including virtualenv provide in a 3.4+ > installation? That was kinda my question Richard From vinay_sajip at yahoo.co.uk Wed Jul 10 11:55:41 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 10 Jul 2013 09:55:41 +0000 (UTC) Subject: [Distutils] PEP 439 and pip bootstrap updated References: <5C868F28-F2E9-4960-9E9C-2C766F7F15D2@stufft.io> Message-ID: Richard Jones python.org> writes: > It makes sense to me (and Nick) to simplify the packaging overhead for > users of Python 2. Currently the story is a bit of a mess (multiple > sites with different approaches). No argument there, but I still don't see the relevance of virtualenv in a 3.4+ context. The PEP states "Hereafter the installation of the 'pip implementation' will imply installation of setuptools and virtualenv." and, a few lines further down, "The bootstrap process will proceed as follows: 1. The user system has Python (3.4+) installed." I don't see any mention of backporting this bootstrap to 2.x. > > Exactly what benefit does including virtualenv provide in a 3.4+ > > installation? > > That was kinda my question Sorry, it didn't come across like a question, more like a fait accompli :-) Regards, Vinay Sajip From richard at python.org Wed Jul 10 12:33:16 2013 From: richard at python.org (Richard Jones) Date: Wed, 10 Jul 2013 20:33:16 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <5C868F28-F2E9-4960-9E9C-2C766F7F15D2@stufft.io> Message-ID: On 10 July 2013 19:55, Vinay Sajip wrote: > Richard Jones python.org> writes: > >> It makes sense to me (and Nick) to simplify the packaging overhead for >> users of Python 2. Currently the story is a bit of a mess (multiple >> sites with different approaches). > > No argument there, but I still don't see the relevance of virtualenv in a > 3.4+ context. The PEP states > > "Hereafter the installation of the 'pip implementation' will imply > installation of setuptools and virtualenv." Sorry I've not made this clearer. Per the discussion here I've removed that from the PEP. That version hasn't been built on the web server yet. >> > Exactly what benefit does including virtualenv provide in a 3.4+ >> > installation? >> >> That was kinda my question > > Sorry, it didn't come across like a question, more like a fait accompli :-) Poorly phrased, my apologies. Richard From brett at python.org Wed Jul 10 14:46:12 2013 From: brett at python.org (Brett Cannon) Date: Wed, 10 Jul 2013 08:46:12 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On Wed, Jul 10, 2013 at 12:54 AM, Richard Jones wrote: > On 10 July 2013 14:19, Carl Meyer wrote: > > They certainly do today, but that's primarily because pyvenv isn't very > > useful yet, since the stdlib has no installer and thus a newly-created > > pyvenv has no way to install anything in it. > > Ah, thanks for clarifying that. > > > > Certainly if the bootstrap is ever ported to 2.7 or 3.2, it would make > > sense for it to install virtualenv there (or, probably even better, for > > pyvenv to be backported along with the bootstrap). > > I intend to create two forks; one for consideration in a 2.7.5 release > as "pip" and the other for users of 2.6+ called "get-pip.py". > Why the specific shift between 2.7 and 2.6 in terms of naming? I realize you are differentiating between the bootstrap being pre-installed with Python vs. not, but is there really anything wrong with the script being called pip (or pip3 for Python 3.3/3.2) if it knows how to do the right thing to get pip up and going? IOW why not make the bootstrap what everyone uses to install pip and it just so happens to come pre-installed with Python 3.4 (and maybe Python 2.7)? -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Jul 10 15:43:59 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 10 Jul 2013 14:43:59 +0100 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On 10 July 2013 13:46, Brett Cannon wrote: > pip (or pip3 for Python 3.3/3.2) Sorry to butt in here, but can I just catch this point. There seems to be an ongoing series of assumptions over whether the bootstrap is called pip or pip3. The pep actually says the bootstrap will be called pip3, but I'm not happy with that - specifically because the *existing* pip is not called pip3. So, at present, if I (as a 100% Python 3 user) want to install a package, I type "pip install XXX". No version suffix. In the same way, to invoke Python, I type "py" (I'm on Windows here) or if I want the currently active virtualenv, "python". I would find it distinctly irritating if in Python 3.4 I have to type "pip3 bootstrap" to bootstrap pip - and even worse if *after* the bootstrap the command I use is still "pip". (And no, there is currently no "pip3" command installed by pip, and even if there were, I would not want to use it, I'm happy with the unsuffixed version). I appreciate that Unix users have different compatibility priorities here, but can I propose that on Windows at least, the bootstrap command is "pip" and that matches the "core" pip that will be downloaded? Oh - and one other thing, on Windows python is often not on the system PATH - that's what the py.exe launcher is for. So where will the pip bootstrap command be installed, and where will it install the real pip? And also, will the venv code be modified to install the pip bootstrap in the venv's Scripts directory? Does virtualenv need to change to do the same? What if pip has already been bootstrapped in the system Python? Maybe I need to properly review the PEP rather than just throwing out random thoughts :-) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Jul 10 16:02:15 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 10 Jul 2013 10:02:15 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: When bundled the script is supposed to mask the fact you don't have pip installed. Basically if you type pip3 install requests it will first install setuptools and pip and then pass the command into the real pip. If it was called get pip then the workflow would be "attempt to install", "run get-pip", "rerun the original install command" On Jul 10, 2013, at 8:46 AM, Brett Cannon wrote: > > > > On Wed, Jul 10, 2013 at 12:54 AM, Richard Jones wrote: >> On 10 July 2013 14:19, Carl Meyer wrote: >> > They certainly do today, but that's primarily because pyvenv isn't very >> > useful yet, since the stdlib has no installer and thus a newly-created >> > pyvenv has no way to install anything in it. >> >> Ah, thanks for clarifying that. >> >> >> > Certainly if the bootstrap is ever ported to 2.7 or 3.2, it would make >> > sense for it to install virtualenv there (or, probably even better, for >> > pyvenv to be backported along with the bootstrap). >> >> I intend to create two forks; one for consideration in a 2.7.5 release >> as "pip" and the other for users of 2.6+ called "get-pip.py". > > Why the specific shift between 2.7 and 2.6 in terms of naming? I realize you are differentiating between the bootstrap being pre-installed with Python vs. not, but is there really anything wrong with the script being called pip (or pip3 for Python 3.3/3.2) if it knows how to do the right thing to get pip up and going? IOW why not make the bootstrap what everyone uses to install pip and it just so happens to come pre-installed with Python 3.4 (and maybe Python 2.7)? > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Jul 10 16:28:12 2013 From: brett at python.org (Brett Cannon) Date: Wed, 10 Jul 2013 10:28:12 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On Wed, Jul 10, 2013 at 9:43 AM, Paul Moore wrote: > On 10 July 2013 13:46, Brett Cannon wrote: > >> pip (or pip3 for Python 3.3/3.2) > > > Sorry to butt in here, but can I just catch this point. There seems to be > an ongoing series of assumptions over whether the bootstrap is called pip > or pip3. The pep actually says the bootstrap will be called pip3, but I'm > not happy with that - specifically because the *existing* pip is not called > pip3. > > So, at present, if I (as a 100% Python 3 user) want to install a package, > I type "pip install XXX". No version suffix. In the same way, to invoke > Python, I type "py" (I'm on Windows here) or if I want the currently active > virtualenv, "python". > But you should be typing python3 here, not python (and PEP 394 is trying to get people to start using python2 as the name to invoke). > > I would find it distinctly irritating if in Python 3.4 I have to type > "pip3 bootstrap" to bootstrap pip - and even worse if *after* the bootstrap > the command I use is still "pip". (And no, there is currently no "pip3" > command installed by pip, and even if there were, I would not want to use > it, I'm happy with the unsuffixed version). > > As Donald pointed out, you would always use pip3. The bootstrapping aspect is a behind-the-scenes thing; just consider the script as "launch pip if installed, else, bootstrap it in and then launch it". > I appreciate that Unix users have different compatibility priorities here, > but can I propose that on Windows at least, the bootstrap command is "pip" > and that matches the "core" pip that will be downloaded? > > There won't be a difference in command-line usage. > Oh - and one other thing, on Windows python is often not on the system > PATH - that's what the py.exe launcher is for. So where will the pip > bootstrap command be installed, and where will it install the real pip? > Covered in the PEP: it will go into the user installation location as if --user had been specified. > And also, will the venv code be modified to install the pip bootstrap in > the venv's Scripts directory? > In the PEP: goes into the venv. > Does virtualenv need to change to do the same? What if pip has already > been bootstrapped in the system Python? > Then nothing special happens; the script just executes pip instead of triggering a bootstrap first. > Maybe I need to properly review the PEP rather than just throwing out > random thoughts :-) > I feel like I just fed a bad habit. =) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Wed Jul 10 16:37:52 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 10 Jul 2013 16:37:52 +0200 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <5C868F28-F2E9-4960-9E9C-2C766F7F15D2@stufft.io> Message-ID: On 10 Jul, 2013, at 11:40, Richard Jones wrote: > On 10 July 2013 19:08, Vinay Sajip wrote: >> Richard Jones gmail.com> writes: >> >>> pip without virtualenv in python 2 contexts is pretty rare (or at >>> least *should* be ) so I think I'll retain it in that bootstrap >>> code. >> >> Perhaps I misunderstand, but what's the relevance of Python 2 contexts here? >> Aren't we talking about Python 3.4 and later? > > It makes sense to me (and Nick) to simplify the packaging overhead for > users of Python 2. Currently the story is a bit of a mess (multiple > sites with different approaches). New features in a bugfix release? You better hope the RM doesn't hear :-) That said, 2.7 will be around for a while and adding a consistent installation experience to both 3.4 and 2.7 does sound attractive and adding a new script shouldn't have that many side effects. What about backporting pyvenv as well? I guess that's way too invasive for a bugfix release. Ronald From erik.m.bray at gmail.com Wed Jul 10 17:35:10 2013 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 10 Jul 2013 11:35:10 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <5C868F28-F2E9-4960-9E9C-2C766F7F15D2@stufft.io> Message-ID: On Wed, Jul 10, 2013 at 12:37 AM, Richard Jones wrote: > pip without virtualenv in python 2 contexts is pretty rare (or at > least *should* be ) so I think I'll retain it in that bootstrap > code. I agree it *should* be rare in most cases but it most assuredly is not. I can tell you from experience that a lot of people in the scientific community, for example, do not use virtualenv (sometimes with good reasons, but more often not). Erik From barry at python.org Wed Jul 10 17:21:41 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 10 Jul 2013 11:21:41 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated References: <51DCE0B5.6030506@oddbird.net> Message-ID: <20130710112141.603dd396@anarchist> On Jul 10, 2013, at 02:43 PM, Paul Moore wrote: >I would find it distinctly irritating if in Python 3.4 I have to type "pip3 >bootstrap" to bootstrap pip - and even worse if *after* the bootstrap the >command I use is still "pip". (And no, there is currently no "pip3" command >installed by pip, and even if there were, I would not want to use it, I'm >happy with the unsuffixed version). I have a lot of sympathy for this, and the general issue has come up in a number of different contexts, e.g. nostests/nosetests3 and so on. On a distro like Debian, this just adds more gunk to /usr/bin, especially since some scripts are also minor-version dependent. One approach is to use `$python -m nose` or in this case `$python -m pip` which cuts down on the number of scripts, is unambiguous, but is far from convenient and may not work in all cases, e.g. for older Python's that don't support -m or don't support it for packages. I think there was a thread on python-ideas about this, but in the back of my mind, I have this horrible idea for a version-aware relauncher you could use in your shebang line. Something like: #! /usr/bin/pylaunch So that you could do something like: $ nosetests -3 $ nosetests -2 $ nosetests -3.3 $ nosetests -2.7 and it would relaunch itself using the correct Python version, consuming the version argument so the actual script wouldn't see it. I'm not sure if the convenience is worth it, and I'm sorry for making you throw up a little in your mouth there. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From p.f.moore at gmail.com Wed Jul 10 18:11:11 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 10 Jul 2013 17:11:11 +0100 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On 10 July 2013 15:28, Brett Cannon wrote: > So, at present, if I (as a 100% Python 3 user) want to install a package, >> I type "pip install XXX". No version suffix. In the same way, to invoke >> Python, I type "py" (I'm on Windows here) or if I want the currently active >> virtualenv, "python". >> > > But you should be typing python3 here, not python (and PEP 394 is trying > to get people to start using python2 as the name to invoke). > So - that's a major behaviour change on Windows. At the moment, Python 3.3 for Windows installs python.exe and pythonw.exe. There are no versioned executables at all. Are you saying that in 3.4 this will change? That will break so many things I have to believe you're wrong or I've misunderstood you. OTOH, adding python3.exe and python3w.exe (or is that pythonw3.exe?) which I can then ignore is fine by me (but in that case, the change doesn't affect my point about the pip command). As I say, I understand Unix is different. This is a purely Windows point - and in the context of the PEP, that's what I'm saying, please can we be careful to be clear whether the plan is for the new pip bootstrap to favour existing platform conventions or uniformity (with the further complication of needing to consider the full pip distribution's behaviour - and there, I will be lobbying hard against any change to require a pip3 command to be used, at least on Windows). As things stand, I can assume the PEP specifies Unix behaviour and is vague or silent on Windows variations, or I can ask for clarification, and for the results to be documented in the PEP. Up to now I was doing the former, but I'm moving towards the latter - hence my question(s). Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Jul 10 20:55:00 2013 From: brett at python.org (Brett Cannon) Date: Wed, 10 Jul 2013 14:55:00 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On Wed, Jul 10, 2013 at 12:11 PM, Paul Moore wrote: > On 10 July 2013 15:28, Brett Cannon wrote: > >> So, at present, if I (as a 100% Python 3 user) want to install a package, >>> I type "pip install XXX". No version suffix. In the same way, to invoke >>> Python, I type "py" (I'm on Windows here) or if I want the currently active >>> virtualenv, "python". >>> >> >> But you should be typing python3 here, not python (and PEP 394 is trying >> to get people to start using python2 as the name to invoke). >> > > So - that's a major behaviour change on Windows. At the moment, Python 3.3 > for Windows installs python.exe and pythonw.exe. There are no versioned > executables at all. Are you saying that in 3.4 this will change? That will > break so many things I have to believe you're wrong or I've misunderstood > you. OTOH, adding python3.exe and python3w.exe (or is that pythonw3.exe?) > which I can then ignore is fine by me (but in that case, the change doesn't > affect my point about the pip command). > Didn't know Windows was never updated to use a versioned binary. That's rather unfortunate. -Brett > > As I say, I understand Unix is different. This is a purely Windows point - > and in the context of the PEP, that's what I'm saying, please can we be > careful to be clear whether the plan is for the new pip bootstrap to favour > existing platform conventions or uniformity (with the further complication > of needing to consider the full pip distribution's behaviour - and there, I > will be lobbying hard against any change to require a pip3 command to be > used, at least on Windows). > > As things stand, I can assume the PEP specifies Unix behaviour and is > vague or silent on Windows variations, or I can ask for clarification, and > for the results to be documented in the PEP. Up to now I was doing the > former, but I'm moving towards the latter - hence my question(s). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Jul 10 22:30:11 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 11 Jul 2013 06:30:11 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On 11 Jul 2013 04:56, "Brett Cannon" wrote: > > > > > On Wed, Jul 10, 2013 at 12:11 PM, Paul Moore wrote: >> >> On 10 July 2013 15:28, Brett Cannon wrote: >>>> >>>> So, at present, if I (as a 100% Python 3 user) want to install a package, I type "pip install XXX". No version suffix. In the same way, to invoke Python, I type "py" (I'm on Windows here) or if I want the currently active virtualenv, "python". >>> >>> >>> But you should be typing python3 here, not python (and PEP 394 is trying to get people to start using python2 as the name to invoke). >> >> >> So - that's a major behaviour change on Windows. At the moment, Python 3.3 for Windows installs python.exe and pythonw.exe. There are no versioned executables at all. Are you saying that in 3.4 this will change? That will break so many things I have to believe you're wrong or I've misunderstood you. OTOH, adding python3.exe and python3w.exe (or is that pythonw3.exe?) which I can then ignore is fine by me (but in that case, the change doesn't affect my point about the pip command). > > > Didn't know Windows was never updated to use a versioned binary. That's rather unfortunate. Hence the PyLauncher project. Paul's right, though - the PEP is currently very *nix-centric. For Windows, we likely need to consider something based on "py -m pip", which then raises the question of whether or not that's what we should be supporting on *nix as well (with pip and pip3 as convenient shorthand). There's also the fact that the Python launcher is *already* available as a separate Windows installer for earlier releases. Perhaps we should just be bundling the bootstrap script with that for earlier Windows releases. Cheers, Nick. > > -Brett > >> >> >> As I say, I understand Unix is different. This is a purely Windows point - and in the context of the PEP, that's what I'm saying, please can we be careful to be clear whether the plan is for the new pip bootstrap to favour existing platform conventions or uniformity (with the further complication of needing to consider the full pip distribution's behaviour - and there, I will be lobbying hard against any change to require a pip3 command to be used, at least on Windows). >> >> As things stand, I can assume the PEP specifies Unix behaviour and is vague or silent on Windows variations, or I can ask for clarification, and for the results to be documented in the PEP. Up to now I was doing the former, but I'm moving towards the latter - hence my question(s). > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Jul 10 22:50:25 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 10 Jul 2013 21:50:25 +0100 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On 10 July 2013 21:30, Nick Coghlan wrote: > > On 11 Jul 2013 04:56, "Brett Cannon" wrote: > > > > Didn't know Windows was never updated to use a versioned binary. That's > rather unfortunate. > > Hence the PyLauncher project. > > Paul's right, though - the PEP is currently very *nix-centric. For > Windows, we likely need to consider something based on "py -m pip", which > then raises the question of whether or not that's what we should be > supporting on *nix as well (with pip and pip3 as convenient shorthand). > > There's also the fact that the Python launcher is *already* available as a > separate Windows installer for earlier releases. Perhaps we should just be > bundling the bootstrap script with that for earlier Windows releases. > Thanks Nick. I was part way through a much more laboured email basically saying the same thing :-) For reference, PEP 394 is the versioned binary PEP. It is explicitly Unix only and defers Windows to PEP 397 (pylauncher) as being "too complex" to cover alongside the Unix proposal :-) I think "python -m pip" should be the canonical form (used in documentation, examples, etc). The unittest module has taken this route, as has timeit. Traditionally, python-dev have been lukewarm about the -m interface, but its key advantage is that it bypasses all the issues around versioned executables, cross-platform issues, the general dreadfulness of script wrappers on Windows, etc, in one fell swoop. Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Jul 10 23:39:42 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 10 Jul 2013 17:39:42 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated References: <51DCE0B5.6030506@oddbird.net> Message-ID: <20130710173942.3cecfa2c@anarchist> On Jul 10, 2013, at 09:50 PM, Paul Moore wrote: >I think "python -m pip" should be the canonical form (used in >documentation, examples, etc). The unittest module has taken this route, as >has timeit. Traditionally, python-dev have been lukewarm about the -m >interface, but its key advantage is that it bypasses all the issues around >versioned executables, cross-platform issues, the general dreadfulness of >script wrappers on Windows, etc, in one fell swoop. +1 -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From donald at stufft.io Thu Jul 11 00:00:59 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 10 Jul 2013 18:00:59 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: <20130710173942.3cecfa2c@anarchist> References: <51DCE0B5.6030506@oddbird.net> <20130710173942.3cecfa2c@anarchist> Message-ID: <8CF28CAB-0D28-4FFC-BCC7-B63932BF37FA@stufft.io> On Jul 10, 2013, at 5:39 PM, Barry Warsaw wrote: > On Jul 10, 2013, at 09:50 PM, Paul Moore wrote: > >> I think "python -m pip" should be the canonical form (used in >> documentation, examples, etc). The unittest module has taken this route, as >> has timeit. Traditionally, python-dev have been lukewarm about the -m >> interface, but its key advantage is that it bypasses all the issues around >> versioned executables, cross-platform issues, the general dreadfulness of >> script wrappers on Windows, etc, in one fell swoop. > > +1 > > -Barry > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig As long as the non -m way exists so I don't have to use it D: ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Thu Jul 11 00:20:56 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 10 Jul 2013 23:20:56 +0100 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: <8CF28CAB-0D28-4FFC-BCC7-B63932BF37FA@stufft.io> References: <51DCE0B5.6030506@oddbird.net> <20130710173942.3cecfa2c@anarchist> <8CF28CAB-0D28-4FFC-BCC7-B63932BF37FA@stufft.io> Message-ID: On 10 July 2013 23:00, Donald Stufft wrote: > As long as the non -m way exists so I don't have to use it D: Fair enough :-) Having a standard method (-m) and a platform-specific Unix method seems fine to me (and the Unix people can debate the merits of pip3 vs pip etc as much or as little as they want). It'll be nice seeing Unix be the non-standard one for a change :-) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Thu Jul 11 03:09:03 2013 From: richard at python.org (Richard Jones) Date: Thu, 11 Jul 2013 11:09:03 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On 11 July 2013 06:50, Paul Moore wrote: > I think "python -m pip" should be the canonical form (used in documentation, > examples, etc). The unittest module has taken this route, as has timeit. > Traditionally, python-dev have been lukewarm about the -m interface, but its > key advantage is that it bypasses all the issues around versioned > executables, cross-platform issues, the general dreadfulness of script > wrappers on Windows, etc, in one fell swoop. "python -m pip" does make the bootstrapping a more complex proposition - the stdlib would have to have something called "pip" that could be overridden (while it is actually *running*) by something installed in site-packages. Not easy. Thanks everyone for your brilliant feedback and discussion - I look forward to being able to say something sensible about Windows in the PEP :-) Richard From brett at python.org Thu Jul 11 14:49:33 2013 From: brett at python.org (Brett Cannon) Date: Thu, 11 Jul 2013 08:49:33 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On Wed, Jul 10, 2013 at 9:09 PM, Richard Jones wrote: > On 11 July 2013 06:50, Paul Moore wrote: > > I think "python -m pip" should be the canonical form (used in > documentation, > > examples, etc). The unittest module has taken this route, as has timeit. > > Traditionally, python-dev have been lukewarm about the -m interface, but > its > > key advantage is that it bypasses all the issues around versioned > > executables, cross-platform issues, the general dreadfulness of script > > wrappers on Windows, etc, in one fell swoop. > > "python -m pip" does make the bootstrapping a more complex proposition > - the stdlib would have to have something called "pip" that could be > overridden (while it is actually *running*) by something installed in > site-packages. Not easy. > It's also fraught with historical baggage; remember xmlplus? That was extremely painful and something I believe everyone was glad to see go away. Having said that, there are two solutions to this. The compatible solution with older Python versions is to have the bootstrap download pip and have it installed as piplib or some other alternative name that is not masked by a pip stub in the stdlib. The dead-simple, extremely elegant solution (starting in Python 3.4) is to make pip a namespace package in the stdlib with nothing more than a __main__.py file that installs pip; no checking if it's installed and then running it, etc, just blindly install pip. Then, if you install pip as a regular package, it takes precedence and what's in the stdlib is completely ignored (this helps with any possible staleness with the stdlib's bootstrap script vs. what's in pip, etc.). You don't even need to change the __main__.py in pip as it stands today since namespace packages only work if no regular package is found. In case that didn't make sense, here is the file structure: python3.4/ pip/ __main__.py # Install pip, nothing more ~/.local/ bin/ pip # Literally a shebang and two lines of Python; see below lib/python3.4/site-packages pip/ # As it stands today __init__.py __main__.py ... This also means pip3 literally becomes ``import runpy; runpy.run_module('pip')``, so that is even easier to maintain (assuming pip's bin/ stub isn't already doing that because of backwards-compatibility concerns or something with __main__.py or runpy not existing far enough back, otherwise it should =). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jul 11 15:33:16 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 11 Jul 2013 14:33:16 +0100 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On 11 July 2013 13:49, Brett Cannon wrote: > The dead-simple, extremely elegant solution (starting in Python 3.4) is to > make pip a namespace package in the stdlib with nothing more than a > __main__.py file that installs pip; no checking if it's installed and then > running it, etc, just blindly install pip. Then, if you install pip as a > regular package, it takes precedence and what's in the stdlib is completely > ignored (this helps with any possible staleness with the stdlib's bootstrap > script vs. what's in pip, etc.). You don't even need to change the > __main__.py in pip as it stands today since namespace packages only work if > no regular package is found. > Wow - that is exceptionally cool. I had never realised namespace packages would work like this. > This also means pip3 literally becomes ``import runpy; > runpy.run_module('pip')``, so that is even easier to maintain (assuming > pip's bin/ stub isn't already doing that because of backwards-compatibility > concerns or something with __main__.py or runpy not existing far enough > back, otherwise it should =). > The pip executable script/wrapper currently uses setuptools entry points and wrapper scripts. I'm not a fan of those, so I'd be happy to see the change you suggest, but OTOH they have been like that since long before I was involved with pip, and I have no idea if there are reasons they need to stay that way. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jul 11 15:39:21 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 11 Jul 2013 09:39:21 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On Jul 11, 2013, at 9:33 AM, Paul Moore wrote: > The pip executable script/wrapper currently uses setuptools entry points and wrapper scripts. I'm not a fan of those, so I'd be happy to see the change you suggest, but OTOH they have been like that since long before I was involved with pip, and I have no idea if there are reasons they need to stay that way. Typically the reasoning is because of the .exe wrapper. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From brett at python.org Thu Jul 11 16:20:26 2013 From: brett at python.org (Brett Cannon) Date: Thu, 11 Jul 2013 10:20:26 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On Thu, Jul 11, 2013 at 9:39 AM, Donald Stufft wrote: > > On Jul 11, 2013, at 9:33 AM, Paul Moore wrote: > > The pip executable script/wrapper currently uses setuptools entry points > and wrapper scripts. I'm not a fan of those, so I'd be happy to see the > change you suggest, but OTOH they have been like that since long before I > was involved with pip, and I have no idea if there are reasons they need to > stay that way. > > > Typically the reasoning is because of the .exe wrapper. > And if people want to promote the -m option then the executable scripts just become a secondary convenience. Plus you can't exactly require setuptools to create those scripts at install-time with Python if that's when they are going to be installed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Jul 11 16:29:06 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 11 Jul 2013 10:29:06 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On Thu, Jul 11, 2013 at 9:33 AM, Paul Moore wrote: > On 11 July 2013 13:49, Brett Cannon wrote: >> >> The dead-simple, extremely elegant solution (starting in Python 3.4) is to >> make pip a namespace package in the stdlib with nothing more than a >> __main__.py file that installs pip; no checking if it's installed and then >> running it, etc, just blindly install pip. Then, if you install pip as a >> regular package, it takes precedence and what's in the stdlib is completely >> ignored (this helps with any possible staleness with the stdlib's bootstrap >> script vs. what's in pip, etc.). You don't even need to change the >> __main__.py in pip as it stands today since namespace packages only work if >> no regular package is found. > > > Wow - that is exceptionally cool. I had never realised namespace packages > would work like this. Not exceptionally cool ... and that's why the namespace_package form is popular, since the first package in a set of namespace packages that gets it wrong breaks everything. From brett at python.org Thu Jul 11 16:47:56 2013 From: brett at python.org (Brett Cannon) Date: Thu, 11 Jul 2013 10:47:56 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On Thu, Jul 11, 2013 at 10:29 AM, Daniel Holth wrote: > On Thu, Jul 11, 2013 at 9:33 AM, Paul Moore wrote: > > On 11 July 2013 13:49, Brett Cannon wrote: > >> > >> The dead-simple, extremely elegant solution (starting in Python 3.4) is > to > >> make pip a namespace package in the stdlib with nothing more than a > >> __main__.py file that installs pip; no checking if it's installed and > then > >> running it, etc, just blindly install pip. Then, if you install pip as a > >> regular package, it takes precedence and what's in the stdlib is > completely > >> ignored (this helps with any possible staleness with the stdlib's > bootstrap > >> script vs. what's in pip, etc.). You don't even need to change the > >> __main__.py in pip as it stands today since namespace packages only > work if > >> no regular package is found. > > > > > > Wow - that is exceptionally cool. I had never realised namespace packages > > would work like this. > > Not exceptionally cool ... and that's why the namespace_package form > is popular, since the first package in a set of namespace packages > that gets it wrong breaks everything. > I'm really not following that sentence. You are saying the idea is bad, but is that in general or for this specific case? And you say it's popular because people get it wrong which breaks everything? And how can namespace packages be popular if they are new to Python 3.3 (the ability to execute them with -m is new in Python 3.4)? Are you talking about pkgutil's extend_path hack because I'm talking about NamespaceLoader in importlib? I'm just not seeing the downside. We control the stdlib and pip, so we know the expected interaction and we are purposefully using the override mechanics so it's not going to get messed up by us if we consciously use it (and obviously have tests for it). -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jul 11 16:52:06 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 11 Jul 2013 10:52:06 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: <80918BCA-78AA-482B-B551-FFDB364A799A@stufft.io> On Jul 11, 2013, at 10:47 AM, Brett Cannon wrote: > I'm just not seeing the downside. We control the stdlib and pip, so we know the expected interaction and we are purposefully using the override mechanics so it's not going to get messed up by us if we consciously use it (and obviously have tests for it). I don't think it's especially a problem for pip. I think Daniel was just speaking how the behavior you suggested we could exploit to make this happen has been a major issue for namespace packages in the past using the other methods. However I'm not sure how it's going to work? python -m pip is going to import the pip namespace package yes? And then when pip is installed it'll shadow that, but in the original process where we ran python -m pip won't the namespace package have been cached in sys.modules already? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From brett at python.org Thu Jul 11 17:50:54 2013 From: brett at python.org (Brett Cannon) Date: Thu, 11 Jul 2013 11:50:54 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: <80918BCA-78AA-482B-B551-FFDB364A799A@stufft.io> References: <51DCE0B5.6030506@oddbird.net> <80918BCA-78AA-482B-B551-FFDB364A799A@stufft.io> Message-ID: On Thu, Jul 11, 2013 at 10:52 AM, Donald Stufft wrote: > > On Jul 11, 2013, at 10:47 AM, Brett Cannon wrote: > > I'm just not seeing the downside. We control the stdlib and pip, so we > know the expected interaction and we are purposefully using the override > mechanics so it's not going to get messed up by us if we consciously use it > (and obviously have tests for it). > > > I don't think it's especially a problem for pip. I think Daniel was just > speaking how the behavior you suggested we could exploit to make this > happen has been a major issue for namespace packages in the past using the > other methods. > > However I'm not sure how it's going to work? python -m pip is going to > import the pip namespace package yes? And then when pip is installed it'll > shadow that, but in the original process where we ran python -m pip won't > the namespace package have been cached in sys.modules already? > Yes, but you can clear it out of sys.modules before executing runpy to get the desired effect of falling through to the regular package (runpy wouldn't import pip.__main__ so you literally just need ``del sys.modules['pip']``). You could also pull the old pkgutil.extend_path() trick and use the append method on the _NamespacePath object to directly add the new directory that pip was installed to and then import pip.runner.main(), but that feels like more of a hack to me (but then again I'm rather comfortable mucking with the import system =). -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Jul 11 17:52:30 2013 From: brett at python.org (Brett Cannon) Date: Thu, 11 Jul 2013 11:52:30 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> <80918BCA-78AA-482B-B551-FFDB364A799A@stufft.io> Message-ID: On Thu, Jul 11, 2013 at 11:50 AM, Brett Cannon wrote: > > > > On Thu, Jul 11, 2013 at 10:52 AM, Donald Stufft wrote: > >> >> On Jul 11, 2013, at 10:47 AM, Brett Cannon wrote: >> >> I'm just not seeing the downside. We control the stdlib and pip, so we >> know the expected interaction and we are purposefully using the override >> mechanics so it's not going to get messed up by us if we consciously use it >> (and obviously have tests for it). >> >> >> I don't think it's especially a problem for pip. I think Daniel was just >> speaking how the behavior you suggested we could exploit to make this >> happen has been a major issue for namespace packages in the past using the >> other methods. >> >> However I'm not sure how it's going to work? python -m pip is going to >> import the pip namespace package yes? And then when pip is installed it'll >> shadow that, but in the original process where we ran python -m pip won't >> the namespace package have been cached in sys.modules already? >> > > Yes, but you can clear it out of sys.modules before executing runpy to get > the desired effect of falling through to the regular package (runpy > wouldn't import pip.__main__ so you literally just need ``del > sys.modules['pip']``). You could also pull the old pkgutil.extend_path() > trick and use the append method on the _NamespacePath object to directly > add the new directory that pip was installed to and then import > pip.runner.main(), but that feels like more of a hack to me (but then again > I'm rather comfortable mucking with the import system =). > And if you're still worried you can always invalidate the cache of the finder representing the parent directory pip got installed to (or all finder caches if you really want to get jumpy). -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jul 11 18:21:13 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 11 Jul 2013 12:21:13 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> <80918BCA-78AA-482B-B551-FFDB364A799A@stufft.io> Message-ID: <064659DF-D9F6-4C7C-ABDC-00935AE35E89@stufft.io> On Jul 11, 2013, at 11:50 AM, Brett Cannon wrote: > Yes, but you can clear it out of sys.modules before executing runpy to get the desired effect of falling through to the regular package (runpy wouldn't import pip.__main__ so you literally just need ``del sys.modules['pip']``). You could also pull the old pkgutil.extend_path() trick and use the append method on the _NamespacePath object to directly add the new directory that pip was installed to and then import pip.runner.main(), but that feels like more of a hack to me (but then again I'm rather comfortable mucking with the import system =). Ok, Just making sure :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From pje at telecommunity.com Thu Jul 11 19:05:33 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 11 Jul 2013 13:05:33 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On Thu, Jul 11, 2013 at 10:20 AM, Brett Cannon wrote: > And if people want to promote the -m option then the executable scripts just > become a secondary convenience. Plus you can't exactly require setuptools to > create those scripts at install-time with Python if that's when they are > going to be installed. You don't need setuptools in order to include .exe wrappers, though: there's nothing setuptools-specific about the .exe files, they just run a matching, adjacent 'foo-script.py', which can contain whatever you want. Just take the appropriate wrapper .exe, and rename it to whatever 'foo' you want. IOW, if you want to ship a pip.exe on windows that just does "from pip import __main__; __main__()" (or whatever), you can do precisely that, no setuptools needed. From p.f.moore at gmail.com Thu Jul 11 19:41:13 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 11 Jul 2013 18:41:13 +0100 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On 11 July 2013 18:05, PJ Eby wrote: > On Thu, Jul 11, 2013 at 10:20 AM, Brett Cannon wrote: > > And if people want to promote the -m option then the executable scripts > just > > become a secondary convenience. Plus you can't exactly require > setuptools to > > create those scripts at install-time with Python if that's when they are > > going to be installed. > > You don't need setuptools in order to include .exe wrappers, though: > there's nothing setuptools-specific about the .exe files, they just > run a matching, adjacent 'foo-script.py', which can contain whatever > you want. Just take the appropriate wrapper .exe, and rename it to > whatever 'foo' you want. > > IOW, if you want to ship a pip.exe on windows that just does "from pip > import __main__; __main__()" (or whatever), you can do precisely that, > no setuptools needed. With the launcher, a .py file with the relevant #! line set pretty much covers things. It's not an exe, although there are very few things I know of that need specifically an exe file, and if you want to omit the ".py" suffix when invoking it you need to add .py to PATHEXT. But actual exe launchers are much less critical nowadays, I believe. What *is* important, though, is some level of consistency. Before setuptools promoted the idea of declaraive entry points, distributions shipped with a ridiculous variety of attempts to make cross-platform launchers (many of which didn't work very well). I care a lot more about promoting a consistent cross-platform approach than about arguing for any particular solution... Paul Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jul 11 23:48:07 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 12 Jul 2013 07:48:07 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: (Oops, started this yesterday, got distracted and never hit send) On 11 July 2013 11:09, Richard Jones wrote: > > On 11 July 2013 06:50, Paul Moore wrote: > > I think "python -m pip" should be the canonical form (used in documentation, > > examples, etc). The unittest module has taken this route, as has timeit. > > Traditionally, python-dev have been lukewarm about the -m interface, but its > > key advantage is that it bypasses all the issues around versioned > > executables, cross-platform issues, the general dreadfulness of script > > wrappers on Windows, etc, in one fell swoop. > > "python -m pip" does make the bootstrapping a more complex proposition > - the stdlib would have to have something called "pip" that could be > overridden (while it is actually *running*) by something installed in > site-packages. Not easy. I was thinking about that, and I'm wondering if the most sensible option may be to claim the "getpip" name on PyPI for ourselves and then do the following: 1. Provide "getpip" in the standard library for 3.4+ (and perhaps in a 2.7.x release) 2. Install it to site-packages in the "Python launcher for Windows" installer for earlier versions getpip would expose at least one function: def bootstrap(index_url=None, system_install=False): ... And executing it as a main module would either: 1. Do nothing, if "import pip" already works 2. Call bootstrap with the appropriate arguments That way, installation instructions can simply say to unconditionally do: python -m getpip And that will either: 1. Report that pip is already installed; 2. Bootstrap pip into the user environment; or 3. Emit a distro-specific message if the distro packagers want to push users to use the system pip instead (since they get to patch the system Python and can tweak the system getpip however they want) The 2.7 change would then be to create a new download that bundles the Windows launcher into the Windows installer. Users aren't stupid - the problem with the status quo is really that the bootstrapping instructions are annoyingly complicated and genuinely confusing, not that an explicit bootstrapping step is needed in the first place. Cheers, Nick. > > Thanks everyone for your brilliant feedback and discussion - I look > forward to being able to say something sensible about Windows in the > PEP :-) > > > > Richard -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia On 11 July 2013 06:50, Paul Moore wrote: > I think "python -m pip" should be the canonical form (used in documentation, > examples, etc). The unittest module has taken this route, as has timeit. > Traditionally, python-dev have been lukewarm about the -m interface, but its > key advantage is that it bypasses all the issues around versioned > executables, cross-platform issues, the general dreadfulness of script > wrappers on Windows, etc, in one fell swoop. "python -m pip" does make the bootstrapping a more complex proposition - the stdlib would have to have something called "pip" that could be overridden (while it is actually *running*) by something installed in site-packages. Not easy. Thanks everyone for your brilliant feedback and discussion - I look forward to being able to say something sensible about Windows in the PEP :-) Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Jul 11 23:57:16 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 11 Jul 2013 17:57:16 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: +1. No magic side effects will make everyone happier. On Jul 11, 2013 5:48 PM, "Nick Coghlan" wrote: > (Oops, started this yesterday, got distracted and never hit send) > > On 11 July 2013 11:09, Richard Jones wrote: > > > > On 11 July 2013 06:50, Paul Moore wrote: > > > I think "python -m pip" should be the canonical form (used in > documentation, > > > examples, etc). The unittest module has taken this route, as has > timeit. > > > Traditionally, python-dev have been lukewarm about the -m interface, > but its > > > key advantage is that it bypasses all the issues around versioned > > > executables, cross-platform issues, the general dreadfulness of script > > > wrappers on Windows, etc, in one fell swoop. > > > > "python -m pip" does make the bootstrapping a more complex proposition > > - the stdlib would have to have something called "pip" that could be > > overridden (while it is actually *running*) by something installed in > > site-packages. Not easy. > > I was thinking about that, and I'm wondering if the most sensible option > may be to claim the "getpip" name on PyPI for ourselves and then do the > following: > > 1. Provide "getpip" in the standard library for 3.4+ (and perhaps in a > 2.7.x release) > 2. Install it to site-packages in the "Python launcher for Windows" > installer for earlier versions > > getpip would expose at least one function: > > def bootstrap(index_url=None, system_install=False): > ... > > And executing it as a main module would either: > > 1. Do nothing, if "import pip" already works > 2. Call bootstrap with the appropriate arguments > > That way, installation instructions can simply say to unconditionally do: > > python -m getpip > > And that will either: > > 1. Report that pip is already installed; > 2. Bootstrap pip into the user environment; or > 3. Emit a distro-specific message if the distro packagers want to push > users to use the system pip instead (since they get to patch the system > Python and can tweak the system getpip however they want) > > The 2.7 change would then be to create a new download that bundles the > Windows launcher into the Windows installer. > > Users aren't stupid - the problem with the status quo is really that the > bootstrapping instructions are annoyingly complicated and genuinely > confusing, not that an explicit bootstrapping step is needed in the first > place. > > Cheers, > Nick. > > > > > > > > Thanks everyone for your brilliant feedback and discussion - I look > > forward to being able to say something sensible about Windows in the > > PEP :-) > > > > > > > > > > Richard > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > On 11 July 2013 06:50, Paul Moore wrote: > > I think "python -m pip" should be the canonical form (used in > documentation, > > examples, etc). The unittest module has taken this route, as has timeit. > > Traditionally, python-dev have been lukewarm about the -m interface, but > its > > key advantage is that it bypasses all the issues around versioned > > executables, cross-platform issues, the general dreadfulness of script > > wrappers on Windows, etc, in one fell swoop. > > "python -m pip" does make the bootstrapping a more complex proposition > - the stdlib would have to have something called "pip" that could be > overridden (while it is actually *running*) by something installed in > site-packages. Not easy. > > Thanks everyone for your brilliant feedback and discussion - I look > forward to being able to say something sensible about Windows in the > PEP :-) > > > Richard > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at oddbird.net Fri Jul 12 00:00:01 2013 From: carl at oddbird.net (Carl Meyer) Date: Thu, 11 Jul 2013 16:00:01 -0600 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: <51DF2AE1.2010803@oddbird.net> On 07/11/2013 03:48 PM, Nick Coghlan wrote: > I was thinking about that, and I'm wondering if the most sensible option > may be to claim the "getpip" name on PyPI for ourselves and then do the > following: > > 1. Provide "getpip" in the standard library for 3.4+ (and perhaps in a > 2.7.x release) > 2. Install it to site-packages in the "Python launcher for Windows" > installer for earlier versions > > getpip would expose at least one function: > > def bootstrap(index_url=None, system_install=False): > ... > > And executing it as a main module would either: > > 1. Do nothing, if "import pip" already works > 2. Call bootstrap with the appropriate arguments > > That way, installation instructions can simply say to unconditionally do: > > python -m getpip > > And that will either: > > 1. Report that pip is already installed; > 2. Bootstrap pip into the user environment; or > 3. Emit a distro-specific message if the distro packagers want to push > users to use the system pip instead (since they get to patch the system > Python and can tweak the system getpip however they want) > > The 2.7 change would then be to create a new download that bundles the > Windows launcher into the Windows installer. > > Users aren't stupid - the problem with the status quo is really that the > bootstrapping instructions are annoyingly complicated and genuinely > confusing, not that an explicit bootstrapping step is needed in the > first place. +1. This sounds far better to me than the implicit bootstrapping. Carl From donald at stufft.io Fri Jul 12 00:06:01 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 11 Jul 2013 18:06:01 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: <51DF2AE1.2010803@oddbird.net> References: <51DCE0B5.6030506@oddbird.net> <51DF2AE1.2010803@oddbird.net> Message-ID: <55CB6E5F-18E6-4A77-BDFE-531FFE842466@stufft.io> On Jul 11, 2013, at 6:00 PM, Carl Meyer wrote: > On 07/11/2013 03:48 PM, Nick Coghlan wrote: >> I was thinking about that, and I'm wondering if the most sensible option >> may be to claim the "getpip" name on PyPI for ourselves and then do the >> following: >> >> 1. Provide "getpip" in the standard library for 3.4+ (and perhaps in a >> 2.7.x release) >> 2. Install it to site-packages in the "Python launcher for Windows" >> installer for earlier versions >> >> getpip would expose at least one function: >> >> def bootstrap(index_url=None, system_install=False): >> ... >> >> And executing it as a main module would either: >> >> 1. Do nothing, if "import pip" already works >> 2. Call bootstrap with the appropriate arguments >> >> That way, installation instructions can simply say to unconditionally do: >> >> python -m getpip >> >> And that will either: >> >> 1. Report that pip is already installed; >> 2. Bootstrap pip into the user environment; or >> 3. Emit a distro-specific message if the distro packagers want to push >> users to use the system pip instead (since they get to patch the system >> Python and can tweak the system getpip however they want) >> >> The 2.7 change would then be to create a new download that bundles the >> Windows launcher into the Windows installer. >> >> Users aren't stupid - the problem with the status quo is really that the >> bootstrapping instructions are annoyingly complicated and genuinely >> confusing, not that an explicit bootstrapping step is needed in the >> first place. > > +1. This sounds far better to me than the implicit bootstrapping. > > Carl > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Generally +1, the one negative point I see is it's kinda a degradation in functionality to need to type ``python -m getpip`` in every PyEnv (coming from virtualenv). Maybe PyEnv can be smart enough to automatically install pip that's installed in the interpreter it's installed from? Maybe that's too much magic and the answer will be that tools like virtualenvwrapper will continue to exist and wrap that for you. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Fri Jul 12 00:12:49 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 11 Jul 2013 23:12:49 +0100 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: +1 Explicit is better than implicit. Amending venv to automatically install pip (as suggested by Donald) may be worth doing. I'm +0 on that (with the proviso that there's a --no-pip option in that case). OTOH, the venv module is very extensible and writing your own wrapper to import getpip and call bootstrap is pretty much trivial. On 11 July 2013 22:48, Nick Coghlan wrote: > (Oops, started this yesterday, got distracted and never hit send) > > On 11 July 2013 11:09, Richard Jones wrote: > > > > On 11 July 2013 06:50, Paul Moore wrote: > > > I think "python -m pip" should be the canonical form (used in > documentation, > > > examples, etc). The unittest module has taken this route, as has > timeit. > > > Traditionally, python-dev have been lukewarm about the -m interface, > but its > > > key advantage is that it bypasses all the issues around versioned > > > executables, cross-platform issues, the general dreadfulness of script > > > wrappers on Windows, etc, in one fell swoop. > > > > "python -m pip" does make the bootstrapping a more complex proposition > > - the stdlib would have to have something called "pip" that could be > > overridden (while it is actually *running*) by something installed in > > site-packages. Not easy. > > I was thinking about that, and I'm wondering if the most sensible option > may be to claim the "getpip" name on PyPI for ourselves and then do the > following: > > 1. Provide "getpip" in the standard library for 3.4+ (and perhaps in a > 2.7.x release) > 2. Install it to site-packages in the "Python launcher for Windows" > installer for earlier versions > > getpip would expose at least one function: > > def bootstrap(index_url=None, system_install=False): > ... > > And executing it as a main module would either: > > 1. Do nothing, if "import pip" already works > 2. Call bootstrap with the appropriate arguments > > That way, installation instructions can simply say to unconditionally do: > > python -m getpip > > And that will either: > > 1. Report that pip is already installed; > 2. Bootstrap pip into the user environment; or > 3. Emit a distro-specific message if the distro packagers want to push > users to use the system pip instead (since they get to patch the system > Python and can tweak the system getpip however they want) > > The 2.7 change would then be to create a new download that bundles the > Windows launcher into the Windows installer. > > Users aren't stupid - the problem with the status quo is really that the > bootstrapping instructions are annoyingly complicated and genuinely > confusing, not that an explicit bootstrapping step is needed in the first > place. > > Cheers, > Nick. > > > > > > > > Thanks everyone for your brilliant feedback and discussion - I look > > forward to being able to say something sensible about Windows in the > > PEP :-) > > > > > > > > > > Richard > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > On 11 July 2013 06:50, Paul Moore wrote: > > I think "python -m pip" should be the canonical form (used in > documentation, > > examples, etc). The unittest module has taken this route, as has timeit. > > Traditionally, python-dev have been lukewarm about the -m interface, but > its > > key advantage is that it bypasses all the issues around versioned > > executables, cross-platform issues, the general dreadfulness of script > > wrappers on Windows, etc, in one fell swoop. > > "python -m pip" does make the bootstrapping a more complex proposition > - the stdlib would have to have something called "pip" that could be > overridden (while it is actually *running*) by something installed in > site-packages. Not easy. > > Thanks everyone for your brilliant feedback and discussion - I look > forward to being able to say something sensible about Windows in the > PEP :-) > > > Richard > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Jul 12 00:37:02 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 11 Jul 2013 18:37:02 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: I hope we will also arrive at a pip that doesn't need to be individually installed per venv... On Jul 11, 2013 6:13 PM, "Paul Moore" wrote: > +1 Explicit is better than implicit. > > Amending venv to automatically install pip (as suggested by Donald) may be > worth doing. I'm +0 on that (with the proviso that there's a --no-pip > option in that case). OTOH, the venv module is very extensible and writing > your own wrapper to import getpip and call bootstrap is pretty much trivial. > > On 11 July 2013 22:48, Nick Coghlan wrote: > >> (Oops, started this yesterday, got distracted and never hit send) >> >> On 11 July 2013 11:09, Richard Jones wrote: >> > >> > On 11 July 2013 06:50, Paul Moore wrote: >> > > I think "python -m pip" should be the canonical form (used in >> documentation, >> > > examples, etc). The unittest module has taken this route, as has >> timeit. >> > > Traditionally, python-dev have been lukewarm about the -m interface, >> but its >> > > key advantage is that it bypasses all the issues around versioned >> > > executables, cross-platform issues, the general dreadfulness of script >> > > wrappers on Windows, etc, in one fell swoop. >> > >> > "python -m pip" does make the bootstrapping a more complex proposition >> > - the stdlib would have to have something called "pip" that could be >> > overridden (while it is actually *running*) by something installed in >> > site-packages. Not easy. >> >> I was thinking about that, and I'm wondering if the most sensible option >> may be to claim the "getpip" name on PyPI for ourselves and then do the >> following: >> >> 1. Provide "getpip" in the standard library for 3.4+ (and perhaps in a >> 2.7.x release) >> 2. Install it to site-packages in the "Python launcher for Windows" >> installer for earlier versions >> >> getpip would expose at least one function: >> >> def bootstrap(index_url=None, system_install=False): >> ... >> >> And executing it as a main module would either: >> >> 1. Do nothing, if "import pip" already works >> 2. Call bootstrap with the appropriate arguments >> >> That way, installation instructions can simply say to unconditionally do: >> >> python -m getpip >> >> And that will either: >> >> 1. Report that pip is already installed; >> 2. Bootstrap pip into the user environment; or >> 3. Emit a distro-specific message if the distro packagers want to push >> users to use the system pip instead (since they get to patch the system >> Python and can tweak the system getpip however they want) >> >> The 2.7 change would then be to create a new download that bundles the >> Windows launcher into the Windows installer. >> >> Users aren't stupid - the problem with the status quo is really that the >> bootstrapping instructions are annoyingly complicated and genuinely >> confusing, not that an explicit bootstrapping step is needed in the first >> place. >> >> Cheers, >> Nick. >> >> >> >> >> > >> > Thanks everyone for your brilliant feedback and discussion - I look >> > forward to being able to say something sensible about Windows in the >> > PEP :-) >> >> >> > >> > >> > >> > Richard >> >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> On 11 July 2013 06:50, Paul Moore wrote: >> > I think "python -m pip" should be the canonical form (used in >> documentation, >> > examples, etc). The unittest module has taken this route, as has timeit. >> > Traditionally, python-dev have been lukewarm about the -m interface, >> but its >> > key advantage is that it bypasses all the issues around versioned >> > executables, cross-platform issues, the general dreadfulness of script >> > wrappers on Windows, etc, in one fell swoop. >> >> "python -m pip" does make the bootstrapping a more complex proposition >> - the stdlib would have to have something called "pip" that could be >> overridden (while it is actually *running*) by something installed in >> site-packages. Not easy. >> >> Thanks everyone for your brilliant feedback and discussion - I look >> forward to being able to say something sensible about Windows in the >> PEP :-) >> >> >> Richard >> > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Fri Jul 12 03:53:29 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 12 Jul 2013 01:53:29 +0000 (UTC) Subject: [Distutils] PEP 439 and pip bootstrap updated References: <51DCE0B5.6030506@oddbird.net> Message-ID: Daniel Holth gmail.com> writes: > I hope we will also arrive at a pip that doesn't need to be individually > installed per venv... You mean, like distil? :-) Regards, Vinay Sajip From richard at python.org Fri Jul 12 04:12:12 2013 From: richard at python.org (Richard Jones) Date: Fri, 12 Jul 2013 12:12:12 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: The point of PEP 439 is that the current situation of "but first do this" for any given 3rd-party package installation was a bad thing and we desire to move away from it. The PEP therefore proposes to allow "just do this" to eventually become the narrative. The direction this conversation is heading is removing that very significant primary benefit, and I'm not convinced there's any point to the PEP in that case. Richard From donald at stufft.io Fri Jul 12 04:19:45 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 11 Jul 2013 22:19:45 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On Jul 11, 2013, at 10:12 PM, Richard Jones wrote: > The point of PEP 439 is that the current situation of "but first do > this" for any given 3rd-party package installation was a bad thing and > we desire to move away from it. The PEP therefore proposes to allow > "just do this" to eventually become the narrative. The direction this > conversation is heading is removing that very significant primary > benefit, and I'm not convinced there's any point to the PEP in that > case. > > > Richard > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Now that I think about it some more I agree (and this was one of my sticking points with PyEnvs). There's already an API given to people who want to run a command to install pip: ``curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python`` Now that's platform dependent obviously but even then I don't see anyone documenting people should do that before installing things and I do think that blessing a script like that in the stdlib seems kind of pointless. The UX of the PEP as written as whenever you want to install something you run ``pip3 install foo``. The fact that pip _isn't_ bundled with Python and is instead fetched from PyPI is an implementation detail. It provides the major benefit of bundling it with Python without tying packaging to the release cycle of the stdlib (which has proven disastrous with distutils). We should remember that in general people have considered PyEnv that ships with Python 3.3 an inferior alternative to virtualenv largely in part because they have to fetch setuptools and pip prior to using it whereas in virtualenv they do not. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Fri Jul 12 07:11:14 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 12 Jul 2013 15:11:14 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On 12 July 2013 12:12, Richard Jones wrote: > The point of PEP 439 is that the current situation of "but first do > this" for any given 3rd-party package installation was a bad thing and > we desire to move away from it. The PEP therefore proposes to allow > "just do this" to eventually become the narrative. The direction this > conversation is heading is removing that very significant primary > benefit, and I'm not convinced there's any point to the PEP in that > case. > That was never the primary benefit to my mind. The status quo sucks because there is *no* simple answer to "first, do this", not because some kind of bootstrapping is needed. The problem in my view is that the "first, do this" step is currently a mess of various arcane platform dependendent incantations that may or may not work (and may even contradict each other) and can't be readily incorporated into an automatic script because they're not idempotent. Accordingly, I consider simplifying that "first, do this" step to "python -m getpip" to be a major upgrade from the status quo: * unlike curl, wget and "python -c" incantations, it's immediately obvious to a reader what it is supposed to do: "Get pip" * unlike curl, wget and "python -c" incantations, it can be easily made platform independent * unlike curl, wget and "python -c" incantations, it can be easily made idempotent (so it does nothing if it has already been run) * through "getpip.bootstrap" it will provide the infrastructure to easily add automatic bootstrapping to other tools In particular, it establishes the infrastructure to have pyvenv automatically bootstrap the installer into each venv, even when it isn't installed system wide (which is the key missing feature of pyvenv relative to virtualenv). Having the retrieval of pip happen automagically as part of an install command initially sounded nice, but I'm now a firm -1 on that because making it work cleanly in a cross-platform way that doesn't conflict with a manual pip install has proven to require several awkward compromises that make it an ugly solution: * we have to introduce a new install command (pip3 vs pip) to avoid packaging problems on Linux distros * this is then inconsistent with Windows (which doesn't have separate versioning for the Python 3 installation) * we have to introduce new bootstrap arguments to pip * we have to special case installation of pip and its dependencies to avoid odd looking warning messages * the implementation is tricky to explain * it doesn't work nicely with the "py" launcher on Windows (or equivalents that may be added to other platforms) If your reaction is "well, in that case, I don't want to write it anymore", I will be disappointed, but that won't stop me from rejecting this approach and waiting for someone else to volunteer to write the more explicit version based on your existing bootstrap code. I'd prefer not to do that though - I'd prefer it if I can persuade you that "python -m getpip" *is* a major upgrade over the status quo that is worth implementing, and one that adheres to the Zen of Python, in particular: * Explicit is better than implicit * Simple is better than complex * Readability counts * Errors should never pass silently, unless explicitly silenced * In the face of ambiguity, refuse the temptation to guess * If the implementation is hard to explain, it's a bad idea. * If the implementation is easy to explain, it may be a good idea. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jul 12 07:19:29 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 12 Jul 2013 15:19:29 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On 12 July 2013 15:11, Nick Coghlan wrote: > In particular, it establishes the infrastructure to have pyvenv > automatically bootstrap the installer into each venv, even when it isn't > installed system wide (which is the key missing feature of pyvenv relative > to virtualenv). > The other thing I will note is that *if* we decide to add an implicit bootstrap later (which I doubt will happen, but you never know), then having "getpip" available as an importable module will also make *that* easier to write (since the heart of the current bootstrap code could be replaced by "from getpip import bootstrap") Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Fri Jul 12 10:35:37 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 12 Jul 2013 08:35:37 +0000 (UTC) Subject: [Distutils] PEP 439 and pip bootstrap updated References: <51DCE0B5.6030506@oddbird.net> Message-ID: Donald Stufft stufft.io> writes: > We should remember that in general people have considered PyEnv that ships > with Python 3.3 an inferior alternative to virtualenv largely in part > because they have to fetch setuptools and pip prior to using it whereas in > virtualenv they do not. Let's remember, that's a consequence of packaging being pulled from 3.3 - the original plan was to have the ability to install stuff in venvs without third- party software being necessary. There is no real barrier to using setuptools/pip with Python 3.3+ venvs: For example, I published the pyvenvex.py script which creates venvs and installs setuptools and pip in a single step: https://gist.github.com/vsajip/4673395 Admittedly it's "only a Gist" and not especially publicised to the wider community, but that could be addressed. The current situation, as I see it, is a transitional one. When distlib-like functionality becomes available in the stdlib, other approaches will be possible, which improve upon what's possible with setuptools and pip. I've demonstrated some of this using distil. When targeting Python 3.4, shouldn't we be looking further than just advancing the status quo a little bit? It's been said numerous times that "executable setup.py" must go. ISTM that, notwithstanding "practicality beats purity", a pip bootstrap in Python would bless executable setup.py and help to extend its lifespan. Regards, Vinay Sajip From ncoghlan at gmail.com Fri Jul 12 15:22:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 12 Jul 2013 23:22:41 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On 12 Jul 2013 18:36, "Vinay Sajip" wrote: > > Donald Stufft stufft.io> writes: > > > We should remember that in general people have considered PyEnv that ships > > with Python 3.3 an inferior alternative to virtualenv largely in part > > because they have to fetch setuptools and pip prior to using it whereas in > > virtualenv they do not. > > Let's remember, that's a consequence of packaging being pulled from 3.3 - the > original plan was to have the ability to install stuff in venvs without third- > party software being necessary. > > There is no real barrier to using setuptools/pip with Python 3.3+ venvs: For > example, I published the pyvenvex.py script which creates venvs and installs > setuptools and pip in a single step: > > https://gist.github.com/vsajip/4673395 > > Admittedly it's "only a Gist" and not especially publicised to the wider > community, but that could be addressed. > > The current situation, as I see it, is a transitional one. When distlib-like > functionality becomes available in the stdlib, other approaches will be > possible, which improve upon what's possible with setuptools and pip. I've > demonstrated some of this using distil. When targeting Python 3.4, shouldn't > we be looking further than just advancing the status quo a little bit? > > It's been said numerous times that "executable setup.py" must go. ISTM that, > notwithstanding "practicality beats purity", a pip bootstrap in Python > would bless executable setup.py and help to extend its lifespan. Some day pip will get a "wheel only" mode, and that's the step that will kill off the need to run setup.py on production machines even when using the Python specific tools. Blessing both setuptools and pip as the "obvious way to do it" is designed to give us the wedge we need to start a gradual transition to that world without facing the initial barriers to adoption that were part of what scuttled the distutils2 effort. Cheers, Nick. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Jul 12 15:24:09 2013 From: brett at python.org (Brett Cannon) Date: Fri, 12 Jul 2013 09:24:09 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: On Fri, Jul 12, 2013 at 4:35 AM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > > > We should remember that in general people have considered PyEnv that > ships > > with Python 3.3 an inferior alternative to virtualenv largely in part > > because they have to fetch setuptools and pip prior to using it whereas > in > > virtualenv they do not. > > Let's remember, that's a consequence of packaging being pulled from 3.3 - > the > original plan was to have the ability to install stuff in venvs without > third- > party software being necessary. I think it's also a consequence of having to remember how to install pip. I don't have the get-pip.py URL memorized in order to pass it to curl to download for executing. At least with Nick's suggestion there is nothing more to remember than to run getpip right after you create your venv. It's also a consequence of habit and laziness, both of which programmers are notorious about holding on two with both hands as tightly as possible. =) > There is no real barrier to using setuptools/pip with Python 3.3+ venvs: > For > example, I published the pyvenvex.py script which creates venvs and > installs > setuptools and pip in a single step: > > https://gist.github.com/vsajip/4673395 > > Admittedly it's "only a Gist" and not especially publicised to the wider > community, but that could be addressed. > > The example in the venv docs actually does something similar but with distribute and pip: http://docs.python.org/3.4/library/venv.html#an-example-of-extending-envbuilder. I have filed a bug to update it to setuptools: http://bugs.python.org/issue18434 . > The current situation, as I see it, is a transitional one. When > distlib-like > functionality becomes available in the stdlib, other approaches will be > possible, which improve upon what's possible with setuptools and pip. I've > demonstrated some of this using distil. When targeting Python 3.4, > shouldn't > we be looking further than just advancing the status quo a little bit? > > It's been said numerous times that "executable setup.py" must go. ISTM > that, > notwithstanding "practicality beats purity", a pip bootstrap in Python > would bless executable setup.py and help to extend its lifespan. > I don't think that analogy is quite fair. It's not like setup.py either runs something if it's installed OR installs it and then continues execution. Having installation code execute arbitrary code is not a good thing, but executing code as part of executing an app make sense. =) But I do see the point you're trying to make. I'm personally +0 on the explicit install and +1 on the implicit bootstrap. I'm fine with adding a --no-bootstrap option that prevents the implicit install if people want to block it, or even prompting by default if people want to install and have a --bootstrap option for those that want to happen automatically for script usage. If this were a library we are talking about then I would feel differently, but since this is an app I don't feel bad about it. Then again as long at the getpip script simply exits quietly if pip is already installed then that's not a big thing either. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Jul 12 17:17:12 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 12 Jul 2013 11:17:12 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> On Jul 12, 2013, at 4:35 AM, Vinay Sajip wrote: > The current situation, as I see it, is a transitional one. When distlib-like > functionality becomes available in the stdlib, other approaches will be > possible, which improve upon what's possible with setuptools and pip. I've > demonstrated some of this using distil. When targeting Python 3.4, shouldn't > we be looking further than just advancing the status quo a little bit? > > It's been said numerous times that "executable setup.py" must go. ISTM that, > notwithstanding "practicality beats purity", a pip bootstrap in Python > would bless executable setup.py and help to extend its lifespan. There's very little reason why a pip bootstrap script couldn't unpack a wheel instead of using setup.py. Infact I've advocated for this and plan on contributing a bare bones wheel installation routine that would work well enough to get pip and setuptools installed. I'm also against adding distlib-like functionality to the stdlib. At least at this point in time. We've seen the far reaching effects that adding a packaging lib directly to the stdlib can have. I don't want to see us repeat the mistakes of the past and add distlib into the stdlib. Maybe in time once the packaging world isn't evolving so rapidly and distlib has had a lot of real world use that can be an option. The benefit for me in the way the pip/setuptools bootstrap is handled is that it's not merely imported into the stdlib and called done. It'll fetch the latest pip during each bootstrap, making it not a point of stagnation like distutils was. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Fri Jul 12 17:25:47 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 12 Jul 2013 11:25:47 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> Message-ID: <038A6971-FB53-497C-B1C0-24C734B4B64F@stufft.io> On Jul 12, 2013, at 1:19 AM, Nick Coghlan wrote: > On 12 July 2013 15:11, Nick Coghlan wrote: > In particular, it establishes the infrastructure to have pyvenv automatically bootstrap the installer into each venv, even when it isn't installed system wide (which is the key missing feature of pyvenv relative to virtualenv). > > The other thing I will note is that *if* we decide to add an implicit bootstrap later (which I doubt will happen, but you never know), then having "getpip" available as an importable module will also make *that* easier to write (since the heart of the current bootstrap code could be replaced by "from getpip import bootstrap") > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig I prefer the implicit bootstrap approach, but if the explicit bootstrap approach is chosen then something special needs to be done for pyvenv. If an explicit bootstrap is required for every pyvenv then I'm going to guess that people are going to just continue using virtualenv. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Fri Jul 12 17:52:46 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 12 Jul 2013 15:52:46 +0000 (UTC) Subject: [Distutils] PEP 439 and pip bootstrap updated References: <51DCE0B5.6030506@oddbird.net> Message-ID: Nick Coghlan gmail.com> writes: > Some day pip will get a "wheel only" mode, and that's the step that will kill > off the need to run setup.py on production machines even when using the > Python specific tools. > Blessing both setuptools and pip as the "obvious way to do it" is designed to > give us the wedge we need to start a gradual transition to that world without > facing the initial barriers to adoption that were part of what scuttled the > distutils2 effort. I think wheel is a good part of that wedge. Considering the barriers to adoption of distutils2: 1. Distutils2 expected people to migrate their setup.py to setup.cfg while providing only minimal help in doing so. I have gotten quite far in addressing the migration issue, in that I already have fully declarative metadata, *automatically* generated from setup.py / setup.cfg, and distil can do dependency resolution and installation using that metadata for a large number of distributions currently existing on PyPI. The automatic process might not be perfected yet, but it already does much of what one might expect given that it doesn't do e.g. exhaustive analysis of a setup.py to determine all possible code paths, etc. so it can't capture all environment- dependent info. 2. Distutils2 did not do any dependency resolution (not having any index metadata it could rely on for dependency information), but that's not the case with distlib. While it's not a full-blown solver, distlib's dependency resolution appears at least as good as setuptools'. 3. Windows seemed to be an afterthought for distutils2 - that's not the case with distlib. Although it may not be necessary because of the existence of the Python launcher for Windows, distlib has provision for e.g. native executables on Windows, just as setuptools does. 4. Distutils2 did not provide some functionality that setuptools users have come to rely on - e.g. entry points and package resources functionality. Distlib makes good many of these omissions, to the point where an implementation of pip using distlib to replace pkg_resources functionality has been developed and passes the pip test suite. 5. Distutils2 did not support the version scheme that setuptools does, but only the PEP 386 version scheme, which was another migration roadblock. Distlib supports PEP 440 versioning *and* setuptools versioning, so that barrier to adoption is no longer present. 6. Distutils2 did not provide "editable" installations for developers, but distil does (using ordinary .pth files, not setuptools-style "executable" ones). 7. Because wheel was not available in the distutils2 world, it would be hard for distutils to provide a build infrastructure as mature as distutils / setuptools extensions as provided by NumPy, SciPy etc. However, now that the wheel specification exists, and wheels can be built using setup.py and installed using distlib, there's much less of a reason to require setuptools and pip at installation time, and more of a reason to give developers reasons to provide their distributions in wheel format. While I'm not claiming that distlib is feature-complete, and while it doesn't have the benefit of being battle-tested through widespread usage, I'm asserting that it removes the barriers to adoption which distutils2 had, at least those identified above. I'm hoping that those who might disagree will speak up by identifying other barriers to adoption which I've failed to identify, or any requirements that I've failed to address satisfactorily in distlib. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Fri Jul 12 18:16:49 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 12 Jul 2013 16:16:49 +0000 (UTC) Subject: [Distutils] PEP 439 and pip bootstrap updated References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > I'm also against adding distlib-like functionality to the stdlib. At least at > this point in time. We've seen the far reaching effects that adding a > packaging lib directly to the stdlib can have. I don't want to see us repeat > the mistakes of the past and add distlib into the stdlib. Maybe in time once > the packaging world isn't evolving so rapidly and distlib has had a lot of > real world use that can be an option. The benefit for me in the way the pip/ On the question of whether distlib should or shouldn't be added to the stdlib, obviously that's for others to decide. My belief is that infrastructure areas like this need *some* stdlib underpinning. Also, distlib is pretty low-level, at the level of mechanism rather than policy, so there's no reason to be too paranoid about it in general terms. There's also some element of chicken and egg - inertia being what it is, I wouldn't expect *any* new packaging software outside the stdlib to gain significant adoption at any reasonable rate while the status quo is good enough for many people. But the status quo doesn't seem to allow any room for innovation. Distil is completely self-contained and does not require distlib to be in the stdlib, but it already does what could reasonably have been expected of packaging (if it had got into 3.3) and then some. What's more, it doesn't require installing into every venv - one copy covers all venvs (2.6+), user site-packages and system site-packages. > setuptools bootstrap is handled is that it's not merely imported into the > stdlib and called done. It'll fetch the latest pip during each bootstrap, > making it not a point of stagnation like distutils was. My pyvenvex script does this now. For venvs, that's the bootstrap right there. Of course, in cases where you want repeatability, getting the latest version each time might not be what you want :-) Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Fri Jul 12 18:23:52 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 12 Jul 2013 16:23:52 +0000 (UTC) Subject: [Distutils] PEP 439 and pip bootstrap updated References: <51DCE0B5.6030506@oddbird.net> <038A6971-FB53-497C-B1C0-24C734B4B64F@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > I prefer the implicit bootstrap approach, but if the explicit bootstrap > approach is chosen then something special needs to be done for pyvenv. The original pyvenv script did install Distribute and pip, but that functionality was removed before beta because Distribute and pip are third-party packages. If that restriction is lifted, we can easily replace the pyvenv script in Python with pyvenvex, and then (as I understand it) that is equivalent to an implicit bootstrap. Regards, Vinay Sajip From donald at stufft.io Fri Jul 12 18:24:10 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 12 Jul 2013 12:24:10 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> Message-ID: <34C42654-A1EF-4444-94C8-19297864A84F@stufft.io> On Jul 12, 2013, at 12:16 PM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > >> I'm also against adding distlib-like functionality to the stdlib. At least at >> this point in time. We've seen the far reaching effects that adding a >> packaging lib directly to the stdlib can have. I don't want to see us repeat >> the mistakes of the past and add distlib into the stdlib. Maybe in time once >> the packaging world isn't evolving so rapidly and distlib has had a lot of >> real world use that can be an option. The benefit for me in the way the pip/ > > On the question of whether distlib should or shouldn't be added to the stdlib, > obviously that's for others to decide. My belief is that infrastructure areas > like this need *some* stdlib underpinning. Also, distlib is pretty low-level, > at the level of mechanism rather than policy, so there's no reason to be too > paranoid about it in general terms. There's also some element of chicken > and egg - inertia being what it is, I wouldn't expect *any* new packaging > software outside the stdlib to gain significant adoption at any reasonable rate > while the status quo is good enough for many people. But the status quo doesn't > seem to allow any room for innovation. Eh, installing a pure Python Wheel is pretty simple. Especially if you restrict the options it can have. I don't see any reason why the bootstrap script can't include that as an internal implementation detail. I think it's kind of funny when folks say that new packaging software *needs* to be in the standard library when setuptools has pretty emphatically shown us that no it doesn't. People have problems with packaging, solve them without throwing away the world and they'll migrate. > > Distil is completely self-contained and does not require distlib to be in the > stdlib, but it already does what could reasonably have been expected of > packaging (if it had got into 3.3) and then some. What's more, it doesn't > require installing into every venv - one copy covers all venvs (2.6+), user > site-packages and system site-packages. pip used to have this and it was removed as a misfeature as it caused more problems then it solved. > >> setuptools bootstrap is handled is that it's not merely imported into the >> stdlib and called done. It'll fetch the latest pip during each bootstrap, >> making it not a point of stagnation like distutils was. > > My pyvenvex script does this now. For venvs, that's the bootstrap right there. > > Of course, in cases where you want repeatability, getting the latest version > each time might not be what you want :-) I haven't read your script in depth. But if that's all that's needed let's make sure it's done automatically for folks. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Fri Jul 12 18:56:55 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 12 Jul 2013 17:56:55 +0100 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> Message-ID: On 12 July 2013 16:17, Donald Stufft wrote: > There's very little reason why a pip bootstrap script couldn't unpack a > wheel instead of using setup.py. Infact I've advocated for this and plan on > contributing a bare bones wheel installation routine that would work well > enough to get pip and setuptools installed. I've written more than one bare-bones wheel installation script myself. They are easy to write (credit to Daniel for developing a format that's very simple to process!). I'm happy to donate any of the code that's useful. Here's one I've used in the past: https://gist.github.com/pfmoore/5985969 Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Fri Jul 12 19:10:54 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 12 Jul 2013 17:10:54 +0000 (UTC) Subject: [Distutils] PEP 439 and pip bootstrap updated References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> <34C42654-A1EF-4444-94C8-19297864A84F@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > Eh, installing a pure Python Wheel is pretty simple. Especially if you > restrict the options it can have. I don't see any reason why the bootstrap > script can't include that as an internal implementation detail. Sorry, I don't understand what you mean here, in terms of which of my points you are responding to. > I think it's kind of funny when folks say that new packaging software *needs* > to be in the standard library when setuptools has pretty emphatically shown > us that no it doesn't. People have problems with packaging, solve them > without throwing away the world and they'll migrate. Inertia definitely is a thing - otherwise why complain that an explicit bootstrap is much worse than an implicit one? The difference in work to use one rather than another isn't that great. I'm not saying that distlib (or any equivalent software) *has* or *needs* to be in the stdlib, merely that adoption will be faster if it is, and also that it is the right kind of software (infrastructure) which could reasonably be expected to be in the stdlib of a language which is acclaimed for (amongst other things) "batteries included". Setuptools, while not itself in the stdlib, built on packaging software that was, so the cases are not quite equivalent. Users did not have to do a major shift away from "executable setup.py", but if we're asking them to do that, it's slightly more work to migrate, even if you don't "throw away the world". And of course I agree that easing migration is important, which is why I've worked on migrating setup.py logic to declarative PEP 426, as far as is practicable. > pip used to have this and it was removed as a misfeature as it caused more > problems then it solved. Was it exactly the same? I don't remember this. I'd be interested in the specifics - can you point me to any more detailed information about this? > I haven't read your script in depth There's not much to it, it shouldn't take too long to review :-) Regards, Vinay Sajip From dholth at gmail.com Fri Jul 12 19:18:13 2013 From: dholth at gmail.com (Daniel Holth) Date: Fri, 12 Jul 2013 13:18:13 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> Message-ID: The goal is that it will be equally easy to install packages built with any build system. We are on our way. Getting rid of an executable build script is no longer a goal. Builds inherently need that often. But we don't want people extending distutils against their will. On Jul 12, 2013 11:59 AM, "Paul Moore" wrote: > > On 12 July 2013 16:17, Donald Stufft wrote: > >> There's very little reason why a pip bootstrap script couldn't unpack a >> wheel instead of using setup.py. Infact I've advocated for this and plan on >> contributing a bare bones wheel installation routine that would work well >> enough to get pip and setuptools installed. > > > I've written more than one bare-bones wheel installation script myself. > They are easy to write (credit to Daniel for developing a format that's > very simple to process!). I'm happy to donate any of the code that's > useful. Here's one I've used in the past: > https://gist.github.com/pfmoore/5985969 > > Paul > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Jul 12 19:27:19 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 12 Jul 2013 13:27:19 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> <34C42654-A1EF-4444-94C8-19297864A84F@stufft.io> Message-ID: <640F3E77-5D43-49F6-9580-27544B3495D2@stufft.io> On Jul 12, 2013, at 1:10 PM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > >> Eh, installing a pure Python Wheel is pretty simple. Especially if you >> restrict the options it can have. I don't see any reason why the bootstrap >> script can't include that as an internal implementation detail. > > Sorry, I don't understand what you mean here, in terms of which of my points > you are responding to. Maybe I misunderstood your point :) I thought you were saying that by installing pip using setup.py install we are "blessing" setup.py install again? I was saying we don't need to do that. > >> I think it's kind of funny when folks say that new packaging software *needs* >> to be in the standard library when setuptools has pretty emphatically shown >> us that no it doesn't. People have problems with packaging, solve them >> without throwing away the world and they'll migrate. > > Inertia definitely is a thing - otherwise why complain that an explicit > bootstrap > is much worse than an implicit one? The difference in work to use one rather > than > another isn't that great. I'm not saying that distlib (or any equivalent > software) *has* or *needs* to be in the stdlib, merely that adoption will be > faster if it is, and also that it is the right kind of software > (infrastructure) which could reasonably be expected to be in the stdlib of a > language which is acclaimed for (amongst other things) "batteries included". > > Setuptools, while not itself in the stdlib, built on packaging software that > was, so the cases are not quite equivalent. Users did not have to do a major > shift away from "executable setup.py", but if we're asking them to do that, > it's slightly more work to migrate, even if you don't "throw away the world". > And of course I agree that easing migration is important, which is why I've > worked on migrating setup.py logic to declarative PEP 426, as far as is > practicable. I'm not overly found of bootstrapping setuptools itself, but I think unless pip comes along and bundles setuptools like it has done distlib it's a nesceary evil right now. Ideally In the future we can move things to where setuptools is just a build tool and isn't something needed at install time unless you're doing a build. I generally agree that a packaging library is the type of item that belongs in a stdlib, I don't think it belongs in there *yet*. We can work around it not being there, and that means we can be more agile about it and evolve the tooling till we are happy with them instead of trying to get it in as quickly as possible to make things easier in the short term and possibly harder in the long term. > >> pip used to have this and it was removed as a misfeature as it caused more >> problems then it solved. > > Was it exactly the same? I don't remember this. I'd be interested in the > specifics - can you point me to any more detailed information about this? > >> I haven't read your script in depth > > There's not much to it, it shouldn't take too long to review :-) > > Regards, > > Vinay Sajip > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Fri Jul 12 19:28:10 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 12 Jul 2013 17:28:10 +0000 (UTC) Subject: [Distutils] PEP 439 and pip bootstrap updated References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> Message-ID: Daniel Holth gmail.com> writes: > Getting rid of an executable build script is no longer a goal. Builds > inherently need that often. But we don't want people extending distutils > against their will. Perhaps I should have been clearer - I meant "executable setup.py install", and as I understand it, it is a goal to get rid of that. Regarding "executable setup.py build", that's less of an issue than for installing, but IIUC, it is still not ideal. Many of the hacks that people have made around distutils/setuptools relate to building, not just installing, or am I wrong? Regards, Vinay Sajip From donald at stufft.io Fri Jul 12 19:34:22 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 12 Jul 2013 13:34:22 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> Message-ID: <19726835-82CC-4133-A161-8E40B590C9BB@stufft.io> On Jul 12, 2013, at 1:28 PM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > >> Getting rid of an executable build script is no longer a goal. Builds >> inherently need that often. But we don't want people extending distutils >> against their will. > > Perhaps I should have been clearer - I meant "executable setup.py install", > and as I understand it, it is a goal to get rid of that. Yes it's a goal to get rid of setup.py install, but I doubt it will ever fully be gone. At least not for a long time. There's almost 150k source dist packages on PyPI and I'm going to assume the vast bulk of them have a setup.py. > > Regarding "executable setup.py build", that's less of an issue than for > installing, but IIUC, it is still not ideal. Many of the hacks that people > have made around distutils/setuptools relate to building, not just > installing, or am I wrong? It's not ideal, but it's also largely only an issue on the machine of the developer who is packaging the software. If they are fine with the hacks then there's not a major reason to move them away from that. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Fri Jul 12 19:42:49 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 12 Jul 2013 17:42:49 +0000 (UTC) Subject: [Distutils] PEP 439 and pip bootstrap updated References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> <19726835-82CC-4133-A161-8E40B590C9BB@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > Yes it's a goal to get rid of setup.py install, but I doubt it will ever > fully be gone. At least not for a long time. There's almost 150k source > dist packages on PyPI and I'm going to assume the vast bulk of them have > a setup.py. True, but distil seems to be able to install a fair few (certainly the ones which don't do significant special processing in their setup.py, such as moving files around and creating files) without ever executing setup.py. > It's not ideal, but it's also largely only an issue on the machine of > the developer who is packaging the software. If they are fine with the > hacks then there's not a major reason to move them away from that. It's a smaller community than the users of those projects, and I don't know what the numbers of affected developers are. Obviously it's up to each project how they do their stuff, but from my understanding the NumPy/SciPy communities aren't especially happy with the extensions they've had to do (else, why Bento?) Regards, Vinay Sajip From brett at python.org Fri Jul 12 20:00:06 2013 From: brett at python.org (Brett Cannon) Date: Fri, 12 Jul 2013 14:00:06 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> Message-ID: On Fri, Jul 12, 2013 at 12:16 PM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > > > I'm also against adding distlib-like functionality to the stdlib. At > least at > > this point in time. We've seen the far reaching effects that adding a > > packaging lib directly to the stdlib can have. I don't want to see us > repeat > > the mistakes of the past and add distlib into the stdlib. Maybe in time > once > > the packaging world isn't evolving so rapidly and distlib has had a lot > of > > real world use that can be an option. The benefit for me in the way the > pip/ > > On the question of whether distlib should or shouldn't be added to the > stdlib, > obviously that's for others to decide. [SNIP] Speaking with my python-dev hat on which has a badge from when I led the stdlib cleanup for Python 3, I would say anything that has a PEP should probably have a module in the stdlib for it. That way standard management of whatever is specified in the PEP will be uniform and expected to be maintained and work. Beyond that code will exist outside the stdlib. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Fri Jul 12 20:11:43 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 12 Jul 2013 18:11:43 +0000 (UTC) Subject: [Distutils] PEP 439 and pip bootstrap updated References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> <34C42654-A1EF-4444-94C8-19297864A84F@stufft.io> <640F3E77-5D43-49F6-9580-27544B3495D2@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > Maybe I misunderstood your point :) I thought you were saying that by > installing pip using setup.py install we are "blessing" setup.py install > again? I was saying we don't need to do that. Okay, I see. I'm used to comments referring to points directly above them, and my comment about blessings was at the end of my post. I meant that pip itself, and not just the bootstrap, uses "setup.py install". I would have thought that pip don't need no steenking blessing from anyone :-), but that's what the PEP is about, after all. > I'm not overly found of bootstrapping setuptools itself, but I think > unless pip comes along and bundles setuptools like it has done distlib > it's a nesceary evil right now. Ideally In the future we can move things But aren't you in favour of getting the latest version of setuptools and pip each time? > to where setuptools is just a build tool and isn't something needed at > install time unless you're doing a build. That "unless" - that stops the clean separation between build and install which wheel enables, and which would be a Good Thing to encourage. > I generally agree that a packaging library is the type of item that > belongs in a stdlib, I don't think it belongs in there *yet*. We can work > around it not being there, and that means we can be more agile about it > and evolve the tooling till we are happy with them instead of trying to > get it in as quickly as possible to make things easier in the short term > and possibly harder in the long term. Oh, I agree there's no sense in rushing things. But how do we know when we're happy enough (or not) with something? When we try it out, that's when we can form an opinion - not before. It's been a good while since I first announced distil, both as a test-bed for distlib, but also as a POC for better user experiences with packaging. Apart from Paul Moore (thanks, Paul!), I've had precious little specific feedback from anyone here (and believe me, I'd welcome adverse feedback if it's warranted). It could all be a steaming pile of the proverbial, or the best thing since sliced proverbials, but there's no way to know. Of course there are good reasons for this - we are all busy people. Inertia, thy ways are many :-) Regards, Vinay Sajip From donald at stufft.io Fri Jul 12 20:16:33 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 12 Jul 2013 14:16:33 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> Message-ID: <1770C961-BC2F-4062-85BB-A82131013FD6@stufft.io> On Jul 12, 2013, at 2:00 PM, Brett Cannon wrote: > Speaking with my python-dev hat on which has a badge from when I led the stdlib cleanup for Python 3, I would say anything that has a PEP should probably have a module in the stdlib for it. That way standard management of whatever is specified in the PEP will be uniform and expected to be maintained and work. Beyond that code will exist outside the stdlib. This is basically the exact opposite of what Nick has said the intent has been (Ecosystem first). Adding packaging tools beyond bootstrapping pip at this point in the game is IMO a huge mistake. If what Nick has said and PEPs are not appropriate for things that don't have a module in the standard lib well that's fine I guess. I just won't worry about trying to write PEPs :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From brett at python.org Fri Jul 12 21:25:04 2013 From: brett at python.org (Brett Cannon) Date: Fri, 12 Jul 2013 15:25:04 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: <1770C961-BC2F-4062-85BB-A82131013FD6@stufft.io> References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> <1770C961-BC2F-4062-85BB-A82131013FD6@stufft.io> Message-ID: On Fri, Jul 12, 2013 at 2:16 PM, Donald Stufft wrote: > > On Jul 12, 2013, at 2:00 PM, Brett Cannon wrote: > > Speaking with my python-dev hat on which has a badge from when I led the > stdlib cleanup for Python 3, I would say anything that has a PEP should > probably have a module in the stdlib for it. That way standard management > of whatever is specified in the PEP will be uniform and expected to be > maintained and work. Beyond that code will exist outside the stdlib. > > > This is basically the exact opposite of what Nick has said the intent has > been (Ecosystem first). > > Not at all as no module will go in immediately until after a PEP has landed and been vetted as needed. > Adding packaging tools beyond bootstrapping pip at this point in the game > is IMO a huge mistake. If what Nick has said and PEPs are not appropriate > for things that don't have a module in the standard lib well that's fine I > guess. > You misunderstand what I mean. I'm just saying that *if* anything were to go into the stdlib it would only be to have code which implements a PEP in the stdlib to prevent everyone from re-implementing a standard. > I just won't worry about trying to write PEPs :) > No, the PEPs are important to prevent version skew and make sure everyone is on the same page. And that's also what a module in the stdlib would do; make sure everyone is on the same page in terms of semantics by using a single code base. I mean I wouldn't expect anything more than maybe code parsing the JSON metadata that does some validation and parsing version numbers that can support comparisons and verifying platform requirements; that's it. Stuff that every installation tool will need to do in order to follow the PEPs properly. And it wouldn't go in until everyone was very happy with the PEPs and ready to commit to code enshrining it in the stdlib. Otherwise I hope distlib moves into PyPA and everyone who develops installation tools, etc. uses that library. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Jul 12 23:21:06 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 12 Jul 2013 17:21:06 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> <1770C961-BC2F-4062-85BB-A82131013FD6@stufft.io> Message-ID: <096D11E0-7859-4466-BA95-FE0BDC43D0B9@stufft.io> On Jul 12, 2013, at 3:25 PM, Brett Cannon wrote: > > > > On Fri, Jul 12, 2013 at 2:16 PM, Donald Stufft wrote: > > On Jul 12, 2013, at 2:00 PM, Brett Cannon wrote: > >> Speaking with my python-dev hat on which has a badge from when I led the stdlib cleanup for Python 3, I would say anything that has a PEP should probably have a module in the stdlib for it. That way standard management of whatever is specified in the PEP will be uniform and expected to be maintained and work. Beyond that code will exist outside the stdlib. > > This is basically the exact opposite of what Nick has said the intent has been (Ecosystem first). > > > Not at all as no module will go in immediately until after a PEP has landed and been vetted as needed. > > Adding packaging tools beyond bootstrapping pip at this point in the game is IMO a huge mistake. If what Nick has said and PEPs are not appropriate for things that don't have a module in the standard lib well that's fine I guess. > > You misunderstand what I mean. I'm just saying that *if* anything were to go into the stdlib it would only be to have code which implements a PEP in the stdlib to prevent everyone from re-implementing a standard. > > I just won't worry about trying to write PEPs :) > > No, the PEPs are important to prevent version skew and make sure everyone is on the same page. And that's also what a module in the stdlib would do; make sure everyone is on the same page in terms of semantics by using a single code base. > > I mean I wouldn't expect anything more than maybe code parsing the JSON metadata that does some validation and parsing version numbers that can support comparisons and verifying platform requirements; that's it. Stuff that every installation tool will need to do in order to follow the PEPs properly. And it wouldn't go in until everyone was very happy with the PEPs and ready to commit to code enshrining it in the stdlib. Otherwise I hope distlib moves into PyPA and everyone who develops installation tools, etc. uses that library. I could probably be convinced about something that makes handling versions easier going into the standard lib, but that's about it. There's a few reasons that I don't want these things added to the stdlib themselves. One of the major ones is that of "agility". We've seen with distutils how impossible it can be to make improvements to the system. Now some of this is made better with the way the new system is being designed with versioned metadata but it doesn't completely go away. We can look at Python's past to see just how long any individual version sticks around and we can assume that if something gets added now that particular version will be around for a long time. Another is because of how long it can take a new version of Python to become "standard", especially in the 3.x series since the entire 3.x series itself isn't standard, any changes made to the standard lib won't be usable for years and years. This can be mitigated by releasing a backport on PyPI, but if every version of Python but the latest one is going to require installing these libs from PyPI in order to usefully interact with the "world", then you might as well just require all versions of Python to install bits from PyPI. Yet another is by blessing a particular implementation, that implementations behaviors become the standard (indeed the way the PEP system generally works for this is once it's been added to the standard lib the PEP is a historical document and the documentation becomes the standard). However packaging is not like Enums or urllibs, or smtp. We are essentially defining a protocol, one that non Python tools will be expected to use (for Debian and RPMs for example). We are using these PEPs more like a RFC than a proposal to include something in the stdlib. There's also the case of usefulness. You mention some code that can parse the JSON metadata and validate it. Weel assumingly we'll have the metadata for 2.0 set down by the time 3.4 comes around. So sure 3.4 could have that, but then maybe we release metadata 2.1 and now 3.4 can only parse _some_ of the metadata. Maybe we release a metadata 3.0 and now it can't parse any metadata. But even if it can parse the metadata what does it do with it? The major places you'd be validating the metadata (other than merely consuming it) is either on the tools that create packages or in PyPI performing checks on a valid file upload. In the build tool case they are going to either need to write their own code for actually creating the package or, more likely, they'll reuse something like distlib. If those tools are already going to be using a distlib-like library then we might as just keep the validation code in there. Now the version parsing stuff which I said I could be convinced is slightly different. It is really sort of it's own thing. It's not dependent on the other pieces of packaging to be useful, and it's not versioned. It's also the only bit that's really useful on it's own. People consuming the (future) PyPI API could use it to fully depict the actual metadata so it's kind of like JSON itself in that regard. The installer side of things the purist side of me doesn't like adding it to the standard library for all the same reasons but the pragmatic side of me wants it there because it enables fetching the other bits that are needed for "pip install X" to be a reasonable official response to these kind of questions. But I pushed for and still believe that if a prerequisite for doing that involves "locking" in pip or any of it's dependencies by adding them to the standard library then I am vehemently against doing it. Wow that was a lot of words... ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Sat Jul 13 01:14:05 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 12 Jul 2013 23:14:05 +0000 (UTC) Subject: [Distutils] PEP 439 and pip bootstrap updated References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> <1770C961-BC2F-4062-85BB-A82131013FD6@stufft.io> <096D11E0-7859-4466-BA95-FE0BDC43D0B9@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > I could probably be convinced about something that makes handling versions > easier going into the standard lib, but that's about it. That seems completely arbitrary to me. Why just versions? Why not, for example, support for the wheel format? Why not agreed metadata formats? > There's a few reasons that I don't want these things added to the stdlib > themselves. > > One of the major ones is that of "agility". We've seen with distutils how > impossible it can be to make improvements to the system. Now some of this You say that, but setuptools, the poster child of packaging, improved quite a lot on distutils. I'm not convinced that it would have been as successful if there were no distutils in the stdlib, but of course you may disagree. I'm well aware of the "the stdlib is where software goes to die" school of thought, and I have considerable sympathy for where it's coming from, but let's not throw the baby out with the bathwater. The agility argument could be made for lots of areas of functionality, to the point where you just basically never add anything new to the stdlib because you're worried about an inability to cope with change. Also, it doesn't seem right to point to particular parts of the stdlib which were hard to adapt to changing requirements and draw the conclusion that all software added to the stdlib would be equally hard to adapt. Of course one could look at a specific piece of software and assess its adaptability, but otherwise, isn't it verging on just arm-waving? > is made better with the way the new system is being designed with versioned > metadata but it doesn't completely go away. We can look at Python's past to > see just how long any individual version sticks around and we can assume that > if something gets added now that particular version will be around for a long > time. That doesn't mean that overall improvements can't take place in the stdlib. For example, getopt -> optparse -> argparse. > Another is because of how long it can take a new version of Python to become > "standard", especially in the 3.x series since the entire 3.x series itself > isn't standard, any changes made to the standard lib won't be usable for > years and years. This can be mitigated by releasing a backport on PyPI, but > if every version of Python but the latest one is going to require installing > these libs from PyPI in order to usefully interact with the "world", then you > might as well just require all versions of Python to install bits from PyPI. Well, other approaches have been looked at - for example, accepting things into the stdlib but warning users about the provisional nature of some APIs. I think that where interoperability between different packaging tools is needed, that's where the argument for something in the stdlib is strongest, as Brett said. > Yet another is by blessing a particular implementation, that implementations > behaviors become the standard (indeed the way the PEP system generally works > for this is once it's been added to the standard lib the PEP is a historical > document and the documentation becomes the standard). However packaging is That's because the PEP is needed to advocate the inclusion in the stdlib and as a record of the discussion and rationale for accepting/rejecting whatever was advocated, but there's no real benefit in keeping the PEP updated as the stdlib component gets refined from its real-world exposure through being in the stdlib. > not like Enums or urllibs, or smtp. We are essentially defining a protocol, > one that non Python tools will be expected to use (for Debian and RPMs for > example). We are using these PEPs more like a RFC than a proposal to include > something in the stdlib. But we can assume that there will either be N different implementations of everything in the RFCs from the ground up, by N different tools, or ideally one canonical implementation in the stdlib that the tool makers can use (but are not forced to use if they don't want to). You might say that if there were some kick-ass implementation of these RFCs on PyPI people would just gravitate to it and the winner would be obvious, but I don't see things working like that. In the web space, look at HTTP Request/Response objects as an example: Pyramid, Werkzeug, Django all have their own, don't really interoperate in practice (though it was a goal of WSGI), and there's very little to choose between them technically. Just a fair amount of duplicated effort on something so low-level, which would have been better spent on truly differentiating features. > There's also the case of usefulness. You mention some code that can parse the > JSON metadata and validate it. Weel assumingly we'll have the metadata for > 2.0 set down by the time 3.4 comes around. So sure 3.4 could have that, but > then maybe we release metadata 2.1 and now 3.4 can only parse _some_ of the > metadata. Maybe we release a metadata 3.0 and now it can't parse any > metadata. But even if it can parse the metadata what does it do with it? The > major places you'd be validating the metadata (other than merely consuming > it) is either on the tools that create packages or in PyPI performing checks > on a valid file upload. In the build tool case they are going to either need > to write their own code for actually creating the package or, more likely, > they'll reuse something like distlib. If those tools are already going to be > using a distlib-like library then we might as just keep the validation code > in there. Is that some blessed-by-being-in-the-stdlib kind of library that everyone uses, or one of several balkanised versions a la HTTP Request / Response? If it's not somehow blessed, why should a particular packaging project use it, even if it's technically up to the job? > Now the version parsing stuff which I said I could be convinced is slightly > different. It is really sort of it's own thing. It's not dependent on the > other pieces of packaging to be useful, and it's not versioned. It's also the > only bit that's really useful on it's own. People consuming the (future) PyPI > API could use it to fully depict the actual metadata so it's kind of like > JSON itself in that regard. That's only because some effort has gone into looking at version comparisons, ordering, pre-/post-/dev-releases, etc. and considering the requirements in some detail. It looks OK now, but so did PEP 386 to many people who hadn't considered the ordering of dev versions of pre-/post-releases. Who's to say that some other issue won't come up that we haven't considered? It's not a reason for doing nothing. > The installer side of things the purist side of me doesn't like adding it to > the standard library for all the same reasons but the pragmatic side of me > wants it there because it enables fetching the other bits that are needed for > "pip install X" to be a reasonable official response to these kind of > questions. But I pushed for and still believe that if a prerequisite for > doing that involves "locking" in pip or any of it's dependencies by adding > them to the standard library then I am vehemently against doing it. Nobody seems to be suggesting doing that, though. Regards, Vinay Sajip From nathaniel at google.com Sat Jul 13 00:58:25 2013 From: nathaniel at google.com (Nathaniel Manista) Date: Fri, 12 Jul 2013 15:58:25 -0700 Subject: [Distutils] Issues with Buildout 2.2.0 and a few other questions Message-ID: Hey Buildout Folks- I'm the lead of the Melange project and we've hit two issues with the recent Buildout release. The first is the setuptools version problem described here, and the second is the node problem described here . I see that a package upgrade is the recommended solution to the first problem, but that's not necessarily easy in our case as we've got a mix of corporate machines and academic machines and otherwise managed platforms - is there anything like a list of supported platforms for Buildout 2.2.0? Is Buildout fully supported on Ubuntu 12.04 (the most recent long-term support release of Ubuntu)? Are there known and unsupported incompatibilities on Ubuntu 12.04? Does Buildout maintain a list of supported platforms and configurations? Is the Buildout team aware of the second (node) issue? In looking for my own answers to these questions, I came across some issues on the buildout.org site. This linkon the front page appears broken, and the latest documentation appears to describe Buildout 1.2.1(with out-of-date references such as "wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py"). What's going on there? Thanks much, -Nathaniel -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Jul 13 02:02:00 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 12 Jul 2013 20:02:00 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> <1770C961-BC2F-4062-85BB-A82131013FD6@stufft.io> <096D11E0-7859-4466-BA95-FE0BDC43D0B9@stufft.io> Message-ID: <95EAEB1E-07CD-416A-8ADA-2C872FF4609B@stufft.io> On Jul 12, 2013, at 7:14 PM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > >> I could probably be convinced about something that makes handling versions >> easier going into the standard lib, but that's about it. > > That seems completely arbitrary to me. Why just versions? Why not, for > example, support for the wheel format? Why not agreed metadata formats? As I said in my email, because it's more or less standalone and it has the greatest utility outside of installers/builders/archivers/indexes. > >> There's a few reasons that I don't want these things added to the stdlib >> themselves. >> >> One of the major ones is that of "agility". We've seen with distutils how >> impossible it can be to make improvements to the system. Now some of this > > You say that, but setuptools, the poster child of packaging, improved quite > a lot on distutils. I'm not convinced that it would have been as successful > if there were no distutils in the stdlib, but of course you may disagree. I've looked at many other languages where they had widely successful packaging tools that weren't added to the standard lib until they were ubiquitous and stable. Something the new tools for Python are not. So I don't think adding it to the standard library is required. And setuptools improved it *outside* of the standard library while distutils itself stagnated. I would venture to guess that if distutils *hadn't* been in the standard library than setuptools could have simply been patches to distutils instead of needing to be essentially "replace" distutils and it just so happened to reuse some it's functionality. So pointing towards setuptools just exposes the fact that improving it in the standard library was hard enough that it was done externally. > > I'm well aware of the "the stdlib is where software goes to die" school of > thought, and I have considerable sympathy for where it's coming from, but > let's not throw the baby out with the bathwater. The agility argument could > be made for lots of areas of functionality, to the point where you just > basically never add anything new to the stdlib because you're worried about > an inability to cope with change. Also, it doesn't seem right to point to > particular parts of the stdlib which were hard to adapt to changing > requirements and draw the conclusion that all software added to the stdlib > would be equally hard to adapt. Of course one could look at a specific piece > of software and assess its adaptability, but otherwise, isn't it verging on > just arm-waving? Well I am of the mind that the standard library is where software goes to die, and I'm also of the mind that a smaller standard library and a strong packaging story and ecosystem is far superior. But that's not what I'm advocating here. A key point to almost every other part of the standard library is if stagnates or falls behind or is unable to adapt then you simply don't use it. This is not a hard thing to do for something like httplib, urllib2, urlib, etc because it's what people have *done* in projects like requests. One persons choice to use url lib in his software has little to no bearing on someone else who might choose to use requests. However a packaging system needs interoperability. My choice to use a particular package software, if there is no interoperability, DRASTICALLY affects you if you want to use my software at all. A huge thing i've been trying to push for is decoupling packaging from a specific implementation so that we have a "protocol" (ala HTTP) and not a "tool" (ala distutils). However the allure of working to the implementation and not the standard is fairly high when there is a singular blessed implementation. > >> is made better with the way the new system is being designed with versioned >> metadata but it doesn't completely go away. We can look at Python's past to >> see just how long any individual version sticks around and we can assume that >> if something gets added now that particular version will be around for a long >> time. > > That doesn't mean that overall improvements can't take place in the stdlib. > For example, getopt -> optparse -> argparse. It's funny you picked and example where improvements *couldn't* take place and the entire system had to be thrown out and a new one written. getopt had to become a new module named opt parse, which had to become a new module named argparse in order to make changes to it. I don't think we need to have distutils, distlib, futurelib, even-further-futurelib and I think that makes packaging even more confusing then it needs to be. This also ties in with the above where one persons use of getopt instead of argparse doesn't drastically affect another person using a different one. > >> Another is because of how long it can take a new version of Python to become >> "standard", especially in the 3.x series since the entire 3.x series itself >> isn't standard, any changes made to the standard lib won't be usable for >> years and years. This can be mitigated by releasing a backport on PyPI, but >> if every version of Python but the latest one is going to require installing >> these libs from PyPI in order to usefully interact with the "world", then you >> might as well just require all versions of Python to install bits from PyPI. > > Well, other approaches have been looked at - for example, accepting things > into the stdlib but warning users about the provisional nature of some APIs. Provisional API's still exist in that version of Python and the only way someone would get a new one is by installing a package. I think that this makes the problem even *worse* because now you're adding API's to the standard library that have a good chance of needing to change and needing to require people to install a package (with no good way to communicate to someone that they need to update it since it's a standard library package and not a versioned installed package). > > I think that where interoperability between different packaging tools is > needed, that's where the argument for something in the stdlib is strongest, > as Brett said. You can gain interoperability in a few ways. One way is to just pick an implementation and make that the standard. Another is to define *actual* standards. The second one is harder, requires more thought and work. But it means that completely different software can work together. It means that something written in Ruby can easily work with a python package without shelling out to Python or without trying to copy all the implementation details and having to guess which ones are significant or not. > >> Yet another is by blessing a particular implementation, that implementations >> behaviors become the standard (indeed the way the PEP system generally works >> for this is once it's been added to the standard lib the PEP is a historical >> document and the documentation becomes the standard). However packaging is > > That's because the PEP is needed to advocate the inclusion in the stdlib and > as a record of the discussion and rationale for accepting/rejecting whatever > was advocated, but there's no real benefit in keeping the PEP updated as the > stdlib component gets refined from its real-world exposure through being in > the stdlib. And that's fine for a certain class of problems. It's not that useful for something where you want interoperability outside of that tool. How terrible would it be if HTTP was "well whatever Apache does, that's what HTTP is". > >> not like Enums or urllibs, or smtp. We are essentially defining a protocol, >> one that non Python tools will be expected to use (for Debian and RPMs for >> example). We are using these PEPs more like a RFC than a proposal to include >> something in the stdlib. > > But we can assume that there will either be N different implementations of > everything in the RFCs from the ground up, by N different tools, or ideally > one canonical implementation in the stdlib that the tool makers can use (but > are not forced to use if they don't want to). You might say that if there > were some kick-ass implementation of these RFCs on PyPI people would just > gravitate to it and the winner would be obvious, but I don't see things > working like that. In the web space, look at HTTP Request/Response objects > as an example: Pyramid, Werkzeug, Django all have their own, don't really > interoperate in practice (though it was a goal of WSGI), and there's very > little to choose between them technically. Just a fair amount of duplicated > effort on something so low-level, which would have been better spent on > truly differentiating features. A singular blessed tool in the standard library incentivizes the standard becoming and implementation detail. I *want* there to be multiple implementations written by different people working on different "slices" of the problem. That incentivizes doing the extra work on PEPs and other documents so that we maintain a highly documented standard. It's true that adding something to the standard library doesn't rule that out but it provides an incentive against properly doing standards because it's easier and simpler to just change it in the implementation. > >> There's also the case of usefulness. You mention some code that can parse the >> JSON metadata and validate it. Weel assumingly we'll have the metadata for >> 2.0 set down by the time 3.4 comes around. So sure 3.4 could have that, but >> then maybe we release metadata 2.1 and now 3.4 can only parse _some_ of the >> metadata. Maybe we release a metadata 3.0 and now it can't parse any >> metadata. But even if it can parse the metadata what does it do with it? The >> major places you'd be validating the metadata (other than merely consuming >> it) is either on the tools that create packages or in PyPI performing checks >> on a valid file upload. In the build tool case they are going to either need >> to write their own code for actually creating the package or, more likely, >> they'll reuse something like distlib. If those tools are already going to be >> using a distlib-like library then we might as just keep the validation code >> in there. > > Is that some blessed-by-being-in-the-stdlib kind of library that everyone > uses, or one of several balkanised versions a la HTTP Request / Response? If > it's not somehow blessed, why should a particular packaging project use it, > even if it's technically up to the job? It's not blessed and a particular packaging project should use it if it fits their needs and they want to use it. Or they shouldn't use it if they don't want. Standards exist for a reason. So you can have multiple implementations that all work together. > >> Now the version parsing stuff which I said I could be convinced is slightly >> different. It is really sort of it's own thing. It's not dependent on the >> other pieces of packaging to be useful, and it's not versioned. It's also the >> only bit that's really useful on it's own. People consuming the (future) PyPI >> API could use it to fully depict the actual metadata so it's kind of like >> JSON itself in that regard. > > That's only because some effort has gone into looking at version > comparisons, ordering, pre-/post-/dev-releases, etc. and considering the > requirements in some detail. It looks OK now, but so did PEP 386 to many > people who hadn't considered the ordering of dev versions of > pre-/post-releases. Who's to say that some other issue won't come up that we > haven't considered? It's not a reason for doing nothing. I didn't make any claims as to it's stability or the amount of testing that went into it. My ability to be convinced of that stems primarily from the fact that it's sort of a side piece of the whole packaging infrastructure and toolchain and it's also a piece that is most likely to be useful on it's own. > >> The installer side of things the purist side of me doesn't like adding it to >> the standard library for all the same reasons but the pragmatic side of me >> wants it there because it enables fetching the other bits that are needed for >> "pip install X" to be a reasonable official response to these kind of >> questions. But I pushed for and still believe that if a prerequisite for >> doing that involves "locking" in pip or any of it's dependencies by adding >> them to the standard library then I am vehemently against doing it. > > Nobody seems to be suggesting doing that, though. I was (trying?) to explain that my belief doesn't extend to only distlib here and instead to the entire toolchain. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Sat Jul 13 07:31:14 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 13 Jul 2013 15:31:14 +1000 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) Message-ID: In addition to the long thread based on Richard's latest set of updates, I've also received a few off-list comments on the current state of the proposal. So, I figured I'd start a new thread summarising my current point of view and see where we want to go from there. 1. However we end up solving the bootstrapping problem, I'm *definitely* a fan of us updating pyvenv in 3.4 to ensure that pip is available by default in new virtual environments created with that tool. I also have an idea for a related import system feature that I'll be sending to import-sig this afternoon (it's a variant on *.pth and *.egg-link files that should be able to address a variety of existing problems, including the one of *selectively* making system and user packages available in a virtual environment in a cross-platform way without needing to copy them) 2. While I was originally a fan of the "implicit bootstrapping on demand" design, I no longer like that notion. While Richard's bootstrap script is a very nice piece of work, the edge cases and "neat tricks" have built up to the point where they trip my "if the implementation is hard to explain, it's a bad idea" filter. Accordingly, I no longer think the implicit bootstrapping is a viable option. 3. That means there are two main options available to us that I still consider viable alternatives (the installer bundling idea was suggested in one of the off list comments I mentioned): * an explicit bootstrapping script * bundling a *full* copy of pip with the Python installers for Windows and Mac OS X, but installing it to site-packages rather than to the standard library directory. That way pip can be used to upgrade itself as normal, rather than making it part of the standard library per se. This is then closer to the "bundled application" model adopted for IDLE in PEP 434 (we could, in fact, move to distributing idle the same way). I'm currently leaning towards offering both, as we're going to need a tool for bootstrapping source builds, but the simplest way to bootstrap pip for Windows and Mac OS X users is to just *bundle a copy with the binary installers*. So long as the bundled copy looks *exactly* the way it would if installed later (so it can update itself), then we avoid the problem of coupling the pip update cycles to the standard library feature release cycle. The bundled version can be updated to the latest available versions when we do a Python maintenance release. For Linux, if you're using the system Python on a Debian or Fedora derivative, then "sudo apt-get python-pip" and "sudo yum install python-pip" are both straightforward, and if you're using something else, then it's unlikely getting pip bootstrapped using the bootstrap script is a task that will bother you :) The "python -m getpip" command is still something we will want to provide, as it is useful to people that build their own copy of Python from source. The bundling idea will obviously need to be discussed with the installer builders, and on python-dev in general, but that was always going to be the case for this PEP anyway (since it *does* touch CPython directly, rather than just being related to the packaging ecosystem). It achieves the aim of allowing people to assume some version of pip will be present on Python 3.4+ installations (or readily available in the case of Linux), while avoiding the problem of coupling pip updates to major Python version updates. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah at coderanger.net Sat Jul 13 07:57:51 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Fri, 12 Jul 2013 22:57:51 -0700 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: <7F3EBECB-1DEC-4187-A33D-31284AB2A944@coderanger.net> On Jul 12, 2013, at 10:31 PM, Nick Coghlan wrote: > In addition to the long thread based on Richard's latest set of updates, I've also received a few off-list comments on the current state of the proposal. So, I figured I'd start a new thread summarising my current point of view and see where we want to go from there. > > 1. However we end up solving the bootstrapping problem, I'm *definitely* a fan of us updating pyvenv in 3.4 to ensure that pip is available by default in new virtual environments created with that tool. I also have an idea for a related import system feature that I'll be sending to import-sig this afternoon (it's a variant on *.pth and *.egg-link files that should be able to address a variety of existing problems, including the one of *selectively* making system and user packages available in a virtual environment in a cross-platform way without needing to copy them) > > 2. While I was originally a fan of the "implicit bootstrapping on demand" design, I no longer like that notion. While Richard's bootstrap script is a very nice piece of work, the edge cases and "neat tricks" have built up to the point where they trip my "if the implementation is hard to explain, it's a bad idea" filter. > > Accordingly, I no longer think the implicit bootstrapping is a viable option. > > 3. That means there are two main options available to us that I still consider viable alternatives (the installer bundling idea was suggested in one of the off list comments I mentioned): > > * an explicit bootstrapping script > * bundling a *full* copy of pip with the Python installers for Windows and Mac OS X, but installing it to site-packages rather than to the standard library directory. That way pip can be used to upgrade itself as normal, rather than making it part of the standard library per se. This is then closer to the "bundled application" model adopted for IDLE in PEP 434 (we could, in fact, move to distributing idle the same way). > > I'm currently leaning towards offering both, as we're going to need a tool for bootstrapping source builds, but the simplest way to bootstrap pip for Windows and Mac OS X users is to just *bundle a copy with the binary installers*. So long as the bundled copy looks *exactly* the way it would if installed later (so it can update itself), then we avoid the problem of coupling the pip update cycles to the standard library feature release cycle. The bundled version can be updated to the latest available versions when we do a Python maintenance release. > > For Linux, if you're using the system Python on a Debian or Fedora derivative, then "sudo apt-get python-pip" and "sudo yum install python-pip" are both straightforward, and if you're using something else, then it's unlikely getting pip bootstrapped using the bootstrap script is a task that will bother you :) > > The "python -m getpip" command is still something we will want to provide, as it is useful to people that build their own copy of Python from source. > > The bundling idea will obviously need to be discussed with the installer builders, and on python-dev in general, but that was always going to be the case for this PEP anyway (since it *does* touch CPython directly, rather than just being related to the packaging ecosystem). It achieves the aim of allowing people to assume some version of pip will be present on Python 3.4+ installations (or readily available in the case of Linux), while avoiding the problem of coupling pip updates to major Python version updates. As someone that has otherwise remained silent on this thread but was talking with people off-list I probably owe them a public +1 for bundling pip as a semi-new category of non-stdlib-but-included project. This would bring us in line with other tools like gem and npm which work out of the box and gives the user experience people want. Care would have to be paid to make sure the final pip binary ends up in the right filename, much in the same way as we do python -> python2 -> python 2.7 and such, but this is a solvable problem. How linux distros adapt to this is certainly another question, but I would absolutely advocate to packagers that installing the main python package results in a working pip install, regardless of how that is accomplished. As someone that has to write system management scripts to install and configure Python, being able to count on both pip and pyenv as standard tools in standard places is near-mind-blowingly awesome (give or take that it would be many years until I could reasonably assume 3.4 as the default python, but a man can dream). While the getpip module is interesting in a few use cases, it is vastly more valuable to me that we focus on the user experience of the majority of Python developers and deployments, and this is somewhere that Ruby and Node are getting it right in having the package tool simply be there by default. Bundling also addresses the myriad ways in which downloading on the fly either during install or afterwards is error-prone and may result in environment fragmentation (servers that don't have internet access, PyPI outages, etc). --Noah From donald at stufft.io Sat Jul 13 08:05:33 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 02:05:33 -0400 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: <55B209B3-9576-4CF0-B58C-2A1E692AFFF1@stufft.io> On Jul 13, 2013, at 1:31 AM, Nick Coghlan wrote: > I'm currently leaning towards offering both, as we're going to need a tool for bootstrapping source builds, but the simplest way to bootstrap pip for Windows and Mac OS X users is to just *bundle a copy with the binary installers*. So long as the bundled copy looks *exactly* the way it would if installed later (so it can update itself), then we avoid the problem of coupling the pip update cycles to the standard library feature release cycle. The bundled version can be updated to the latest available versions when we do a Python maintenance release. We could simply check it into the site-packages inside the CPython source tree could we not? *Not* providing a bootstrap script and merely checking it into the default site-packages means it's available for everyone. No matter how python installed. If Linux packagers really don't want it installed by default they could simply just remove it and either install it along with Python, or continue to keep it how it is today as a separate package? There are a number of things that have to be taken into account when downloading pip from the internet that are completely side stepped when we well don't download it from the internet :) And bundling it as a pre-installed python module and not in the standard library solves basically all of the problems I have with putting it inside of Python. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sat Jul 13 08:07:13 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 02:07:13 -0400 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: <7F3EBECB-1DEC-4187-A33D-31284AB2A944@coderanger.net> References: <7F3EBECB-1DEC-4187-A33D-31284AB2A944@coderanger.net> Message-ID: <3A9FAB73-6AE4-4E26-959B-5346F24ED306@stufft.io> On Jul 13, 2013, at 1:57 AM, Noah Kantrowitz wrote: > I would absolutely advocate to packagers that installing the main python package results in a working pip install, regardless of how that is accomplished. +1 to this as well. Ideally, if we go down this route, installing python just comes with pip preinstalled. However that takes place :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Sat Jul 13 08:28:42 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 13 Jul 2013 16:28:42 +1000 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> <1770C961-BC2F-4062-85BB-A82131013FD6@stufft.io> Message-ID: On 13 July 2013 05:25, Brett Cannon wrote: > > On Fri, Jul 12, 2013 at 2:16 PM, Donald Stufft wrote: > >> >> On Jul 12, 2013, at 2:00 PM, Brett Cannon wrote: >> >> Speaking with my python-dev hat on which has a badge from when I led the >> stdlib cleanup for Python 3, I would say anything that has a PEP should >> probably have a module in the stdlib for it. That way standard management >> of whatever is specified in the PEP will be uniform and expected to be >> maintained and work. Beyond that code will exist outside the stdlib. >> >> >> This is basically the exact opposite of what Nick has said the intent has >> been (Ecosystem first). >> >> > Not at all as no module will go in immediately until after a PEP has > landed and been vetted as needed. > > >> Adding packaging tools beyond bootstrapping pip at this point in the game >> is IMO a huge mistake. If what Nick has said and PEPs are not appropriate >> for things that don't have a module in the standard lib well that's fine I >> guess. >> > > You misunderstand what I mean. I'm just saying that *if* anything were to > go into the stdlib it would only be to have code which implements a PEP in > the stdlib to prevent everyone from re-implementing a standard. > What Brett is saying is closer to what I was thinking when we were discussing this at the language summit. However, I'm no longer sure distlib will be quite baked enough to suggest bundling it in 3.4, in which case it will only be a "pip install distlib" away (that's the entire point of PEP 439). > I just won't worry about trying to write PEPs :) >> > > No, the PEPs are important to prevent version skew and make sure everyone > is on the same page. And that's also what a module in the stdlib would do; > make sure everyone is on the same page in terms of semantics by using a > single code base. > > I mean I wouldn't expect anything more than maybe code parsing the JSON > metadata that does some validation and parsing version numbers that can > support comparisons and verifying platform requirements; that's it. Stuff > that every installation tool will need to do in order to follow the PEPs > properly. And it wouldn't go in until everyone was very happy with the PEPs > and ready to commit to code enshrining it in the stdlib. Otherwise I hope > distlib moves into PyPA and everyone who develops installation tools, etc. > uses that library. > Vinay already moved both distlib and pylauncher over to the PyPA account on BitBucket: https://bitbucket.org/pypa/ PEP 439 is the critical piece, since that's the one that takes the pressure off getting the other components (including distlib and pkg_resources) into the base installers. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Sat Jul 13 08:29:20 2013 From: nad at acm.org (Ned Deily) Date: Fri, 12 Jul 2013 23:29:20 -0700 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) References: <55B209B3-9576-4CF0-B58C-2A1E692AFFF1@stufft.io> Message-ID: In article <55B209B3-9576-4CF0-B58C-2A1E692AFFF1 at stufft.io>, Donald Stufft wrote: > On Jul 13, 2013, at 1:31 AM, Nick Coghlan wrote: > > I'm currently leaning towards offering both, as we're going to need a tool > > for bootstrapping source builds, but the simplest way to bootstrap pip for > > Windows and Mac OS X users is to just *bundle a copy with the binary > > installers*. So long as the bundled copy looks *exactly* the way it would > > if installed later (so it can update itself), then we avoid the problem of > > coupling the pip update cycles to the standard library feature release > > cycle. The bundled version can be updated to the latest available versions > > when we do a Python maintenance release. Off the top of my head, including a copy of pip as a pre-installed global site-package seems like a very reasonable suggestion. For the python.org OS X installer, it should be no problem to implement. It would be equally easy to implement for future 2.7 and 3.3 maintenance releases. > We could simply check it into the site-packages inside the CPython source > tree could we not? *Not* providing a bootstrap script and merely checking it > into the default site-packages means it's available for everyone. No matter > how python installed. If Linux packagers really don't want it installed by > default they could simply just remove it and either install it along with > Python, or continue to keep it how it is today as a separate package? This sounds an unnecessary complication. I suspect that there is a small minority of users who actually build Python from source. And they should know what they are doing. I believe most users either use a distribution-provided Python (via their OS) or a third-party package provider (including python.org binary installers and their derivatives). The OS distributors are going to do what they currently do; the only change needed is to persuade them to include their pip package as a mandatory dependency. Trying to hack the Python source build process to include a copy of pip is just not worth the effort. -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Sat Jul 13 08:31:16 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 13 Jul 2013 16:31:16 +1000 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: <55B209B3-9576-4CF0-B58C-2A1E692AFFF1@stufft.io> References: <55B209B3-9576-4CF0-B58C-2A1E692AFFF1@stufft.io> Message-ID: On 13 July 2013 16:05, Donald Stufft wrote: > > On Jul 13, 2013, at 1:31 AM, Nick Coghlan wrote: > > I'm currently leaning towards offering both, as we're going to need a tool > for bootstrapping source builds, but the simplest way to bootstrap pip for > Windows and Mac OS X users is to just *bundle a copy with the binary > installers*. So long as the bundled copy looks *exactly* the way it would > if installed later (so it can update itself), then we avoid the problem of > coupling the pip update cycles to the standard library feature release > cycle. The bundled version can be updated to the latest available versions > when we do a Python maintenance release. > > > We could simply check it into the site-packages inside the CPython source > tree could we not? *Not* providing a bootstrap script and merely checking > it into the default site-packages means it's available for everyone. No > matter how python installed. > Source code that isn't maintained through bugs.python.org isn't getting checked into the CPython repo - see PEP 360. Getting the latest released version of something from PyPI is a different story, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Jul 13 08:34:49 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 02:34:49 -0400 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: <55B209B3-9576-4CF0-B58C-2A1E692AFFF1@stufft.io> Message-ID: <4376358C-69DC-4F8B-A23C-0C6E63BD14B0@stufft.io> On Jul 13, 2013, at 2:29 AM, Ned Deily wrote: >> We could simply check it into the site-packages inside the CPython source >> tree could we not? *Not* providing a bootstrap script and merely checking it >> into the default site-packages means it's available for everyone. No matter >> how python installed. If Linux packagers really don't want it installed by >> default they could simply just remove it and either install it along with >> Python, or continue to keep it how it is today as a separate package? > > This sounds an unnecessary complication. I suspect that there is a > small minority of users who actually build Python from source. And they > should know what they are doing. I believe most users either use a > distribution-provided Python (via their OS) or a third-party package > provider (including python.org binary installers and their derivatives). > The OS distributors are going to do what they currently do; the only > change needed is to persuade them to include their pip package as a > mandatory dependency. Trying to hack the Python source build process to > include a copy of pip is just not worth the effort. Okies, thought it might be simpler :) Doesn't matter to me where in the process it happens at :) I don't install from source. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From regebro at gmail.com Sat Jul 13 10:02:17 2013 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 13 Jul 2013 10:02:17 +0200 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: <3A9FAB73-6AE4-4E26-959B-5346F24ED306@stufft.io> References: <7F3EBECB-1DEC-4187-A33D-31284AB2A944@coderanger.net> <3A9FAB73-6AE4-4E26-959B-5346F24ED306@stufft.io> Message-ID: On Sat, Jul 13, 2013 at 8:07 AM, Donald Stufft wrote: > +1 to this as well. Ideally, if we go down this route, installing python > just comes with pip preinstalled. However that takes place :) Well, I don't want a "however that takes place" that causes more packaging problems in the future, so I'm -1 on that. :-) I think including it in binary installers in such a way that pip can upgrade itself is definitely a good idea. Including it in the source code distributions seems less of a benefit. In fact, I don't even think I want it as things stand right now. Story time: This week a lot of people had problems with zc.buildout, because of the new releases of setuptools and distribute. Essentially they tried to upgrade themselves and failed for various reasons, such as distribute or setuptools being installed in the system python and not being upgradeable, or not being upgradeable for other reasons, like permissions. Mostly, I had no problems, because the Python installs I use have only one package installed: Virtualenv. That means that for most buildouts I used, buildout had istelf installed setuptools or distribute in its own isolated way. I only got errors where an overly complex build tool insisted on making virtualenvs, installing distribute in them and then running buildout with those virtualenvs. In those cases I had to run bin/pip -U distribute, so it was also an easy fix. So by default I prefer to have no packages except virtualenv for my source-built Pythons. This isn't a big issue, and I'm not gonna throw a hissyfit or anything, but I think I'll -1 the idea of including pip in a source install. Having pyvenv install pip by default in the new environment is +lots from me though. Also, having pyvenv be able to install itself in a non-empty directory would be great. ;-) //Lennart From p.f.moore at gmail.com Sat Jul 13 10:54:38 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 13 Jul 2013 09:54:38 +0100 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: <95EAEB1E-07CD-416A-8ADA-2C872FF4609B@stufft.io> References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> <1770C961-BC2F-4062-85BB-A82131013FD6@stufft.io> <096D11E0-7859-4466-BA95-FE0BDC43D0B9@stufft.io> <95EAEB1E-07CD-416A-8ADA-2C872FF4609B@stufft.io> Message-ID: On 13 July 2013 01:02, Donald Stufft wrote: > >> The installer side of things the purist side of me doesn't like adding > it to > >> the standard library for all the same reasons but the pragmatic side of > me > >> wants it there because it enables fetching the other bits that are > needed for > >> "pip install X" to be a reasonable official response to these kind of > >> questions. But I pushed for and still believe that if a prerequisite for > >> doing that involves "locking" in pip or any of it's dependencies by > adding > >> them to the standard library then I am vehemently against doing it. > > > > Nobody seems to be suggesting doing that, though. > > I was (trying?) to explain that my belief doesn't extend to only distlib > here and > instead to the entire toolchain. (The above quote may not be the best point to comment on, but I wanted to avoid quoting the whole text just to make a general point on this subject) In my view packaging (specifically install) tools are a bit different from other things, because generally packaging tools with dependencies suck. Look at pip's reliance on setuptools, for example. Managing setuptools with pip is a pain, and bootstrapping pip involves getting setuptools installed without already having pip available. I'm +1 on having basic infrastructure in the stdlib, because that way people can concentrate on innovating in more important areas of packaging rather than endlessly reinventing basic stuff. The trick is knowing what counts as basic infrastructure. Things I have *regularly* come up against, over a long period of writing little tools: 1. Version parsing and ordering (I usually use distutils.LooseVersion, just because it's available and "close enough" :-() 2. Reading metadata from distributions (not just parsing it, but getting it out of dist-info files or sdists and the like as well) 3. Installing wheels 4. Locating distributions on PyPI or local indexes At the moment, my choices are to write my own code (usually meaning far more code than the actual functionality I want to write!) require distlib (not good in a tool for building zero-dependency venvs from the ground up, for example) or vendoring in distlib (impractical in a one-file script). Having something in the stdlib (even if it's only able to bootstrap distlib or an alternative) solves all of these issues. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Jul 13 11:05:24 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 13 Jul 2013 10:05:24 +0100 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: On 13 July 2013 06:31, Nick Coghlan wrote: > * bundling a *full* copy of pip with the Python installers for Windows and > Mac OS X, but installing it to site-packages rather than to the standard > library directory. That way pip can be used to upgrade itself as normal, > rather than making it part of the standard library per se. This is then > closer to the "bundled application" model adopted for IDLE in PEP 434 (we > could, in fact, move to distributing idle the same way). How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable. Personally, I have never upgraded pip using itself, because I only ever install pip in virtualenvs, which don't have a lifespan as long as a pip release cycle :-) It would be easy to imagine a new pip release resulting in a *lot* of bugs raised against Python (rather than pip) saying that the upgrade fails. And of course if an upgrade fails, we can't just release a new version of pip that fixes the issue, because it's the *old* version that is installed and has to do the upgrade. So there's manual fiddling to do. Not a good experience for Python users. My current workflow is to have absolutely nothing installed in the system Python and use virtualenvs for everything. This is a bit extreme, but the issues I've hit in the past when package management has gone wrong have made me very cautious. If the pip upgrade process is rock-solid, this isn't an issue, but I'm not sure that it is, myself. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Sat Jul 13 14:12:01 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 13 Jul 2013 13:12:01 +0100 (BST) Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: <95EAEB1E-07CD-416A-8ADA-2C872FF4609B@stufft.io> References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> <1770C961-BC2F-4062-85BB-A82131013FD6@stufft.io> <096D11E0-7859-4466-BA95-FE0BDC43D0B9@stufft.io> <95EAEB1E-07CD-416A-8ADA-2C872FF4609B@stufft.io> Message-ID: <1373717521.98532.YahooMailNeo@web171402.mail.ir2.yahoo.com> > From: Donald Stufft >As I said in my email, because it's more or less standalone and it has the >greatest utility outside of installers/builders/archivers/indexes. Even if that were true, it doesn't mean that it's the *only* thing that's worth considering. >I've looked at many other languages where they had widely successful >packaging tools that weren't added to the standard lib until they were >ubiquitous and stable. Something the new tools for Python are not. So I >don't think adding it to the standard library is required. As I said earlier, I'm not arguing for *premature* inclusion of distlib or anything else in the stdlib. I'm only saying that there's less likelihood that any one approach outside the stdlib will get univerally adopted, leading to balkanisation. >to reuse some it's functionality. So pointing towards setuptools just exposes >the fact that improving it in the standard library was hard enough that it was >done externally. It seems like it wasn't for technical reasons that this approach was taken, just as Distribute wasn't forked from setuptools for technical reasons. >Well I am of the mind that the standard library is where software goes to die, and No kidding? :-) >want to use my software at all. A huge thing i've been trying to push for is decoupling >packaging from a specific implementation so that we have a "protocol" (ala HTTP) >and not a "tool" (ala distutils). However the allure of working to the implementation >and not the standard is fairly high when there is a singular blessed implementation. I'm not aware of this - have you published any protocols around the work you're doing on warehouse, which Nick said was going to be the next-generation PyPI? >It's funny you picked and example where improvements *couldn't* take place and >the entire system had to be thrown out and a new one written. getopt had to become a >new module named opt parse, which had to become a new module named argparse I picked that example specifically to show that even if things go wrong, it's not the end of the world. >You can gain interoperability in a few ways. One way is to just pick an implementation If that were done, it wouldn't make any difference whether the thing picked were in the stdlib or not. But people have a tendency to roll their own stuff, whether there's a good technical reason or not. >and make that the standard. Another is to define *actual* standards. The second >one is harder, requires more thought and work. But it means that completely >different software can work together. It means that something written in Ruby >can easily work with a python package without shelling out to Python or without That's exactly why there are all these packaging PEPs around, isn't it? >And that's fine for a certain class of problems. It's not that useful for something >where you want interoperability outside of that tool. How terrible would it be if >HTTP was "well whatever Apache does, that's what HTTP is". That wouldn't have been so terrible if you replace "Apache" with "W3C", since you would have a reference implementation by the creators of the standard. >A singular blessed tool in the standard library incentivizes the standard becoming >and implementation detail. I *want* there to be multiple implementations written by >different people working on different "slices" of the problem. That incentivizes doing >the extra work on PEPs and other documents so that we maintain a highly documented >standard. It's true that adding something to the standard library doesn't rule that out >but it provides an incentive against properly doing standards because it's easier and >simpler to just change it in the implementation. Are you planning to produce any standards relating to PyPI-like functionality? This is important for the dependency resolution "slice", amongst others. The flip side of this coin is, talking in the abstract without any working code is sub-optimal. It's reasonable for standards and implementations of them to grow together, because each informs the other, at least in the early stages. Most standards PEPs are accepted with a reference implementation in place. >It's not blessed and a particular packaging project should use it if it fits their >needs and they want to use it. Or they shouldn't use it if they don't want. >Standards exist for a reason. So you can have multiple implementations that >all work together. That's true independent of whether one particular implementation of the standard is blessed in some way. >I didn't make any claims as to it's stability or the amount of testing that went into >it. My ability to be convinced of that stems primarily from the fact that it's sort of >a side piece of the whole packaging infrastructure and toolchain and it's also >a piece that is most likely to be useful on it's own. But the arguments about agility and stability apply to any software - version-handling doesn't get a special pass. Proper version handling is central to dependency resolution and is hardly a side issue, though it's not especially complicated. I'll just finish by re-iterating that I think there should be some stdlib underpinning for packaging in general, and that there should be some focus on exactly what that underpinning should be, and that I'm by no means saying that distlib is it. I consider distlib as still in its early days but showing some promise (and deserving of more peer review than it has received to date). Regards, Vinay Sajip From ncoghlan at gmail.com Sat Jul 13 14:25:30 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 13 Jul 2013 22:25:30 +1000 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: On 13 Jul 2013 19:05, "Paul Moore" wrote: > > On 13 July 2013 06:31, Nick Coghlan wrote: >> >> * bundling a *full* copy of pip with the Python installers for Windows and Mac OS X, but installing it to site-packages rather than to the standard library directory. That way pip can be used to upgrade itself as normal, rather than making it part of the standard library per se. This is then closer to the "bundled application" model adopted for IDLE in PEP 434 (we could, in fact, move to distributing idle the same way). > > > How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable. > > Personally, I have never upgraded pip using itself, because I only ever install pip in virtualenvs, which don't have a lifespan as long as a pip release cycle :-) It would be easy to imagine a new pip release resulting in a *lot* of bugs raised against Python (rather than pip) saying that the upgrade fails. And of course if an upgrade fails, we can't just release a new version of pip that fixes the issue, because it's the *old* version that is installed and has to do the upgrade. So there's manual fiddling to do. Not a good experience for Python users. > > My current workflow is to have absolutely nothing installed in the system Python and use virtualenvs for everything. This is a bit extreme, but the issues I've hit in the past when package management has gone wrong have made me very cautious. > > If the pip upgrade process is rock-solid, this isn't an issue, but I'm not sure that it is, myself. I think we need to flip the dependencies so that pip as the installer has all the essential code for installation from PyPI and then setuptools and distlib depend on that pip infrastructure. No need to add anything to the standard library prematurely when we can add it to pip instead. Cheers, Nick. > > Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Jul 13 15:23:02 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 13 Jul 2013 14:23:02 +0100 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: <1373717521.98532.YahooMailNeo@web171402.mail.ir2.yahoo.com> References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> <1770C961-BC2F-4062-85BB-A82131013FD6@stufft.io> <096D11E0-7859-4466-BA95-FE0BDC43D0B9@stufft.io> <95EAEB1E-07CD-416A-8ADA-2C872FF4609B@stufft.io> <1373717521.98532.YahooMailNeo@web171402.mail.ir2.yahoo.com> Message-ID: On 13 July 2013 13:12, Vinay Sajip wrote: > I'll just finish by re-iterating that I think there should be some stdlib > underpinning for packaging in general, and that there should be some focus > on exactly what that underpinning should be, and that I'm by no means > saying that distlib is it. I consider distlib as still in its early days > but showing some promise (and deserving of more peer review than it has > received to date). +1 to all of this Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jul 13 15:31:01 2013 From: brett at python.org (Brett Cannon) Date: Sat, 13 Jul 2013 09:31:01 -0400 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: On Sat, Jul 13, 2013 at 8:25 AM, Nick Coghlan wrote: > > On 13 Jul 2013 19:05, "Paul Moore" wrote: > > > > On 13 July 2013 06:31, Nick Coghlan wrote: > >> > >> * bundling a *full* copy of pip with the Python installers for Windows > and Mac OS X, but installing it to site-packages rather than to the > standard library directory. That way pip can be used to upgrade itself as > normal, rather than making it part of the standard library per se. This is > then closer to the "bundled application" model adopted for IDLE in PEP 434 > (we could, in fact, move to distributing idle the same way). > > > > > > How robust is the process of upgrading pip using itself? Specifically on > Windows, where these things typically seem less reliable. > > > > Personally, I have never upgraded pip using itself, because I only ever > install pip in virtualenvs, which don't have a lifespan as long as a pip > release cycle :-) It would be easy to imagine a new pip release resulting > in a *lot* of bugs raised against Python (rather than pip) saying that the > upgrade fails. And of course if an upgrade fails, we can't just release a > new version of pip that fixes the issue, because it's the *old* version > that is installed and has to do the upgrade. So there's manual fiddling to > do. Not a good experience for Python users. > > > > My current workflow is to have absolutely nothing installed in the > system Python and use virtualenvs for everything. This is a bit extreme, > but the issues I've hit in the past when package management has gone wrong > have made me very cautious. > > > > If the pip upgrade process is rock-solid, this isn't an issue, but I'm > not sure that it is, myself. > > I think we need to flip the dependencies so that pip as the installer has > all the essential code for installation from PyPI and then setuptools and > distlib depend on that pip infrastructure. No need to add anything to the > standard library prematurely when we can add it to pip instead. > +1 on the inversion. I don't know what that will do to pip, it makes sense to have the installer self-contained and the packaging/building libraries be something that you grab using the installer. Having to grab the packaging infrastructure to get an installer is the more painful route. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Jul 13 15:32:57 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 13 Jul 2013 14:32:57 +0100 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: On 13 July 2013 13:25, Nick Coghlan wrote: > I think we need to flip the dependencies so that pip as the installer has > all the essential code for installation from PyPI and then setuptools and > distlib depend on that pip infrastructure. No need to add anything to the > standard library prematurely when we can add it to pip instead. Agreed, up to a point. What I've worked on in the past is things like automated wheel/sdist installers for systems with no internet access at all (distributions copied onto the PC via portable disk) or behind broken proxies (Internet Explorer works to download files from the net, but nothing else does). For those environments, the key to me is that I *only* use stuff that is available in a stock Python build. Something like pyton -m getpip *will not work*, so I have to roll my own means of bootstrapping. Of course I could copy a pip sdist to the machine, unpack it and run python setup.py install. More likely, I write bootstrap code in my script to do that automatically. It's a very specialised use case, certainly, and there are plenty of ways around the issues, but I have seen "give up and just use VBScript./perl" used as the fallback, as well, and that makes me sad... Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jul 13 15:35:08 2013 From: brett at python.org (Brett Cannon) Date: Sat, 13 Jul 2013 09:35:08 -0400 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: <55B209B3-9576-4CF0-B58C-2A1E692AFFF1@stufft.io> Message-ID: On Sat, Jul 13, 2013 at 2:29 AM, Ned Deily wrote: > In article <55B209B3-9576-4CF0-B58C-2A1E692AFFF1 at stufft.io>, > Donald Stufft wrote: > > On Jul 13, 2013, at 1:31 AM, Nick Coghlan wrote: > > > I'm currently leaning towards offering both, as we're going to need a > tool > > > for bootstrapping source builds, but the simplest way to bootstrap pip > for > > > Windows and Mac OS X users is to just *bundle a copy with the binary > > > installers*. So long as the bundled copy looks *exactly* the way it > would > > > if installed later (so it can update itself), then we avoid the > problem of > > > coupling the pip update cycles to the standard library feature release > > > cycle. The bundled version can be updated to the latest available > versions > > > when we do a Python maintenance release. > > Off the top of my head, including a copy of pip as a pre-installed > global site-package seems like a very reasonable suggestion. For the > python.org OS X installer, it should be no problem to implement. It > would be equally easy to implement for future 2.7 and 3.3 maintenance > releases. > Does Apple just install the python.org OS X installer for distribution, or do they build their own thing? My only worry is that Apple will not get the message about including pip and we will end up with an odd skew on OS X (I'm not worried about Linux distros as they all seem to follow Python development closely). And we obviously need to know if Martin is okay with doing something similar on Windows. > > > We could simply check it into the site-packages inside the CPython source > > tree could we not? *Not* providing a bootstrap script and merely > checking it > > into the default site-packages means it's available for everyone. No > matter > > how python installed. If Linux packagers really don't want it installed > by > > default they could simply just remove it and either install it along with > > Python, or continue to keep it how it is today as a separate package? > > This sounds an unnecessary complication. I suspect that there is a > small minority of users who actually build Python from source. And they > should know what they are doing. I believe most users either use a > distribution-provided Python (via their OS) or a third-party package > provider (including python.org binary installers and their derivatives). > The OS distributors are going to do what they currently do; the only > change needed is to persuade them to include their pip package as a > mandatory dependency. Trying to hack the Python source build process to > include a copy of pip is just not worth the effort. > And I doubt that will take much convincing, e.g. ActiveState includes their own installer -- PyPM -- in their distro. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Jul 13 15:38:29 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 13 Jul 2013 14:38:29 +0100 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: On 13 July 2013 14:31, Brett Cannon wrote: > +1 on the inversion. I don't know what that will do to pip, it makes sense > to have the installer self-contained and the packaging/building libraries > be something that you grab using the installer. Having to grab the > packaging infrastructure to get an installer is the more painful route. TBH, I don't understand what "the inversion" implies. If it means pip taking all of the distlib/setuptools code that it currently uses, and making it part of pip and maintained within pip (essentially as a fork while the "inversion" is going on) then I'm not keen on that. Personally, I don't want to have to maintain that code myself - I guess if Vinay and Jason were pip maintainers and looked after that code, then that's an option. If it means pip vendoring distlib and setuptools, then OK (we do that for distlib already) but I don't see the benefit - no-ione should be doing "from pip.vendor.distlib.version import Version". I'd need to know better what it means for pip, I guess... Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jul 13 15:46:57 2013 From: brett at python.org (Brett Cannon) Date: Sat, 13 Jul 2013 09:46:57 -0400 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: On Sat, Jul 13, 2013 at 1:31 AM, Nick Coghlan wrote: > In addition to the long thread based on Richard's latest set of updates, > I've also received a few off-list comments on the current state of the > proposal. So, I figured I'd start a new thread summarising my current point > of view and see where we want to go from there. > > 1. However we end up solving the bootstrapping problem, I'm *definitely* a > fan of us updating pyvenv in 3.4 to ensure that pip is available by default > in new virtual environments created with that tool. I also have an idea for > a related import system feature that I'll be sending to import-sig this > afternoon (it's a variant on *.pth and *.egg-link files that should be able > to address a variety of existing problems, including the one of > *selectively* making system and user packages available in a virtual > environment in a cross-platform way without needing to copy them) > > 2. While I was originally a fan of the "implicit bootstrapping on demand" > design, I no longer like that notion. While Richard's bootstrap script is a > very nice piece of work, the edge cases and "neat tricks" have built up to > the point where they trip my "if the implementation is hard to explain, > it's a bad idea" filter. > > Accordingly, I no longer think the implicit bootstrapping is a viable > option. > > 3. That means there are two main options available to us that I still > consider viable alternatives (the installer bundling idea was suggested in > one of the off list comments I mentioned): > > * an explicit bootstrapping script > * bundling a *full* copy of pip with the Python installers for Windows and > Mac OS X, but installing it to site-packages rather than to the standard > library directory. That way pip can be used to upgrade itself as normal, > rather than making it part of the standard library per se. This is then > closer to the "bundled application" model adopted for IDLE in PEP 434 (we > could, in fact, move to distributing idle the same way). > > I'm currently leaning towards offering both, as we're going to need a tool > for bootstrapping source builds, but the simplest way to bootstrap pip for > Windows and Mac OS X users is to just *bundle a copy with the binary > installers*. So long as the bundled copy looks *exactly* the way it would > if installed later (so it can update itself), then we avoid the problem of > coupling the pip update cycles to the standard library feature release > cycle. The bundled version can be updated to the latest available versions > when we do a Python maintenance release. > > For Linux, if you're using the system Python on a Debian or Fedora > derivative, then "sudo apt-get python-pip" and "sudo yum install > python-pip" are both straightforward, and if you're using something else, > then it's unlikely getting pip bootstrapped using the bootstrap script is a > task that will bother you :) > > The "python -m getpip" command is still something we will want to provide, > as it is useful to people that build their own copy of Python from source. > But is it going to make a difference? If we shift to using included copies of pip in binary installers over a bootstrap I say leave out the bootstrap as anyone building from source should know how to get pip installed on their machine or venv. The only reason I see it worth considering is if pyvenv starts bootstrapping pip and we want to support the case of pip not being installed. But if we are including it in the binary installer and are going to assume it's available through OS distros, then there isn't a need to as pip can then install pip for us into the venv and skip any initial pip bootstrap. If pip isn't found we can simply either point to the docs in the failure message or print out the one-liner it takes to install pip (and obviously there can be a --no-pip flag to skip this for people who want to install it manually like me who build from source). IOW I think taking the worldview in Python 3.4 that pip will come installed with Python unless you build from source negates the need for the bootstrap script beyond just saying ``curl https://pypi.python.org/get-pip.py | python`` if pip isn't found. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jul 13 15:52:01 2013 From: brett at python.org (Brett Cannon) Date: Sat, 13 Jul 2013 09:52:01 -0400 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: On Sat, Jul 13, 2013 at 9:38 AM, Paul Moore wrote: > On 13 July 2013 14:31, Brett Cannon wrote: > >> +1 on the inversion. I don't know what that will do to pip, it makes >> sense to have the installer self-contained and the packaging/building >> libraries be something that you grab using the installer. Having to grab >> the packaging infrastructure to get an installer is the more painful route. > > > TBH, I don't understand what "the inversion" implies. If it means pip > taking all of the distlib/setuptools code that it currently uses, and > making it part of pip and maintained within pip (essentially as a fork > while the "inversion" is going on) then I'm not keen on that. Personally, I > don't want to have to maintain that code myself - I guess if Vinay and > Jason were pip maintainers and looked after that code, then that's an > option. If it means pip vendoring distlib and setuptools, then OK (we do > that for distlib already) > The point is you shouldn't have to grab a packaging tool just to install stuff if you never need the packaging tool. Since pip is supposed to be *the* first thing you install for Python you don't want that to have its own dependencies, muddying up the installation process. > but I don't see the benefit - no-ione should be doing "from > pip.vendor.distlib.version import Version". > > That's just asking for trouble if someone did that (plus if you did that it would be pip._vendor to get the privacy point across). > I'd need to know better what it means for pip, I guess... > I suspect we all do. =) -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Sat Jul 13 16:39:45 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 13 Jul 2013 14:39:45 +0000 (UTC) Subject: [Distutils] Current status of PEP 439 (pip boostrapping) References: Message-ID: Nick Coghlan gmail.com> writes: > 1. However we end up solving the bootstrapping problem, I'm *definitely* > a fan of us updating pyvenv in 3.4 to ensure that pip is available by > default in new virtual environments created with that tool. Will that need green-lighting on python-dev? As events have shown, that script has needed updating between Python releases. OTOH, I'm not sure anyone could have predicted at 3.3 release time that setuptools and Distribute would kiss and make up :-) We should probably ensure that the pip and setuptools URLs used in that script (pyvenvex.py, to be renamed to pyvenv.py) are formally agreed with the relevant maintainers. Or, since those URLs just fetch the latest releases, perhaps some different URLs should be used which refer to more stable releases (for some definition of "more stable") - perhaps those should be python.org URLs, rather than BitBucket and GitHub as they are at present. That way, changing the resources which those URLs reference would have to be an active decision by someone, rather than just following the latest developments on setuptools and pip. There are pluses and minuses either way, of course. > 2. While I was originally a fan of the "implicit bootstrapping on demand" > design, I no longer like that notion. While Richard's bootstrap script is > a very nice piece of work, the edge cases and "neat tricks" have built up > to the point where they trip my "if the implementation is hard to > explain, it's a bad idea" filter. > Accordingly, I no longer think the implicit bootstrapping is a viable > option. But if your reservation stems from one specific implementation of the idea, then might not an alternative implementation fit the bill? Consider: the pyvenvex.script merely runs bootstrapping scripts from setuptools and pip in the venv - there's no magic. I couldn't see Richard's script referenced in the PEP, which just referred to the PIP issue tracker which had no obvious link to any commit or script. So I don't know what the edge cases and neat tricks are that you're referring to. I adapted the pyvenvex.py script into a getpip.py script, available here: https://gist.github.com/vsajip/5990837 82 lines all told - what cases does it not cover? It installs setuptools and pip into the system site-packages for the invoking Python, if not already present. It can, of course, be refined to e.g. install even if the packages are already present, which is tantamount to upgrading. I smoke-tested the script on vanilla Python 3.3 installations on Windows and OS X. On OS X, the pip script was written to the appropriate Frameworks folder, but not to /usr/local/bin - I assume it would be simple enough to arrange that? On Windows, the pip script (including Windows native launcher) were written to c:\Python33\Scripts. > The bundling idea will obviously need to be discussed with the installer > builders, and on python-dev in general If python-dev agrees to the updated pyvenv.py script, then this type of addition should be uncontentious, as it basically does the same thing. It seems a whole lot less work than bundling, to me. Regards, Vinay Sajip From p.f.moore at gmail.com Sat Jul 13 16:54:53 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 13 Jul 2013 15:54:53 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 Message-ID: This issue has been skirted round for some time now, and I think it needs explicit discussion, as I am not at all sure everyone has the same expectations. We're talking about Python 3.4 installations having pip as the default package manager - whether by bundling, having a bootstrap process or whatever. Regardless of the means, pip will be *the* installer for Python 3.4+. And yet, I don't think pip 1.4 currently does what people want "the Python 3.4 pip" to do in some ways - and we need to make sure that any work on the pip side is understood, agreed to, and planned to match the Python 3.4 timescales. So, here's my initial list of things that I think people might be expecting to happen. This is just my impressions, and I don't necessarily have a view on the individual items. And if anyone else can think of other things to add to the list, please do so! 1. Install to user-packages by default. 2. Not depend on setuptools (??? - Nick's "inversion" idea) 3. Possibly change the wrapper command name from pip to pip3 on Unix. 4. Ensure that pip upgrading itself in-place is sufficiently robust and reliable that users don't get "stuck" on the Python-supplied version. I'm sure I've seen people say other things that have made me think "are you expecting the pip maintainers to make that change?" in the various threads, so I doubt this list is definitive. Comments anyone? Is this discussion premature? The pip maintainers team is not huge, so we'll need time (or assistance!) to plan in and make changes like this, if they are needed... At a minimum, can we get the key items logged on the pip issue tracker with a milestone of Python 3.4? Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Jul 13 17:00:27 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 13 Jul 2013 16:00:27 +0100 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: On 13 July 2013 13:25, Nick Coghlan wrote: > I think we need to flip the dependencies so that pip as the installer has > all the essential code for installation from PyPI and then setuptools and > distlib depend on that pip infrastructure. No need to add anything to the > standard library prematurely when we can add it to pip instead. If we do this, I think people will start to expect to be able to code scripts to the pip API. (We've had people ask this on the pip tracker already). If we don't want pip to end up like distutils (with people depending on all sorts of random bits of the internals, because there's no documented API) as a backward-compatibility nightmare, we need to consider how to handle this. Of course, saying explicitly "only the python -m pip command line interface is stable and supported" may well be enough. But didn't we just say that setuptools and distlib depend on the pip API? So either they have special privileges (presumably because they are under the umbrella of the PyPA) or we can't avoid documenting/supporting some API... I don't believe that pip is currently in a state to offer a solid documented internal API. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Jul 13 17:01:22 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 13 Jul 2013 16:01:22 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: Also (see my reply to Nick's "inversion" proposal) we can add: 5. Provide a stable documented programming interface. Paul On 13 July 2013 15:54, Paul Moore wrote: > This issue has been skirted round for some time now, and I think it needs > explicit discussion, as I am not at all sure everyone has the same > expectations. > > We're talking about Python 3.4 installations having pip as the default > package manager - whether by bundling, having a bootstrap process or > whatever. Regardless of the means, pip will be *the* installer for Python > 3.4+. And yet, I don't think pip 1.4 currently does what people want "the > Python 3.4 pip" to do in some ways - and we need to make sure that any work > on the pip side is understood, agreed to, and planned to match the Python > 3.4 timescales. > > So, here's my initial list of things that I think people might be > expecting to happen. This is just my impressions, and I don't necessarily > have a view on the individual items. And if anyone else can think of other > things to add to the list, please do so! > > 1. Install to user-packages by default. > 2. Not depend on setuptools (??? - Nick's "inversion" idea) > 3. Possibly change the wrapper command name from pip to pip3 on Unix. > 4. Ensure that pip upgrading itself in-place is sufficiently robust and > reliable that users don't get "stuck" on the Python-supplied version. > > I'm sure I've seen people say other things that have made me think "are > you expecting the pip maintainers to make that change?" in the various > threads, so I doubt this list is definitive. > > Comments anyone? Is this discussion premature? The pip maintainers team is > not huge, so we'll need time (or assistance!) to plan in and make changes > like this, if they are needed... > > At a minimum, can we get the key items logged on the pip issue tracker > with a milestone of Python 3.4? > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Jul 13 17:03:53 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 11:03:53 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: On Jul 13, 2013, at 10:54 AM, Paul Moore wrote: > This issue has been skirted round for some time now, and I think it needs explicit discussion, as I am not at all sure everyone has the same expectations. > > We're talking about Python 3.4 installations having pip as the default package manager - whether by bundling, having a bootstrap process or whatever. Regardless of the means, pip will be *the* installer for Python 3.4+. And yet, I don't think pip 1.4 currently does what people want "the Python 3.4 pip" to do in some ways - and we need to make sure that any work on the pip side is understood, agreed to, and planned to match the Python 3.4 timescales. > > So, here's my initial list of things that I think people might be expecting to happen. This is just my impressions, and I don't necessarily have a view on the individual items. And if anyone else can think of other things to add to the list, please do so! > > 1. Install to user-packages by default. Do people really want this? I hadn't seen it (other than if pip was installed to user by default). I think it's a bad idea to switch this on people. I doubt the user-packages is going to be in people's default PATH so they'll easily get into cases where things are installed but they don't know where it was installed too. > 2. Not depend on setuptools (??? - Nick's "inversion" idea) I wanted to do this anyways. It will still "depend" on it, but it will just bundle setuptools itself like its other dependencies. For pip dependencies are an implementation detail not an actual thing it can/should have. > 3. Possibly change the wrapper command name from pip to pip3 on Unix. Not sure on this. Ideally i'd want the commands to be pipX.Y, pipX, and pip all available and not install the less specific ones if they already exist but that might be too hard? > 4. Ensure that pip upgrading itself in-place is sufficiently robust and reliable that users don't get "stuck" on the Python-supplied version. I've always used pip to upgrade pip. The only time i've had problems is when setuptools messes up (which would be prevented if bundled). > > I'm sure I've seen people say other things that have made me think "are you expecting the pip maintainers to make that change?" in the various threads, so I doubt this list is definitive. > > Comments anyone? Is this discussion premature? The pip maintainers team is not huge, so we'll need time (or assistance!) to plan in and make changes like this, if they are needed... > > At a minimum, can we get the key items logged on the pip issue tracker with a milestone of Python 3.4? > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tseaver at palladion.com Sat Jul 13 17:05:29 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 13 Jul 2013 11:05:29 -0400 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/13/2013 08:25 AM, Nick Coghlan wrote: > I think we need to flip the dependencies so that pip as the installer > has all the essential code for installation from PyPI and then > setuptools and distlib depend on that pip infrastructure. No need to > add anything to the standard library prematurely when we can add it to > pip instead. - -1. That would effectively mean inlining the bulk of setuptools' code into pip (which is just a UI / policy shim over it). You might as well just have your bootstrapper install both pip and setuptools and be done Unless distlib (or something like it) lands in the stdlib with enough features to support a setuptools-less pip, of course. At-which-point-the-State-will-wither-away'ly, Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHhbLkACgkQ+gerLs4ltQ51fQCfZrmZN5mJKrtoGFTk0YqQrBHd F/IAnRp6XjoU4SpXZ4v3Uz6iOBrCZZZn =gSA5 -----END PGP SIGNATURE----- From donald at stufft.io Sat Jul 13 17:06:23 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 11:06:23 -0400 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: On Jul 13, 2013, at 11:00 AM, Paul Moore wrote: > Of course, saying explicitly "only the python -m pip command line interface is stable and supported" may well be enough. But didn't we just say that setuptools and distlib depend on the pip API? So either they have special privileges (presumably because they are under the umbrella of the PyPA) or we can't avoid documenting/supporting some API... > I don't think we need this any more than if pip was not pre-installed. So still nice to have for things like chef but not a requirement. setuptools and distlib won't depend on pip, pip will just bundle them (like it already does for distlib). The idea should be that pip itself has no dependencies because a package manager with dependencies is kind of strange and can easily lead to issues where the package manager breaks and is unifiable (e.g. setuptools breaks and pip can't be used to fix it). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sat Jul 13 17:09:00 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 11:09:00 -0400 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: <5AB14BB8-3C8E-473C-A52F-2F7EF2D965ED@stufft.io> On Jul 13, 2013, at 11:05 AM, Tres Seaver wrote: > Signed PGP part > On 07/13/2013 08:25 AM, Nick Coghlan wrote: > > I think we need to flip the dependencies so that pip as the installer > > has all the essential code for installation from PyPI and then > > setuptools and distlib depend on that pip infrastructure. No need to > > add anything to the standard library prematurely when we can add it to > > pip instead. > > - -1. That would effectively mean inlining the bulk of setuptools' code > into pip (which is just a UI / policy shim over it). You might as well > just have your bootstrapper install both pip and setuptools and be done > > Unless distlib (or something like it) lands in the stdlib with enough > features to support a setuptools-less pip, of course. > > At-which-point-the-State-will-wither-away'ly, > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig I was planning on doing this to pip anyways. pip should not have any dependencies setuptools or otherwise. So regardless of what happens with this PEP I want pip to be inlining setuptools and providing the code to make that transparent during install. This would mean that for people who are _just_ installing packages they don't need setuptools installed. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sat Jul 13 17:12:35 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 11:12:35 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: <14A2CCC3-305B-4CE3-82E7-D85903F9F258@stufft.io> On Jul 13, 2013, at 11:01 AM, Paul Moore wrote: > 5. Provide a stable documented programming interface. As I said in the other thread I don't think this is required any more than it does normally. I do think we need to have testing infrastructure in pip that tests against the development branch of CPython though. If pip is going to be included in the releases we need to make sure it works prior to it being released. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Jul 13 17:15:22 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 13 Jul 2013 16:15:22 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: On 13 July 2013 16:03, Donald Stufft wrote: > > > 1. Install to user-packages by default. > > Do people really want this? I hadn't seen it (other than if pip was > installed to user by default). I think it's a bad idea to switch this on > people. I doubt the user-packages is going to be in people's default PATH > so they'll easily get into cases where things are installed but they don't > know where it was installed too. > I believe Nick wants to make user-packages the default. I know at least some of the pip maintainers (yourself included) have reservations. Personally, I've never used user-packages, so I don't know what issues might arise. But I hope to try it out sometime when I get the chance, just to get some specific information. > > 2. Not depend on setuptools (??? - Nick's "inversion" idea) > > I wanted to do this anyways. It will still "depend" on it, but it will > just bundle setuptools itself like its other dependencies. For pip > dependencies are an implementation detail not an actual thing it can/should > have. > Bundling is not the same as Nick's suggestion. I personally have no problem with bundling, but pip install with a bundled setuptools might not work because the setup subprocess won't see the bundled setuptools when it imports it in setup.py. But either way, it's doable, I just want to know if it's on the critical path... > > 3. Possibly change the wrapper command name from pip to pip3 on Unix. > > Not sure on this. Ideally i'd want the commands to be pipX.Y, pipX, and > pip all available and not install the less specific ones if they already > exist but that might be too hard? > > > 4. Ensure that pip upgrading itself in-place is sufficiently robust and > reliable that users don't get "stuck" on the Python-supplied version. > > I've always used pip to upgrade pip. The only time i've had problems is > when setuptools messes up (which would be prevented if bundled). > I've never tried myself, but I'm on Windows and I expect in-place stuff like this to fail. Maybe I'm paranoid :-) Again I need to check. Thanks for the comments. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Jul 13 17:23:35 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 11:23:35 -0400 Subject: [Distutils] PEP 439 and pip bootstrap updated In-Reply-To: <1373717521.98532.YahooMailNeo@web171402.mail.ir2.yahoo.com> References: <51DCE0B5.6030506@oddbird.net> <49DEE7D4-9223-4B8C-B235-E057F3B2DCE3@stufft.io> <1770C961-BC2F-4062-85BB-A82131013FD6@stufft.io> <096D11E0-7859-4466-BA95-FE0BDC43D0B9@stufft.io> <95EAEB1E-07CD-416A-8ADA-2C872FF4609B@stufft.io> <1373717521.98532.YahooMailNeo@web171402.mail.ir2.yahoo.com> Message-ID: <5F1F3DAC-1BD5-40D4-8671-465A0781E90D@stufft.io> On Jul 13, 2013, at 8:12 AM, Vinay Sajip wrote: > I'm not aware of this - have you published any protocols around the work you're doing on warehouse, which Nick said was going to be the next-generation PyPI? I think we're talking past each other at this point but I wanted to respond to this point. Warehouse will evolve by publishing standards yes. Currently its not making API changes and is primarily working on taking the existing APIs and porting them to a modern framework, adding tests, etc. I do have some changes I want to make to the API and I've started a PEP to propose it that once it's done will be published for discussion here at distutils-sig. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jaraco at jaraco.com Sat Jul 13 18:28:51 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sat, 13 Jul 2013 16:28:51 +0000 Subject: [Distutils] beginner ticket for code base modernization of setuptools Message-ID: I've created the following ticket for Setuptools: https://bitbucket.org/pypa/setuptools/issue/35/code-base-modernization I'm inviting junior developers or students or other less experienced enthusiasts who are interested in learning more about the inner workings of Setuptools to assist in a code modernization effort to bring Setuptools more in line with the expectations of modern Python developers. If this effort might interest you, please follow the link for more details. Regards, Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From jaraco at jaraco.com Sat Jul 13 18:37:56 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sat, 13 Jul 2013 16:37:56 +0000 Subject: [Distutils] buildout/setuptools/distribute unhelpful error message (0.7.x issue?) In-Reply-To: <51D913F5.3080804@simplistix.co.uk> References: <51D913F5.3080804@simplistix.co.uk> Message-ID: Hi Chris, It looks like something is trying to install Setuptools 0.7.2, possibly with a temporary version of distribute or one that's not visible by default in your Python environment. When you get that error message, I suggest you upgrade away from distribute. The easiest way to do this if you have distribute installed is to 'easy_install -U distribute', which will grab distribute 0.7.3 and install setuptools>=0.7. If this doesn't work (as it may not if you in fact don't have Distribute), you may be able to pro-actively avoid the problem by installing the latest Setuptools (0.9 at the time of this writing) using the published installation instructions: https://pypi.python.org/pypi/setuptools/0.9 I hope that helps. Please report back if that doesn't get you going. Regards, Jason > -----Original Message----- > From: Distutils-SIG [mailto:distutils-sig- > bounces+jaraco=jaraco.com at python.org] On Behalf Of Chris Withers > Sent: Sunday, 07 July, 2013 03:09 > To: distutils sig > Subject: [Distutils] buildout/setuptools/distribute unhelpful error message > (0.7.x issue?) > > Hi All, > > What is this exception trying to tell me? > > Downloading > https://pypi.python.org/packages/source/s/setuptools/setuptools- > 0.7.2.tar.gz > Extracting in /tmp/tmpJNVsOY > Now working in /tmp/tmpJNVsOY/setuptools-0.7.2 Building a Setuptools egg > in /tmp/tmpBLZGeg /tmp/tmpBLZGeg/setuptools-0.7.2-py2.6.egg > Traceback (most recent call last): > File "bootstrap.py", line 91, in > pkg_resources.working_set.add_entry(path) > File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 451, in > add_entry > self.add(dist, entry, False) > File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 542, in > add > self._added_new(dist) > File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 705, in > _added_new > callback(dist) > File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2727, in > > add_activation_listener(lambda dist: dist.activate()) > File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2227, in > activate > self.insert_on(path) > File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2328, in > insert_on > "with distribute. Found one at %s" % str(self.location)) > ValueError: A 0.7-series setuptools cannot be installed with distribute. > Found one at /tmp/tmpBLZGeg/setuptools-0.7.2-py2.6.egg > > I don't see any distribute in there, and I don't know where it might be... > > $ python2.6 > Python 2.6.8 (unknown, Jan 29 2013, 10:05:44) [GCC 4.7.2] on linux2 Type > "help", "copyright", "credits" or "license" for more information. > >>> import setuptools > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named setuptools > > cheers, > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From brett at python.org Sat Jul 13 18:59:18 2013 From: brett at python.org (Brett Cannon) Date: Sat, 13 Jul 2013 12:59:18 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: On Sat, Jul 13, 2013 at 11:15 AM, Paul Moore wrote: > On 13 July 2013 16:03, Donald Stufft wrote: > >> >> > 1. Install to user-packages by default. >> >> Do people really want this? I hadn't seen it (other than if pip was >> installed to user by default). I think it's a bad idea to switch this on >> people. I doubt the user-packages is going to be in people's default PATH >> so they'll easily get into cases where things are installed but they don't >> know where it was installed too. >> > > I believe Nick wants to make user-packages the default. I know at least > some of the pip maintainers (yourself included) have reservations. > Personally, I've never used user-packages, so I don't know what issues > might arise. But I hope to try it out sometime when I get the chance, just > to get some specific information. > I would assume the executable script was installed next to the python binary but the library parts went into user-packages. That way -m would work for all binaries of the same version. > > >> > 2. Not depend on setuptools (??? - Nick's "inversion" idea) >> >> I wanted to do this anyways. It will still "depend" on it, but it will >> just bundle setuptools itself like its other dependencies. For pip >> dependencies are an implementation detail not an actual thing it can/should >> have. >> > > Bundling is not the same as Nick's suggestion. I personally have no > problem with bundling, but pip install with a bundled setuptools might not > work because the setup subprocess won't see the bundled setuptools when it > imports it in setup.py. But either way, it's doable, I just want to know if > it's on the critical path... > > >> > 3. Possibly change the wrapper command name from pip to pip3 on Unix. >> >> Not sure on this. Ideally i'd want the commands to be pipX.Y, pipX, and >> pip all available and not install the less specific ones if they already >> exist but that might be too hard? >> > Could we just start to move away from an executable script and start promoting rather aggressively -m instead? It truly solves this problem and since the results are tied to the Python executable used (i.e. where something gets installed) it disambiguates what Python binary pip is going to work with (something I have trouble with thanks to Python 2 and 3 both being installed and each with their own pip installation). I realize older Python versions can't do this (I believe 2.6 and older can't for packages) but at least in the situation we are discussing here of bundling pip it's not an issue. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sat Jul 13 19:21:58 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 13 Jul 2013 10:21:58 -0700 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: > 1. Install to user-packages by default. > there was a thread a few weeks back on this. everyone seemed to agree at the end that just better error messages were enough. changing the default install location is a huge leap. http://mail.python.org/pipermail/distutils-sig/2013-May/020673.html > 2. Not depend on setuptools (??? - Nick's "inversion" idea) > with the bootstrap installing setuptools, it's not necessary, but I plan on considering/helping/working on one or multiple of these for pip v1.5 anyway: 1) "bundling" setuptools (Donald's idea). it might not work, but interesting to try. lotta pros to doing this 2) replacing pkg_resources with distlib (vinay posted a PR for this) 3) if not #1, pip installing setuptools on-demand when building is needed (this was the old plan I think for PEP439 until the recent changes, and get's us closer to the "MEB"s model) Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sat Jul 13 19:26:02 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 13 Jul 2013 10:26:02 -0700 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: 2) replacing pkg_resources with distlib (vinay posted a PR for this) > 3) if not #1, pip installing setuptools on-demand when building is needed > (this was the old plan I think for PEP439 until the recent changes, and > get's us closer to the "MEB"s model) > > to be clearer for everyone, #3 depends on #2, so that pip could install setuptools from wheel (without needing setuptools) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Jul 13 19:30:29 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 13:30:29 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> On Jul 13, 2013, at 12:59 PM, Brett Cannon wrote: > Could we just start to move away from an executable script and start promoting rather aggressively -m instead? It truly solves this problem and since the results are tied to the Python executable used (i.e. where something gets installed) it disambiguates what Python binary pip is going to work with (something I have trouble with thanks to Python 2 and 3 both being installed and each with their own pip installation). I realize older Python versions can't do this (I believe 2.6 and older can't for packages) but at least in the situation we are discussing here of bundling pip it's not an issue. I find the -m interface ugly as a primary cli api. It's ok for bonus functionality (ala json.tool) and debugging utilities (ala SimpleServer) but as a developer of user facing tools I don't think I'd ever want to tell them that they should use ``python -m`` to execute my tool. It's also a massive change in functionality from the existing pip interface. ``pip install`` is what everyone uses. The point is more or less moot though unless you're advocating not including an executable script at all. Because pip is already able to be executed with ``python -m pip`` however I don't believe i've seen anyone use that in practice. It also provides the "pip" and "pip-X.Y" commands which should probably be normalized to "pip", "pipX", and "pipX.Y". ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From noah at coderanger.net Sat Jul 13 19:31:32 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sat, 13 Jul 2013 10:31:32 -0700 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: <24297A81-EE5C-4150-860C-FFB484E6B468@coderanger.net> On Jul 13, 2013, at 9:59 AM, Brett Cannon wrote: > > > > On Sat, Jul 13, 2013 at 11:15 AM, Paul Moore wrote: > On 13 July 2013 16:03, Donald Stufft wrote: > > > 1. Install to user-packages by default. > > Do people really want this? I hadn't seen it (other than if pip was installed to user by default). I think it's a bad idea to switch this on people. I doubt the user-packages is going to be in people's default PATH so they'll easily get into cases where things are installed but they don't know where it was installed too. > > I believe Nick wants to make user-packages the default. I know at least some of the pip maintainers (yourself included) have reservations. Personally, I've never used user-packages, so I don't know what issues might arise. But I hope to try it out sometime when I get the chance, just to get some specific information. > > I would assume the executable script was installed next to the python binary but the library parts went into user-packages. That way -m would work for all binaries of the same version. > > > > 2. Not depend on setuptools (??? - Nick's "inversion" idea) > > I wanted to do this anyways. It will still "depend" on it, but it will just bundle setuptools itself like its other dependencies. For pip dependencies are an implementation detail not an actual thing it can/should have. > > Bundling is not the same as Nick's suggestion. I personally have no problem with bundling, but pip install with a bundled setuptools might not work because the setup subprocess won't see the bundled setuptools when it imports it in setup.py. But either way, it's doable, I just want to know if it's on the critical path... > > > 3. Possibly change the wrapper command name from pip to pip3 on Unix. > > Not sure on this. Ideally i'd want the commands to be pipX.Y, pipX, and pip all available and not install the less specific ones if they already exist but that might be too hard? > > Could we just start to move away from an executable script and start promoting rather aggressively -m instead? It truly solves this problem and since the results are tied to the Python executable used (i.e. where something gets installed) it disambiguates what Python binary pip is going to work with (something I have trouble with thanks to Python 2 and 3 both being installed and each with their own pip installation). I realize older Python versions can't do this (I believe 2.6 and older can't for packages) but at least in the situation we are discussing here of bundling pip it's not an issue. No, this is not how any user ever will expect unix programs to work. I know that python -m is very cute, and I use it myself for some debug and helper functionality at times, but it can never replace normal scripts. This is a user experience expectation, and we will have to meet it. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From noah at coderanger.net Sat Jul 13 19:36:30 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sat, 13 Jul 2013 10:36:30 -0700 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: <59573653-32BF-44DE-BCB8-6A38EE8613B6@coderanger.net> On Jul 13, 2013, at 6:46 AM, Brett Cannon wrote: > > > > On Sat, Jul 13, 2013 at 1:31 AM, Nick Coghlan wrote: > In addition to the long thread based on Richard's latest set of updates, I've also received a few off-list comments on the current state of the proposal. So, I figured I'd start a new thread summarising my current point of view and see where we want to go from there. > > 1. However we end up solving the bootstrapping problem, I'm *definitely* a fan of us updating pyvenv in 3.4 to ensure that pip is available by default in new virtual environments created with that tool. I also have an idea for a related import system feature that I'll be sending to import-sig this afternoon (it's a variant on *.pth and *.egg-link files that should be able to address a variety of existing problems, including the one of *selectively* making system and user packages available in a virtual environment in a cross-platform way without needing to copy them) > > 2. While I was originally a fan of the "implicit bootstrapping on demand" design, I no longer like that notion. While Richard's bootstrap script is a very nice piece of work, the edge cases and "neat tricks" have built up to the point where they trip my "if the implementation is hard to explain, it's a bad idea" filter. > > Accordingly, I no longer think the implicit bootstrapping is a viable option. > > 3. That means there are two main options available to us that I still consider viable alternatives (the installer bundling idea was suggested in one of the off list comments I mentioned): > > * an explicit bootstrapping script > * bundling a *full* copy of pip with the Python installers for Windows and Mac OS X, but installing it to site-packages rather than to the standard library directory. That way pip can be used to upgrade itself as normal, rather than making it part of the standard library per se. This is then closer to the "bundled application" model adopted for IDLE in PEP 434 (we could, in fact, move to distributing idle the same way). > > I'm currently leaning towards offering both, as we're going to need a tool for bootstrapping source builds, but the simplest way to bootstrap pip for Windows and Mac OS X users is to just *bundle a copy with the binary installers*. So long as the bundled copy looks *exactly* the way it would if installed later (so it can update itself), then we avoid the problem of coupling the pip update cycles to the standard library feature release cycle. The bundled version can be updated to the latest available versions when we do a Python maintenance release. > > For Linux, if you're using the system Python on a Debian or Fedora derivative, then "sudo apt-get python-pip" and "sudo yum install python-pip" are both straightforward, and if you're using something else, then it's unlikely getting pip bootstrapped using the bootstrap script is a task that will bother you :) > > The "python -m getpip" command is still something we will want to provide, as it is useful to people that build their own copy of Python from source. > > But is it going to make a difference? If we shift to using included copies of pip in binary installers over a bootstrap I say leave out the bootstrap as anyone building from source should know how to get pip installed on their machine or venv. > > The only reason I see it worth considering is if pyvenv starts bootstrapping pip and we want to support the case of pip not being installed. But if we are including it in the binary installer and are going to assume it's available through OS distros, then there isn't a need to as pip can then install pip for us into the venv and skip any initial pip bootstrap. If pip isn't found we can simply either point to the docs in the failure message or print out the one-liner it takes to install pip (and obviously there can be a --no-pip flag to skip this for people who want to install it manually like me who build from source). > > IOW I think taking the worldview in Python 3.4 that pip will come installed with Python unless you build from source negates the need for the bootstrap script beyond just saying ``curl https://pypi.python.org/get-pip.py | python`` if pip isn't found. This is highly unhelpful for dealing with systems automation. For the foreseeable future, the bulk of Python 3.4 installations will either be source installs, or homegrown packages based on source installs. The bundled pip doesn't need to be included with, say, an hg clone that you then build and install, but it does have to come with an install from an official release source tarball. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sat Jul 13 19:51:48 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 13:51:48 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: On Jul 13, 2013, at 12:59 PM, Brett Cannon wrote: > Could we just start to move away from an executable script and start promoting rather aggressively -m instead? It truly solves this problem and since the results are tied to the Python executable used (i.e. where something gets installed) it disambiguates what Python binary pip is going to work with (something I have trouble with thanks to Python 2 and 3 both being installed and each with their own pip installation). I realize older Python versions can't do this (I believe 2.6 and older can't for packages) but at least in the situation we are discussing here of bundling pip it's not an issue. Also looking at what already ships with Python. idle, idle2, idle2.7 smtpd.py, smtpd2.py, smptd2.7.py pydoc, pydoc2, pydoc2.7 2to3, 2to3-2, 2to3-2.7 This is also the convention anywhere someone does versioned scripts in a Python package in the ecosystem. PEP439 is there to streamline the process so that python dependencies are much easier to install and there's a smaller barrier to "entry" so that projects like Django can give simple instructions for dependencies instead of needing to opt not to have dependencies or have to give instructions on how to install the installer. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Sat Jul 13 19:55:24 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 13 Jul 2013 18:55:24 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> Message-ID: On 13 July 2013 18:30, Donald Stufft wrote: > On Jul 13, 2013, at 12:59 PM, Brett Cannon wrote: > > Could we just start to move away from an executable script and start > promoting rather aggressively -m instead? It truly solves this problem and > since the results are tied to the Python executable used (i.e. where > something gets installed) it disambiguates what Python binary pip is going > to work with (something I have trouble with thanks to Python 2 and 3 both > being installed and each with their own pip installation). I realize older > Python versions can't do this (I believe 2.6 and older can't for packages) > but at least in the situation we are discussing here of bundling pip it's > not an issue. > > > I find the -m interface ugly as a primary cli api. It's ok for bonus > functionality (ala json.tool) and debugging utilities (ala SimpleServer) > but as a developer of user facing tools I don't think I'd ever want to tell > them that they should use ``python -m`` to execute my tool. > > It's also a massive change in functionality from the existing pip > interface. ``pip install`` is what everyone uses. > > The point is more or less moot though unless you're advocating not > including an executable script at all. Because pip is already able to be > executed with ``python -m pip`` however I don't believe i've seen anyone > use that in practice. It also provides the "pip" and "pip-X.Y" commands > which should probably be normalized to "pip", "pipX", and "pipX.Y". > That's the point of "aggressively promote". We'd advocate "python -m pip" as the primary means of running pip. I agree it's less convenient for users than having a simple "pip" command, but there are a number of downsides to "pip" being the primary interface (note that you can always alias pip to "python -m pip" in your shell - it's no harder than managing PATH, which is what many people need to do at the moment). 1. It's not *actually* the case that the command is always "pip". Maybe it's "pip3" if your system makes the default Python be python 2, but you want to use python 3. Maybe you're creating a virtualenv and you haven't activated it yet. In that case a plain "pip" will quietly do the wrong thing (at the moment, I don't install pip in my system python precisely to avoid this issue). 2. On Windows, ...\Python34\Scripts is not on PATH by default. Even if python is (as python.exe is in a different directory to the one distutils installs executables in). Again, you can change your own PATH. 3. There's a lot of clutter. On Windows, you have 3 executables (pip.exe, pip3.exe and pip3.4.exe) and 3 scripts alongside them. For one command. Apart from the first of these, the issues are all Windows ones, and it's reasonable to say "well, fix the Windows setup, then, it's silly". I have some sympathy with that view, but backward compatibility and many, many years of history will make that extremely difficult. Also, some of it may simply not be fixable because people won't agree on the solution (that's been the case in the past). I assume, perhaps naively, that improving the experience on Windows is just as key as improving it on Unix. In my view, the key initial userbase for the new packaging tools will be Windows users wanting access to binary wheels. Paul. PS I actually *do* prefer just having a pip command. It's just that I doubt I'll get that on Windows, no matter *what* approach people take - I'll have to "roll my own" solution. Obviously, aliasing "pip" to "python -m pip" is the easiest, and probably the one I'd choose, but then again, I know what I'm doing :-) A "normal" Windows user will just say "these instructions are c**p, there's no pip command" and either have to resort to Google (probably with a question like "why doesn't Python's packaging work?" which isn't good PR for us), or give up. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Jul 13 19:58:43 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 13 Jul 2013 18:58:43 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: On 13 July 2013 18:51, Donald Stufft wrote: > > On Jul 13, 2013, at 12:59 PM, Brett Cannon wrote: > > Could we just start to move away from an executable script and start > promoting rather aggressively -m instead? It truly solves this problem and > since the results are tied to the Python executable used (i.e. where > something gets installed) it disambiguates what Python binary pip is going > to work with (something I have trouble with thanks to Python 2 and 3 both > being installed and each with their own pip installation). I realize older > Python versions can't do this (I believe 2.6 and older can't for packages) > but at least in the situation we are discussing here of bundling pip it's > not an issue. > > > Also looking at what already ships with Python. > > idle, idle2, idle2.7 > smtpd.py, smtpd2.py, smptd2.7.py > pydoc, pydoc2, pydoc2.7 > 2to3, 2to3-2, 2to3-2.7 > > This is also the convention anywhere someone does versioned scripts in a > Python package in the ecosystem. PEP439 is there to streamline the process > so that python dependencies are much easier to install and there's a > smaller barrier to "entry" so that projects like Django can give simple > instructions for dependencies instead of needing to opt not to have > dependencies or have to give instructions on how to install the installer. > None of these commands work at the command line in a base Python install on Windows. They are all obscure enough that nobody cares (or they are GUI apps that have a start menu entry provided, i.e. Idle) - that won't be the case for pip. Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Jul 13 20:24:24 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 14:24:24 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> Message-ID: On Jul 13, 2013, at 1:55 PM, Paul Moore wrote: > 1. It's not *actually* the case that the command is always "pip". Maybe it's "pip3" if your system makes the default Python be python 2, but you want to use python 3. Maybe you're creating a virtualenv and you haven't activated it yet. In that case a plain "pip" will quietly do the wrong thing (at the moment, I don't install pip in my system python precisely to avoid this issue). > 2. On Windows, ...\Python34\Scripts is not on PATH by default. Even if python is (as python.exe is in a different directory to the one distutils installs executables in). Again, you can change your own PATH. > 3. There's a lot of clutter. On Windows, you have 3 executables (pip.exe, pip3.exe and pip3.4.exe) and 3 scripts alongside them. For one command. > > Apart from the first of these, the issues are all Windows ones, and it's reasonable to say "well, fix the Windows setup, then, it's silly". I have some sympathy with that view, but backward compatibility and many, many years of history will make that extremely difficult. Also, some of it may simply not be fixable because people won't agree on the solution (that's been the case in the past). > 1. There's no good way to make it so you don't have to modify your command depending on what python you want to install into. In the case of both ``pip`` and ``python -m pip`` the changes a person would need to make is equivalent. They need to add a version number. The virtualenv case I don't see how that's relevant at all because if pip is preinstalled then pip will be available in both the virtualenv and the system environment. So both ``python -m pip`` and ``pip`` will be operating on the system python if you don't have it activated. 2. This sounds like something that needs fixed on Windows. Even if you say ``-m`` for pip then things are still broken by default for any other package on PyPI that installs a script. So this feels like something wrong with Python on windows not wrong with the script approach. 3. I don't really get the clutter argument. Does it *hurt* to have extra files there? I don't think i've ever looked at the directories on my $PATH and gone "wow I wish there was _less_ things in here". Is this an actual problem people have? Even if it was I think user experience trumps this case. I'm not sure what Brett is exactly advocating for. If he just wants to document it as ``python -m pip`` well whatever. I have absolutely zero faith that method of invocation will ever become popular. Every single piece of documentation that i've ever seen out there for installing things with pip tells people to use ``pip install``. Every developer that I've ever seen out there is using ``pip install``. An explicit command is shorter, easier to type, and already has basically all of the mindshare behind it. People gravitate towards what's easiest and in my opinion ``-m`` is easier only for the folks implementing this, not for the end users. Even if ``python -m pip`` is documented we still need to handle the CLI case, and I think that following the convention used by most other programs on *Nix and by windows itself of making the commands, "pip", "pipX", and "pipX.Y" makes the most sense. If Brett is advocating we _remove_ the command line options and expose only the ``python -m pip`` command that I am vehemently against that and in my opinion that makes for a far worst experience than users have now. The very first thing I would do, if it did happen this way, is create a package "pip-sanity" on PyPI that did nothing but restored the commands and then we end up with a similar situation we have now. That people need to run some bullshit before they can start using pip in the way they want to. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From qwcode at gmail.com Sat Jul 13 20:30:40 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 13 Jul 2013 11:30:40 -0700 Subject: [Distutils] flip the pip dependencies (was Current status of PEP 439) Message-ID: > I think we need to flip the dependencies so that pip as the installer has > all the essential code for installation from PyPI and then setuptools and > distlib depend on that pip infrastructure. No need to add anything to the > standard library prematurely when we can add it to pip instead. > not sure about the flip, but let me break some things down a bit for those who don't know: what pip has internally already (i.e. literally in it's package namespace): - pypi crawling/downloading - wheel installing (does not require the pypi wheel project; only building wheels requires that) what pip has "bundled' already: - distlib (in 'pip.vendor'; currently only used for some --pre version logic) what pip still needs to be self-sufficient to do wheel installs: - something bundled or internal that does what pkg_resources does theoretical options: 1) bundle setuptools/pkg_resources 2) use the bundled distlib to replace our use of pkg_resources 3) internalize pkg_resources as pip.pkg_resources (i.e. fork off pkg_resources) Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Jul 13 21:14:48 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 15:14:48 -0400 Subject: [Distutils] flip the pip dependencies (was Current status of PEP 439) In-Reply-To: References: Message-ID: On Jul 13, 2013, at 2:30 PM, Marcus Smith wrote: > > I think we need to flip the dependencies so that pip as the installer has all the essential code for installation from PyPI and then setuptools and distlib depend on that pip infrastructure. No need to add anything to the standard library prematurely when we can add it to pip instead. > > not sure about the flip, but let me break some things down a bit for those who don't know: > > what pip has internally already (i.e. literally in it's package namespace): > - pypi crawling/downloading > - wheel installing (does not require the pypi wheel project; only building wheels requires that) > > what pip has "bundled' already: > - distlib (in 'pip.vendor'; currently only used for some --pre version logic) > > what pip still needs to be self-sufficient to do wheel installs: > - something bundled or internal that does what pkg_resources does > > theoretical options: > 1) bundle setuptools/pkg_resources > 2) use the bundled distlib to replace our use of pkg_resources > 3) internalize pkg_resources as pip.pkg_resources (i.e. fork off pkg_resources) > > Marcus > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig As you're aware I think it makes the most sense to just bundle setuptools wholesale. This makes it impossible to "break" pip by something going wrong in setuptools causing it to be uninstalled and means that for users who are only doing installs, they don't need setuptools installed just pip. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From qwcode at gmail.com Sat Jul 13 21:35:05 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 13 Jul 2013 12:35:05 -0700 Subject: [Distutils] flip the pip dependencies (was Current status of PEP 439) In-Reply-To: References: Message-ID: > As you're aware I think it makes the most sense to just bundle setuptools > wholesale. This makes it impossible to "break" pip by something going wrong > in setuptools causing it to be uninstalled and means that for users who are > only doing installs, they don't need setuptools installed just pip. > I'm a fan of bundling too (if it works), but the "dynamic install of setuptools" idea also offers what you mention, although admittedly with more fragility. If a user uninstalled setuptools, it would be installed again when needed, and users only need pip to get started, and don't have to think about the setuptools dependency themselves. The drawbacks of bundling setuptools: 1) maybe some weird bug/side-effect shows up after we do it (ok, maybe that's FUD) 2) users can't upgrade themselves (for use in pip) 3) more tedium in our release process. 4) feels odd to bundle it knowing we'd likely drop it later, if we do the MEBs thing. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Jul 13 21:50:01 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 15:50:01 -0400 Subject: [Distutils] flip the pip dependencies (was Current status of PEP 439) In-Reply-To: References: Message-ID: <24660B6A-78A8-41E8-AA30-8F60DF6E5C72@stufft.io> On Jul 13, 2013, at 3:35 PM, Marcus Smith wrote: > > As you're aware I think it makes the most sense to just bundle setuptools wholesale. This makes it impossible to "break" pip by something going wrong in setuptools causing it to be uninstalled and means that for users who are only doing installs, they don't need setuptools installed just pip. > > I'm a fan of bundling too (if it works), but the "dynamic install of setuptools" idea also offers what you mention, although admittedly with more fragility. If a user uninstalled setuptools, it would be installed again when needed, and users only need pip to get started, and don't have to think about the setuptools dependency themselves. > > The drawbacks of bundling setuptools: > 1) maybe some weird bug/side-effect shows up after we do it (ok, maybe that's FUD) > 2) users can't upgrade themselves (for use in pip) > 3) more tedium in our release process. > 4) feels odd to bundle it knowing we'd likely drop it later, if we do the MEBs thing. > > Marcus 1) That's kinda FUD-y yea ;) But I'd say it's equally as likely to have weird bugs/side effects due to people using different combinations of pip/setuptools with pip than we've tested. 2) This much is true, the question then becomes how important is that? If there's a major regression in setuptools that needs fixed I'd think we'd release an updated pip. If there's new functionality I would guess we'd need to expose that in pip anyways. 3) I think this isn't as big of a deal as it sounds. Especially given we can write tooling to make it simpler :) 4) Even if MEBs were here *right now* we'd still have nearly 150k source dists that required setuptools. So either in the MEB system we'd be grabbing setuptools *a lot* or we could just bundle it to provide a better UX for people using the large corpus of existing software. I think it will be a long time once the MEBs exist before they gain enough traction that even the bulk of installs are using that system. MEBs depend on sdist 2.0 which hasn't even been started yet ;) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Sat Jul 13 21:59:44 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 13 Jul 2013 20:59:44 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> Message-ID: On 13 July 2013 19:24, Donald Stufft wrote: > 2. This sounds like something that needs fixed on Windows. Even if you say > ``-m`` for pip then things are still broken by default for any other > package on PyPI that installs a script. So this feels like something wrong > with Python on windows not wrong with the script approach. It is, and it should be fixed. But in many years, nobody has managed to come up with an acceptable solution. The debates seem to be largely around what happens if you install multiple versions of Python and then remove some of them, and how badly your system PATH gets messed up by this. I don't know how many people actually do things like that, but nevertheless it's never been sorted out. (Not all of the arguments are trivial, either, there are some genuinely difficult issues to resolve, IIRC). Ultimately, I guess there are a few options: * Accept that Windows is a problem in this regard, but don't worry about it - install executable wrappers/scripts and let the user deal with path issues. * Promote "python -m pip" as a least common denominator approach, and mildly irritate people who don't use Windows (they can still use the commands, but the docs look odd to them). * Only provide "python -m pip" and seriously annoy people who don't use Windows. * Document the difference, which implies either a certain level of repetitious "pip install (or py -m pip install on Windows)" type of thing, or a high level "For Windows, the pip command is not available directly, you should use ``python -m pip`` in its place (or wrap this in the shell if you prefer)" which people may miss. It would be nice to get feedback from "normal users" on this. I suspect that the scientific community would make a good cross-section (AIUI there's quite a lot of Windows use, and for many people in the community Python is very much a tool, rather than a way of life :-)). Does anyone have links into the scipy groups? I lurk on the IPython lists, so I could ask there, at a pinch... Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Jul 13 22:08:13 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 13 Jul 2013 21:08:13 +0100 Subject: [Distutils] flip the pip dependencies (was Current status of PEP 439) In-Reply-To: References: Message-ID: On 13 July 2013 20:35, Marcus Smith wrote: > The drawbacks of bundling setuptools: > 1) maybe some weird bug/side-effect shows up after we do it (ok, maybe > that's FUD) > One possible issue is with the "install from sdist" code, which runs Python in a subprocess with the "import setuptools, etc etc" incantation to force always using setuptools. That may break (or at least need changing) for a bundled setuptools, which won't be visible from the top level by default. Worth checking, anyway. Of course, this code path becomes less important as we move towards installing from wheels, but it's going to be a while before that's the norm. Paul PS Apologies if I already said this. I'm losing track of what I've replied to and what I've just thought about on this thread :-( -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Jul 13 22:14:22 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 16:14:22 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> Message-ID: <2BDA2CD1-CEC9-4EC0-A879-EE6A4D92D33C@stufft.io> On Jul 13, 2013, at 3:59 PM, Paul Moore wrote: > On 13 July 2013 19:24, Donald Stufft wrote: > 2. This sounds like something that needs fixed on Windows. Even if you say ``-m`` for pip then things are still broken by default for any other package on PyPI that installs a script. So this feels like something wrong with Python on windows not wrong with the script approach. > > It is, and it should be fixed. But in many years, nobody has managed to come up with an acceptable solution. The debates seem to be largely around what happens if you install multiple versions of Python and then remove some of them, and how badly your system PATH gets messed up by this. I don't know how many people actually do things like that, but nevertheless it's never been sorted out. (Not all of the arguments are trivial, either, there are some genuinely difficult issues to resolve, IIRC). > > Ultimately, I guess there are a few options: > * Accept that Windows is a problem in this regard, but don't worry about it - install executable wrappers/scripts and let the user deal with path issues. Ultimately I think this is what the community is going to do regardless of what happens here unless we remove the command line tools all together. > * Promote "python -m pip" as a least common denominator approach, and mildly irritate people who don't use Windows (they can still use the commands, but the docs look odd to them). This also has the problem where the existing documentation (project READMEs etc) are pointing to ``pip``. So it fractures the documentation about what the command "should" be. > * Only provide "python -m pip" and seriously annoy people who don't use Windows. Also invalidate all the existing documentation :) > * Document the difference, which implies either a certain level of repetitious "pip install (or py -m pip install on Windows)" type of thing, or a high level "For Windows, the pip command is not available directly, you should use ``python -m pip`` in its place (or wrap this in the shell if you prefer)" which people may miss. This is probably the most realistic approach, at least in my eyes. If the Scripts directory isn't available on windows people are going to need to know to either add it or execute it with python -m pip and that's going to include documentation outside our control. So given that there s a lot of existing documentation around ``pip`` and people are likely to continue that practice windows users will need to know that when random projects say to do ``pip install foo`` they need to translate that too ``python -m pip foo``. > > It would be nice to get feedback from "normal users" on this. I suspect that the scientific community would make a good cross-section (AIUI there's quite a lot of Windows use, and for many people in the community Python is very much a tool, rather than a way of life :-)). Does anyone have links into the scipy groups? I lurk on the IPython lists, so I could ask there, at a pinch? I don't know any windows users off hand except for you ;) (And you already said you use ``pip`` and not ``python -m pip`` which already works with pip :) > > Paul. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From qwcode at gmail.com Sat Jul 13 22:15:33 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 13 Jul 2013 13:15:33 -0700 Subject: [Distutils] flip the pip dependencies (was Current status of PEP 439) In-Reply-To: <24660B6A-78A8-41E8-AA30-8F60DF6E5C72@stufft.io> References: <24660B6A-78A8-41E8-AA30-8F60DF6E5C72@stufft.io> Message-ID: yea, all those comebacks make sense to me. we should try the bundle and see if it works. we already do some fancy footwork when working with setup.py https://github.com/pypa/pip/blob/develop/pip/req.py#L602 https://github.com/pypa/pip/blob/develop/pip/req.py#L687 https://github.com/pypa/pip/blob/develop/pip/req.py#L269 https://github.com/pypa/pip/blob/develop/pip/wheel.py#L291 I guess we'd be doing some additional override work in sys.modules. Marcus On Sat, Jul 13, 2013 at 12:50 PM, Donald Stufft wrote: > > On Jul 13, 2013, at 3:35 PM, Marcus Smith wrote: > > > As you're aware I think it makes the most sense to just bundle setuptools >> wholesale. This makes it impossible to "break" pip by something going wrong >> in setuptools causing it to be uninstalled and means that for users who are >> only doing installs, they don't need setuptools installed just pip. >> > > I'm a fan of bundling too (if it works), but the "dynamic install of > setuptools" idea also offers what you mention, although admittedly with > more fragility. If a user uninstalled setuptools, it would be installed > again when needed, and users only need pip to get started, and don't have > to think about the setuptools dependency themselves. > > The drawbacks of bundling setuptools: > 1) maybe some weird bug/side-effect shows up after we do it (ok, maybe > that's FUD) > 2) users can't upgrade themselves (for use in pip) > 3) more tedium in our release process. > 4) feels odd to bundle it knowing we'd likely drop it later, if we do the > MEBs thing. > > Marcus > > > > 1) That's kinda FUD-y yea ;) But I'd say it's equally as likely to have > weird bugs/side effects due to people using different combinations of > pip/setuptools with pip than we've tested. > > 2) This much is true, the question then becomes how important is that? If > there's a major regression in setuptools that needs fixed I'd think we'd > release an updated pip. If there's new functionality I would guess we'd > need to expose that in pip anyways. > > 3) I think this isn't as big of a deal as it sounds. Especially given we > can write tooling to make it simpler :) > > 4) Even if MEBs were here *right now* we'd still have nearly 150k source > dists that required setuptools. So either in the MEB system we'd be > grabbing setuptools *a lot* or we could just bundle it to provide a better > UX for people using the large corpus of existing software. I think it will > be a long time once the MEBs exist before they gain enough traction that > even the bulk of installs are using that system. MEBs depend on sdist 2.0 > which hasn't even been started yet ;) > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Jul 13 22:16:40 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 16:16:40 -0400 Subject: [Distutils] flip the pip dependencies (was Current status of PEP 439) In-Reply-To: References: Message-ID: <1FEBEA1D-E166-4F30-8FA6-B9913BA304A4@stufft.io> On Jul 13, 2013, at 4:08 PM, Paul Moore wrote: > On 13 July 2013 20:35, Marcus Smith wrote: > The drawbacks of bundling setuptools: > 1) maybe some weird bug/side-effect shows up after we do it (ok, maybe that's FUD) > > One possible issue is with the "install from sdist" code, which runs Python in a subprocess with the "import setuptools, etc etc" incantation to force always using setuptools. That may break (or at least need changing) for a bundled setuptools, which won't be visible from the top level by default. Worth checking, anyway. > > Of course, this code path becomes less important as we move towards installing from wheels, but it's going to be a while before that's the norm. > > Paul > > PS Apologies if I already said this. I'm losing track of what I've replied to and what I've just thought about on this thread :-( Pip already wraps that code to force things to use setuptools even if they use distutils. So in that case pip would just need to modify it's own code to add setuptools to sys.modules (or extend sys.path in that sub process). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Sat Jul 13 23:35:55 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 13 Jul 2013 22:35:55 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: <2BDA2CD1-CEC9-4EC0-A879-EE6A4D92D33C@stufft.io> References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> <2BDA2CD1-CEC9-4EC0-A879-EE6A4D92D33C@stufft.io> Message-ID: On 13 July 2013 21:14, Donald Stufft wrote: > I don't know any windows users off hand except for you ;) (And you already > said you use ``pip`` and not ``python -m pip`` which already works with pip > :) You caught me :-) My problem is that I'm pretty sure I'm seriously atypical in never installing anything into my system Python. (And so I only use the pip command in activated virtualenvs, which *do* add the scripts directory to PATH). Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Jul 14 00:46:10 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 18:46:10 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> <2BDA2CD1-CEC9-4EC0-A879-EE6A4D92D33C@stufft.io>, Message-ID: <3AFF5308-D205-4887-8CD3-35EFBA8C3C50@stufft.io> On Jul 13, 2013, at 6:44 PM, Steve Dower wrote: > Because of the issues around compilation on Windows, we believe that most users avoid pip in favor of precompiled installers. The model of "download an executable that matches my Python version and run it" is more familiar than a command line tool, and unlikely to go away anytime soon. Luckily for them the upcoming pip 1.4 includes support for compiled packages called Wheels ;) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From Steve.Dower at microsoft.com Sun Jul 14 00:44:50 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 13 Jul 2013 22:44:50 +0000 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> <2BDA2CD1-CEC9-4EC0-A879-EE6A4D92D33C@stufft.io>, Message-ID: Because of the issues around compilation on Windows, we believe that most users avoid pip in favor of precompiled installers. The model of "download an executable that matches my Python version and run it" is more familiar than a command line tool, and unlikely to go away anytime soon. Those who use pip are going to be quite capable of managing their PATH variable to ensure the correct one is used (or they'll do what I do and use a full path). There are also GUI apps, but I have no idea how widely used they are. Sent from my Windows Phone ________________________________ From: Paul Moore Sent: ?7/?13/?2013 14:36 To: Donald Stufft Cc: Distutils Subject: Re: [Distutils] Expectations on how pip needs to change for Python 3.4 On 13 July 2013 21:14, Donald Stufft > wrote: I don't know any windows users off hand except for you ;) (And you already said you use ``pip`` and not ``python -m pip`` which already works with pip :) You caught me :-) My problem is that I'm pretty sure I'm seriously atypical in never installing anything into my system Python. (And so I only use the pip command in activated virtualenvs, which *do* add the scripts directory to PATH). Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Sun Jul 14 01:45:06 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 13 Jul 2013 23:45:06 +0000 (UTC) Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 References: Message-ID: Paul Moore gmail.com> writes: > 4. Ensure that pip upgrading itself in-place is sufficiently robust and > reliable that users don't get "stuck" on the Python-supplied version. Perhaps one could add to your list, the ability to downgrade to the previous version should there be a problem with a newly-upgraded version. Regards, Vinay Sajip From qwcode at gmail.com Sun Jul 14 02:06:41 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 13 Jul 2013 17:06:41 -0700 Subject: [Distutils] pip and virtualenv release candidates In-Reply-To: References: Message-ID: pip-1.4rc4 and virtualenv-1.10rc6 are now available the changes from the previous RCs: - virtualenv now contains setuptools v0.9 (which enables indexes to use md5, sha1, or one of the sha2 variants in their urls) - the new "pip install --pre" option now applies to all packages installed in the command, not just top-level requirements - pip support for building and installing pybundles is now noted as deprecated (our plan is to remove it in v1.5) here's the RC install instructions again: $ curl -L -O https://github.com/pypa/virtualenv/archive/1.10rc6.tar.gz $ tar zxf 1.10rc6.tar.gz $ python virtualenv-1.10rc6/virtualenv.py myVE $ myVE/bin/pip --version pip 1.4rc4 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jul 14 04:20:31 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jul 2013 12:20:31 +1000 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: On 14 July 2013 00:54, Paul Moore wrote: > This issue has been skirted round for some time now, and I think it needs > explicit discussion, as I am not at all sure everyone has the same > expectations. > > We're talking about Python 3.4 installations having pip as the default > package manager - whether by bundling, having a bootstrap process or > whatever. Regardless of the means, pip will be *the* installer for Python > 3.4+. And yet, I don't think pip 1.4 currently does what people want "the > Python 3.4 pip" to do in some ways - and we need to make sure that any work > on the pip side is understood, agreed to, and planned to match the Python > 3.4 timescales. > Good point. We also need to start to articulate the relevant questions for the core development side of the fence - Richard, it would be good if PEP 439 could be the vehicle for this, even though it does mean I'm shifting the goal posts on you fairly substantially. Then we can turn it into a set of tracker issues for pip and CPython. As a reminder, here's the current deadlines as per PEP 429 (the 3.4 release schedule): Changes to CPython: November 23, 2013 (3.4 beta 1) Changes to pip: January 18, 2014 (3.4 rc 1) (allowing changes to bundled applications up until the first RC is what I think we *should* do, but that will require agreement from Larry Hastings as release manager) So, here's my initial list of things that I think people might be expecting > to happen. This is just my impressions, and I don't necessarily have a view > on the individual items. And if anyone else can think of other things to > add to the list, please do so! > > 1. Install to user-packages by default. > I made this suggestion at one point, but Marcus and others convinced me it was a bad idea. Issue for better error message filed as https://github.com/pypa/pip/issues/1048 (I don't have the power to set milestones) > 2. Not depend on setuptools (??? - Nick's "inversion" idea) > I think Donald is right that bundling a vendor'ed copy of setuptools is the most sensible near-term option - there's too much risk of upgrade/downgrade issues when allowing arbitrary combinations of pip with setuptools on target systems for source based installs. > 3. Possibly change the wrapper command name from pip to pip3 on Unix. > I think there's a bit more to it than that. Really, what we want to try to ensure is that the following commands are available across Windows, Mac OS X and *nix (ignoring, for the moment, the behaviour of vendor provided installations for Mac OS X and Linux): * python, python3, python3.4 * pip, pip3, pip3.4 That is, the version qualifier on the pip executable would relate to the *default Python version* associated with that executable/script, rather than the version of pip. We would take whatever steps were needed in our Windows and Mac OS X installers to ensure all these wrappers were provided. The reason why I think we still want to offer "python -m getpip" is because I think that sends a clearer message to repackagers that we *do* consider pip a part of Python now, but we keep the source control and issue management for the two projects separate for pragmatic reasons (notably, the different update lifecycles). It's still going to map to two separate source tarballs (and hence SRPMs) for While I acknowledge you *can* invoke the Python launcher directly on Windows, I think it is better to leave that in the background as a tool for advanced users, as well the engine that lets us base shebang line processing on Windows file associations. If/when we start offering a "py" style launcher on POSIX systems as well, then we can revisit that question. > 4. Ensure that pip upgrading itself in-place is sufficiently robust and > reliable that users don't get "stuck" on the Python-supplied version. > As Vinay noted, we also need to ensure downgrades work. However, must of these have been related to depending on an external setuptools, so eliminating that should help a lot. > I'm sure I've seen people say other things that have made me think "are > you expecting the pip maintainers to make that change?" in the various > threads, so I doubt this list is definitive. > The other big one is the one you noted about pip *not* offering a stable API, *but* exposing an apparently stable API to introspection. Introspection currently tells me that pip exports *at least* 32 public names (and this is without checking for public submodules that aren't implicitly imported by pip/__init__.py): >>> import pip; public = set(k for k, v in pip.__dict__.items() if not k.startswith('_') and (not hasattr(v, "__name__") or hasattr(v, "__module__") or v.__name__.startswith("pip."))); print(len(public)) 32 If pip really has no stable public API, then it should properly indicate this under introspection (if it already uses relative imports correctly, then the easiest ways to achieve that are to just shove everything under a "pip._impl" subpackage or shuffle it sideways into a "_pip" package). > Comments anyone? Is this discussion premature? The pip maintainers team is > not huge, so we'll need time (or assistance!) to plan in and make changes > like this, if they are needed... > Agreed, I think refocusing the discussion on "What do we need to do in pip?" and "What do we need to do in CPython?" is a very necessary step at this point. > At a minimum, can we get the key items logged on the pip issue tracker > with a milestone of Python 3.4? > The existing 1.5 milestone is probably usable - I expect 1.5 is the version that would be bundled with 3.4. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Jul 14 04:46:22 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 22:46:22 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> On Jul 13, 2013, at 10:20 PM, Nick Coghlan wrote: > On 14 July 2013 00:54, Paul Moore wrote: > This issue has been skirted round for some time now, and I think it needs explicit discussion, as I am not at all sure everyone has the same expectations. > > We're talking about Python 3.4 installations having pip as the default package manager - whether by bundling, having a bootstrap process or whatever. Regardless of the means, pip will be *the* installer for Python 3.4+. And yet, I don't think pip 1.4 currently does what people want "the Python 3.4 pip" to do in some ways - and we need to make sure that any work on the pip side is understood, agreed to, and planned to match the Python 3.4 timescales. > > Good point. We also need to start to articulate the relevant questions for the core development side of the fence - Richard, it would be good if PEP 439 could be the vehicle for this, even though it does mean I'm shifting the goal posts on you fairly substantially. > > Then we can turn it into a set of tracker issues for pip and CPython. As a reminder, here's the current deadlines as per PEP 429 (the 3.4 release schedule): > > Changes to CPython: November 23, 2013 (3.4 beta 1) > Changes to pip: January 18, 2014 (3.4 rc 1) Good dates to have! > > (allowing changes to bundled applications up until the first RC is what I think we *should* do, but that will require agreement from Larry Hastings as release manager) > > So, here's my initial list of things that I think people might be expecting to happen. This is just my impressions, and I don't necessarily have a view on the individual items. And if anyone else can think of other things to add to the list, please do so! > > 1. Install to user-packages by default. > > I made this suggestion at one point, but Marcus and others convinced me it was a bad idea. Issue for better error message filed as https://github.com/pypa/pip/issues/1048 (I don't have the power to set milestones) Added this to the 1.5 milestone and mentioned my agreement to the implementation on the ticket. > > 2. Not depend on setuptools (??? - Nick's "inversion" idea) > > I think Donald is right that bundling a vendor'ed copy of setuptools is the most sensible near-term option - there's too much risk of upgrade/downgrade issues when allowing arbitrary combinations of pip with setuptools on target systems for source based installs. https://github.com/pypa/pip/issues/1049 > > 3. Possibly change the wrapper command name from pip to pip3 on Unix. > > I think there's a bit more to it than that. Really, what we want to try to ensure is that the following commands are available across Windows, Mac OS X and *nix (ignoring, for the moment, the behaviour of vendor provided installations for Mac OS X and Linux): > > * python, python3, python3.4 > * pip, pip3, pip3.4 > > That is, the version qualifier on the pip executable would relate to the *default Python version* associated with that executable/script, rather than the version of pip. We would take whatever steps were needed in our Windows and Mac OS X installers to ensure all these wrappers were provided. https://github.com/pypa/pip/issues/1050 > > The reason why I think we still want to offer "python -m getpip" is because I think that sends a clearer message to repackagers that we *do* consider pip a part of Python now, but we keep the source control and issue management for the two projects separate for pragmatic reasons (notably, the different update lifecycles). It's still going to map to two separate source tarballs (and hence SRPMs) for I don't care if getpip is available especially if that's the command that is actually executed to pre-install pip for the CPython releases. (To be clear, I agree with Noah that pip should be pre-installed for every type of official release Python makes. However It does not need to be there from a hg.python.org checkout). > > While I acknowledge you *can* invoke the Python launcher directly on Windows, I think it is better to leave that in the background as a tool for advanced users, as well the engine that lets us base shebang line processing on Windows file associations. If/when we start offering a "py" style launcher on POSIX systems as well, then we can revisit that question. > > 4. Ensure that pip upgrading itself in-place is sufficiently robust and reliable that users don't get "stuck" on the Python-supplied version. > > As Vinay noted, we also need to ensure downgrades work. However, must of these have been related to depending on an external setuptools, so eliminating that should help a lot. In the years of using pip the only time i've ever had any issue upgrading or downgrading pip was related to setuptools screw ups. But possibly we do want to have some explicit testing around this? > > I'm sure I've seen people say other things that have made me think "are you expecting the pip maintainers to make that change?" in the various threads, so I doubt this list is definitive. > > The other big one is the one you noted about pip *not* offering a stable API, *but* exposing an apparently stable API to introspection. Introspection currently tells me that pip exports *at least* 32 public names (and this is without checking for public submodules that aren't implicitly imported by pip/__init__.py): > > >>> import pip; public = set(k for k, v in pip.__dict__.items() if not k.startswith('_') and (not hasattr(v, "__name__") or hasattr(v, "__module__") or v.__name__.startswith("pip."))); print(len(public)) > 32 > > If pip really has no stable public API, then it should properly indicate this under introspection (if it already uses relative imports correctly, then the easiest ways to achieve that are to just shove everything under a "pip._impl" subpackage or shuffle it sideways into a "_pip" package). Pip does not use relative imports. Is simply documenting the fact there is no public API enough? Pushing everything into a _impl or _pip directory makes me nervous because that's a lot of code churn (and I know there are people using those APIs, and while they aren't technically stable it feels like moving things around just for the sake of an _ in the name is unfriendly to those people. > > Comments anyone? Is this discussion premature? The pip maintainers team is not huge, so we'll need time (or assistance!) to plan in and make changes like this, if they are needed... > > Agreed, I think refocusing the discussion on "What do we need to do in pip?" and "What do we need to do in CPython?" is a very necessary step at this point. Agreed. > > At a minimum, can we get the key items logged on the pip issue tracker with a milestone of Python 3.4? > > The existing 1.5 milestone is probably usable - I expect 1.5 is the version that would be bundled with 3.4. Instead of a milestone I added a PEP439 tag so that we can differentiate between 1.5 milestone items for PEP439 and not. Ideally we don't need to drop anything from 1.5 but just in case we do. I think we should probably target pip 1.5 to release in the beginning of December? Would need to see what the other team members think, that's a shorter release cycle then we normally have but I think it'd be good to have 1.5 out for a month or so to get real world use to make sure it doesn't need a patch release before inclusion in CPython (assuming the dates you mentioned are correct). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Sun Jul 14 04:59:02 2013 From: dholth at gmail.com (Daniel Holth) Date: Sat, 13 Jul 2013 21:59:02 -0500 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> Message-ID: It is easy to forget that pip only needs the "package database" part of setuptools (pkg_resources.py) to install things. With the small catch that the rest of setuptools is required to install anything besides wheels. MEBS is just about implementing build requirements properly and giving pip a consistent interface to build traditional sdists *or* any new (sdist 2.0, distil, bento) kinds of packages that may come along. From donald at stufft.io Sun Jul 14 05:00:21 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 23:00:21 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> Message-ID: <42ECB620-5A0F-4C77-8451-159E4D88504C@stufft.io> On Jul 13, 2013, at 10:59 PM, Daniel Holth wrote: > It is easy to forget that pip only needs the "package database" part > of setuptools (pkg_resources.py) to install things. With the small > catch that the rest of setuptools is required to install anything > besides wheels. > > MEBS is just about implementing build requirements properly and giving > pip a consistent interface to build traditional sdists *or* any new > (sdist 2.0, distil, bento) kinds of packages that may come along. Where "besides wheels" represents almost every single pip installable package on PyPI. Wheels are great, but reality is we need something sane for sdists where setuptools is expected to be there. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sun Jul 14 05:08:01 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 13 Jul 2013 23:08:01 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: I've gone through the pip issue tracker and attempted to identify issues related to the things we want to get in for PEP439. Some of them are duplicates but I left them opened. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From qwcode at gmail.com Sun Jul 14 07:00:50 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 13 Jul 2013 22:00:50 -0700 Subject: [Distutils] pip and virtualenv release candidates In-Reply-To: References: Message-ID: pip-1.4rc4 and virtualenv-1.10rc7 are now available the changes from the previous RCs: - virtualenv now contains setuptools v0.9.1 here's the RC install instructions again: $ curl -L -O https://github.com/pypa/virtualenv/archive/1.10rc7.tar.gz $ tar zxf 1.10rc7.tar.gz $ python virtualenv-1.10rc7/virtualenv.py myVE $ myVE/bin/pip --version pip 1.4rc4 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jul 14 07:58:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jul 2013 15:58:02 +1000 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> Message-ID: On 14 July 2013 12:46, Donald Stufft wrote: > I'm sure I've seen people say other things that have made me think "are >> you expecting the pip maintainers to make that change?" in the various >> threads, so I doubt this list is definitive. >> > > The other big one is the one you noted about pip *not* offering a stable > API, *but* exposing an apparently stable API to introspection. > Introspection currently tells me that pip exports *at least* 32 public > names (and this is without checking for public submodules that aren't > implicitly imported by pip/__init__.py): > > >>> import pip; public = set(k for k, v in pip.__dict__.items() if not > k.startswith('_') and (not hasattr(v, "__name__") or hasattr(v, > "__module__") or v.__name__.startswith("pip."))); print(len(public)) > 32 > > If pip really has no stable public API, then it should properly indicate > this under introspection (if it already uses relative imports correctly, > then the easiest ways to achieve that are to just shove everything under a > "pip._impl" subpackage or shuffle it sideways into a "_pip" package). > > > Pip does not use relative imports. Is simply documenting the fact there is > no public API enough? Pushing everything into a _impl or _pip directory > makes me nervous because that's a lot of code churn (and I know there are > people using those APIs, and while they aren't technically stable it feels > like moving things around just for the sake of an _ in the name is > unfriendly to those people. > Either the existing APIs are moved to a different name, or they get declared stable and pip switches to "internally forked" APIs any time a backwards incompatible change is needed for refactoring purposes (see runpy._run_module_as_main for an example of needing to do this in the standard library). I've had to directly deal with too many issues arising from getting this wrong in the past for me to endorse bundling of a module that doesn't follow this practice with CPython - if introspection indicates an API is public, then it's public and subject to all standard library backwards compatibility guarantees, or else we take the pain *once* and explicitly mark it private by adding a leading underscore rather than leaving it in limbo (contextlib._GeneratorContextManager is a standard library example of the latter approach - it used to lack the leading underscore, suggesting it was a public API when it's really just an implementation detail of contextlib.contextmanager). Mere documentation of public vs private generally doesn't cut it, as too many people use dir(), help() and inspect() rather than the published docs to explore APIs. The only general exception I'm aware of is "test" packages, including the standard library's test package, and for those you can make the case that having "test" or "tests" as a name segment is just as clear an indicator of something being private as at least one name segment starting with a leading underscore. I really this is a fairly big ask for the pip maintainers, but I *don't* consider "Oh, don't use our module API, it isn't stable" to be an adequate answer for something that is bundled with the standard installers. Beyond that, I don't mind if the answer is to declare the 1.5 API stable or to sprinkle underscore where appropriate or moving everything to a private package - the documentation and the naming conventions just need to be consistent in their private vs public distinctions (although your points do suggest heavily that the right answer is to accept the burden of backwards compatibility for all APIs currently marked public, and move towards the introduction of appropriate private APIs over time through refactoring). Instead of a milestone I added a PEP439 tag so that we can differentiate between 1.5 milestone items for PEP439 and not. Ideally we don't need to drop anything from 1.5 but just in case we do. > I think we should probably target pip 1.5 to release in the beginning of > December? Would need to see what the other team members think, that's a > shorter release cycle then we normally have but I think it'd be good to > have 1.5 out for a month or so to get real world use to make sure it > doesn't need a patch release before inclusion in CPython (assuming the > dates you mentioned are correct). > Just to confuse matters a little bit, Richard has suggested explicitly creating a bundling PEP as a *competitor* to PEP 439, thus making it easier to be explicit about our reasons for rejecting bootstrapping in favour of bundling. I think that's a good way to move this forward, but I won't actually reject 439 until the competing bundling PEP has been posted (otherwise people might get the wrong impression that we're moving away from the idea of making "pip install X" work out of the box, when we're really just changing our tactics for achieving that goal). I also realised what is probably a better idea than "python -m getpip" for dealing with the "How do I get pip after doing a source build?": add a "get-pip.py" utility to Tools/scripts in the cpython repo, rather than adding anything to the standard library. This also puts us on a more solid footing for getting pip bundled with 2.7.x at some point: we're not touching the standard library, just the installers and the utility scripts. The bundling PEP should also suggest to Linux packagers that pip be considered an essential part of a fully functional Python installation. Exactly how that is handled will be up to the distro packagers, but could include noting pip as a recommended dependency for Python (Debian), or rearranging the packaging to make "cpython" a package in its own right, with "python" requiring both "cpython" and "python-pip" (while the latter would just require cpython). I'll post an explicit call for a PEP champion in a separate thread. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jul 14 08:13:00 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jul 2013 16:13:00 +1000 Subject: [Distutils] Call for PEP author/champion: Bundling pip with CPython installers Message-ID: Based on the recent discussions, I now plan to reject the pip bootstrapping-on-first-invocation approach described in PEP 439 in favour of a new PEP that proposes: * bundling the latest version of pip with the CPython binary installers for Mac OS X and Windows for all future CPython releases (including maintenance releases) * aligns the proposal with the Python 3.4 release schedule by noting that CPython changes must be completed by the first 3.4 beta, while pip changes must be completed by the first 3.4 release candidate. * ensuring that, for Python 3.4, "python3" and "python3.4" are available for command line invocation of Python, even on Windows * ensuring that the bundled pip, for Python 3.4, ensures "pip", "pip3" and "pip3.4" are available for command line invocation of Python, even on Windows * ensuring that the bundled pip is able to upgrade/downgrade itself independent of the CPython release cycle * ensuring that pip is automatically available in virtual environments created with pyvenv * adding a "get-pip.py" script to Tools/scripts in the CPython repo for bootstrapping the latest pip release from PyPI into custom CPython source builds Note that there are still open questions to be resolved, which is why an author/champion is needed: * what guidance, if any, should we provide to Linux distro packagers? * how should maintenance updates handle the presence of an existing pip installation? Automatically upgrade older versions to the bundled version, while leaving newer versions alone? Force installation of the bundled version? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah at coderanger.net Sun Jul 14 08:19:58 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sat, 13 Jul 2013 23:19:58 -0700 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> Message-ID: <28DB7C9B-5978-44FD-B713-C4A17FEC7B16@coderanger.net> On Jul 13, 2013, at 10:58 PM, Nick Coghlan wrote: > On 14 July 2013 12:46, Donald Stufft wrote: >> I'm sure I've seen people say other things that have made me think "are you expecting the pip maintainers to make that change?" in the various threads, so I doubt this list is definitive. >> >> The other big one is the one you noted about pip *not* offering a stable API, *but* exposing an apparently stable API to introspection. Introspection currently tells me that pip exports *at least* 32 public names (and this is without checking for public submodules that aren't implicitly imported by pip/__init__.py): >> >> >>> import pip; public = set(k for k, v in pip.__dict__.items() if not k.startswith('_') and (not hasattr(v, "__name__") or hasattr(v, "__module__") or v.__name__.startswith("pip."))); print(len(public)) >> 32 >> >> If pip really has no stable public API, then it should properly indicate this under introspection (if it already uses relative imports correctly, then the easiest ways to achieve that are to just shove everything under a "pip._impl" subpackage or shuffle it sideways into a "_pip" package). > > Pip does not use relative imports. Is simply documenting the fact there is no public API enough? Pushing everything into a _impl or _pip directory makes me nervous because that's a lot of code churn (and I know there are people using those APIs, and while they aren't technically stable it feels like moving things around just for the sake of an _ in the name is unfriendly to those people. > > Either the existing APIs are moved to a different name, or they get declared stable and pip switches to "internally forked" APIs any time a backwards incompatible change is needed for refactoring purposes (see runpy._run_module_as_main for an example of needing to do this in the standard library). I've had to directly deal with too many issues arising from getting this wrong in the past for me to endorse bundling of a module that doesn't follow this practice with CPython - if introspection indicates an API is public, then it's public and subject to all standard library backwards compatibility guarantees, or else we take the pain *once* and explicitly mark it private by adding a leading underscore rather than leaving it in limbo (contextlib._GeneratorContextManager is a standard library example of the latter approach - it used to lack the leading underscore, suggesting it was a public API when it's really just an implementation detail of contextlib.contextmanager). > Respectfully, I disagree. Pip is not going in to the stdlib, and as such should not be subject to the same API stability policies as the stdlib. If the PyPA team wants to break the API every release, that is their call as the subject matter experts. Pip is not being included as a library at all. What should be subject to compat is the defined command line interface, because pip is a CLI tool. Independently of this discussion I've already been talking to the PyPA team about what they want to consider a stable API, but that is a discussion to be had over in pip-land, not here and not now. This new category of "bundled for your convenience but still external" applications will need new standards, and we should be clear about them for sure, but I think this is going too far and puts undue burden on the PyPA team. Remember the end goal is simply to get an installer in the hands of users easier. --Noah From ncoghlan at gmail.com Sun Jul 14 08:34:57 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jul 2013 16:34:57 +1000 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: <28DB7C9B-5978-44FD-B713-C4A17FEC7B16@coderanger.net> References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> <28DB7C9B-5978-44FD-B713-C4A17FEC7B16@coderanger.net> Message-ID: On 14 July 2013 16:19, Noah Kantrowitz wrote: > On Jul 13, 2013, at 10:58 PM, Nick Coghlan wrote: > > On 14 July 2013 12:46, Donald Stufft wrote: > > Either the existing APIs are moved to a different name, or they get > declared stable and pip switches to "internally forked" APIs any time a > backwards incompatible change is needed for refactoring purposes (see > runpy._run_module_as_main for an example of needing to do this in the > standard library). I've had to directly deal with too many issues arising > from getting this wrong in the past for me to endorse bundling of a module > that doesn't follow this practice with CPython - if introspection indicates > an API is public, then it's public and subject to all standard library > backwards compatibility guarantees, or else we take the pain *once* and > explicitly mark it private by adding a leading underscore rather than > leaving it in limbo (contextlib._GeneratorContextManager is a standard > library example of the latter approach - it used to lack the leading > underscore, suggesting it was a public API when it's really just an > implementation detail of contextlib.contextmanager). > > > > Respectfully, I disagree. Pip is not going in to the stdlib, and as such > should not be subject to the same API stability policies as the stdlib. If > the PyPA team wants to break the API every release, that is their call as > the subject matter experts. Pip is not being included as a library at all. > What should be subject to compat is the defined command line interface, > because pip is a CLI tool. Independently of this discussion I've already > been talking to the PyPA team about what they want to consider a stable > API, but that is a discussion to be had over in pip-land, not here and not > now. This new category of "bundled for your convenience but still external" > applications will need new standards, and we should be clear about them for > sure, but I think this is going too far and puts undue burden on the PyPA > team. Remember the end goal is simply to get an installer in the hands of > users easier. > I would also be fine with a solution where "import pip" issues a warning about API instability if sys.argv[0] indicates the main executable is something other than the pip CLI. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jul 14 08:35:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jul 2013 16:35:41 +1000 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> <28DB7C9B-5978-44FD-B713-C4A17FEC7B16@coderanger.net> Message-ID: On 14 July 2013 16:34, Nick Coghlan wrote: > On 14 July 2013 16:19, Noah Kantrowitz wrote: > >> On Jul 13, 2013, at 10:58 PM, Nick Coghlan wrote: >> > On 14 July 2013 12:46, Donald Stufft wrote: >> > Either the existing APIs are moved to a different name, or they get >> declared stable and pip switches to "internally forked" APIs any time a >> backwards incompatible change is needed for refactoring purposes (see >> runpy._run_module_as_main for an example of needing to do this in the >> standard library). I've had to directly deal with too many issues arising >> from getting this wrong in the past for me to endorse bundling of a module >> that doesn't follow this practice with CPython - if introspection indicates >> an API is public, then it's public and subject to all standard library >> backwards compatibility guarantees, or else we take the pain *once* and >> explicitly mark it private by adding a leading underscore rather than >> leaving it in limbo (contextlib._GeneratorContextManager is a standard >> library example of the latter approach - it used to lack the leading >> underscore, suggesting it was a public API when it's really just an >> implementation detail of contextlib.contextmanager). >> > >> >> Respectfully, I disagree. Pip is not going in to the stdlib, and as such >> should not be subject to the same API stability policies as the stdlib. If >> the PyPA team wants to break the API every release, that is their call as >> the subject matter experts. Pip is not being included as a library at all. >> What should be subject to compat is the defined command line interface, >> because pip is a CLI tool. Independently of this discussion I've already >> been talking to the PyPA team about what they want to consider a stable >> API, but that is a discussion to be had over in pip-land, not here and not >> now. This new category of "bundled for your convenience but still external" >> applications will need new standards, and we should be clear about them for >> sure, but I think this is going too far and puts undue burden on the PyPA >> team. Remember the end goal is simply to get an installer in the hands of >> users easier. >> > > I would also be fine with a solution where "import pip" issues a warning > about API instability if sys.argv[0] indicates the main executable is > something other than the pip CLI. > Oops, meant to add a point to https://github.com/pypa/pip/issues/1052 for that one. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Jul 14 08:43:35 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 14 Jul 2013 02:43:35 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> Message-ID: On Jul 14, 2013, at 1:58 AM, Nick Coghlan wrote: > Either the existing APIs are moved to a different name, or they get declared stable and pip switches to "internally forked" APIs any time a backwards incompatible change is needed for refactoring purposes (see runpy._run_module_as_main for an example of needing to do this in the standard library). I've had to directly deal with too many issues arising from getting this wrong in the past for me to endorse bundling of a module that doesn't follow this practice with CPython - if introspection indicates an API is public, then it's public and subject to all standard library backwards compatibility guarantees, or else we take the pain *once* and explicitly mark it private by adding a leading underscore rather than leaving it in limbo (contextlib._GeneratorContextManager is a standard library example of the latter approach - it used to lack the leading underscore, suggesting it was a public API when it's really just an implementation detail of contextlib.contextmanager). > > Mere documentation of public vs private generally doesn't cut it, as too many people use dir(), help() and inspect() rather than the published docs to explore APIs. The only general exception I'm aware of is "test" packages, including the standard library's test package, and for those you can make the case that having "test" or "tests" as a name segment is just as clear an indicator of something being private as at least one name segment starting with a leading underscore. > > I really this is a fairly big ask for the pip maintainers, but I *don't* consider "Oh, don't use our module API, it isn't stable" to be an adequate answer for something that is bundled with the standard installers. Beyond that, I don't mind if the answer is to declare the 1.5 API stable or to sprinkle underscore where appropriate or moving everything to a private package - the documentation and the naming conventions just need to be consistent in their private vs public distinctions (although your points do suggest heavily that the right answer is to accept the burden of backwards compatibility for all APIs currently marked public, and move towards the introduction of appropriate private APIs over time through refactoring). I agree with Noah here. In my eyes pip is either an external project with it's own polices on backwards compatibility and governance or it's part of the standard library and under the domain of Python core. I'm completely against moving it into the standard library for all the reasons I've given in the past. Maybe I'm reading too much into this but one of my primary fears here is that including pip with the Python distribution is going to lead to Python core dictating to pip what pip must do with itself. Now I don't mean to say that Python core should have no sway over pip either as they are officially blessing it and including it as part of the official releases. If I am reading too much into it then I apologize. I just want to make sure that the boundaries between the governance of Python and pip are clearly defined and the expectations on both sides are laid out and agreed upon before it happens. And I think this raises a good point about how the two projects are going to interact. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ben at bendarnell.com Sat Jul 13 15:14:59 2013 From: ben at bendarnell.com (Ben Darnell) Date: Sat, 13 Jul 2013 09:14:59 -0400 Subject: [Distutils] Best practices for optional C extensions Message-ID: I'd like to add a C extension to speed up a small bit of code in a package (Tornado), but make it optional both for compatibility with non-cpython implementations and for ease of installation on systems without a C compiler available. Ideally any user who runs "pip install tornado" on a system capable of compiling extensions would get the extensions; if this capability cannot be detected automatically I'd prefer the opt-out case to be the one that requires non-default arguments. Are there any packages that provide a good example to follow for this? PEP 426 uses "c-accelerators" as an example of an "extra", but it's unclear how this would work (based on the equivalent setuptools feature). There doesn't appear to be a way to know what extras are requested at build time. If the extra required a package like cython then you could build the extension whenever that package is present, but what about hand-written extensions? Extras are also opt-in instead of opt-out, so I'd have to recommend that most people use "pip install tornado[fast]" instead of "pip install tornado" (with "tornado[slow]" available as an option for limited environments). Thanks, -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at bendarnell.com Sat Jul 13 18:29:20 2013 From: ben at bendarnell.com (Ben Darnell) Date: Sat, 13 Jul 2013 12:29:20 -0400 Subject: [Distutils] Best practices for optional C extensions Message-ID: I'd like to add a C extension to speed up a small bit of code in a package (Tornado), but make it optional both for compatibility with non-cpython implementations and for ease of installation on systems without a C compiler available. Ideally any user who runs "pip install tornado" on a system capable of compiling extensions would get the extensions; if this capability cannot be detected automatically I'd prefer the opt-out case to be the one that requires non-default arguments. Are there any packages that provide a good example to follow for this? PEP 426 uses "c-accelerators" as an example of an "extra", but it's unclear how this would work (based on the equivalent setuptools feature). There doesn't appear to be a way to know what extras are requested at build time. If the extra required a package like cython then you could build the extension whenever that package is present, but what about hand-written extensions? Extras are also opt-in instead of opt-out, so I'd have to recommend that most people use "pip install tornado[fast]" instead of "pip install tornado" (with "tornado[slow]" available as an option for limited environments). Thanks, -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jul 14 09:01:58 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jul 2013 17:01:58 +1000 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> Message-ID: On 14 July 2013 16:43, Donald Stufft wrote: > I just want to make sure that the boundaries between the governance of > Python and pip are clearly defined and the expectations on both sides are > laid out and agreed upon before it happens. And I think this raises a good > point about how the two projects are going to interact. > Agreed, I think the boundaries need to be clear. If something installed by default is *only* support code for a bundled application, then it should either adhere to the standard library's backwards compatibility policies (by appropriately marking private APIs as private), or else it should issue a warning when imported by any other application. Either of those options sounds good to me. However, I consider expecting people to "just know" (or to look at documentation to determine) which provided modules are public or private without adhering to standard naming conventions or providing an explicit runtime warning to be unreasonable. (and yes, if "pip" goes down the runtime warning path, we should probably look into providing a runtime warning for at least the "test" namespace and possibly even the "idlelib" namespace, too) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Jul 14 09:13:50 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 14 Jul 2013 03:13:50 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> Message-ID: <09947DB4-FD58-469B-85DA-971B05B18799@stufft.io> On Jul 14, 2013, at 3:01 AM, Nick Coghlan wrote: > On 14 July 2013 16:43, Donald Stufft wrote: > I just want to make sure that the boundaries between the governance of Python and pip are clearly defined and the expectations on both sides are laid out and agreed upon before it happens. And I think this raises a good point about how the two projects are going to interact. > > Agreed, I think the boundaries need to be clear. If something installed by default is *only* support code for a bundled application, then it should either adhere to the standard library's backwards compatibility policies (by appropriately marking private APIs as private), or else it should issue a warning when imported by any other application. Either of those options sounds good to me. > > However, I consider expecting people to "just know" (or to look at documentation to determine) which provided modules are public or private without adhering to standard naming conventions or providing an explicit runtime warning to be unreasonable. > > (and yes, if "pip" goes down the runtime warning path, we should probably look into providing a runtime warning for at least the "test" namespace and possibly even the "idlelib" namespace, too) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia Yea I forget to talk about the *actual* change that prompted that email when I started feeling dictated to which touched upon one of my fears in this process :) I'm not against either renaming or emitting a warning. I was actually asking if just documenting the fact would be ok because I fear bugs from the code churn that renaming would cause :) I think we'd need to rename things because emitting a warning is an all or nothing ordeal and we've had requests to make certain parts of the API public for Chef and other tools like it. A question that certainly raises in my mind though is "standard library's backwards compatibility policies". What affect does this have on *actual* public API exposed from pip? Does it mean we cannot break compatibility for them until Python 4.x? That sounds very onerous for something that is installed in a way that allows easy upgrade and downgrading separately from Python to match the version requirements of someone using that library. Pip has it's own versions and develops at it's own speed. I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's "None") using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Sun Jul 14 09:35:34 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jul 2013 17:35:34 +1000 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: <09947DB4-FD58-469B-85DA-971B05B18799@stufft.io> References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> <09947DB4-FD58-469B-85DA-971B05B18799@stufft.io> Message-ID: On 14 July 2013 17:13, Donald Stufft wrote: > I think it would be reasonable for the pip maintainers to be asked to > declare a public API (even if that's "None") using the naming scheme or an > import warning and declare a backwards compatibility policy for pip itself > so that people can know what to expect from pip. I do not however, believe > it is reasonable to bind pip to the same policy that CPython uses nor the > same schedule. (If you weren't suggesting that I apologize). > The main elements of CPython's backwards compatibility policy that I consider relevant are: * Use leading underscores to denote private APIs with no backwards compatibility guarantees * Be conservative with deprecating public APIs that aren't fundamentally broken * Use DeprecationWarning to give at least one (pip) release notice of an upcoming backwards incompatible change We *are* sometimes quite aggressive with deprecation and removal even in the standard library - we removed contextlib.nested from Python 3.2 as a problematic bug magnet well before I came up with the contextlib.ExitStack API as a less error prone replacement in Python 3.3. It's only when it comes to core syntax and builtin behaviour that we're likely to hit issues that simply don't have a sensible deprecation strategy, so we decide we have to live with them indefinitely. That said, I think the answer to this discussion also affects the answer to whether or not CPython maintenance releases should update to newer versions of pip: if pip chooses to adopt a faster deprecation cycle than CPython, then our maintenance releases shouldn't bundle updated versions. Instead, they should follow the policy: * if this is a new major release, or the first maintenance release to bundle pip, bundle the latest available version of pip * otherwise, bundle the same version of pip as the previous release This would mean we'd be asking the pip team to help out by providing security releases for the bundled version, so we can get that without breaking the public API that's available by default. On the other hand, if the pip team are willing to use long deprecation cycles then we can just bundle the updated versions and not worry about security releases (I'd prefer that, but it only works if the pip team are willing to put up with keeping old APIs around for a couple of years before killing them off once the affected CPython branches go into security fix only mode). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Jul 14 09:50:52 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 14 Jul 2013 03:50:52 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> <09947DB4-FD58-469B-85DA-971B05B18799@stufft.io> Message-ID: I do think pip is pretty conservative about backwards compat other than the security related changes I've been doing. I think we can find the middle ground that lets things work smoothly here :). I was just making sure that we wernt going to have to keep things around for really long times like python 4 ;) On Jul 14, 2013, at 3:35 AM, Nick Coghlan wrote: > On 14 July 2013 17:13, Donald Stufft wrote: >> I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's "None") using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize). > > The main elements of CPython's backwards compatibility policy that I consider relevant are: > > * Use leading underscores to denote private APIs with no backwards compatibility guarantees > * Be conservative with deprecating public APIs that aren't fundamentally broken > * Use DeprecationWarning to give at least one (pip) release notice of an upcoming backwards incompatible change > > We *are* sometimes quite aggressive with deprecation and removal even in the standard library - we removed contextlib.nested from Python 3.2 as a problematic bug magnet well before I came up with the contextlib.ExitStack API as a less error prone replacement in Python 3.3. It's only when it comes to core syntax and builtin behaviour that we're likely to hit issues that simply don't have a sensible deprecation strategy, so we decide we have to live with them indefinitely. > > That said, I think the answer to this discussion also affects the answer to whether or not CPython maintenance releases should update to newer versions of pip: if pip chooses to adopt a faster deprecation cycle than CPython, then our maintenance releases shouldn't bundle updated versions. Instead, they should follow the policy: > > * if this is a new major release, or the first maintenance release to bundle pip, bundle the latest available version of pip > * otherwise, bundle the same version of pip as the previous release > > This would mean we'd be asking the pip team to help out by providing security releases for the bundled version, so we can get that without breaking the public API that's available by default. > > On the other hand, if the pip team are willing to use long deprecation cycles then we can just bundle the updated versions and not worry about security releases (I'd prefer that, but it only works if the pip team are willing to put up with keeping old APIs around for a couple of years before killing them off once the affected CPython branches go into security fix only mode). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Jul 14 09:55:12 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 14 Jul 2013 08:55:12 +0100 Subject: [Distutils] Best practices for optional C extensions In-Reply-To: References: Message-ID: On 13 July 2013 14:14, Ben Darnell wrote: > I'd like to add a C extension to speed up a small bit of code in a package > (Tornado), but make it optional both for compatibility with non-cpython > implementations and for ease of installation on systems without a C > compiler available. Ideally any user who runs "pip install tornado" on a > system capable of compiling extensions would get the extensions; if this > capability cannot be detected automatically I'd prefer the opt-out case to > be the one that requires non-default arguments. Are there any packages > that provide a good example to follow for this? I believe that coverage has an optional C extension like this. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jul 14 10:12:22 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jul 2013 18:12:22 +1000 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> <09947DB4-FD58-469B-85DA-971B05B18799@stufft.io> Message-ID: On 14 July 2013 17:50, Donald Stufft wrote: > I do think pip is pretty conservative about backwards compat other than > the security related changes I've been doing. > > I think we can find the middle ground that lets things work smoothly here > :). I was just making sure that we wernt going to have to keep things > around for really long times like python 4 ;) > Even most of the standard library isn't that conservative - it usually only happens when we can't find a sensible place to hook up DeprecationWarning. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah at coderanger.net Sun Jul 14 10:23:36 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sun, 14 Jul 2013 01:23:36 -0700 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> <09947DB4-FD58-469B-85DA-971B05B18799@stufft.io> Message-ID: <6E6C1AA7-365E-4FD9-8213-A1E809473B74@coderanger.net> On Jul 14, 2013, at 12:35 AM, Nick Coghlan wrote: > On 14 July 2013 17:13, Donald Stufft wrote: > I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's "None") using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize). > > The main elements of CPython's backwards compatibility policy that I consider relevant are: > > * Use leading underscores to denote private APIs with no backwards compatibility guarantees > * Be conservative with deprecating public APIs that aren't fundamentally broken > * Use DeprecationWarning to give at least one (pip) release notice of an upcoming backwards incompatible change > > We *are* sometimes quite aggressive with deprecation and removal even in the standard library - we removed contextlib.nested from Python 3.2 as a problematic bug magnet well before I came up with the contextlib.ExitStack API as a less error prone replacement in Python 3.3. It's only when it comes to core syntax and builtin behaviour that we're likely to hit issues that simply don't have a sensible deprecation strategy, so we decide we have to live with them indefinitely. > > That said, I think the answer to this discussion also affects the answer to whether or not CPython maintenance releases should update to newer versions of pip: if pip chooses to adopt a faster deprecation cycle than CPython, then our maintenance releases shouldn't bundle updated versions. Instead, they should follow the policy: > > * if this is a new major release, or the first maintenance release to bundle pip, bundle the latest available version of pip > * otherwise, bundle the same version of pip as the previous release > > This would mean we'd be asking the pip team to help out by providing security releases for the bundled version, so we can get that without breaking the public API that's available by default. > > On the other hand, if the pip team are willing to use long deprecation cycles then we can just bundle the updated versions and not worry about security releases (I'd prefer that, but it only works if the pip team are willing to put up with keeping old APIs around for a couple of years before killing them off once the affected CPython branches go into security fix only mode). If I can surmise your worry here, it is that people will open an interactive terminal, import pip, reflect out the classes/methods/etc, see that despite being mentioned no-where in the Python or pip documentation the methods and classes don't start with an underscore, and thus conclude that this is a stable API to build against? I agree that conventions are good, but I have to say this sounds like a bit of a stretch and certainly anyone complaining that their undocumented API that they only found via reflection (or reading the pip source) was broken basically gets what they deserve. The point I was trying to make is that a major shift in thinking is needed here. pip is not part of CPython, regardless of this bundling neither this mailing list nor the CPython team will have any control (aside from the nuclear option that the CPython team can elect to stop bundling pip). If you think it would be good for the code-health of pip to be clearer about what their public API is, I will support that all the way and in fact have an open ticket against pip to that effect already, but that is something for the pip team to decide. This does very much mean that the CPython team is not just backing the pip codebase, but the PyPA/pip team. I think the past few years have shown them deserving of this trust, and they should be allowed to run things as they see fit. These lines get blurry since several people move back and forth between CPython and PyPA (and distutils and PyPI, etc) hats, so I think this must be stated clearly up front that what the CPython team thinks is "reasonable" for an API policy will be nothing more than a recommendation from very knowledgable colleagues and will be given the appropriate consideration and respect it deserves based on that. Hopefully that makes my point-of-view a little clearer. --Noah From ncoghlan at gmail.com Sun Jul 14 12:41:31 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jul 2013 20:41:31 +1000 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: <6E6C1AA7-365E-4FD9-8213-A1E809473B74@coderanger.net> References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> <09947DB4-FD58-469B-85DA-971B05B18799@stufft.io> <6E6C1AA7-365E-4FD9-8213-A1E809473B74@coderanger.net> Message-ID: On 14 Jul 2013 18:24, "Noah Kantrowitz" wrote: > > > On Jul 14, 2013, at 12:35 AM, Nick Coghlan wrote: > > > On 14 July 2013 17:13, Donald Stufft wrote: > > I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's "None") using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize). > > > > The main elements of CPython's backwards compatibility policy that I consider relevant are: > > > > * Use leading underscores to denote private APIs with no backwards compatibility guarantees > > * Be conservative with deprecating public APIs that aren't fundamentally broken > > * Use DeprecationWarning to give at least one (pip) release notice of an upcoming backwards incompatible change > > > > We *are* sometimes quite aggressive with deprecation and removal even in the standard library - we removed contextlib.nested from Python 3.2 as a problematic bug magnet well before I came up with the contextlib.ExitStack API as a less error prone replacement in Python 3.3. It's only when it comes to core syntax and builtin behaviour that we're likely to hit issues that simply don't have a sensible deprecation strategy, so we decide we have to live with them indefinitely. > > > > That said, I think the answer to this discussion also affects the answer to whether or not CPython maintenance releases should update to newer versions of pip: if pip chooses to adopt a faster deprecation cycle than CPython, then our maintenance releases shouldn't bundle updated versions. Instead, they should follow the policy: > > > > * if this is a new major release, or the first maintenance release to bundle pip, bundle the latest available version of pip > > * otherwise, bundle the same version of pip as the previous release > > > > This would mean we'd be asking the pip team to help out by providing security releases for the bundled version, so we can get that without breaking the public API that's available by default. > > > > On the other hand, if the pip team are willing to use long deprecation cycles then we can just bundle the updated versions and not worry about security releases (I'd prefer that, but it only works if the pip team are willing to put up with keeping old APIs around for a couple of years before killing them off once the affected CPython branches go into security fix only mode). > > If I can surmise your worry here, it is that people will open an interactive terminal, import pip, reflect out the classes/methods/etc, see that despite being mentioned no-where in the Python or pip documentation the methods and classes don't start with an underscore, and thus conclude that this is a stable API to build against? I agree that conventions are good, but I have to say this sounds like a bit of a stretch and certainly anyone complaining that their undocumented API that they only found via reflection (or reading the pip source) was broken basically gets what they deserve. The point I was trying to make is that a major shift in thinking is needed here. pip is not part of CPython, regardless of this bundling neither this mailing list nor the CPython team will have any control (aside from the nuclear option that the CPython team can elect to stop bundling pip). If you think it would be good for the code-health of pip to be clearer about what their public API is, I will suppor > t that all the way and in fact have an open ticket against pip to that effect already, but that is something for the pip team to decide. This does very much mean that the CPython team is not just backing the pip codebase, but the PyPA/pip team. I think the past few years have shown them deserving of this trust, and they should be allowed to run things as they see fit. These lines get blurry since several people move back and forth between CPython and PyPA (and distutils and PyPI, etc) hats, so I think this must be stated clearly up front that what the CPython team thinks is "reasonable" for an API policy will be nothing more than a recommendation from very knowledgable colleagues and will be given the appropriate consideration and respect it deserves based on that. Hopefully that makes my point-of-view a little clearer. I started a thread on python-dev proposing strengthened wording in PEP 8 regarding marking of private interfaces, but beyond that, yes, I now agree that this isn't a blocker for bundling pip with CPython. Cheers, Nick. > > --Noah > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Jul 14 13:09:51 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 14 Jul 2013 12:09:51 +0100 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs Message-ID: I don't think there is any doubt that we need to document to at least some extent how to use pip, in the Python documentation for 3.4. The obvious place is "Installing Python Modules" as a chapter to itself. What is less clear to me is how much to document - just basic "pip install XXX", or the whole pip command set, or somewhere in between? Should there be some type of "refer to www.pip-installer.org for the complete documentation" reference? Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jul 14 13:42:32 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jul 2013 21:42:32 +1000 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: On 14 July 2013 21:09, Paul Moore wrote: > I don't think there is any doubt that we need to document to at least some > extent how to use pip, in the Python documentation for 3.4. The obvious > place is "Installing Python Modules" as a chapter to itself. > > What is less clear to me is how much to document - just basic "pip install > XXX", or the whole pip command set, or somewhere in between? Should there > be some type of "refer to www.pip-installer.org for the complete > documentation" reference? > This is the hole https://python-packaging-user-guide.readthedocs.org/en/latest/ is supposed to fill - once it's "ready" (i.e. things have stabilised sufficiently , then I'd like to replace the "Installing Python Modules" and "Distributing Python Modules" sections for 2.7 and 3.3 with some *very* abbreviated quick start guides that then reference that site. The 3.3 changes would then carry over into 3.4. I spent some time at PyCon AU talking to Matthew Iverson ( https://bitbucket.org/Ivoz/python-packaging-user-guide) who I believe Marcus was hoping to get back to that after pip 1.4 was out the door, but anyone on the PyPA list on BitBucket actually has full access to accept pull requests, etc. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Jul 14 15:01:50 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 14 Jul 2013 14:01:50 +0100 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: On 14 July 2013 12:42, Nick Coghlan wrote: > This is the hole > https://python-packaging-user-guide.readthedocs.org/en/latest/ is > supposed to fill - once it's "ready" (i.e. things have stabilised > sufficiently , then I'd like to replace the "Installing Python Modules" and > "Distributing Python Modules" sections for 2.7 and 3.3 with some *very* > abbreviated quick start guides that then reference that site. The 3.3 > changes would then carry over into 3.4. > Hmm, OK. I've no problem with that (although I do find the packaging guide pretty hard to get into for an end user who only wants to *use* packages, not *create* them, but that's a separate issue for me to address by providing some pull requests). I was more thinking in terms of your quick start guides. I think we should explain *in the core documentation* how to (a) install a new package, (b) uninstall a package, (c) list what is installed and (d) upgrade pip itself. That translates to the pip install, uninstall, and list commands at a minimum. I could offer some text, if that's the way you want to go with this. How about if I provide a new (short) document called something like "Python package management" and we work out how to integrate it into the docs as things settle down? Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Jul 14 15:29:22 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 14 Jul 2013 14:29:22 +0100 Subject: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping)) Message-ID: On 13 July 2013 10:05, Paul Moore wrote: > How robust is the process of upgrading pip using itself? Specifically on > Windows, where these things typically seem less reliable. OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it. The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but "a little bit broken" is still broken :-( On the other hand, "python -m pip install -U pip" works fine because it avoids the exe wrappers. There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend "don't do that" - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython. Does anyone have any suggestions? Paul. PS In better news, apart from this issue, pip upgrades of pip and setuptools seem fine. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jul 14 15:43:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jul 2013 23:43:46 +1000 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: On 14 July 2013 23:01, Paul Moore wrote: > > On 14 July 2013 12:42, Nick Coghlan wrote: > >> This is the hole >> https://python-packaging-user-guide.readthedocs.org/en/latest/ is >> supposed to fill - once it's "ready" (i.e. things have stabilised >> sufficiently , then I'd like to replace the "Installing Python Modules" and >> "Distributing Python Modules" sections for 2.7 and 3.3 with some *very* >> abbreviated quick start guides that then reference that site. The 3.3 >> changes would then carry over into 3.4. >> > > Hmm, OK. I've no problem with that (although I do find the packaging guide > pretty hard to get into for an end user who only wants to *use* packages, > not *create* them, but that's a separate issue for me to address by > providing some pull requests). I was more thinking in terms of your quick > start guides. I think we should explain *in the core documentation* how to > (a) install a new package, (b) uninstall a package, (c) list what is > installed and (d) upgrade pip itself. That translates to the pip install, > uninstall, and list commands at a minimum. > > I could offer some text, if that's the way you want to go with this. How > about if I provide a new (short) document called something like "Python > package management" and we work out how to integrate it into the docs as > things settle down? > That sounds great - so far it's mostly just been myself and Marcus thinking about it (mostly Marcus, to be honest, along with a couple of folks that submitted pull requests and BitBucket issues), and it keeps getting bumped down the todo list by other things. I think we're getting closer to having something stable enough to document clearly, though - with distribute merged back into setuptools and pip not far away, the bootstrapping seems to be the only remaining slightly messy part (since the 3.4 discussions aren't relevant to the user guide as yet). As far as your first point goes, I agree the "Installation Tutorial" part should probably come first and definitely needs more content. I did just accept a pull request earlier that at least makes that page more than just a list of headings (see https://python-packaging-user-guide.readthedocs.org/en/latest/installation_tutorial.html- courtesy of https://bitbucket.org/alexjeffburke) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Jul 14 16:25:25 2013 From: brett at python.org (Brett Cannon) Date: Sun, 14 Jul 2013 10:25:25 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: <6E6C1AA7-365E-4FD9-8213-A1E809473B74@coderanger.net> References: <29FDBDCE-EC77-40BD-823B-A291C5C4EE49@stufft.io> <09947DB4-FD58-469B-85DA-971B05B18799@stufft.io> <6E6C1AA7-365E-4FD9-8213-A1E809473B74@coderanger.net> Message-ID: On Sun, Jul 14, 2013 at 4:23 AM, Noah Kantrowitz wrote: > > On Jul 14, 2013, at 12:35 AM, Nick Coghlan wrote: > > > On 14 July 2013 17:13, Donald Stufft wrote: > > I think it would be reasonable for the pip maintainers to be asked to > declare a public API (even if that's "None") using the naming scheme or an > import warning and declare a backwards compatibility policy for pip itself > so that people can know what to expect from pip. I do not however, believe > it is reasonable to bind pip to the same policy that CPython uses nor the > same schedule. (If you weren't suggesting that I apologize). > > > > The main elements of CPython's backwards compatibility policy that I > consider relevant are: > > > > * Use leading underscores to denote private APIs with no backwards > compatibility guarantees > > * Be conservative with deprecating public APIs that aren't fundamentally > broken > > * Use DeprecationWarning to give at least one (pip) release notice of an > upcoming backwards incompatible change > > > > We *are* sometimes quite aggressive with deprecation and removal even in > the standard library - we removed contextlib.nested from Python 3.2 as a > problematic bug magnet well before I came up with the contextlib.ExitStack > API as a less error prone replacement in Python 3.3. It's only when it > comes to core syntax and builtin behaviour that we're likely to hit issues > that simply don't have a sensible deprecation strategy, so we decide we > have to live with them indefinitely. > > > > That said, I think the answer to this discussion also affects the answer > to whether or not CPython maintenance releases should update to newer > versions of pip: if pip chooses to adopt a faster deprecation cycle than > CPython, then our maintenance releases shouldn't bundle updated versions. > Instead, they should follow the policy: > > > > * if this is a new major release, or the first maintenance release to > bundle pip, bundle the latest available version of pip > > * otherwise, bundle the same version of pip as the previous release > > > > This would mean we'd be asking the pip team to help out by providing > security releases for the bundled version, so we can get that without > breaking the public API that's available by default. > > > > On the other hand, if the pip team are willing to use long deprecation > cycles then we can just bundle the updated versions and not worry about > security releases (I'd prefer that, but it only works if the pip team are > willing to put up with keeping old APIs around for a couple of years before > killing them off once the affected CPython branches go into security fix > only mode). > > If I can surmise your worry here, it is that people will open an > interactive terminal, import pip, reflect out the classes/methods/etc, see > that despite being mentioned no-where in the Python or pip documentation > the methods and classes don't start with an underscore, and thus conclude > that this is a stable API to build against? Yes, and to make that statement even stronger: all of that happening with a freshly installed copy of Python with no external packages installed. Nick's worry stems from experience (which I have also had) where people simply don't check docs as to whether something is public in the stdlib and so it ends up being considered such by users to the point that we feel obliged to support it. > I agree that conventions are good, but I have to say this sounds like a > bit of a stretch and certainly anyone complaining that their undocumented > API that they only found via reflection (or reading the pip source) was > broken basically gets what they deserve. That's what we typically say for older modules, especially when we are cranky. =) But it doesn't stop the wingeing and bug reports. Luckily I think we have gotten better about this. > The point I was trying to make is that a major shift in thinking is needed > here. pip is not part of CPython, regardless of this bundling neither this > mailing list nor the CPython team will have any control (aside from the > nuclear option that the CPython team can elect to stop bundling pip). If > you think it would be good for the code-health of pip to be clearer about > what their public API is, I will suppor > t that all the way and in fact have an open ticket against pip to that > effect already, but that is something for the pip team to decide. This does > very much mean that the CPython team is not just backing the pip codebase, > but the PyPA/pip team. I think the past few years have shown them deserving > of this trust, and they should be allowed to run things as they see fit. > These lines get blurry since several people move back and forth between > CPython and PyPA (and distutils and PyPI, etc) hats, so I think this must > be stated clearly up front that what the CPython team thinks is > "reasonable" for an API policy will be nothing more than a recommendation > from very knowledgable colleagues and will be given the appropriate > consideration and respect it deserves based on that. Hopefully that makes > my point-of-view a little clearer. > I think it's all going to come down to messaging. It will have to be yelled form the top of every mountain that pip is being bundled with Python as a convenience to the community, but that it is **NOT** part of the (C)Python project and thus has it's own development process, issue tracker, etc. If Python bundles pip then python-dev will make promises about what versions we include in bugfix releases, how it's bundled, etc., but otherwise it's a separate project with its own rules. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Jul 14 16:28:54 2013 From: brett at python.org (Brett Cannon) Date: Sun, 14 Jul 2013 10:28:54 -0400 Subject: [Distutils] Call for PEP author/champion: Bundling pip with CPython installers In-Reply-To: References: Message-ID: On Sun, Jul 14, 2013 at 2:13 AM, Nick Coghlan wrote: > Based on the recent discussions, I now plan to reject the pip > bootstrapping-on-first-invocation approach described in PEP 439 in favour > of a new PEP that proposes: > [SNIP] > * ensuring that, for Python 3.4, "python3" and "python3.4" are available > for command line invocation of Python, even on Windows > Can I ask why this is part of the PEP? -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Sun Jul 14 16:31:22 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 14 Jul 2013 10:31:22 -0400 Subject: [Distutils] Best practices for optional C extensions In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/13/2013 09:14 AM, Ben Darnell wrote: > I'd like to add a C extension to speed up a small bit of code in a > package (Tornado), but make it optional both for compatibility with > non-cpython implementations and for ease of installation on systems > without a C compiler available. Ideally any user who runs "pip > install tornado" on a system capable of compiling extensions would get > the extensions; if this capability cannot be detected automatically > I'd prefer the opt-out case to be the one that requires non-default > arguments. Are there any packages that provide a good example to > follow for this? > > PEP 426 uses "c-accelerators" as an example of an "extra", but it's > unclear how this would work (based on the equivalent setuptools > feature). There doesn't appear to be a way to know what extras are > requested at build time. If the extra required a package like cython > then you could build the extension whenever that package is present, > but what about hand-written extensions? Extras are also opt-in > instead of opt-out, so I'd have to recommend that most people use "pip > install tornado[fast]" instead of "pip install tornado" (with > "tornado[slow]" available as an option for limited environments). zope.interface subclasses the 'build_ext' command so: - ---------------------------- %< ----------------------------- from distutils.command.build_ext import build_ext from distutils.errors import CCompilerError from distutils.errors import DistutilsExecError from distutils.errors import DistutilsPlatformError class optional_build_ext(build_ext): """Allow the building of C extensions to fail. """ def run(self): try: build_ext.run(self) except DistutilsPlatformError as e: self._unavailable(e) def build_extension(self, ext): try: build_ext.build_extension(self, ext) except (CCompilerError, DistutilsExecError) as e: self._unavailable(e) def _unavailable(self, e): print('*' * 80) print("""WARNING: An optional code optimization (C extension) could not be compiled. Optimizations for this package will not be available!""") print() print(e) print('*' * 80) - ---------------------------- %< ----------------------------- Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHitjoACgkQ+gerLs4ltQ7SJQCgrhgN58g9ztFPEkFAOM49Wu4p RpQAoLnboKietjTx3eXho1kyRvH3r2uN =qWRK -----END PGP SIGNATURE----- From ncoghlan at gmail.com Sun Jul 14 16:33:45 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 00:33:45 +1000 Subject: [Distutils] Call for PEP author/champion: Bundling pip with CPython installers In-Reply-To: References: Message-ID: On 15 July 2013 00:28, Brett Cannon wrote: > > > > On Sun, Jul 14, 2013 at 2:13 AM, Nick Coghlan wrote: >> >> Based on the recent discussions, I now plan to reject the pip >> bootstrapping-on-first-invocation approach described in PEP 439 in favour of >> a new PEP that proposes: >> [SNIP] >> * ensuring that, for Python 3.4, "python3" and "python3.4" are available >> for command line invocation of Python, even on Windows > > > Can I ask why this is part of the PEP? Mostly because "pip3" makes no sense without it. If we *don't* bring Windows into conformance with the way other platforms handle parallel Python 2 and Python 3 installs, then the PEP should document an explicit rationale for not doing it (the py launcher doesn't count in that regard, since the real purpose of that is to handle shebang lines through file associations, with the direct command line usage as an added bonus). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From setuptools at bugs.python.org Sun Jul 14 17:10:31 2013 From: setuptools at bugs.python.org (richard) Date: Sun, 14 Jul 2013 15:10:31 +0000 Subject: [Distutils] [issue154] file protection group/world writeable on egg-info files since 0.7x Message-ID: <1373814631.49.0.577934070272.issue154@psf.upfronthosting.co.za> New submission from richard: since 0.7.x, there are some issues in the tarball setuptools.egg-info directory...namely, 1. the file protection of the files is group/world writeable (should be group/world read-only) 2. there are additional .orig files 3. there is an additional directory EGG-INFO containing an obsolete copy of PKG-INFO. This should be trivial to fix in the source tree. extract below from the 0.8 tarball: drwxrwxrwx 0/0 0 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/ -rw-rw-rw- 0/0 654 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/dependency_links.txt drwxrwxrwx 0/0 0 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/EGG-INFO/ -rw-rw-rw- 0/0 153 2013-07-02 17:30 setuptools-0.8/setuptools.egg-info/EGG-INFO/PKG-INFO -rw-rw-rw- 0/0 2773 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/entry_points.txt -rw-rw-rw- 0/0 2773 2013-07-03 14:12 setuptools-0.8/setuptools.egg-info/entry_points.txt.orig -rw-rw-rw- 0/0 45823 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/PKG-INFO -rw-rw-rw- 0/0 186 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/requires.txt -rw-rw-rw- 0/0 186 2013-07-03 14:12 setuptools-0.8/setuptools.egg-info/requires.txt.orig -rw-rw-rw- 0/0 3663 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/SOURCES.txt -rw-rw-rw- 0/0 49 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/top_level.txt -rw-rw-rw- 0/0 2 2013-07-02 17:48 setuptools-0.8/setuptools.egg-info/zip-safe ---------- messages: 737 nosy: richard priority: bug status: unread title: file protection group/world writeable on egg-info files since 0.7x _______________________________________________ Setuptools tracker _______________________________________________ From pnasrat at gmail.com Sun Jul 14 17:10:28 2013 From: pnasrat at gmail.com (Paul Nasrat) Date: Sun, 14 Jul 2013 11:10:28 -0400 Subject: [Distutils] Call for PEP author/champion: Bundling pip with CPython installers In-Reply-To: References: Message-ID: On 14 July 2013 02:13, Nick Coghlan wrote: > Based on the recent discussions, I now plan to reject the pip > bootstrapping-on-first-invocation approach described in PEP 439 in favour > of a new PEP that proposes: > > * bundling the latest version of pip with the CPython binary installers > for Mac OS X and Windows for all future CPython releases (including > maintenance releases) > * aligns the proposal with the Python 3.4 release schedule by noting that > CPython changes must be completed by the first 3.4 beta, while pip changes > must be completed by the first 3.4 release candidate. > * ensuring that, for Python 3.4, "python3" and "python3.4" are available > for command line invocation of Python, even on Windows > * ensuring that the bundled pip, for Python 3.4, ensures "pip", "pip3" and > "pip3.4" are available for command line invocation of Python, even on > Windows > * ensuring that the bundled pip is able to upgrade/downgrade itself > independent of the CPython release cycle > * ensuring that pip is automatically available in virtual environments > created with pyvenv > * adding a "get-pip.py" script to Tools/scripts in the CPython repo for > bootstrapping the latest pip release from PyPI into custom CPython source > builds > > Note that there are still open questions to be resolved, which is why an > author/champion is needed: > I've a bunch of contacts in various distros still - I've not championed a PEP before but I would be happy to take this on. > * what guidance, if any, should we provide to Linux distro packagers? > * how should maintenance updates handle the presence of an existing pip > installation? > There are various distro packaging specific ways of handling this. Hard requirements, recommends, obsoleting the standalone package and providing it virtually as part of Automatically upgrade older versions to the bundled version, while leaving > newer versions alone? Force installation of the bundled version? > My personal experience is that forcing the bundled version to OS with strong in-built packaging (Linux, BSD, other *NIX) is likely to meet with some resistance. I can certainly talk with some people, my instinct is it's likely to be only bundle with installers, allow optional install as part of the cPython build which can then be subpackaged/ignored for seperate pip/bundled as distros so desire. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Sun Jul 14 18:45:31 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sun, 14 Jul 2013 16:45:31 +0000 Subject: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping)) In-Reply-To: References: Message-ID: From: Paul Moore > On 13 July 2013 10:05, Paul Moore wrote: > How robust is the process of upgrading pip using itself? Specifically on > Windows, where these things typically seem less reliable. > > OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason > for the failure is simple enough to explain - the pip.exe wrapper is held open > by the OS while it's in use, so that the upgrade cannot replace it. > > The result is a failed upgrade and a partially installed new version of pip. In > practice, the exe stubs are probably added fairly late in the install (at least > when installing from sdist, with a wheel that depends on the order of the files > in the wheel), so it's probably only a little bit broken, but "a little bit > broken" is still broken :-( > > On the other hand, "python -m pip install -U pip" works fine because it avoids > the exe wrappers. > > There's a lot of scope for user confusion and frustration in all this. For > standalone pip I've tended to recommend "don't do that" - manually uninstall and > reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. > That sort of advice isn't going to be realistic for a pip bundled with CPython. > > Does anyone have any suggestions? Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. Cheers, Steve From qwcode at gmail.com Sun Jul 14 19:02:20 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 14 Jul 2013 10:02:20 -0700 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: > Hmm, OK. I've no problem with that (although I do find the packaging guide > pretty hard to get into for an end user who only wants to *use* packages, > not *create* them > there's a section marked "Installation Tutorial". Someone wanting to install packages should be able to get into that, once it actually has content : ) > I think we should explain *in the core documentation* > How about if I provide a new (short) document called something like "Python > package management" > IMO, we should put the energy into one comprehensive document (the new user guide) and link out to it, not duplicate effort into the core docs. The "Installing Python Modules"/"Distributing Python Modules" documents should be integrated into the new User Guide. It will be confusing and anti-zen for a user to come upon those documents and have to reconcile it with the user guide Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Jul 14 19:03:21 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 14 Jul 2013 18:03:21 +0100 Subject: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping)) In-Reply-To: References: Message-ID: On 14 July 2013 17:45, Steve Dower wrote: > So unless the exe wrapper is changing with each version, I think the best > way of handling this is to not force them to be replaced when they have not > changed. Thanks. That sounds annoyingly complex (pretty much as I expected, though). My feeling is that I'd like to remove the exe wrapper altogether, and use a .py file. I need to check what issues there might be with that before recommending it, though. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Jul 14 19:06:57 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 14 Jul 2013 13:06:57 -0400 Subject: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping)) In-Reply-To: References: Message-ID: <6EACCA35-1901-4D60-9BFF-FCBF94480112@stufft.io> On Jul 14, 2013, at 1:03 PM, Paul Moore wrote: > On 14 July 2013 17:45, Steve Dower wrote: > So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. > > Thanks. That sounds annoyingly complex (pretty much as I expected, though). My feeling is that I'd like to remove the exe wrapper altogether, and use a .py file. I need to check what issues there might be with that before recommending it, though. > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Wouldn't a .py file make the command `pip.py`` and not ``pip`` ? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Sun Jul 14 19:11:21 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 14 Jul 2013 18:11:21 +0100 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: On 14 July 2013 18:02, Marcus Smith wrote: > there's a section marked "Installation Tutorial". Someone wanting to > install packages should be able to get into that, once it actually has > content : ) > Yes, I see that. I'm not sure I like the up-front "decide between user or system or virtualenv" presentation, though. I'm working on something I prefer, see what you think when it's done. My perspective is that someone who's used to writing Python code wants to install requests (and get on with his latest script). How does he do that? He doesn't want to be presented with decisions to make, he just wants to know how to do it. "pip install --user requests" is the answer he is looking for, frankly. (Missing out --user gets permission issues to deal with, and virtualenvs are more than he's looking to deal with at the moment). The guide should get this person to that command as quickly as possible. I think we should explain *in the core documentation* >> > How about if I provide a new (short) document called something like >> "Python package management" >> > > IMO, we should put the energy into one comprehensive document (the new > user guide) and link out to it, not duplicate effort into the core docs. > OK... But remember that most users are consumers of packages, not creators of them. The packaging guide should reflect that with "install and maintain packages" on page 1, and creating packages squarely in "advanced" usage. Is that in line with the goals of the packaging guide? (If not, let's just get a user-only starter page in the Python docs and leave the packaging guide as the "more comprehensive" documentation it can refer to). Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah at coderanger.net Sun Jul 14 19:12:28 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sun, 14 Jul 2013 10:12:28 -0700 Subject: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping)) In-Reply-To: References: Message-ID: <18A62404-30AE-41E6-A7BC-B7061DB0C4EE@coderanger.net> On Jul 14, 2013, at 9:45 AM, Steve Dower wrote: > From: Paul Moore >> On 13 July 2013 10:05, Paul Moore wrote: >> How robust is the process of upgrading pip using itself? Specifically on >> Windows, where these things typically seem less reliable. >> >> OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason >> for the failure is simple enough to explain - the pip.exe wrapper is held open >> by the OS while it's in use, so that the upgrade cannot replace it. >> >> The result is a failed upgrade and a partially installed new version of pip. In >> practice, the exe stubs are probably added fairly late in the install (at least >> when installing from sdist, with a wheel that depends on the order of the files >> in the wheel), so it's probably only a little bit broken, but "a little bit >> broken" is still broken :-( >> >> On the other hand, "python -m pip install -U pip" works fine because it avoids >> the exe wrappers. >> >> There's a lot of scope for user confusion and frustration in all this. For >> standalone pip I've tended to recommend "don't do that" - manually uninstall and >> reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. >> That sort of advice isn't going to be realistic for a pip bundled with CPython. >> >> Does anyone have any suggestions? > > Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? > > Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. > > MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. > > If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. > > So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done. --Noah From graffatcolmingov at gmail.com Sun Jul 14 19:31:20 2013 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Sun, 14 Jul 2013 13:31:20 -0400 Subject: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping)) In-Reply-To: <18A62404-30AE-41E6-A7BC-B7061DB0C4EE@coderanger.net> References: <18A62404-30AE-41E6-A7BC-B7061DB0C4EE@coderanger.net> Message-ID: On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz wrote: > > On Jul 14, 2013, at 9:45 AM, Steve Dower wrote: > >> From: Paul Moore >>> On 13 July 2013 10:05, Paul Moore wrote: >>> How robust is the process of upgrading pip using itself? Specifically on >>> Windows, where these things typically seem less reliable. >>> >>> OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason >>> for the failure is simple enough to explain - the pip.exe wrapper is held open >>> by the OS while it's in use, so that the upgrade cannot replace it. >>> >>> The result is a failed upgrade and a partially installed new version of pip. In >>> practice, the exe stubs are probably added fairly late in the install (at least >>> when installing from sdist, with a wheel that depends on the order of the files >>> in the wheel), so it's probably only a little bit broken, but "a little bit >>> broken" is still broken :-( >>> >>> On the other hand, "python -m pip install -U pip" works fine because it avoids >>> the exe wrappers. >>> >>> There's a lot of scope for user confusion and frustration in all this. For >>> standalone pip I've tended to recommend "don't do that" - manually uninstall and >>> reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. >>> That sort of advice isn't going to be realistic for a pip bundled with CPython. >>> >>> Does anyone have any suggestions? >> >> Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? >> >> Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. >> >> MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. >> >> If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. >> >> So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. > > The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done. I accidentally only emailed Paul earlier, but why can't we upgrade the pip module with the exe and then replace the process (using something in the os.exec* family) with `python -m pip update-exe` which could then succeed since the OS isn't holding onto the exe file? I could be missing something entirely obvious since I haven't developed (directly) on or for Windows in at least 5 years. From noah at coderanger.net Sun Jul 14 19:39:36 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sun, 14 Jul 2013 10:39:36 -0700 Subject: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping)) In-Reply-To: References: <18A62404-30AE-41E6-A7BC-B7061DB0C4EE@coderanger.net> Message-ID: <5C4267BE-2D69-4FC6-9ED4-C023213CAE77@coderanger.net> On Jul 14, 2013, at 10:31 AM, Ian Cordasco wrote: > On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz wrote: >> >> On Jul 14, 2013, at 9:45 AM, Steve Dower wrote: >> >>> From: Paul Moore >>>> On 13 July 2013 10:05, Paul Moore wrote: >>>> How robust is the process of upgrading pip using itself? Specifically on >>>> Windows, where these things typically seem less reliable. >>>> >>>> OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason >>>> for the failure is simple enough to explain - the pip.exe wrapper is held open >>>> by the OS while it's in use, so that the upgrade cannot replace it. >>>> >>>> The result is a failed upgrade and a partially installed new version of pip. In >>>> practice, the exe stubs are probably added fairly late in the install (at least >>>> when installing from sdist, with a wheel that depends on the order of the files >>>> in the wheel), so it's probably only a little bit broken, but "a little bit >>>> broken" is still broken :-( >>>> >>>> On the other hand, "python -m pip install -U pip" works fine because it avoids >>>> the exe wrappers. >>>> >>>> There's a lot of scope for user confusion and frustration in all this. For >>>> standalone pip I've tended to recommend "don't do that" - manually uninstall and >>>> reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. >>>> That sort of advice isn't going to be realistic for a pip bundled with CPython. >>>> >>>> Does anyone have any suggestions? >>> >>> Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? >>> >>> Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. >>> >>> MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. >>> >>> If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. >>> >>> So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. >> >> The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done. > > I accidentally only emailed Paul earlier, but why can't we upgrade the > pip module with the exe and then replace the process (using something > in the os.exec* family) with `python -m pip update-exe` which could > then succeed since the OS isn't holding onto the exe file? I could be > missing something entirely obvious since I haven't developed > (directly) on or for Windows in at least 5 years. Unfortunately windows doesn't actually offer the equivalent of a POSIX exec(). The various functions in os don't actually replace the current process, they just create a new one and terminate the old one. This means the controlling terminal would see the pip process as ended, so it makes showing output difficult at best. --Noah From noah at coderanger.net Sun Jul 14 19:43:18 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sun, 14 Jul 2013 10:43:18 -0700 Subject: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping)) In-Reply-To: <5C4267BE-2D69-4FC6-9ED4-C023213CAE77@coderanger.net> References: <18A62404-30AE-41E6-A7BC-B7061DB0C4EE@coderanger.net> <5C4267BE-2D69-4FC6-9ED4-C023213CAE77@coderanger.net> Message-ID: On Jul 14, 2013, at 10:39 AM, Noah Kantrowitz wrote: > > On Jul 14, 2013, at 10:31 AM, Ian Cordasco wrote: > >> On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz wrote: >>> >>> On Jul 14, 2013, at 9:45 AM, Steve Dower wrote: >>> >>>> From: Paul Moore >>>>> On 13 July 2013 10:05, Paul Moore wrote: >>>>> How robust is the process of upgrading pip using itself? Specifically on >>>>> Windows, where these things typically seem less reliable. >>>>> >>>>> OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason >>>>> for the failure is simple enough to explain - the pip.exe wrapper is held open >>>>> by the OS while it's in use, so that the upgrade cannot replace it. >>>>> >>>>> The result is a failed upgrade and a partially installed new version of pip. In >>>>> practice, the exe stubs are probably added fairly late in the install (at least >>>>> when installing from sdist, with a wheel that depends on the order of the files >>>>> in the wheel), so it's probably only a little bit broken, but "a little bit >>>>> broken" is still broken :-( >>>>> >>>>> On the other hand, "python -m pip install -U pip" works fine because it avoids >>>>> the exe wrappers. >>>>> >>>>> There's a lot of scope for user confusion and frustration in all this. For >>>>> standalone pip I've tended to recommend "don't do that" - manually uninstall and >>>>> reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. >>>>> That sort of advice isn't going to be realistic for a pip bundled with CPython. >>>>> >>>>> Does anyone have any suggestions? >>>> >>>> Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? >>>> >>>> Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. >>>> >>>> MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. >>>> >>>> If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. >>>> >>>> So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. >>> >>> The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done. >> >> I accidentally only emailed Paul earlier, but why can't we upgrade the >> pip module with the exe and then replace the process (using something >> in the os.exec* family) with `python -m pip update-exe` which could >> then succeed since the OS isn't holding onto the exe file? I could be >> missing something entirely obvious since I haven't developed >> (directly) on or for Windows in at least 5 years. > > Unfortunately windows doesn't actually offer the equivalent of a POSIX exec(). The various functions in os don't actually replace the current process, they just create a new one and terminate the old one. This means the controlling terminal would see the pip process as ended, so it makes showing output difficult at best. Check that, maybe I'm wrong, does anyone know if the P_OVERLAY flag unlocks the original binary? /me drags out a windows VM ? --Noah From qwcode at gmail.com Sun Jul 14 19:49:13 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 14 Jul 2013 10:49:13 -0700 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: > On 14 July 2013 18:02, Marcus Smith wrote: > >> there's a section marked "Installation Tutorial". Someone wanting to >> install packages should be able to get into that, once it actually has >> content : ) >> > > Yes, I see that. I'm not sure I like the up-front "decide between user or > system or virtualenv" presentation, though. I'm working on something I > prefer, see what you think when it's done. > yea, don't take the current install tutorial TOC that seriously. The user/global/virtualenv content came from a recent merge Nick referred to above. We just need knowledgeable people to get in there and start working/changing OK... But remember that most users are consumers of packages, not creators > of them. The packaging guide should reflect that > Like Nick said, we can put the "Installation Tutorial" above the "Packaging Tutorial". Neither tutorial should be considered "Advanced" IMO. The tutorials should be crisp and fast. Also, we *could* put a "Quickstart" above both tutorials, that just lists the frequent commands with one-liner descriptions, but we have to be really careful that it doesn't end up duping the tutorials, which are also intended to be quick as well. > If not, let's just get a user-only starter page in the Python docs and > leave the packaging guide as the "more comprehensive" documentation it can > refer to). > anything that results in feeling like the user guide needs a "user-only starter page in Python docs" means we're doing it wrong IMO : ) I admit the title itself concerns me: "Python Packaging User Guide", like it should have the word "Installation" in it. "Python Installation and Packaging User Guide"? (its soooo long though....) Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Jul 14 19:55:18 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 14 Jul 2013 10:55:18 -0700 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: > > I believe Marcus was hoping to get back to that after pip 1.4 was out the > door, but anyone on the PyPA list on BitBucket actually has full access to > accept pull requests, etc. > yes, the time has come to get this thing going or risk getting a "yet another dead document" rep, which it's starting to get. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah at coderanger.net Sun Jul 14 20:10:43 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sun, 14 Jul 2013 11:10:43 -0700 Subject: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping)) In-Reply-To: References: <18A62404-30AE-41E6-A7BC-B7061DB0C4EE@coderanger.net> <5C4267BE-2D69-4FC6-9ED4-C023213CAE77@coderanger.net> Message-ID: On Jul 14, 2013, at 10:43 AM, Noah Kantrowitz wrote: > > On Jul 14, 2013, at 10:39 AM, Noah Kantrowitz wrote: > >> >> On Jul 14, 2013, at 10:31 AM, Ian Cordasco wrote: >> >>> On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz wrote: >>>> >>>> On Jul 14, 2013, at 9:45 AM, Steve Dower wrote: >>>> >>>>> From: Paul Moore >>>>>> On 13 July 2013 10:05, Paul Moore wrote: >>>>>> How robust is the process of upgrading pip using itself? Specifically on >>>>>> Windows, where these things typically seem less reliable. >>>>>> >>>>>> OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason >>>>>> for the failure is simple enough to explain - the pip.exe wrapper is held open >>>>>> by the OS while it's in use, so that the upgrade cannot replace it. >>>>>> >>>>>> The result is a failed upgrade and a partially installed new version of pip. In >>>>>> practice, the exe stubs are probably added fairly late in the install (at least >>>>>> when installing from sdist, with a wheel that depends on the order of the files >>>>>> in the wheel), so it's probably only a little bit broken, but "a little bit >>>>>> broken" is still broken :-( >>>>>> >>>>>> On the other hand, "python -m pip install -U pip" works fine because it avoids >>>>>> the exe wrappers. >>>>>> >>>>>> There's a lot of scope for user confusion and frustration in all this. For >>>>>> standalone pip I've tended to recommend "don't do that" - manually uninstall and >>>>>> reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. >>>>>> That sort of advice isn't going to be realistic for a pip bundled with CPython. >>>>>> >>>>>> Does anyone have any suggestions? >>>>> >>>>> Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? >>>>> >>>>> Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. >>>>> >>>>> MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. >>>>> >>>>> If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. >>>>> >>>>> So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. >>>> >>>> The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done. >>> >>> I accidentally only emailed Paul earlier, but why can't we upgrade the >>> pip module with the exe and then replace the process (using something >>> in the os.exec* family) with `python -m pip update-exe` which could >>> then succeed since the OS isn't holding onto the exe file? I could be >>> missing something entirely obvious since I haven't developed >>> (directly) on or for Windows in at least 5 years. >> >> Unfortunately windows doesn't actually offer the equivalent of a POSIX exec(). The various functions in os don't actually replace the current process, they just create a new one and terminate the old one. This means the controlling terminal would see the pip process as ended, so it makes showing output difficult at best. > > Check that, maybe I'm wrong, does anyone know if the P_OVERLAY flag unlocks the original binary? /me drags out a windows VM ? Ignore my ignoring, with os.execl command flow does return back to the controlling terminal process (the new process continues in the background) and with os.spawnl(os.P_OVERLAY, 'python-2') I just get a segfault on 3.3. Yay for not completely misremembering, boo for this being so complicated. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From graffatcolmingov at gmail.com Sun Jul 14 20:12:34 2013 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Sun, 14 Jul 2013 14:12:34 -0400 Subject: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping)) In-Reply-To: References: <18A62404-30AE-41E6-A7BC-B7061DB0C4EE@coderanger.net> <5C4267BE-2D69-4FC6-9ED4-C023213CAE77@coderanger.net> Message-ID: On Sun, Jul 14, 2013 at 2:10 PM, Noah Kantrowitz wrote: > > On Jul 14, 2013, at 10:43 AM, Noah Kantrowitz wrote: > >> >> On Jul 14, 2013, at 10:39 AM, Noah Kantrowitz wrote: >> >>> >>> On Jul 14, 2013, at 10:31 AM, Ian Cordasco wrote: >>> >>>> On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz wrote: >>>>> >>>>> On Jul 14, 2013, at 9:45 AM, Steve Dower wrote: >>>>> >>>>>> From: Paul Moore >>>>>>> On 13 July 2013 10:05, Paul Moore wrote: >>>>>>> How robust is the process of upgrading pip using itself? Specifically on >>>>>>> Windows, where these things typically seem less reliable. >>>>>>> >>>>>>> OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason >>>>>>> for the failure is simple enough to explain - the pip.exe wrapper is held open >>>>>>> by the OS while it's in use, so that the upgrade cannot replace it. >>>>>>> >>>>>>> The result is a failed upgrade and a partially installed new version of pip. In >>>>>>> practice, the exe stubs are probably added fairly late in the install (at least >>>>>>> when installing from sdist, with a wheel that depends on the order of the files >>>>>>> in the wheel), so it's probably only a little bit broken, but "a little bit >>>>>>> broken" is still broken :-( >>>>>>> >>>>>>> On the other hand, "python -m pip install -U pip" works fine because it avoids >>>>>>> the exe wrappers. >>>>>>> >>>>>>> There's a lot of scope for user confusion and frustration in all this. For >>>>>>> standalone pip I've tended to recommend "don't do that" - manually uninstall and >>>>>>> reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. >>>>>>> That sort of advice isn't going to be realistic for a pip bundled with CPython. >>>>>>> >>>>>>> Does anyone have any suggestions? >>>>>> >>>>>> Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? >>>>>> >>>>>> Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. >>>>>> >>>>>> MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. >>>>>> >>>>>> If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. >>>>>> >>>>>> So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. >>>>> >>>>> The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done. >>>> >>>> I accidentally only emailed Paul earlier, but why can't we upgrade the >>>> pip module with the exe and then replace the process (using something >>>> in the os.exec* family) with `python -m pip update-exe` which could >>>> then succeed since the OS isn't holding onto the exe file? I could be >>>> missing something entirely obvious since I haven't developed >>>> (directly) on or for Windows in at least 5 years. >>> >>> Unfortunately windows doesn't actually offer the equivalent of a POSIX exec(). The various functions in os don't actually replace the current process, they just create a new one and terminate the old one. This means the controlling terminal would see the pip process as ended, so it makes showing output difficult at best. >> >> Check that, maybe I'm wrong, does anyone know if the P_OVERLAY flag unlocks the original binary? /me drags out a windows VM ? > > Ignore my ignoring, with os.execl command flow does return back to the controlling terminal process (the new process continues in the background) and with os.spawnl(os.P_OVERLAY, 'python-2') I just get a segfault on 3.3. Yay for not completely misremembering, boo for this being so complicated. I expected I was wrong, but I appreciate you looking at it to be certain. From qwcode at gmail.com Sun Jul 14 20:34:06 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 14 Jul 2013 11:34:06 -0700 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: > I admit the title itself concerns me: "Python Packaging User Guide", like > it should have the word "Installation" in it. > "Python Installation and Packaging User Guide"? (its soooo long > though....) > other ideas: "Python Installation and Packaging Guide" "Python Installation and Distribution Guide" -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Sun Jul 14 20:59:18 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Sun, 14 Jul 2013 19:59:18 +0100 Subject: [Distutils] buildout/setuptools/distribute unhelpful error message (0.7.x issue?) In-Reply-To: References: <51D913F5.3080804@simplistix.co.uk> Message-ID: <51E2F506.1090504@simplistix.co.uk> On 13/07/2013 17:37, Jason R. Coombs wrote: > It looks like something is trying to install Setuptools 0.7.2, possibly with > a temporary version of distribute or one that's not visible by default in your > Python environment. That would have been buildout's bootstrap.py. Jim, is this fixed in the latest 1 and 2 bootstrap.pys? > When you get that error message, I suggest you upgrade away from distribute. The problem is that this is the stable python-pkg-resources in Debian, so likely Ubuntu to. This problem has the potential to affect many many users so I hope a sensible story is being developed to deal with it... > The easiest way to do this if you have distribute installed is to > 'easy_install -U distribute', which will grab distribute 0.7.3 and install > setuptools>=0.7. What's the OS-packages plan for dealing with this? I would consider doing the above to an OS-installed package to be pretty anti-social... > I hope that helps. Please report back if that doesn't get you going. I fixed it before you replied, but it was pretty horrific: aptitude remove python-pkg-resources aptitude remove python-pkg-resources python-bzrlib python-pygments aptitude remove python-pkg-resources bzr python-bzrlib python-pygments aptitude remove python-pkg-resources bzr bzr-svn python-docutils python-bzrlib python-pygments ...which removed a lot of packages. I'm in a position where I can manage that, but I'd imagine a "civilian user" could end up in a lot of trouble. Also, don't forget the weird and crappy setuptools versions that Apple will have baked in to the various Pythons that ship with the various Mac OS's. While this merge is a good thing, it's causing a lot of pain from what I've heard and experienced. Chris > > Regards, > Jason > >> -----Original Message----- >> From: Distutils-SIG [mailto:distutils-sig- >> bounces+jaraco=jaraco.com at python.org] On Behalf Of Chris Withers >> Sent: Sunday, 07 July, 2013 03:09 >> To: distutils sig >> Subject: [Distutils] buildout/setuptools/distribute unhelpful error message >> (0.7.x issue?) >> >> Hi All, >> >> What is this exception trying to tell me? >> >> Downloading >> https://pypi.python.org/packages/source/s/setuptools/setuptools- >> 0.7.2.tar.gz >> Extracting in /tmp/tmpJNVsOY >> Now working in /tmp/tmpJNVsOY/setuptools-0.7.2 Building a Setuptools egg >> in /tmp/tmpBLZGeg /tmp/tmpBLZGeg/setuptools-0.7.2-py2.6.egg >> Traceback (most recent call last): >> File "bootstrap.py", line 91, in >> pkg_resources.working_set.add_entry(path) >> File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 451, in >> add_entry >> self.add(dist, entry, False) >> File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 542, in >> add >> self._added_new(dist) >> File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 705, in >> _added_new >> callback(dist) >> File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2727, in >> >> add_activation_listener(lambda dist: dist.activate()) >> File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2227, in >> activate >> self.insert_on(path) >> File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2328, in >> insert_on >> "with distribute. Found one at %s" % str(self.location)) >> ValueError: A 0.7-series setuptools cannot be installed with distribute. >> Found one at /tmp/tmpBLZGeg/setuptools-0.7.2-py2.6.egg >> >> I don't see any distribute in there, and I don't know where it might be... >> >> $ python2.6 >> Python 2.6.8 (unknown, Jan 29 2013, 10:05:44) [GCC 4.7.2] on linux2 Type >> "help", "copyright", "credits" or "license" for more information. >> >>> import setuptools >> Traceback (most recent call last): >> File "", line 1, in >> ImportError: No module named setuptools >> >> cheers, >> >> Chris >> >> -- >> Simplistix - Content Management, Batch Processing & Python Consulting >> - http://www.simplistix.co.uk >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From p.f.moore at gmail.com Sun Jul 14 21:44:08 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 14 Jul 2013 20:44:08 +0100 Subject: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping)) In-Reply-To: <6EACCA35-1901-4D60-9BFF-FCBF94480112@stufft.io> References: <6EACCA35-1901-4D60-9BFF-FCBF94480112@stufft.io> Message-ID: On 14 July 2013 18:06, Donald Stufft wrote: > Wouldn't a .py file make the command `pip.py`` and not ``pip`` ? Not if .py is a registered extension. What I can't remember is whether it needs to be in PATHEXT (which it isn't by default). The big problem here is that the behaviour isn't very well documented (if at all) so the various command shells act subtly differently. That's why I want to test, and why it won't be a 5-minute job to do so... But the various "replace the exe afterwards" hacks sound awfully complicated to me - particularly as pip doesn't control the exes in the first place, they are part of the setuptools console script entry point infrastructure. My strong preference here is to remove the current use of setuptools entry points, simply because I don't think the problem is solvable while pip doesn't control the exe management at all. That's a non-trivial change, but longer term maybe the best. Question for Nick, Brett and any other core devs around: Would python-dev be willing to include in the stdlib some sort of package for managing exe-wrappers? I don't really want pip to manage exe wrappers any more than I like setuptools doing so. Maybe the existing launcher can somehow double up in that role? Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Jul 14 21:51:49 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 14 Jul 2013 20:51:49 +0100 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: On 14 July 2013 18:49, Marcus Smith wrote: > I admit the title itself concerns me: "Python Packaging User Guide", like > it should have the word "Installation" in it. > "Python Installation and Packaging User Guide"? (its soooo long > though....) > I like "Package Management Guide". Strong hints of managing existing 3rd party packages, which is good because it's what the majority want, but general enough to allow for including managing the build & distribution of your own packages. BTW, I'm sick and tired of agonising every time I use the word "package" over whether I should be "correct" and use "distribution". Can the guide just come right out and bless the occasionally-ambiguous but commonly-used dual nature of the word "package"? If not, can people start actually *using* "distribution" consistently for what pip downloads and installs, so I can find a few more examples for me to copy when I end up with awkward phrases like "distributing your distribution"...? (You'd never believe English was my native language, would you? :-)) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Jul 14 22:05:05 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 14 Jul 2013 13:05:05 -0700 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: > > > BTW, I'm sick and tired of agonising every time I use the word "package" > over whether I should be "correct" and use "distribution". Can the guide > just come right out and bless the occasionally-ambiguous but commonly-used > dual nature of the word "package"? If not, can people start actually > *using* "distribution" consistently for what pip downloads and installs, so > I can find a few more examples for me to copy when I end up with awkward > phrases like "distributing your distribution"...? (You'd never believe > English was my native language, would you? :-)) > I hear you. I feel the same agony. It think we should use the word "Distribution", but it's hard to compete when PyPI uses "Package". Nick, what do we do? : ) Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Jul 14 23:22:51 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 14 Jul 2013 14:22:51 -0700 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: Donald: thoughts on changing our use of "Package" on pypi to "Distribution"? (except for the title of course) If we're not going to do that, we should explain and bless the double use of "Package" and drop using "Distribution" in any docs. Our fundamental concepts shouldn't be confusing and conflicted. On Sun, Jul 14, 2013 at 1:05 PM, Marcus Smith wrote: > >> BTW, I'm sick and tired of agonising every time I use the word "package" >> over whether I should be "correct" and use "distribution". Can the guide >> just come right out and bless the occasionally-ambiguous but commonly-used >> dual nature of the word "package"? If not, can people start actually >> *using* "distribution" consistently for what pip downloads and installs, so >> I can find a few more examples for me to copy when I end up with awkward >> phrases like "distributing your distribution"...? (You'd never believe >> English was my native language, would you? :-)) >> > > I hear you. I feel the same agony. It think we should use the word > "Distribution", but it's hard to compete when PyPI uses "Package". > Nick, what do we do? : ) > > Marcus > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Jul 14 23:24:46 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 14 Jul 2013 17:24:46 -0400 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: On Jul 14, 2013, at 5:22 PM, Marcus Smith wrote: > Donald: > thoughts on changing our use of "Package" on pypi to "Distribution"? (except for the title of course) > If we're not going to do that, we should explain and bless the double use of "Package" and drop using "Distribution" in any docs. > Our fundamental concepts shouldn't be confusing and conflicted. > > On Sun, Jul 14, 2013 at 1:05 PM, Marcus Smith wrote: > > BTW, I'm sick and tired of agonising every time I use the word "package" over whether I should be "correct" and use "distribution". Can the guide just come right out and bless the occasionally-ambiguous but commonly-used dual nature of the word "package"? If not, can people start actually *using* "distribution" consistently for what pip downloads and installs, so I can find a few more examples for me to copy when I end up with awkward phrases like "distributing your distribution"...? (You'd never believe English was my native language, would you? :-)) > > I hear you. I feel the same agony. It think we should use the word "Distribution", but it's hard to compete when PyPI uses "Package". > Nick, what do we do? : ) > > Marcus > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig PyPI will eventually move to using the definitions as defined in PEP426 http://www.python.org/dev/peps/pep-0426/#supporting-definitions . ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From qwcode at gmail.com Sun Jul 14 23:29:11 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 14 Jul 2013 14:29:11 -0700 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: no reason not to change "Package" to "Distribution" now, right? this (http://docs.python.org/2/install/index.html) already uses the "Distribution" word. which means changing sentences like "There are currently 32660 packages here." to "There are currently 32660 distributions here." and the main table listing would use the word "Distribution" On Sun, Jul 14, 2013 at 2:24 PM, Donald Stufft wrote: > > On Jul 14, 2013, at 5:22 PM, Marcus Smith wrote: > > Donald: > thoughts on changing our use of "Package" on pypi to "Distribution"? > (except for the title of course) > If we're not going to do that, we should explain and bless the double use > of "Package" and drop using "Distribution" in any docs. > Our fundamental concepts shouldn't be confusing and conflicted. > > On Sun, Jul 14, 2013 at 1:05 PM, Marcus Smith wrote: > >> >>> BTW, I'm sick and tired of agonising every time I use the word "package" >>> over whether I should be "correct" and use "distribution". Can the guide >>> just come right out and bless the occasionally-ambiguous but commonly-used >>> dual nature of the word "package"? If not, can people start actually >>> *using* "distribution" consistently for what pip downloads and installs, so >>> I can find a few more examples for me to copy when I end up with awkward >>> phrases like "distributing your distribution"...? (You'd never believe >>> English was my native language, would you? :-)) >>> >> >> I hear you. I feel the same agony. It think we should use the word >> "Distribution", but it's hard to compete when PyPI uses "Package". >> Nick, what do we do? : ) >> >> Marcus >> >> > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > > PyPI will eventually move to using the definitions as defined in PEP426 > http://www.python.org/dev/peps/pep-0426/#supporting-definitions . > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Jul 14 23:31:43 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 14 Jul 2013 17:31:43 -0400 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: <63522A4F-C104-4FE3-8F4F-5F6DE016E896@stufft.io> On Jul 14, 2013, at 5:29 PM, Marcus Smith wrote: > no reason not to change "Package" to "Distribution" now, right? > this (http://docs.python.org/2/install/index.html) already uses the "Distribution" word. > which means changing sentences like "There are currently 32660 packages here." to "There are currently 32660 distributions here." > and the main table listing would use the word "Distribution" > > That's 32600 projects, There are almost 200k distributions. And the main table listing would be "Releases" or "Project Releases". > On Sun, Jul 14, 2013 at 2:24 PM, Donald Stufft wrote: > > On Jul 14, 2013, at 5:22 PM, Marcus Smith wrote: > >> Donald: >> thoughts on changing our use of "Package" on pypi to "Distribution"? (except for the title of course) >> If we're not going to do that, we should explain and bless the double use of "Package" and drop using "Distribution" in any docs. >> Our fundamental concepts shouldn't be confusing and conflicted. >> >> On Sun, Jul 14, 2013 at 1:05 PM, Marcus Smith wrote: >> >> BTW, I'm sick and tired of agonising every time I use the word "package" over whether I should be "correct" and use "distribution". Can the guide just come right out and bless the occasionally-ambiguous but commonly-used dual nature of the word "package"? If not, can people start actually *using* "distribution" consistently for what pip downloads and installs, so I can find a few more examples for me to copy when I end up with awkward phrases like "distributing your distribution"...? (You'd never believe English was my native language, would you? :-)) >> >> I hear you. I feel the same agony. It think we should use the word "Distribution", but it's hard to compete when PyPI uses "Package". >> Nick, what do we do? : ) >> >> Marcus >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > PyPI will eventually move to using the definitions as defined in PEP426 http://www.python.org/dev/peps/pep-0426/#supporting-definitions . > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From qwcode at gmail.com Sun Jul 14 23:32:14 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 14 Jul 2013 14:32:14 -0700 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: Message-ID: although some of the uses could be argued as needing "Project" e.g "Package documentation is hosted on its own domain" On Sun, Jul 14, 2013 at 2:29 PM, Marcus Smith wrote: > no reason not to change "Package" to "Distribution" now, right? > this (http://docs.python.org/2/install/index.html) already uses the > "Distribution" word. > which means changing sentences like "There are currently 32660 packages > here." to "There are currently 32660 distributions here." > and the main table listing would use the word "Distribution" > > > > On Sun, Jul 14, 2013 at 2:24 PM, Donald Stufft wrote: > >> >> On Jul 14, 2013, at 5:22 PM, Marcus Smith wrote: >> >> Donald: >> thoughts on changing our use of "Package" on pypi to "Distribution"? >> (except for the title of course) >> If we're not going to do that, we should explain and bless the double use >> of "Package" and drop using "Distribution" in any docs. >> Our fundamental concepts shouldn't be confusing and conflicted. >> >> On Sun, Jul 14, 2013 at 1:05 PM, Marcus Smith wrote: >> >>> >>>> BTW, I'm sick and tired of agonising every time I use the word >>>> "package" over whether I should be "correct" and use "distribution". Can >>>> the guide just come right out and bless the occasionally-ambiguous but >>>> commonly-used dual nature of the word "package"? If not, can people start >>>> actually *using* "distribution" consistently for what pip downloads and >>>> installs, so I can find a few more examples for me to copy when I end up >>>> with awkward phrases like "distributing your distribution"...? (You'd never >>>> believe English was my native language, would you? :-)) >>>> >>> >>> I hear you. I feel the same agony. It think we should use the word >>> "Distribution", but it's hard to compete when PyPI uses "Package". >>> Nick, what do we do? : ) >>> >>> Marcus >>> >>> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig >> >> >> PyPI will eventually move to using the definitions as defined in PEP426 >> http://www.python.org/dev/peps/pep-0426/#supporting-definitions . >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 >> DCFA >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Jul 14 23:36:26 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 14 Jul 2013 14:36:26 -0700 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: <63522A4F-C104-4FE3-8F4F-5F6DE016E896@stufft.io> References: <63522A4F-C104-4FE3-8F4F-5F6DE016E896@stufft.io> Message-ID: > That's 32600 projects, There are almost 200k distributions. And the main > table listing would be "Releases" or "Project Releases". > ok, got it. anything I do from here on will follow this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Jul 14 23:38:14 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 14 Jul 2013 22:38:14 +0100 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: <63522A4F-C104-4FE3-8F4F-5F6DE016E896@stufft.io> Message-ID: On 14 July 2013 22:36, Marcus Smith wrote: > > That's 32600 projects, There are almost 200k distributions. And the main >> table listing would be "Releases" or "Project Releases". >> > > ok, got it. anything I do from here on will follow this. > That's much better. I apologise, I hadn't noticed that PEP 426 included terminology definitions. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jul 15 00:06:15 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 08:06:15 +1000 Subject: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping)) In-Reply-To: References: <6EACCA35-1901-4D60-9BFF-FCBF94480112@stufft.io> Message-ID: On 15 Jul 2013 05:44, "Paul Moore" wrote: > > On 14 July 2013 18:06, Donald Stufft wrote: >> >> Wouldn't a .py file make the command `pip.py`` and not ``pip`` ? > > > Not if .py is a registered extension. What I can't remember is whether it needs to be in PATHEXT (which it isn't by default). The big problem here is that the behaviour isn't very well documented (if at all) so the various command shells act subtly differently. That's why I want to test, and why it won't be a 5-minute job to do so... > > But the various "replace the exe afterwards" hacks sound awfully complicated to me - particularly as pip doesn't control the exes in the first place, they are part of the setuptools console script entry point infrastructure. > > My strong preference here is to remove the current use of setuptools entry points, simply because I don't think the problem is solvable while pip doesn't control the exe management at all. That's a non-trivial change, but longer term maybe the best. > > Question for Nick, Brett and any other core devs around: Would python-dev be willing to include in the stdlib some sort of package for managing exe-wrappers? I don't really want pip to manage exe wrappers any more than I like setuptools doing so. Maybe the existing launcher can somehow double up in that role? Not sure it fits the launcher, but having something along those lines in the stdlib makes sense (especially in the context of a pip bundling PEP). Another option we may want to consider is an actual msi installer for pip (I'm not sure that would actually help, but it's worth looking into), as well as investigating what other self-updating Windows apps (like Firefox) do to handle this problem. Cheers, Nick. > > Paul > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah at coderanger.net Mon Jul 15 00:09:51 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sun, 14 Jul 2013 15:09:51 -0700 Subject: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping)) In-Reply-To: References: <6EACCA35-1901-4D60-9BFF-FCBF94480112@stufft.io> Message-ID: <3E40F838-1970-4792-A96C-BDF282D152BB@coderanger.net> On Jul 14, 2013, at 3:06 PM, Nick Coghlan wrote: > > On 15 Jul 2013 05:44, "Paul Moore" wrote: > > > > On 14 July 2013 18:06, Donald Stufft wrote: > >> > >> Wouldn't a .py file make the command `pip.py`` and not ``pip`` ? > > > > > > Not if .py is a registered extension. What I can't remember is whether it needs to be in PATHEXT (which it isn't by default). The big problem here is that the behaviour isn't very well documented (if at all) so the various command shells act subtly differently. That's why I want to test, and why it won't be a 5-minute job to do so... > > > > But the various "replace the exe afterwards" hacks sound awfully complicated to me - particularly as pip doesn't control the exes in the first place, they are part of the setuptools console script entry point infrastructure. > > > > My strong preference here is to remove the current use of setuptools entry points, simply because I don't think the problem is solvable while pip doesn't control the exe management at all. That's a non-trivial change, but longer term maybe the best. > > > > Question for Nick, Brett and any other core devs around: Would python-dev be willing to include in the stdlib some sort of package for managing exe-wrappers? I don't really want pip to manage exe wrappers any more than I like setuptools doing so. Maybe the existing launcher can somehow double up in that role? > > Not sure it fits the launcher, but having something along those lines in the stdlib makes sense (especially in the context of a pip bundling PEP). > > Another option we may want to consider is an actual msi installer for pip (I'm not sure that would actually help, but it's worth looking into), as well as investigating what other self-updating Windows apps (like Firefox) do to handle this problem. They do the "exec a helper executable that replaces the original" approach, which works fine for non-console apps since there isn't the problem of the shell getting confused :-/ --Noah From p.f.moore at gmail.com Mon Jul 15 00:17:59 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 14 Jul 2013 23:17:59 +0100 Subject: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping)) In-Reply-To: <3E40F838-1970-4792-A96C-BDF282D152BB@coderanger.net> References: <6EACCA35-1901-4D60-9BFF-FCBF94480112@stufft.io> <3E40F838-1970-4792-A96C-BDF282D152BB@coderanger.net> Message-ID: On 14 July 2013 23:09, Noah Kantrowitz wrote: > > Another option we may want to consider is an actual msi installer for > pip (I'm not sure that would actually help, but it's worth looking into), > as well as investigating what other self-updating Windows apps (like > Firefox) do to handle this problem. > > They do the "exec a helper executable that replaces the original" > approach, which works fine for non-console apps since there isn't the > problem of the shell getting confused :-/ Generally, I don't think that going down the route of MSIs is a good move. They aren't a good fit for this problem. Apart from anything else, they won't support installing into a virtualenv. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jul 15 00:18:23 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 08:18:23 +1000 Subject: [Distutils] Another conversation starter - pip documentation in the Python docs In-Reply-To: References: <63522A4F-C104-4FE3-8F4F-5F6DE016E896@stufft.io> Message-ID: On 15 Jul 2013 07:45, "Paul Moore" wrote: > > On 14 July 2013 22:36, Marcus Smith wrote: >> >> >>> That's 32600 projects, There are almost 200k distributions. And the main table listing would be "Releases" or "Project Releases". >> >> >> ok, got it. anything I do from here on will follow this. > > > That's much better. I apologise, I hadn't noticed that PEP 426 included terminology definitions. They were necessary for me to keep them straight in my own head. The original set were a bit odd, but I think the project/release/distribution/archive split has ended up in a reasonable place after a few iterations (it's still open to revisions if anything seems too awkward, though). I don't think the ambiguity of "package" will ever go away entirely, but we can justify that to some degree by noting that distributions mainly serve to get Python packages and modules installed and available for import. Cheers, Nick. > Paul > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Mon Jul 15 11:30:20 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 15 Jul 2013 11:30:20 +0200 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: <78D8BC3C-C638-422A-9D8D-D89FFA917E60@mac.com> On 13 Jul, 2013, at 7:31, Nick Coghlan wrote: > > 3. That means there are two main options available to us that I still consider viable alternatives (the installer bundling idea was suggested in one of the off list comments I mentioned): > > * an explicit bootstrapping script > * bundling a *full* copy of pip with the Python installers for Windows and Mac OS X, but installing it to site-packages rather than to the standard library directory. That way pip can be used to upgrade itself as normal, rather than making it part of the standard library per se. This is then closer to the "bundled application" model adopted for IDLE in PEP 434 (we could, in fact, move to distributing idle the same way). Or automaticly invoke the bootstrap script during installation (for the Python installers), that we the installers don't end up with a stale version of pip. Either option should be easy enough to add to the OSX installers. Ronald From ronaldoussoren at mac.com Mon Jul 15 11:35:13 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 15 Jul 2013 11:35:13 +0200 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: <55B209B3-9576-4CF0-B58C-2A1E692AFFF1@stufft.io> Message-ID: <849E042D-F8A4-46D1-A15A-2C6661A58821@mac.com> On 13 Jul, 2013, at 15:35, Brett Cannon wrote: > > > > On Sat, Jul 13, 2013 at 2:29 AM, Ned Deily wrote: > In article <55B209B3-9576-4CF0-B58C-2A1E692AFFF1 at stufft.io>, > Donald Stufft wrote: > > On Jul 13, 2013, at 1:31 AM, Nick Coghlan wrote: > > > I'm currently leaning towards offering both, as we're going to need a tool > > > for bootstrapping source builds, but the simplest way to bootstrap pip for > > > Windows and Mac OS X users is to just *bundle a copy with the binary > > > installers*. So long as the bundled copy looks *exactly* the way it would > > > if installed later (so it can update itself), then we avoid the problem of > > > coupling the pip update cycles to the standard library feature release > > > cycle. The bundled version can be updated to the latest available versions > > > when we do a Python maintenance release. > > Off the top of my head, including a copy of pip as a pre-installed > global site-package seems like a very reasonable suggestion. For the > python.org OS X installer, it should be no problem to implement. It > would be equally easy to implement for future 2.7 and 3.3 maintenance > releases. > > Does Apple just install the python.org OS X installer for distribution, or do they build their own thing? They do their own thing. > My only worry is that Apple will not get the message about including pip and we will end up with an odd skew on OS X (I'm not worried about Linux distros as they all seem to follow Python development closely). That will happen anyway, pip won't get magically installed on current OSX releases and adding it to the upcoming 10.9 release is probably not possible either unless it already happens to be in the current beta (they appear to have a fairly early cutoff point for including new software at the Unix layer). Ronald From ronaldoussoren at mac.com Mon Jul 15 11:38:46 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 15 Jul 2013 11:38:46 +0200 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: References: Message-ID: <5661DE4D-D148-44B5-9C40-C61D1E4A8AE1@mac.com> On 13 Jul, 2013, at 16:39, Vinay Sajip wrote: > > > I smoke-tested the script on vanilla Python 3.3 installations on Windows and > OS X. On OS X, the pip script was written to the appropriate Frameworks > folder, but not to /usr/local/bin - I assume it would be simple enough to > arrange that? Not installing in /usr/local/bin is a feature that makes it easier to maintain several python installs without worrying about contamination (for example Python 3 and Python 2, or even 2.6 and 2.7). The installer by default adds the framework 'bin' directory to the environment for the shell of the user that installed the framework. Ronald From ncoghlan at gmail.com Mon Jul 15 11:54:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 19:54:24 +1000 Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: <78D8BC3C-C638-422A-9D8D-D89FFA917E60@mac.com> References: <78D8BC3C-C638-422A-9D8D-D89FFA917E60@mac.com> Message-ID: On 15 July 2013 19:30, Ronald Oussoren wrote: > > On 13 Jul, 2013, at 7:31, Nick Coghlan wrote: >> >> 3. That means there are two main options available to us that I still consider viable alternatives (the installer bundling idea was suggested in one of the off list comments I mentioned): >> >> * an explicit bootstrapping script >> * bundling a *full* copy of pip with the Python installers for Windows and Mac OS X, but installing it to site-packages rather than to the standard library directory. That way pip can be used to upgrade itself as normal, rather than making it part of the standard library per se. This is then closer to the "bundled application" model adopted for IDLE in PEP 434 (we could, in fact, move to distributing idle the same way). > > Or automaticly invoke the bootstrap script during installation (for the Python installers), that we the installers don't end up with a stale version of pip. Yeah, I see pros and cons to either approach, with the main con of install time bootstrapping being the requirement for network access, while the main con of bundling is that you may end up needing to do "pip install --upgrade pip" immediately after installing Python anyway. I currently have a slight preference for actual bundling, but could probably be persuaded to endorse an install time bootstrap instead. It's only the bootstrap-on-first-use approach that I've decided is asking for trouble. I don't believe either Martin (von L?wis) or Brian (Curtin) is on this list, so I'll email them directly to see if they have a preference. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Mon Jul 15 13:03:48 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 15 Jul 2013 12:03:48 +0100 (BST) Subject: [Distutils] Current status of PEP 439 (pip boostrapping) In-Reply-To: <5661DE4D-D148-44B5-9C40-C61D1E4A8AE1@mac.com> References: <5661DE4D-D148-44B5-9C40-C61D1E4A8AE1@mac.com> Message-ID: <1373886228.36583.YahooMailNeo@web171401.mail.ir2.yahoo.com> > Not installing in /usr/local/bin is a feature that makes it easier to maintain > several python installs without worrying about contamination (for example Python > 3 > and Python 2, or even 2.6 and 2.7). I thought it might be behaving as designed, but wasn't sure. Regards, Vinay Sajip From p.f.moore at gmail.com Mon Jul 15 13:24:10 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 15 Jul 2013 12:24:10 +0100 Subject: [Distutils] Call for PEP author/champion: Bundling pip with CPython installers In-Reply-To: References: Message-ID: On 14 July 2013 07:13, Nick Coghlan wrote: > * ensuring that pip is automatically available in virtual environments > created with pyvenv > Is the proposal here for pyvenv to download pip or to install a locally stored copy? Using a locally stored copy is what virtualenv does, and I'd prefer it to avoid issues where the user's PC has no internet access (as well as avoiding the need to worry about secure downloads and bundling certs, which was why virtualenv took this route). Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jul 15 15:12:07 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 23:12:07 +1000 Subject: [Distutils] Call for PEP author/champion: Bundling pip with CPython installers In-Reply-To: References: Message-ID: On 15 July 2013 21:24, Paul Moore wrote: > > On 14 July 2013 07:13, Nick Coghlan wrote: >> >> * ensuring that pip is automatically available in virtual environments >> created with pyvenv > > > Is the proposal here for pyvenv to download pip or to install a locally > stored copy? Using a locally stored copy is what virtualenv does, and I'd > prefer it to avoid issues where the user's PC has no internet access (as > well as avoiding the need to worry about secure downloads and bundling > certs, which was why virtualenv took this route). Using a locally stored copy. I'm also considering a new trick for the import system (in the general vein of *.egg-link files) that would let us achieve that without copying, and perhaps even let us eventually deprecate *.pth files entirely. If that scheme comes to fruition it will be as an independent PEP, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Mon Jul 15 15:39:30 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 15 Jul 2013 14:39:30 +0100 Subject: [Distutils] Replacing pip.exe with a Python script Message-ID: I'm looking at the possibility of replacing the current setuptools entry point based pip executables with Python scripts. The biggest problem is that a script "pip.py" shadows the pip package, making "import pip" fail. I can get round this by deleting sys.path[0] (the location of the currently running script) but how robust is that? Are there any corner cases where it would break? Or alternatively, is there a better way to do this rather than messing directly with sys.path? I suspect this is a fairly common question, but my Google-fu is failing me :-( Sorry, I know this is a basic Python coding question - in my defence, it's for something related to the current pip discussions :-) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Mon Jul 15 15:44:43 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 15 Jul 2013 14:44:43 +0100 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: Message-ID: On 15 July 2013 14:39, Paul Moore wrote: > > I'm looking at the possibility of replacing the current setuptools entry > point based pip executables with Python scripts. The biggest problem is that > a script "pip.py" shadows the pip package, making "import pip" fail. > > I can get round this by deleting sys.path[0] (the location of the currently > running script) but how robust is that? Are there any corner cases where it > would break? Or alternatively, is there a better way to do this rather than > messing directly with sys.path? I suspect this is a fairly common question, > but my Google-fu is failing me :-( Can you not just rename the pip module to _pip? Oscar From brett at python.org Mon Jul 15 15:53:52 2013 From: brett at python.org (Brett Cannon) Date: Mon, 15 Jul 2013 09:53:52 -0400 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: Message-ID: On Mon, Jul 15, 2013 at 9:39 AM, Paul Moore wrote: > I'm looking at the possibility of replacing the current setuptools entry > point based pip executables with Python scripts. The biggest problem is > that a script "pip.py" shadows the pip package, making "import pip" fail. > > I can get round this by deleting sys.path[0] (the location of the > currently running script) but how robust is that? Are there any corner > cases where it would break? Or alternatively, is there a better way to do > this rather than messing directly with sys.path? I suspect this is a fairly > common question, but my Google-fu is failing me :-( > > Sorry, I know this is a basic Python coding question - in my defence, it's > for something related to the current pip discussions :-) > As long as you make sure that sys.path[0] is actually the script location then it will work (other things like .pth files, PYTHONSTARTUP, etc. could have changed things before your script started execution). But realize that a) in Python 3.3 the scripts location will be ./pip.py, not just pip.py, and b) if I get my way all paths will be absolute for __file__, so you will have to just associate '' with os.getcwd() and then search for the proper directory on sys.path. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Jul 15 16:08:10 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 15 Jul 2013 15:08:10 +0100 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: Message-ID: On 15 July 2013 14:44, Oscar Benjamin wrote: > Can you not just rename the pip module to _pip? That's a far more intrusive change, and I'm not sure I want to go there. It *will* break existing code that (rightly or wrongly) imports pip. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Jul 15 16:16:56 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 15 Jul 2013 15:16:56 +0100 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: Message-ID: On 15 July 2013 14:53, Brett Cannon wrote: > As long as you make sure that sys.path[0] is actually the script location > then it will work (other things like .pth files, PYTHONSTARTUP, etc. could > have changed things before your script started execution). But realize that > a) in Python 3.3 the scripts location will be ./pip.py, not just pip.py, > and b) if I get my way all paths will be absolute for __file__, so you will > have to just associate '' with os.getcwd() and then search for the proper > directory on sys.path. OK, that pretty much tells me that this is a bad idea. It's never going to be robust enough to work. I'm amazed actually that there's no way to say "don't add the script location to sys.path", even as a command line option. It seems like the sort of thing you'd want to make scripts robust, a bit like -S and -E. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Mon Jul 15 16:16:38 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 15 Jul 2013 15:16:38 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> Message-ID: On 13 July 2013 20:59, Paul Moore wrote: > It would be nice to get feedback from "normal users" on this. I suspect that > the scientific community would make a good cross-section (AIUI there's quite > a lot of Windows use, and for many people in the community Python is very > much a tool, rather than a way of life :-)). Does anyone have links into the > scipy groups? I lurk on the IPython lists, so I could ask there, at a > pinch... I don't know if I really count as a normal user but I can describe how Python is installed on the Windows machines in my faculty for scientific use. All our Windows machines have the Enthought Python Distribution (EPD) installed. This bundles CPython with numpy, scipy, matplotlib, wxpython, setuptools, pip and a whole load more. Ordinary users do not need to install numpy etc. since these are pre-installed. The bootstrap process is probably irrelevant since EPD installs easy_install and that can be used to install pip if desired. Ordinary users do not have write access to the EPD installation directory and so can only use pip/easy_install with --user anyway. On my own desktop machine which runs Windows all of the Python installations I use are inside my user directory so there is no meaningful difference between 'pip install' and 'pip install --user'. The real problem for us with using e.g. pip to install something like numpy is that it will not install the appropriate non-Python external libraries. For example, numpy ships with a just functional BLAS library but you really want to install and have it link against proper BLAS/LAPACK libraries. The good free libraries should be compiled on the target machine and pypi/pip/distutils do not currently help much with doing this. Debian (or at least Ubuntu) provides for example the ATLAS library as a source only package. This means that you can compile it on the target machine and get the most out of your CPU capabilities while still using the Debian tools to obtain the dependencies and manage the build process. The new wheel format will not help with this since even if there were an ATLAS wheel it would probably be a generic 686 binary without e.g. SSE. This is another advantage of using EPD which ships the non-free Intel MKL library. Python(x, y) is similar to EPD but is GPL'd and ships with OpenBLAS. Both distributions also ship MinGW which is useful since it's likely that our Windows machines will not have the appropriate MSVC version to match up with the CPython version. (They also don't suffer from Issue12641 so MinGW works). Oscar From p.f.moore at gmail.com Mon Jul 15 16:21:58 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 15 Jul 2013 15:21:58 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> Message-ID: On 15 July 2013 15:16, Oscar Benjamin wrote: > I don't know if I really count as a normal user but I can describe how > Python is installed on the Windows machines in my faculty for > scientific use. > Thanks, that's interesting. Do people typically write command-line Python scripts? If so, do they expect to be able to put them on PATH and run them? What command processor is typically used? Powershell or cmd? I suspect that there isn't much that's "typical" about anyone :-) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jul 15 16:32:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 00:32:49 +1000 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: Message-ID: On 16 July 2013 00:16, Paul Moore wrote: > > On 15 July 2013 14:53, Brett Cannon wrote: >> >> As long as you make sure that sys.path[0] is actually the script location >> then it will work (other things like .pth files, PYTHONSTARTUP, etc. could >> have changed things before your script started execution). But realize that >> a) in Python 3.3 the scripts location will be ./pip.py, not just pip.py, and >> b) if I get my way all paths will be absolute for __file__, so you will have >> to just associate '' with os.getcwd() and then search for the proper >> directory on sys.path. > > > OK, that pretty much tells me that this is a bad idea. It's never going to > be robust enough to work. Most of the stuff Brett mentioned there shouldn't be relevant for a directly executed script - doing sys.path.remove(os.path.dirname(os.path.abspath(__file__)) should be pretty robust in any scenario. > I'm amazed actually that there's no way to say > "don't add the script location to sys.path", even as a command line option. > It seems like the sort of thing you'd want to make scripts robust, a bit > like -S and -E. You'd think that, but then you'd look at getpath.c and run away (or write something like PEP 432, as I did) :P Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Mon Jul 15 16:41:15 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 15 Jul 2013 15:41:15 +0100 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: Message-ID: On 15 July 2013 15:32, Nick Coghlan wrote: > Most of the stuff Brett mentioned there shouldn't be relevant for a > directly executed script - doing > sys.path.remove(os.path.dirname(os.path.abspath(__file__)) should be > pretty robust in any scenario. > OK, well apart from the shadowing issue, my initial tests have looked relatively positive. So the questions are now more around whether this is how we want to go with pip, what backward compatibility issues it may have (the launcher isn't available in Python < 3.3) etc. So I'll work up a pull request for discussion by the pip devs. > > I'm amazed actually that there's no way to say > > "don't add the script location to sys.path", even as a command line > option. > > It seems like the sort of thing you'd want to make scripts robust, a bit > > like -S and -E. > > You'd think that, but then you'd look at getpath.c and run away (or > write something like PEP 432, as I did) :P > You're a better man than I, Gunga Din :-) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Mon Jul 15 17:21:29 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 15 Jul 2013 16:21:29 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> Message-ID: On 15 July 2013 15:21, Paul Moore wrote: > > On 15 July 2013 15:16, Oscar Benjamin wrote: >> >> I don't know if I really count as a normal user but I can describe how >> Python is installed on the Windows machines in my faculty for >> scientific use. > > Thanks, that's interesting. > > Do people typically write command-line Python scripts? A lot of researchers would run their scripts in an IDE such as Spyder (pre-installed by Enthought). This is the way that people are used to working if they are more familiar with Matlab. It's a bad idea in may ways but essentially rather than passing command line arguments a lot of people will just edit the variables at the top of their script and rerun it. Another method used is ipython which you can use to edit/run your code in a semi-interactive/semi-manual manner using the magic %edit command; this is similar to spyder. We also have a number of Linux clusters that are used to farm out big computational jobs and for this people do need to write proper command line scripts and submit the jobs via ssh (using putty rather than a real terminal) but they probably edit/test this code on the target machines. > If so, do they expect to be able to put them on PATH and run them? Probably not. I think that most people make a folder full of scripts and either run them from an IDE or cd into the folder and run them there. Again this is basically how you would do it in Matlab. In scientific work the end user is someone who spends a lot of time writing quite small scripts that are often not really reusable and are tied in some sense to a wider project. I think that maybe 80% of the .py files I have written are command line scripts under 100 lines that produce a single figure with matplotlib. The majority of those scripts are tied to a LaTeX document and invoked by a Makefile with e.g. 'python scripts/fig1.py images/fig1.pdf'. Most of my colleagues would probably have a more manual/interactive approach than me though. > What command processor is typically used? Powershell or cmd? I haven't seen anyone use Powershell but I assume that it is installed. It's not on my machine but I use Console2/git-bash which means that shebang lines already work for me. I do often see (unfortunate) people using cmd.exe though. Oscar From alexis at notmyidea.org Mon Jul 15 18:27:26 2013 From: alexis at notmyidea.org (=?ISO-8859-1?Q?Alexis_M=E9taireau?=) Date: Mon, 15 Jul 2013 18:27:26 +0200 Subject: [Distutils] Request to add a trove classifier for pelican plugins Message-ID: <51E422EE.8060106@notmyidea.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I hope this is the right place to ask for this. I would like to have a trove classifier for pelican [0] plugins. We plan to release them on PyPI and having a classifier to distinguish them from all the other packages sounds the way to go. We're not really a framework, but following the already established pattern, I guess "Framework :: Pelican" makes sense. Thanks! Alexis [0] http://getpelican.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJR5CLuAAoJEAeIBdhn9W8SyqwH/3e64+X6KJ4WxX/zeO9isKqw F5XfeRieO2rthUd6ALREF+VYhgsIwrU6B6gwLjyDe5tFA8a0sPkXFIg7pCEXjNxt ufX7W8BhdjRcVOx//9/zP4v4+HeU9OUZBwpiNuMnE7N9jbq4iWtt3OQ0GtfVIl1h d9WJoxb+8aDGbes/jgNuTh4B/Jm9XjIm8fXP5mcyLkfj0vnyHJUTgm/GjnuYEF6K 9mtWMAMoZ6RgIS41JgI7yUt81pLBQrCJc5yVuG7lE3hdaFstKAdBZhOVSrIzj0eJ KN4TMABexVCsVfwklDPIifAneKQZEmONUJLWfzleU367kRGb5YNavrtpW/kQomY= =FomK -----END PGP SIGNATURE----- From chris.barker at noaa.gov Mon Jul 15 18:47:05 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 15 Jul 2013 09:47:05 -0700 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> Message-ID: On Sat, Jul 13, 2013 at 12:59 PM, Paul Moore wrote: >> 2. This sounds like something that needs fixed on Windows. Even if you say >> ``-m`` for pip then things are still broken by default for any other package >> on PyPI that installs a script. So this feels like something wrong with >> Python on windows not wrong with the script approach. > > It is, and it should be fixed. But in many years, nobody has managed to come > up with an acceptable solution. I don't _think_ this is just Windows Bashing: MS has done very very little to improve the whole command line experience on Windows over the years. For example, as far as I now, even with Windows 7 (8), the standard system tool to edit PATH is a very, very old little text box that only holds maybe 50 characters -- it's really painful and pathetic. That, and I think the really, really old way of editing autoexec.bat is dead (editing a text file is easier than a really lousy GUI) All that is a way to say that Python can only make it so easy for Windows users, but what's in place is not bad, and it really makes sense for pip to use what's been there for ages, i.e. a command called "pip" (and pip2, pip3...) that sits in the same place that all other third-party Pyton "scripts" are installed. No matter how you slice it, a user will need to put that on their PATH one way or another. Of course, what MS is telling us is: don't rely on the command line! So a really nice thing to do for Windows users would be to provide a little GUI pip tool that's part of the standard install. (not that I'm volunteering to write it...has no none yet written a tkInter-bsed pip front-end?) the current setuptools exe-wrapper feels really kludgy, but it works -- it seems the only real problematic issue is the self-update problem -- maybe there is a Windows guru somewhere that can fix that.... > The debates seem to be largely around what > happens if you install multiple versions of Python and then remove some of > them, and how badly your system PATH gets messed up by this. Is this any better on *nix? When I use the OS-X installers, after a while, I get a pretty klunky pile-up of PATH-manipulating stuf in my .bash_profile... > * Accept that Windows is a problem in this regard, but don't worry about it > - install executable wrappers/scripts and let the user deal with path > issues. not so bad, really > It would be nice to get feedback from "normal users" on this. I suspect that > the scientific community would make a good cross-section (AIUI there's quite > a lot of Windows use, and for many people in the community Python is very > much a tool, rather than a way of life :-)). True -- note that there are now two commercial pyton distributions (Enthought Canopy, and Continuum Anaconda) that heavily used by the scipy community -- they both provide their own package distribution solutions (though ship pip, too, i'm pretty sure). The demand for those tells us something about packaging.... > Does anyone have links into the > scipy groups? Yes, but I don't know that post from me would get you anything that that post from a core pip-developer wouldn't get -- I'd post on the numpy list for best access to developers, maybe scipy and/or matplotlib for more it's-just-a-tool-to-me users. iPython's not a bad option for folks concerned about user experience, as well. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From p.f.moore at gmail.com Mon Jul 15 19:11:30 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 15 Jul 2013 18:11:30 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> Message-ID: On 15 July 2013 17:47, Chris Barker - NOAA Federal wrote: > I don't _think_ this is just Windows Bashing: MS has done very very > little to improve the whole command line experience on Windows over > the years. > It's not MS-bashing. I agree with you, and I'm one of the least likely people around here to indulge in arbitrary MS-bashing. (With the exception of Steve Dower, I guess :-)) Powershell is a *great* step up from cmd, but there are still a lot of dodgy bits round the edges (mostly because of the whole "console vs GUI subsystems" thing). > All that is a way to say that Python can only make it so easy for > Windows users, but what's in place is not bad, and it really makes > sense for pip to use what's been there for ages, i.e. a command called > "pip" (and pip2, pip3...) that sits in the same place that all other > third-party Pyton "scripts" are installed. No matter how you slice it, > a user will need to put that on their PATH one way or another. > Agreed, PATH manipulation is a fact of life for everyone, both Unix and Windows. Of course, what MS is telling us is: don't rely on the command line! > So a really nice thing to do for Windows users would be to provide a > little GUI pip tool that's part of the standard install. (not that I'm > volunteering to write it...has no none yet written a tkInter-bsed pip > front-end?) > I don't think a GUI-based tool is the answer here - the command line is orders of magnitude more powerful. For simple cases yes, but we have bdist_wininst and bdist_msi for those, and they are clearly not enough. > the current setuptools exe-wrapper feels really kludgy, but it works > -- it seems the only real problematic issue is the self-update problem > The self-update issue is the big one. There are others (completely incomprehensible errors if the #! line in the script is wrong, for example) but it's certainly pretty much the best solution we have at the moment. Most of my problems with the setuptools wrappers are not actually with the exe, but rather with the actual script (and its dependency on pkg_resources) that lies behind it - and that's not a Windows problem per se. > -- maybe there is a Windows guru somewhere that can fix that.... > I think I'm that guru, unfortunately, and I'm not having a whole lot of luck :-) The real problem is not technical, actually - it's knowing what Windows users will actually be comfortable with. Unix users tend to assume Windows users are very uncomfortable on the command line (no offence meant to anyone by that) whereas the reality is that some are, some (like me...) really are not, and some are simply unfamiliar with the capabilities of the Windows command line through lack of need to use it (many of my colleagues, for example). I'm actually tempted to give up trying to please everyone, and just put together a solution that suits *me* and see how that flies. Second guessing what other people want makes my head hurt :-) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Jul 15 19:26:00 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 15 Jul 2013 13:26:00 -0400 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: Message-ID: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> On Jul 15, 2013, at 9:39 AM, Paul Moore wrote: > I'm looking at the possibility of replacing the current setuptools entry point based pip executables with Python scripts. The biggest problem is that a script "pip.py" shadows the pip package, making "import pip" fail. > > I can get round this by deleting sys.path[0] (the location of the currently running script) but how robust is that? Are there any corner cases where it would break? Or alternatively, is there a better way to do this rather than messing directly with sys.path? I suspect this is a fairly common question, but my Google-fu is failing me :-( > > Sorry, I know this is a basic Python coding question - in my defence, it's for something related to the current pip discussions :-) > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Maybe this is a crazy idea, but would a windows only extension work? .pye(executable) Then just associate .pye with the launcher. Python won't see .pye as importable so there's no shadow issues. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Mon Jul 15 20:23:37 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 15 Jul 2013 19:23:37 +0100 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> References: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> Message-ID: On 15 July 2013 18:26, Donald Stufft wrote: > Maybe this is a crazy idea, but would a windows only extension work? > .pye(executable) Then just associate .pye with the launcher. Python won't > see .pye as importable so there's no shadow issues. That's actually a very good idea. The only downside is the proliferation of extensions involved, and the need to register them. That puts it into the territory of things the installer needs to do if we're to be able to assume it. But I may propose it to python-dev (Daniel proposed a "zipped Python app" extension a while back, as well. I'm not sure what happened with that one...) Actually, this and many of the other ideas fall foul of backward compatibility issues - we can't assume the Python launcher is available prior to Python 3.3, so #! support in .py files isn't available either. I think I'm coming to the conclusion that the best way forward is: 1. Continue using the setuptools exe launcher, but bundle our own copy. 2. Modify setup.py to install our own scripts run via the exe launcher, which don't rely on entry points and pkg_resources. 3. Special case the heck out of pip upgrading itself to ignore errors from trying to replace the exe (as long as the exe is unchanged, based on a size/checksum check) This covers the replacing-the-exe issue and the entry point script problems of vendoring setuptools. I don't *like* this option, but at least it's not going to break big chunks of our userbase... There are some other options I'd still like to explore before settling on something, for example making pip install from sdist *always* build a temporary wheel and then always install from wheels - we can then introspect the wheel before installing and catch this type of issue before starting. That lets us easily avoid the "overwriting the exe" issue, as well as letting us cleanly roll back failed installs. One thing is clear - this is a longer term effort, not a quick fix... Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From setuptools at bugs.python.org Mon Jul 15 21:16:37 2013 From: setuptools at bugs.python.org (Naftuli Tzvi Kay) Date: Mon, 15 Jul 2013 19:16:37 +0000 Subject: [Distutils] [issue155] 0.7 breaks existing packages with hyphens in 'name' Message-ID: <1373915797.18.0.489917380748.issue155@psf.upfronthosting.co.za> New submission from Naftuli Tzvi Kay: Please see this on StackOverflow for a full explanation: http://stackoverflow.com/questions/17659561/hyphens-in-project-names-in-setuptools-0-7 I have a nice little script library called [buildout-starter](https://github.com/rfkrocktk/buildout-starter) which makes creating Buildout projects really easy. Then, the latest Buildout declared a dependency on `setuptools>=0.7`, so I had to upgrade my `setuptools` here to be compliant. After the upgrade, `setuptools` now fails along with my Buildout. Whereas the following would work before 0.7, it now fails: from setuptools import setup, find_packages setup( name = "tornado-chat-example", version = "0.0.1-SNAPSHOT", packages = find_packages('src'), package_dir = { '': 'src'}, install_requires = [ 'setuptools', ], ) My `src` directory looks like this: src ??? tornadochatexample ??? tornado_chat_example.egg-info Here's the error I get: Develop: '/home/naftuli/tornado-chat-example/.' Installing python_section. Couldn't find index page for 'tornadochatexample' (maybe misspelled?) Getting distribution for 'tornadochatexample'. Couldn't find index page for 'tornadochatexample' (maybe misspelled?) While: Installing python_section. Getting distribution for 'tornadochatexample'. Error: Couldn't find a distribution for 'tornadochatexample'. Like I mentioned before, this example would seemingly run on `setuptools` 0.6, but now fails in the latest `setuptools` 0.7. How can I get this to work? I'd like my project to be named `tornado-chat-example` but have a package of `tornadochatexample`. Is this a bug, as it used to work before? ---------- messages: 739 nosy: rfkrocktk priority: bug status: unread title: 0.7 breaks existing packages with hyphens in 'name' _______________________________________________ Setuptools tracker _______________________________________________ From oscar.j.benjamin at gmail.com Mon Jul 15 21:47:23 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 15 Jul 2013 20:47:23 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> Message-ID: On 15 July 2013 18:11, Paul Moore wrote: > The real problem is not technical, actually - it's knowing what Windows > users will actually be comfortable with. Unix users tend to assume Windows > users are very uncomfortable on the command line (no offence meant to anyone > by that) whereas the reality is that some are, some (like me...) really are > not, and some are simply unfamiliar with the capabilities of the Windows > command line through lack of need to use it (many of my colleagues, for > example). There's another significant group of users including myself who use the command line extensively but infect every Windows machine they touch with unixy tools. I use Console2 as my terminal GUI and git-bash (msys bash) as the shell. I don't really get on with it but a lot of other people use Cygwin for similar reasons. One consequence of this setup is that git-bash considers any file with a shebang to be executable and also .exe files but not .bat files. I commonly write a python script with a shebang (and no .py extension) and place a bat file with the same name next to it if, for example, I need to invoke it from something that doesn't understand my unixy setup (e.g. os.system). PATHEXT wouldn't solve the problem for me since the .py extension would still be there when I invoke the script from git-bash. > I'm actually tempted to give up trying to please everyone, and just put > together a solution that suits *me* and see how that flies. Second guessing > what other people want makes my head hurt :-) That's probably a good strategy :) Oscar From chris.barker at noaa.gov Tue Jul 16 00:09:31 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 15 Jul 2013 15:09:31 -0700 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> Message-ID: On Mon, Jul 15, 2013 at 10:11 AM, Paul Moore wrote: > of Steve Dower, I guess :-)) Powershell is a *great* step up from cmd, could we use a powershell script to launch python scripts? Maybe it wouldn't be any easier to update than an exe, but it might be more accessible. >> Of course, what MS is telling us is: don't rely on the command line! >> So a really nice thing to do for Windows users would be to provide a >> little GUI pip tool that's part of the standard install. (not that I'm >> volunteering to write it...has no none yet written a tkInter-bsed pip >> front-end?) > I don't think a GUI-based tool is the answer here - the command line is > orders of magnitude more powerful. For simple cases yes, but we have > bdist_wininst and bdist_msi for those, and they are clearly not enough. but they are really widely used -- maybe when binary wheels become ubiqitous, I'll stop using them, but I'm no command line phobic, and I usually go first to look for an installer on Windows. Over in the Mac world, we have similar issues (except a proper command line under there if you want it...), and eggs were a real issue because there was nothing to launch if you point and clicked on one. That being said, you're only going to get so far programmin python if you can't run a simple command on the command line. So maybe any GUI front-end should be part of a larger tool -- perhaps provided by IDE developers, for instance. > Most > of my problems with the setuptools wrappers are not actually with the exe, > but rather with the actual script (and its dependency on pkg_resources) that > lies behind it - and that's not a Windows problem per se. I've never liked pkg_resources..... ;-) > The real problem is not technical, actually - it's knowing what Windows > users will actually be comfortable with. Unix users tend to assume Windows > users are very uncomfortable on the command line (no offence meant to anyone > by that) whereas the reality is that some are, some (like me...) really are > not, and some are simply unfamiliar with the capabilities of the Windows > command line through lack of need to use it (many of my colleagues, for > example). true -- and a simple command line solution is fine for most -- as I said, they'll need to deal with that one way or another if they are going to program..(and I say this as an instructor of intro to pyton classes...) > I'm actually tempted to give up trying to please everyone, and just put > together a solution that suits *me* and see how that flies. Second guessing > what other people want makes my head hurt :-) fine plan! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From vinay_sajip at yahoo.co.uk Tue Jul 16 00:21:56 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 15 Jul 2013 22:21:56 +0000 (UTC) Subject: [Distutils] Replacing pip.exe with a Python script References: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> Message-ID: Paul Moore gmail.com> writes: > I think I'm coming to the conclusion that the best way forward is: > 1. Continue using the setuptools exe launcher, but bundle our own copy. Wouldn't that be the case with a bundled setuptools/pip anyway? The launcher executables are part of setuptools AFAIK. > 2. Modify setup.py to install our own scripts run via the exe launcher, which don't rely on entry points and pkg_resources. FYI the distlib executable launchers (not based on setuptools, but work the same way) and the distlib script generation approach does not require distlib to run. You might be able to make use of that in some way :-) Regards, Vinay Sajip From p.f.moore at gmail.com Tue Jul 16 00:22:26 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 15 Jul 2013 23:22:26 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> Message-ID: On 15 July 2013 23:09, Chris Barker - NOAA Federal wrote: > > For simple cases yes, but we have > > bdist_wininst and bdist_msi for those, and they are clearly not enough. > > but they are really widely used -- maybe when binary wheels become > ubiqitous, I'll stop using them, but I'm no command line phobic, and I > usually go first to look for an installer on Windows. The killer issue with bdist_wininst and bdist_msi is that they don't work with virtualenvs. I was a fan of them till I started using virtualenv, at which point they become totally useless :-( I'm not against someone writing a GUI. But it won't be me :-) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Jul 16 00:24:40 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 15 Jul 2013 18:24:40 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> Message-ID: <7D576421-E423-4FD5-A39F-BF3072A5B362@stufft.io> On Jul 15, 2013, at 6:22 PM, Paul Moore wrote: > > On 15 July 2013 23:09, Chris Barker - NOAA Federal wrote: > > For simple cases yes, but we have > > bdist_wininst and bdist_msi for those, and they are clearly not enough. > > but they are really widely used -- maybe when binary wheels become > ubiqitous, I'll stop using them, but I'm no command line phobic, and I > usually go first to look for an installer on Windows. > > The killer issue with bdist_wininst and bdist_msi is that they don't work with virtualenvs. I was a fan of them till I started using virtualenv, at which point they become totally useless :-( > > I'm not against someone writing a GUI. But it won't be me :-) > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig There is something like 200 total bdist_msi on PyPI and 5k bdist_wininst. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Tue Jul 16 00:28:41 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 15 Jul 2013 18:28:41 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: <7D576421-E423-4FD5-A39F-BF3072A5B362@stufft.io> References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> <7D576421-E423-4FD5-A39F-BF3072A5B362@stufft.io> Message-ID: On Jul 15, 2013, at 6:24 PM, Donald Stufft wrote: > > On Jul 15, 2013, at 6:22 PM, Paul Moore wrote: > >> >> On 15 July 2013 23:09, Chris Barker - NOAA Federal wrote: >> > For simple cases yes, but we have >> > bdist_wininst and bdist_msi for those, and they are clearly not enough. >> >> but they are really widely used -- maybe when binary wheels become >> ubiqitous, I'll stop using them, but I'm no command line phobic, and I >> usually go first to look for an installer on Windows. >> >> The killer issue with bdist_wininst and bdist_msi is that they don't work with virtualenvs. I was a fan of them till I started using virtualenv, at which point they become totally useless :-( >> >> I'm not against someone writing a GUI. But it won't be me :-) >> >> Paul >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > There is something like 200 total bdist_msi on PyPI and 5k bdist_wininst. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig To put numbers into perspective, there are ~180k total files uploaded to PyPI. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From chris.barker at noaa.gov Tue Jul 16 00:39:07 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 15 Jul 2013 15:39:07 -0700 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> <7D576421-E423-4FD5-A39F-BF3072A5B362@stufft.io> Message-ID: On Mon, Jul 15, 2013 at 3:28 PM, Donald Stufft wrote: > There is something like 200 total bdist_msi on PyPI and 5k bdist_wininst. > To put numbers into perspective, there are ~180k total files uploaded to > PyPI. I don't hink this means that the installers aren't widely used, I think it mean they aren't distributed on PyPI. Installers are really useful for packages that require compiled code that depends on external libs -- and most of the major such package maintainers provide them. Also, numbers aren't as important as the handful of widely used, but hard to build, packages.... But they are useless with virtualenv, so I'm looking forward to binary wheels... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oscar.j.benjamin at gmail.com Tue Jul 16 01:12:55 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 16 Jul 2013 00:12:55 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> <7D576421-E423-4FD5-A39F-BF3072A5B362@stufft.io> Message-ID: On 15 July 2013 23:39, Chris Barker - NOAA Federal wrote: > On Mon, Jul 15, 2013 at 3:28 PM, Donald Stufft wrote: >> There is something like 200 total bdist_msi on PyPI and 5k bdist_wininst. > >> To put numbers into perspective, there are ~180k total files uploaded to >> PyPI. > > I don't hink this means that the installers aren't widely used, I > think it mean they aren't distributed on PyPI. > > Installers are really useful for packages that require compiled code > that depends on external libs -- and most of the major such package > maintainers provide them. I second this. I use pip all the time for pure Python packages on Linux and Windows because it works very well for these. However when it comes to numpy, matplotlib, wxpython, PyQT4 et al. I wouldn't even attempt to use pip on Windows. Most commonly I would do the standard Windows thing of going to the project website and looking for the downloads page. I've also used Christian Gohlke's index of science-related Windows binaries [1] which are supplied as .exe files. He says that "Most binaries are built from source code found on PyPI..." or in other words if it were easy to build these with pip then his index would be unnecessary. When wheel distribution becomes common I hope that this situation will improve substantially though. Oscar [1] http://www.lfd.uci.edu/~gohlke/pythonlibs/ From ncoghlan at gmail.com Tue Jul 16 01:33:48 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 09:33:48 +1000 Subject: [Distutils] Call for PEP author/champion: Bundling pip with CPython installers In-Reply-To: References: Message-ID: On 15 Jul 2013 01:10, "Paul Nasrat" wrote: > > > > > On 14 July 2013 02:13, Nick Coghlan wrote: >> >> Based on the recent discussions, I now plan to reject the pip bootstrapping-on-first-invocation approach described in PEP 439 in favour of a new PEP that proposes: >> >> * bundling the latest version of pip with the CPython binary installers for Mac OS X and Windows for all future CPython releases (including maintenance releases) >> * aligns the proposal with the Python 3.4 release schedule by noting that CPython changes must be completed by the first 3.4 beta, while pip changes must be completed by the first 3.4 release candidate. >> * ensuring that, for Python 3.4, "python3" and "python3.4" are available for command line invocation of Python, even on Windows >> * ensuring that the bundled pip, for Python 3.4, ensures "pip", "pip3" and "pip3.4" are available for command line invocation of Python, even on Windows >> * ensuring that the bundled pip is able to upgrade/downgrade itself independent of the CPython release cycle >> * ensuring that pip is automatically available in virtual environments created with pyvenv >> * adding a "get-pip.py" script to Tools/scripts in the CPython repo for bootstrapping the latest pip release from PyPI into custom CPython source builds >> >> Note that there are still open questions to be resolved, which is why an author/champion is needed: > > > I've a bunch of contacts in various distros still - I've not championed a PEP before but I would be happy to take this on. Thanks, Paul, that sounds great. Once we have something written up I can run it by Fedora's python-devel list, too. Probably the easiest way to get started is to grab the PEP 439 source from https://hg.python.org/peps, file the serial numbers off and edit that into a proposal for bundling pip with the installers rather than using runtime bootstrapping. PEP 1 has more general guidance on the PEP process (although in this case feel free to send updates directly to me for posting). >> >> * what guidance, if any, should we provide to Linux distro packagers? >> >> * how should maintenance updates handle the presence of an existing pip installation? > > > There are various distro packaging specific ways of handling this. Hard requirements, recommends, obsoleting the standalone package and providing it virtually as part of I suspect we'll end up being fairly agnostic on the Linux details, and merely make it clear that at the very least "pip install --user " should be readily available. > >> Automatically upgrade older versions to the bundled version, while leaving newer versions alone? Force installation of the bundled version? > > > My personal experience is that forcing the bundled version to OS with strong in-built packaging (Linux, BSD, other *NIX) is likely to meet with some resistance. I can certainly talk with some people, my instinct is it's likely to be only bundle with installers, allow optional install as part of the cPython build which can then be subpackaged/ignored for seperate pip/bundled as distros so desire. Yes, you can take all my bundling comments as relating specifically to the Windows and Mac OS X installers we provide. Cheers, Nick. > > Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Jul 16 06:31:59 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 15 Jul 2013 21:31:59 -0700 Subject: [Distutils] pip and virtualenv release candidates In-Reply-To: References: Message-ID: pip-1.4rc5 and virtualenv-1.10rc8 are now available the changes from the previous RCs: - Applied security patch to pip's ssl support related to certificate DNS wildcard matching (http://bugs.python.org/issue17980) - Fixed index header processing bug: https://github.com/pypa/pip/pull/1047 here's the RC install instructions again: $ curl -L -O https://github.com/pypa/virtualenv/archive/1.10rc8.tar.gz $ tar zxf 1.10rc8.tar.gz $ python virtualenv-1.10rc8/virtualenv.py myVE $ myVE/bin/pip --version pip 1.4rc5 -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Tue Jul 16 11:08:10 2013 From: holger at merlinux.eu (holger krekel) Date: Tue, 16 Jul 2013 09:08:10 +0000 Subject: [Distutils] devpi-0.9.3: new list/remove commands, bugfixes Message-ID: <20130716090810.GK3125@merlinux.eu> I just released new versions of the devpi system, which provides a self-updating pypi caching and index server and a ``devpi`` command line tool to help with common upload/test/release activities. devpi-0.9.3 comes with some bug fixes and two new sub commands to view and remove release files from a private index. For general docs see: http://doc.devpi.net and for the changelog see below. Special thanks to Anthon van der Neut for his contributions, in particular the "argcomplete" support allowing for completion on options and subcommands. best, holger 0.9.3 ---------------------------- server: - fixed issue9: caching of packages where upstream provides no last-modified header now works. - fixed issue8: only http/https archives are allowed and other schemes (such as ftp) are silently skipped - added support for REST DELETE methods of projects and versions on an index - added "argcomplete" support for tab completion on options (thanks to Anthon van der Neut) client: - new "devpi list" command to show projects of the in-use index or all release files of a project with "devpi list PROJECTNAME". - new "devpi remove" command to remove releases from the current index, including any contained release files - added "argcomplete" support for tab completion on options (thanks to Anthon van der Neut) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: Digital signature URL: From holger at merlinux.eu Tue Jul 16 11:19:00 2013 From: holger at merlinux.eu (holger krekel) Date: Tue, 16 Jul 2013 09:19:00 +0000 Subject: [Distutils] vetting, signing, verification of release files Message-ID: <20130716091900.GL3125@merlinux.eu> I am considering implementing gpg-signing and verification of release files for devpi. Rather than requiring package authors to sign their release files, i am pondering a scheme where anyone can vet for a particular published release file by publishing a signature about it. This aims to help responsible companies to work together. I've heart from devops/admins that they manually download and check release files and then install it offline after some vetting. Wouldn't it be useful to turn this into a more collaborative effort? Any thoughts or pointers to existing efforts within the (Python) packaging ecologies? best, holger From jannis at leidel.info Tue Jul 16 12:21:41 2013 From: jannis at leidel.info (Jannis Leidel) Date: Tue, 16 Jul 2013 12:21:41 +0200 Subject: [Distutils] vetting, signing, verification of release files In-Reply-To: <20130716091900.GL3125@merlinux.eu> References: <20130716091900.GL3125@merlinux.eu> Message-ID: <8AD857B0-F41E-448C-B639-B0EE033E5E7E@leidel.info> On 16.07.2013, at 11:19, holger krekel wrote: > Any thoughts or pointers to existing efforts within the (Python) > packaging ecologies? Erik Rose just released peep the other day [1], which admittedly doesn't use gpg but at least allows pip users to simplify the manual vetting process. Jannis 1: https://pypi.python.org/pypi/peep From p.f.moore at gmail.com Tue Jul 16 12:28:19 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Jul 2013 11:28:19 +0100 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> Message-ID: On 15 July 2013 23:21, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > > > I think I'm coming to the conclusion that the best way forward is: > > 1. Continue using the setuptools exe launcher, but bundle our own copy. > > Wouldn't that be the case with a bundled setuptools/pip anyway? The > launcher > executables are part of setuptools AFAIK. > > > 2. Modify setup.py to install our own scripts run via the exe launcher, > which don't rely on entry points and pkg_resources. > > FYI the distlib executable launchers (not based on setuptools, but work the > same way) and the distlib script generation approach does not require > distlib to run. You might be able to make use of that in some way :-) Yes, the physical executables aren't that much of an issue - grabbing them from somewhere and bundling them is easy enough and as you say we may already have them (I actually thought distlib included the exes, but the version bundled with pip at the moment doesn't have them). The fun bit is having to modify setuptools to do our own script wrapper management, because setuptools doesn't let us customise its process to remove the runtime dependency on a top-level pkg_resources. Having to make project-specific customisations to distutils feels like we're going in precisely the opposite direction from what the whole packaging process is trying to achieve, and it makes me feel vaguely unclean having to consider it :-) Two thoughts for the wider audience: 1. Should pip re-vendor a newer version of distlib, so we have the exe wrappers? We currently have 0.1.1 and 0.1.2 is on PyPI. 2. Would writing a distutils extension class in setup.py to make the exe wrappers using the vendored distlib.scripts package be acceptable to remove the runtime dependency on pkg_resources from the wrappers? Note: This is just one relatively small step towards removing some of our dependencies on an external setuptools. It's not the whole story, and it still leaves the "upgrading a running exe wrapper" problem to address. (This discussion may be worth migrating to pypa-dev, I'm not sure how much the wider distutils audience cares about this level of detail - feel free to switch lists anyone who thinks it's appropriate). Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Jul 16 12:37:21 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Jul 2013 11:37:21 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> <7D576421-E423-4FD5-A39F-BF3072A5B362@stufft.io> Message-ID: On 16 July 2013 00:12, Oscar Benjamin wrote: > On 15 July 2013 23:39, Chris Barker - NOAA Federal > wrote: > > On Mon, Jul 15, 2013 at 3:28 PM, Donald Stufft wrote: > >> There is something like 200 total bdist_msi on PyPI and 5k > bdist_wininst. > > > >> To put numbers into perspective, there are ~180k total files uploaded to > >> PyPI. > > > > I don't hink this means that the installers aren't widely used, I > > think it mean they aren't distributed on PyPI. > > > > Installers are really useful for packages that require compiled code > > that depends on external libs -- and most of the major such package > > maintainers provide them. > > I second this. I use pip all the time for pure Python packages on > Linux and Windows because it works very well for these. However when > it comes to numpy, matplotlib, wxpython, PyQT4 et al. I wouldn't even > attempt to use pip on Windows. > > Most commonly I would do the standard Windows thing of going to the > project website and looking for the downloads page. I've also used > Christian Gohlke's index of science-related Windows binaries [1] which > are supplied as .exe files. He says that "Most binaries are built from > source code found on PyPI..." or in other words if it were easy to > build these with pip then his index would be unnecessary. > > When wheel distribution becomes common I hope that this situation will > improve substantially though. Precisely. At the moment, unless you need to compile code with external dependencies, pip install works fine (it's even fine for C code without dependencies if you have a compiler). But once the build process is even slightly complex, wininst installers are the only real answer. The fact that they don't work on virtualenvs is a pain, but there are 2 ways round this: 1. I believe that easy_install will install wininst installers. I've not tried myself. 2. You can use wheel convert to make wheels out of wininsts, and then pip install those. This is what I do, and it's really effective. I keep a local index of converted wheels to limit the download/convert overhead. I'd like to see more wheels on PyPI and sites like Christoph Gohlke's move to providing wheels, and preferably in a PyPI index style format, so people can pip install *anything*. But obviously that's the authors' (and people like Christoph's) choice. MSI is a lousy format in this context, because it's near-impossible to introspect, and hence convert to a wheel or anything similar. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Tue Jul 16 12:38:03 2013 From: christian at python.org (Christian Heimes) Date: Tue, 16 Jul 2013 12:38:03 +0200 Subject: [Distutils] vetting, signing, verification of release files In-Reply-To: <8AD857B0-F41E-448C-B639-B0EE033E5E7E@leidel.info> References: <20130716091900.GL3125@merlinux.eu> <8AD857B0-F41E-448C-B639-B0EE033E5E7E@leidel.info> Message-ID: <51E5228B.3000904@python.org> Am 16.07.2013 12:21, schrieb Jannis Leidel: > On 16.07.2013, at 11:19, holger krekel wrote: > >> Any thoughts or pointers to existing efforts within the (Python) >> packaging ecologies? > > Erik Rose just released peep the other day [1], which admittedly doesn't use gpg but at least allows pip users to simplify the manual vetting process. Peep is a bit scary because the author doesn't have much confidence in his own crypto fu: "Proof of concept. Does all the crypto stuff. Should be secure." Peep doesn't protect you from at least on DoS attack scenario. The tool does neither verify nor limit the size of a downloaded file. In theory an active attacker could make you download an arbitrarily large file in order to clog your network pipes. Eventually your machine runs out of disk space, too. I'd feel much better if such a tool would verify both hashsum and file size. Christian From oscar.j.benjamin at gmail.com Tue Jul 16 13:04:39 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 16 Jul 2013 12:04:39 +0100 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> Message-ID: On 16 July 2013 11:28, Paul Moore wrote: > Two thoughts for the wider audience: > > 1. Should pip re-vendor a newer version of distlib, so we have the exe > wrappers? We currently have 0.1.1 and 0.1.2 is on PyPI. In what way would that affect anyone? > 2. Would writing a distutils extension class in setup.py to make the exe > wrappers using the vendored distlib.scripts package be acceptable to remove > the runtime dependency on pkg_resources from the wrappers? Does that mean that an end user would need a C compiler in a situation where they previously didn't? Oscar From holger at merlinux.eu Tue Jul 16 13:17:12 2013 From: holger at merlinux.eu (holger krekel) Date: Tue, 16 Jul 2013 11:17:12 +0000 Subject: [Distutils] vetting, signing, verification of release files In-Reply-To: <8AD857B0-F41E-448C-B639-B0EE033E5E7E@leidel.info> References: <20130716091900.GL3125@merlinux.eu> <8AD857B0-F41E-448C-B639-B0EE033E5E7E@leidel.info> Message-ID: <20130716111712.GB1668@merlinux.eu> On Tue, Jul 16, 2013 at 12:21 +0200, Jannis Leidel wrote: > On 16.07.2013, at 11:19, holger krekel wrote: > > > Any thoughts or pointers to existing efforts within the (Python) > > packaging ecologies? > > Erik Rose just released peep the other day [1], which admittedly doesn't use gpg but at least allows pip users to simplify the manual vetting process. > > Jannis > > 1: https://pypi.python.org/pypi/peep thanks for the pointer, i actually saw that earlier. If i see it correctly it does not target "vetting sharing": if a 1000 careful people want to install Django-1.5.1.tar.gz they each need to do the verification work individually, each creating their particular "requirements.txt" with extra hashes. best, holger From p.f.moore at gmail.com Tue Jul 16 13:25:15 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Jul 2013 12:25:15 +0100 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> Message-ID: On 16 July 2013 12:04, Oscar Benjamin wrote: > On 16 July 2013 11:28, Paul Moore wrote: > > Two thoughts for the wider audience: > > > > 1. Should pip re-vendor a newer version of distlib, so we have the exe > > wrappers? We currently have 0.1.1 and 0.1.2 is on PyPI. > > In what way would that affect anyone? > Sorry, you're right - that's really for the pip developers. > > > 2. Would writing a distutils extension class in setup.py to make the exe > > wrappers using the vendored distlib.scripts package be acceptable to > remove > > the runtime dependency on pkg_resources from the wrappers? > > Does that mean that an end user would need a C compiler in a situation > where they previously didn't? > I don't believe so - distlib bundles the compiled code. On the other hand, I'm missing something, as I don't see how the *current* exe wrappers avoid meaning that there need to be separate 32-bit and 64-bit versions of pip... Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Tue Jul 16 13:42:31 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 16 Jul 2013 12:42:31 +0100 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> Message-ID: On 16 July 2013 12:25, Paul Moore wrote: >> >> > 2. Would writing a distutils extension class in setup.py to make the exe >> > wrappers using the vendored distlib.scripts package be acceptable to >> > remove >> > the runtime dependency on pkg_resources from the wrappers? >> >> Does that mean that an end user would need a C compiler in a situation >> where they previously didn't? > > I don't believe so - distlib bundles the compiled code. > > On the other hand, I'm missing something, as I don't see how the *current* > exe wrappers avoid meaning that there need to be separate 32-bit and 64-bit > versions of pip... I thought that 64 bit Windows could run 32 bit Windows .exe files (although I don't have a way to check this). Oscar From p.f.moore at gmail.com Tue Jul 16 14:23:02 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Jul 2013 13:23:02 +0100 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> Message-ID: On 16 July 2013 12:42, Oscar Benjamin wrote: > I thought that 64 bit Windows could run 32 bit Windows .exe files > (although I don't have a way to check this). > Yes, but there are 32-bit and 64-bit exe wrappers, which I suspect is because a 32-bit exe can't load a 64-bit DLL (and may be vice versa). As I said, I don't know for sure at the moment, but it needs investigating. Grumble. Next time the label on the can says "worms" I need to leave the can opener alone :-) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Tue Jul 16 14:42:06 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 16 Jul 2013 14:42:06 +0200 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> Message-ID: <1F12F353-7907-474E-9CAD-11D36A0A463A@mac.com> On 16 Jul, 2013, at 13:25, Paul Moore wrote: > > On the other hand, I'm missing something, as I don't see how the *current* exe wrappers avoid meaning that there need to be separate 32-bit and 64-bit versions of pip... Couldn't you just ship both variants of the exe wrappers in a single distribution and then use the correct one for the current installation? That's what I'm doing in py2app. Ronald From ncoghlan at gmail.com Tue Jul 16 14:39:37 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 22:39:37 +1000 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> Message-ID: On 16 July 2013 22:23, Paul Moore wrote: > > On 16 July 2013 12:42, Oscar Benjamin wrote: >> >> I thought that 64 bit Windows could run 32 bit Windows .exe files >> (although I don't have a way to check this). > > > Yes, but there are 32-bit and 64-bit exe wrappers, which I suspect is > because a 32-bit exe can't load a 64-bit DLL (and may be vice versa). As I > said, I don't know for sure at the moment, but it needs investigating. > > Grumble. Next time the label on the can says "worms" I need to leave the can > opener alone :-) I, for one, am happy you opened it now rather than in a few months time! If Paul Nasrat takes up the challenge of documenting all the practical issues associated with bundling pip with the CPython binary installers as a PEP, I expect he will need to work closely with you to capture the details on the Windows side of things. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Tue Jul 16 15:08:43 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Jul 2013 14:08:43 +0100 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) Message-ID: On 16 July 2013 13:42, Ronald Oussoren wrote: > > On the other hand, I'm missing something, as I don't see how the > *current* exe wrappers avoid meaning that there need to be separate 32-bit > and 64-bit versions of pip... > > Couldn't you just ship both variants of the exe wrappers in a single > distribution and then use the correct one for the current installation? > That's what I'm doing in py2app. That's OK for source-style installs (which is what setuptools does, and what pip mostly cares about right now). But for bundling with Python it needs to be considered (although it's just getting the right one in the right installer). But for wheels it's a pain, because instead of having just a single pip wheel, we need 32-bit and 64-bit wheels solely for the wrappers. That sucks. Hard. (And it's not just for pip, nose will have the same problem, as will many other projects that use exe wrappers). And the bdist_wheel command currently doesn't recognise this *at all*. So wheels using wrappers are potentially broken. I think the correct solution is to explicitly have declarative support for "console script entry point" metadata in PEP 426, as well as having tools like bdist_wheel and distil do some explicit backward compatibility hacking to remove legacy-style exe wrappers. The wheel install code should then explicitly install appropriate wrappers for the target platform (which may be exe wrappers similar to the current ones, but moving forward may be some other mechanism if one is found). This is complex enough that I'm now concerned that we need reference "wheel install" code in the stdlib, just so that people don't make up their own on the basis that "wheel is simple to install manually" and screw it up. Also so that we only have one style of command line script wrapper to deal with going forward, not a multitude of mostly-compatible solutions. Nick: See the above point re PEP 426 - do you agree that this needs addressing in Metadata 2.0? Paul PS There is still the proviso that I haven't tested my assumption that the separate 32 and 64 bit wrappers are *needed* (setuptools and distlib use them, so I think it's a valid assumption, but I need to test). I will try to get time to check that ASAP. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Jul 16 15:21:17 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 23:21:17 +1000 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: Message-ID: On 16 July 2013 23:08, Paul Moore wrote: > Nick: See the above point re PEP 426 - do you agree that this needs > addressing in Metadata 2.0? I believe Daniel already covered it in PEP 427 - rather than baking the entry point wrappers into the wheel, installers can generate any needed entry point wrappers if the wheel includes Python scripts in {distribution}-{version}.data/scripts/ (see http://www.python.org/dev/peps/pep-0427/#recommended-installer-features) Now, there may be holes in that scheme, but it seemed solid enough when I approved the PEP. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Tue Jul 16 15:29:01 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Jul 2013 14:29:01 +0100 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: Message-ID: On 16 July 2013 14:21, Nick Coghlan wrote: > On 16 July 2013 23:08, Paul Moore wrote: > > Nick: See the above point re PEP 426 - do you agree that this needs > > addressing in Metadata 2.0? > > I believe Daniel already covered it in PEP 427 - rather than baking > the entry point wrappers into the wheel, installers can generate any > needed entry point wrappers if the wheel includes Python scripts in > {distribution}-{version}.data/scripts/ (see > http://www.python.org/dev/peps/pep-0427/#recommended-installer-features) > > Now, there may be holes in that scheme, but it seemed solid enough > when I approved the PEP. > The big problem is that implementations don't do that :-( AFAIK, none of distlib, pip or wheel itself do anything with script wrappers except rewrite #! lines (which is the other, much easier, item in that section). At the moment, bdist_wheel just puts the exe wrappers generated from the source into the wheel itself, which again is probably wrong in the context what the PEP suggests. As I said in my other email, I think this is subtle enough that we need a stdlib implementation to stop people making mistakes like this. It's certainly not fair to expect a mostly-Unix development team to get this sort of Windows arcana right without some help. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Tue Jul 16 15:30:56 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 16 Jul 2013 15:30:56 +0200 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: Message-ID: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> On 16 Jul, 2013, at 15:08, Paul Moore wrote: > On 16 July 2013 13:42, Ronald Oussoren wrote: > > On the other hand, I'm missing something, as I don't see how the *current* exe wrappers avoid meaning that there need to be separate 32-bit and 64-bit versions of pip... > > Couldn't you just ship both variants of the exe wrappers in a single distribution and then use the correct one for the current installation? That's what I'm doing in py2app. > > That's OK for source-style installs (which is what setuptools does, and what pip mostly cares about right now). But for bundling with Python it needs to be considered (although it's just getting the right one in the right installer). But for wheels it's a pain, because instead of having just a single pip wheel, we need 32-bit and 64-bit wheels solely for the wrappers. That sucks. Hard. (And it's not just for pip, nose will have the same problem, as will many other projects that use exe wrappers). And the bdist_wheel command currently doesn't recognise this *at all*. So wheels using wrappers are potentially broken. You could just have a wheel that contains two data files: wrapper-win32.exe and wrapper-win64.exe, then select the one that gets used as the wrapper for a specific script when you create that wrapper. That's assuming that the wrapper .exe gets "created" when a wheel is installed and is not included in the wheel. > > I think the correct solution is to explicitly have declarative support for "console script entry point" metadata in PEP 426, as well as having tools like bdist_wheel and distil do some explicit backward compatibility hacking to remove legacy-style exe wrappers. The wheel install code should then explicitly install appropriate wrappers for the target platform (which may be exe wrappers similar to the current ones, but moving forward may be some other mechanism if one is found). Yikes, that means my assumption is wrong. The section on "Recommended installer features" in the wheel spec[1] says that the wrapper executable should be created on installation, does pip not do this? > > This is complex enough that I'm now concerned that we need reference "wheel install" code in the stdlib, just so that people don't make up their own on the basis that "wheel is simple to install manually" and screw it up. Also so that we only have one style of command line script wrapper to deal with going forward, not a multitude of mostly-compatible solutions. I'd love to see comprehensive wheel support in the stdlib, but that may have to wait for 3.5 because the entire packaging systeem (wheels, metadata, ...) is moving forward quickly at the moment. That said, it would be nice if distutils would grow support for creating wheels and modern metadata in sdists as that would mean I could drop usage of setuptools for most of my software (for python 3.4). > > Nick: See the above point re PEP 426 - do you agree that this needs addressing in Metadata 2.0? > > Paul > > PS There is still the proviso that I haven't tested my assumption that the separate 32 and 64 bit wrappers are *needed* (setuptools and distlib use them, so I think it's a valid assumption, but I need to test). I will try to get time to check that ASAP. That depends on what the wrapper does, if it launches a regular python with the right command-line you might be able to get away with a single wrapper, if it loads python.dll and executes the script directory you do need separate wrappers for 32 and 64 bit. [1] http://www.python.org/dev/peps/pep-0427/#recommended-installer-features From p.f.moore at gmail.com Tue Jul 16 15:32:27 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Jul 2013 14:32:27 +0100 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: Message-ID: On 16 July 2013 14:08, Paul Moore wrote: > PS There is still the proviso that I haven't tested my assumption that the > separate 32 and 64 bit wrappers are *needed* (setuptools and distlib use > them, so I think it's a valid assumption, but I need to test). I will try > to get time to check that ASAP. Hmm. I just did a quick test, and then based on the results checked the setuptools source code. I can see no reason why there needs to be 32 and 64 bit launcher exes. The launchers simply use CreateProcess to launch a separate Python process using the #! line of the script. So there's no DLL loading going on, and no reason that I can see for needing separate 32 and 64 bit builds. Jason - can you shed any light on why there are separate builds for 32 and 64 bits? Actually, the launcher is essentially identical to the "py" launcher for Windows, except that it gets a script name to execute from the name of the launcher. I'm wondering whether the correct approach here would be to enhance the launcher one more time to look for a suitably named script and auto-run it if it's present (i.e. merge the wrapper functionality into the launcher). Then we have a standard wrapper that everyone can use and not reinvent their own. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Jul 16 15:38:23 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Jul 2013 14:38:23 +0100 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> Message-ID: On 16 July 2013 14:30, Ronald Oussoren wrote: > > I think the correct solution is to explicitly have declarative support > for "console script entry point" metadata in PEP 426, as well as having > tools like bdist_wheel and distil do some explicit backward compatibility > hacking to remove legacy-style exe wrappers. The wheel install code should > then explicitly install appropriate wrappers for the target platform (which > may be exe wrappers similar to the current ones, but moving forward may be > some other mechanism if one is found). > > Yikes, that means my assumption is wrong. The section on "Recommended > installer features" in the wheel spec[1] says that the wrapper executable > should be created on installation, does pip not do this? > Yes, Nick pointed me at that part of the PEP. Nobody's doing that at the moment, and exes are being added to the wheels at wheel build time, which is also wrong. That'll teach me to work from reality rather than specs :-( Daniel, Vinay, pip developers - it looks like we need to do some work in this area to make the code conform to the specs. The PEP only says this is a "recommended" feature, but frankly I think it needs to be mandatory, or script wrappers are going to be a mess we'll be dealing with for some time :-( > PS There is still the proviso that I haven't tested my assumption that > the separate 32 and 64 bit wrappers are *needed* (setuptools and distlib > use them, so I think it's a valid assumption, but I need to test). I will > try to get time to check that ASAP. > > That depends on what the wrapper does, if it launches a regular python > with the right command-line you might be able to get away with a single > wrapper, if it loads python.dll and executes the script directory you do > need separate wrappers for 32 and 64 bit. As I said in another message, looks like there's no real reason for separate wrappers. A 32-bit one should work regardless [1]. But wheels built on 64-bit systems at the moment won't work on 32-bit ones (because the wrappers will be 64-bit). [1] With the possible exception that Windows' magic shadowing of 32 and 64 bit "stuff" - the WOW64 magic that I know nothing about - could cause odd results in obscure cases. I propose ignoring this in the absence of actual bug reports :-) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Jul 16 15:40:12 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 23:40:12 +1000 Subject: [Distutils] PEP 426 updated based on last round of discussion Message-ID: I actually pushed this to python.org on the weekend but forgot to announce it on the list. The latest version of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/ It was a net deletion of content despite going into more depth in a few areas, so I'm counting that as a win for clarity :) Change details are at http://hg.python.org/peps/rev/067d3c3c1351 Significant changes: * serialisation prefix changed to "pydist". This metadata is the metadata that exists at the sdist level. Wheels and installation may add other metadata files, and PyPI will publish additional metadata extracted from the archive contents, so I decided "pymeta" didn't feel write (the name of the schema file changed as well, but I did that in a separate commit so the diff stayed readable) * all the *_may_require fields are gone (as previously discussed) * rather than "install specifiers" I went with "requirement specifiers" (install turned out not to read very well) * accordingly, the subfields of dependency specifiers are "requires", "extra" and "environment" * the abbreviated form (which has "requires" as a list) was easy enough to specify, so that's what I used. The unpacked form (where multiple entries in the same dependency list have the same extra/environment combination) is explicitly disallowed in order to encourage consistent formatting. * clarified that internal whitespace is permitted in requirement specifiers (there may be a simpler way to specify this, such as "all whitespace in requirement specifiers is ignored") * I made the change to explicitly distrust "Provides" data retrieved from a public index server, and noted that projects defining a virtual dependency should claim that name to define the default provider. * noted that Debian packagers may want to map extras to Recommended or Suggested packages * noted some possible use cases for metadata extensions * fixed and clarified various things in the reference copy of the JSON schema (it could still do with an audit against the current PEP text, though) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Jul 16 15:41:43 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 23:41:43 +1000 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: Message-ID: On 16 July 2013 23:29, Paul Moore wrote: > As I said in my other email, I think this is subtle enough that we need a > stdlib implementation to stop people making mistakes like this. It's > certainly not fair to expect a mostly-Unix development team to get this sort > of Windows arcana right without some help. Are we talking about the pip developers or python-dev, here? I think Martin and Brian feel pretty lonely, too :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Tue Jul 16 15:53:17 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 16 Jul 2013 13:53:17 +0000 (UTC) Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> Message-ID: Paul Moore gmail.com> writes: > Yes, Nick pointed me at that part of the PEP. Nobody's doing that at the > moment, and exes are being added to the wheels at wheel build time, which > is also wrong. Not true for distlib - it doesn't add .exe wrappers to wheels at build time :-) It adds them to the target directory when installing under Windows. (You can also choose not to install any wrappers.) Regards, Vinay Sajip From dholth at gmail.com Tue Jul 16 15:59:57 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 16 Jul 2013 09:59:57 -0400 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> Message-ID: Need a script that visits all the console-script entry points to regenerate the wrappers. Then there are also the non-console-scripts scripts... I consider scripts as an optional convenience, but I suppose that isn't always the case. On Tue, Jul 16, 2013 at 9:53 AM, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > >> Yes, Nick pointed me at that part of the PEP. Nobody's doing that at the >> moment, and exes are being added to the wheels at wheel build time, which >> is also wrong. > > Not true for distlib - it doesn't add .exe wrappers to wheels at build time > :-) It adds them to the target directory when installing under Windows. (You > can also choose not to install any wrappers.) > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From vinay_sajip at yahoo.co.uk Tue Jul 16 16:01:56 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 16 Jul 2013 14:01:56 +0000 (UTC) Subject: [Distutils] =?utf-8?q?Wheels_and_console_script_entry_point_wrapp?= =?utf-8?q?ers=09=28Was=3A_Replacing_pip=2Eexe_with_a_Python_script?= =?utf-8?q?=29?= References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> Message-ID: Vinay Sajip yahoo.co.uk> writes: > > Not true for distlib - it doesn't add .exe wrappers to wheels at build time > It adds them to the target directory when installing under Windows. (You > can also choose not to install any wrappers.) Sorry, some misinformation there - distlib does do this when invoked via distil. However, this can be turned off at the distlib level - I will update distil to not do this when adding wheels. Regards, Vinay Sajip From oscar.j.benjamin at gmail.com Tue Jul 16 17:09:58 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 16 Jul 2013 16:09:58 +0100 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> Message-ID: On 16 July 2013 14:38, Paul Moore wrote: > On 16 July 2013 14:30, Ronald Oussoren wrote: >> >> > I think the correct solution is to explicitly have declarative support >> > for "console script entry point" metadata in PEP 426, as well as having >> > tools like bdist_wheel and distil do some explicit backward compatibility >> > hacking to remove legacy-style exe wrappers. The wheel install code should >> > then explicitly install appropriate wrappers for the target platform (which >> > may be exe wrappers similar to the current ones, but moving forward may be >> > some other mechanism if one is found). There are many other uses for console script entry point metadata. For example, it would be good to be able to query pip/pypi in order to find out which packages supply a particular console command. One feature of Ubuntu that I really like is the way that it automatically tells you how to install any missing command: oscar:~$ pypy The program 'pypy' is currently not installed. You can install it by typing: sudo apt-get install pypy Obviously it's not as useful when the command and the package have exactly the same name :) >> >> Yikes, that means my assumption is wrong. The section on "Recommended >> installer features" in the wheel spec[1] says that the wrapper executable >> should be created on installation, does pip not do this? > > > Yes, Nick pointed me at that part of the PEP. Nobody's doing that at the > moment, and exes are being added to the wheels at wheel build time, which is > also wrong. > > That'll teach me to work from reality rather than specs :-( > > Daniel, Vinay, pip developers - it looks like we need to do some work in > this area to make the code conform to the specs. The PEP only says this is a > "recommended" feature, but frankly I think it needs to be mandatory, or > script wrappers are going to be a mess we'll be dealing with for some time > :-( I think that it should be mandatory. It should be possible for someone releasing a script via pypi to ensure that their program can be invoked after installation under a name of their choosing (assuming the user has the Python Scripts/bin directory in PATH). AFAIK the only bullet-proof way to do this on Windows is with an .exe wrapper. If you only want the program to be invokable from cmd and PowerShell* then a .bat file should be fine. Depending on file extension to invoke .py files with py.exe is subject to input/output redirection bugs on some windows systems (this is solveable when using .py in PATHEXT instead of file associations for cmd at least). However, if you also want the program name to be invokable from e.g. subprocess with shell=False or from git-bash or Cygwin or many other things then neither .bat files nor PATHEXT are sufficient. Wrapper .exes are necessary to ensure that this works properly. Oscar * I don't actually use PowerShell and cannot confirm that running .bat files works fully i.e. without screwing up sys.argv encoding or input/output redirection or anything else. From p.f.moore at gmail.com Tue Jul 16 17:12:41 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Jul 2013 16:12:41 +0100 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> Message-ID: On 16 July 2013 15:01, Vinay Sajip wrote: > Vinay Sajip yahoo.co.uk> writes: > > > > Not true for distlib - it doesn't add .exe wrappers to wheels at build > time > > It adds them to the target directory when installing under Windows. (You > > can also choose not to install any wrappers.) > > Sorry, some misinformation there - distlib does do this when invoked via > distil. However, this can be turned off at the distlib level - I will > update > distil to not do this when adding wheels. OK. That sounds good. I'm starting to become uncertain as to whether we actually have an issue here. I think: (1) Builders can put anything they like into the scripts directory. That's more or less out of our control. From what I know of distil's approach, it doesn't actually execute setup.py so it keeps a lot more control than (say) bdist_wheel, which effectively runs setup.py install to a temporary directory then bundles what it finds. But in essence, as things stand right now, the scripts directory of an arbitrary wheel could contain arbitrary stuff. (2) Wheel installers should "make Python scripts work". All that I'm aware of fix up shebang lines and add execute bits. Only distlib adds exe wrappers, I believe. Others work on Windows because the builders put exe wrappers in place (see (1)) but that has some issues. I suspect there may be some "accidental" successes on Unix where setuptools scripts do or don't get a .py extension depending on the target platform, but I have no evidence of this myself. So there's a lot of potentially platform dependent stuff in "scripts" and wheel builders don't (can't) recognise and encode this in the tags. Wheels mostly work at the moment because not many people use them cross platform. But there's potential for issues down the line. FWIW, I believe that the whole "scripts" directory as a concept is too platform-specific. The only real use for it is to expose CLI applications, and a declarative approach like setuptools console_scripts entry points would be better. So longer term I'd argue for deprecating "scripts" altogether and replacing it with some form of "CLI entry point" metadata which may be exposed as part of metadata 2.0 or 3.0, or may simply be internal metadata communicated between the builder and the installer, but not exposed at the PyPI level. Oscar's email argues for exposing it as project metadata, though, and I can see the benefit. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Jul 16 17:22:16 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Jul 2013 16:22:16 +0100 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> Message-ID: On 16 July 2013 16:09, Oscar Benjamin wrote: > If you only want the program to be invokable from cmd and PowerShell* > then a .bat file should be fine. Depending on file extension to invoke > .py files with py.exe is subject to input/output redirection bugs on > some windows systems (this is solveable when using .py in PATHEXT > instead of file associations for cmd at least). > bat files have many, many problems. The worst ones are: * Not nestable. If pip is a bat file, saying "pip install foo" from within another bat file will fail silently (control never returns to the line after the pip command). * If you interrupt them you get the obnoxious "Do you want to terminate the batch file?" prompt. If anyone suggests using bat files, I'll cry :-) However, if you also want the program name to be invokable from e.g. > subprocess with shell=False or from git-bash or Cygwin or many other > things then neither .bat files nor PATHEXT are sufficient. Wrapper > .exes are necessary to ensure that this works properly. > Yes. I have been convinced that ultimately, wrapper exes are the only "transparent" means of writing command-line applications on Windows. Because of this, I'd quite like it if wrapper functionality were added to the py launcher (most of the functionality is already present, it would probably be a pretty small change) so that we had a "one obvious way" of writing wrappers. I may try to put together a patch for CPython to this effect... Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Tue Jul 16 17:40:18 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 16 Jul 2013 16:40:18 +0100 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> Message-ID: On 16 July 2013 16:22, Paul Moore wrote: > On 16 July 2013 16:09, Oscar Benjamin wrote: > >> However, if you also want the program name to be invokable from e.g. >> subprocess with shell=False or from git-bash or Cygwin or many other >> things then neither .bat files nor PATHEXT are sufficient. Wrapper >> .exes are necessary to ensure that this works properly. > > Yes. I have been convinced that ultimately, wrapper exes are the only > "transparent" means of writing command-line applications on Windows. > > Because of this, I'd quite like it if wrapper functionality were added to > the py launcher (most of the functionality is already present, it would > probably be a pretty small change) so that we had a "one obvious way" of > writing wrappers. I may try to put together a patch for CPython to this > effect... I don't know whether or not you intend to have wrappers also work for Python 2.7 (in a third-party package perhaps) but there is a slightly subtle point to watch out for when non-ASCII characters in sys.argv come into play. Python 2.x uses GetCommandLineA and 3.x uses GetCommandLineW. A wrapper to launch 2.x should use GetCommandLineA and CreateProcessA to ensure that the 8-bit argument strings are passed through unaltered. To launch 3.x it should use the W versions. If not then the MSVC runtime (or the OS?) will convert between the 8-bit and 16-bit encodings using its own lossy routines. Oscar From vinay_sajip at yahoo.co.uk Tue Jul 16 17:51:09 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 16 Jul 2013 15:51:09 +0000 (UTC) Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> Message-ID: Paul Moore gmail.com> writes: > FWIW, I believe that the whole "scripts" directory as a concept is too > platform-specific. The only real use for it is to expose CLI applications Well, you can also expose GUI applications this way, though the applications are fewer - Qt applications could easily be cross-platform, for example. Also, you can put in scripts which are not entry-point related, which would be essential if you e.g. have existing scripts to bundle as part of an application, which could be written in other languages, say. Not common, perhaps, but not a case you want to arbitrarily restrict given that it works now. Of course, there the distributor of the package is responsible for ensuring cross-platform workability of such scripts using e.g. including .cmd files for Windows or whatever. Regards, Vinay Sajip From p.f.moore at gmail.com Tue Jul 16 18:44:54 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Jul 2013 17:44:54 +0100 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> Message-ID: On 16 July 2013 16:51, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > > > FWIW, I believe that the whole "scripts" directory as a concept is too > > platform-specific. The only real use for it is to expose CLI applications > > Well, you can also expose GUI applications this way, though the > applications > are fewer - Qt applications could easily be cross-platform, for example. > Also, you can put in scripts which are not entry-point related, which would > be essential if you e.g. have existing scripts to bundle as part of an > application, which could be written in other languages, say. Not common, > perhaps, but not a case you want to arbitrarily restrict given that it > works > now. Of course, there the distributor of the package is responsible for > ensuring cross-platform workability of such scripts using e.g. including > .cmd files for Windows or whatever. You're right, I should have included GUI apps. I'm not aware of any cases of scripts that aren't entry-point related (and which couldn't be converted to entry points fairly easily) although that certainly doesn't mean there aren't any. OTOH, I do know that there used to be a *lot* of examples of scripts doing what entry points do that were either absolutely not cross-platform (Unix shell scripts, bat files) or were problematic/badly implemented (Python scripts without a py extension, etc) Many of these have now gone, thank goodness, migrated to setuptools entry points. If I were writing a firm proposal, I'd go for something like entry points as metadata, managed by installers, as the primary interface (with backward compatibility code to automatically migrate setuptools entry points so the impact on developers is minimal) and the scripts directory/setup argument being solely for backward compatibility, no management by installers at all, whatever is in there just gets dumped onto the target unchanged (and the contents of scripts is explicitly defined as *not* affecting the compatibility tags). So setuptools users get new entry point metadata automatically, which is processed at install time, and anyone still using the distutils scripts parameter still works as before but gets no particular support from the new tools. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Jul 16 19:34:41 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 16 Jul 2013 13:34:41 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <8BF0BD6D-93A0-4478-B8D5-F810F128A415@stufft.io> <7D576421-E423-4FD5-A39F-BF3072A5B362@stufft.io> Message-ID: <99D5AEF3-045C-4F04-9378-5EE4B6F48F21@stufft.io> On Jul 15, 2013, at 6:39 PM, Chris Barker - NOAA Federal wrote: > On Mon, Jul 15, 2013 at 3:28 PM, Donald Stufft wrote: >> There is something like 200 total bdist_msi on PyPI and 5k bdist_wininst. > >> To put numbers into perspective, there are ~180k total files uploaded to >> PyPI. > > I don't hink this means that the installers aren't widely used, I > think it mean they aren't distributed on PyPI. > > Installers are really useful for packages that require compiled code > that depends on external libs -- and most of the major such package > maintainers provide them. > > Also, numbers aren't as important as the handful of widely used, but > hard to build, packages.... > > But they are useless with virtualenv, so I'm looking forward to binary wheels... > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Sorry I should be more clear :) I wasn't claiming they weren't used (In fact I would guess but without looking that those 5k probably have a good bit of downloads). Just making a statement as to how many of them exist on PyPI. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Tue Jul 16 19:40:25 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 16 Jul 2013 13:40:25 -0400 Subject: [Distutils] PEP 426 updated based on last round of discussion In-Reply-To: References: Message-ID: <5E706F7D-0128-4FCB-8C93-7098331A0683@stufft.io> On Jul 16, 2013, at 9:40 AM, Nick Coghlan wrote: > I actually pushed this to python.org on the weekend but forgot to > announce it on the list. > > The latest version of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/ > > It was a net deletion of content despite going into more depth in a > few areas, so I'm counting that as a win for clarity :) > > Change details are at http://hg.python.org/peps/rev/067d3c3c1351 > > Significant changes: > > * serialisation prefix changed to "pydist". This metadata is the > metadata that exists at the sdist level. Wheels and installation may > add other metadata files, and PyPI will publish additional metadata > extracted from the archive contents, so I decided "pymeta" didn't feel > write (the name of the schema file changed as well, but I did that in > a separate commit so the diff stayed readable) > > * all the *_may_require fields are gone (as previously discussed) > > * rather than "install specifiers" I went with "requirement > specifiers" (install turned out not to read very well) > > * accordingly, the subfields of dependency specifiers are "requires", > "extra" and "environment" > > * the abbreviated form (which has "requires" as a list) was easy > enough to specify, so that's what I used. The unpacked form (where > multiple entries in the same dependency list have the same > extra/environment combination) is explicitly disallowed in order to > encourage consistent formatting. So to be clear, this means it's { "requires": [ "foo", "bar" ] } ? And it means that having multiple combinations of the same extra/envs is disallowed so I'm going to have to collapse everything back down since it's not stored that way at all? > > * clarified that internal whitespace is permitted in requirement > specifiers (there may be a simpler way to specify this, such as "all > whitespace in requirement specifiers is ignored") > > * I made the change to explicitly distrust "Provides" data retrieved > from a public index server, and noted that projects defining a virtual > dependency should claim that name to define the default provider. > > * noted that Debian packagers may want to map extras to Recommended or > Suggested packages > > * noted some possible use cases for metadata extensions > > * fixed and clarified various things in the reference copy of the JSON > schema (it could still do with an audit against the current PEP text, > though) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From doko at ubuntu.com Tue Jul 16 19:36:42 2013 From: doko at ubuntu.com (Matthias Klose) Date: Tue, 16 Jul 2013 19:36:42 +0200 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: Message-ID: <51E584AA.5060308@ubuntu.com> Am 13.07.2013 16:54, schrieb Paul Moore: > 1. Install to user-packages by default. > 2. Not depend on setuptools (??? - Nick's "inversion" idea) > 3. Possibly change the wrapper command name from pip to pip3 on Unix. > 4. Ensure that pip upgrading itself in-place is sufficiently robust and > reliable that users don't get "stuck" on the Python-supplied version. 5. Support cross-compilation of extensions by default. From vinay_sajip at yahoo.co.uk Tue Jul 16 19:54:21 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 16 Jul 2013 17:54:21 +0000 (UTC) Subject: [Distutils] PEP 426 updated based on last round of discussion References: <5E706F7D-0128-4FCB-8C93-7098331A0683@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > So to be clear, this means it's > > { > "requires": [ > "foo", > "bar" > ] > } > > ? > > And it means that having multiple combinations of the same > extra/envs is disallowed so I'm going to have to collapse everything > back down since it's not stored that way at all? > I posted a working example [1] showing how there's no need to have the same structure at the RDBMS layer and the JSON layer. I asked for more information about modelling difficulties you said you had encountered, but didn't hear anything more about it. AFAICT the code you were talking about isn't public - at least, I couldn't see it in the branches on your GitHub repo. As my example shows, it's possible to have a sensible RDBMS structure which interoperates with multiple entries in "requires". If I've misunderstood something, please let me know what it is. Regards, Vinay Sajip [1] https://gist.github.com/vsajip/5929707 From donald at stufft.io Tue Jul 16 19:57:45 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 16 Jul 2013 13:57:45 -0400 Subject: [Distutils] vetting, signing, verification of release files In-Reply-To: <20130716091900.GL3125@merlinux.eu> References: <20130716091900.GL3125@merlinux.eu> Message-ID: <70D36543-935E-4749-9D0F-7B106E2D04E3@stufft.io> On Jul 16, 2013, at 5:19 AM, holger krekel wrote: > > I am considering implementing gpg-signing and verification of release files > for devpi. Rather than requiring package authors to sign their release > files, i am pondering a scheme where anyone can vet for a particular > published release file by publishing a signature about it. This aims > to help responsible companies to work together. I've heart from devops/admins > that they manually download and check release files and then install > it offline after some vetting. Wouldn't it be useful to turn this > into a more collaborative effort? > > Any thoughts or pointers to existing efforts within the (Python) > packaging ecologies? > > best, > holger > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig So I'm not entirely sure what your goals are here. What exactly are you verifying? What is going to verify signatures once you have a (theoretically) trusted set? What is going to keep a malicious actor from poisoning the well? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Tue Jul 16 20:18:24 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 16 Jul 2013 19:18:24 +0100 (BST) Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> Message-ID: <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> >If I were writing a firm proposal, I'd go for something like entry points as metadata My extended metadata already covers this, though I use the name "exports" (suggested by PJE) because you can share not just code but data, and "entry points" generally implies code. The current version of distil creates wrappers for both gui and console scripts, and adds the appropriate native executable wrappers (32- or 64-bit, according to the running Python) on Windows. Example of the metadata for setuptools 0.8: "exports": { "distutils.commands": [ "alias = setuptools.command.alias:alias", "bdist_egg = setuptools.command.bdist_egg:bdist_egg", "bdist_rpm = setuptools.command.bdist_rpm:bdist_rpm", "build_ext = setuptools.command.build_ext:build_ext", "build_py = setuptools.command.build_py:build_py", "develop = setuptools.command.develop:develop", "easy_install = setuptools.command.easy_install:easy_install", "egg_info = setuptools.command.egg_info:egg_info", "install = setuptools.command.install:install", "install_lib = setuptools.command.install_lib:install_lib", "rotate = setuptools.command.rotate:rotate", "saveopts = setuptools.command.saveopts:saveopts", "sdist = setuptools.command.sdist:sdist", "setopt = setuptools.command.setopt:setopt", "test = setuptools.command.test:test", "install_egg_info = setuptools.command.install_egg_info:install_egg_info", "install_scripts = setuptools.command.install_scripts:install_scripts", "register = setuptools.command.register:register", "bdist_wininst = setuptools.command.bdist_wininst:bdist_wininst", "upload_docs = setuptools.command.upload_docs:upload_docs" ], "scripts": { "console": [ "easy_install = setuptools.command.easy_install:main" ] }, "egg_info.writers": [ "PKG-INFO = setuptools.command.egg_info:write_pkg_info", "requires.txt = setuptools.command.egg_info:write_requirements", "entry_points.txt = setuptools.command.egg_info:write_entries", "eager_resources.txt = setuptools.command.egg_info:overwrite_arg", "namespace_packages.txt = setuptools.command.egg_info:overwrite_arg", "top_level.txt = setuptools.command.egg_info:write_toplevel_names", "depends.txt = setuptools.command.egg_info:warn_depends_obsolete", "dependency_links.txt = setuptools.command.egg_info:overwrite_arg" ], "setuptools.file_finders": [ "svn_cvs = setuptools.command.sdist:_default_revctrl" ], "distutils.setup_keywords": [ "eager_resources = setuptools.dist:assert_string_list", "namespace_packages = setuptools.dist:check_nsp", "extras_require = setuptools.dist:check_extras", "install_requires = setuptools.dist:check_requirements", "tests_require = setuptools.dist:check_requirements", "entry_points = setuptools.dist:check_entry_points", "test_suite = setuptools.dist:check_test_suite", "zip_safe = setuptools.dist:assert_bool", "package_data = setuptools.dist:check_package_data", "exclude_package_data = setuptools.dist:check_package_data", "include_package_data = setuptools.dist:assert_bool", "packages = setuptools.dist:check_packages", "dependency_links = setuptools.dist:assert_string_list", "test_loader = setuptools.dist:check_importable", "use_2to3 = setuptools.dist:assert_bool", "convert_2to3_doctests = setuptools.dist:assert_string_list", "use_2to3_fixers = setuptools.dist:assert_string_list", "use_2to3_exclude_fixers = setuptools.dist:assert_string_list" ], "setuptools.installation": [ "eggsecutable = setuptools.command.easy_install:bootstrap" ] } Regards, Vinay Sajip From p.f.moore at gmail.com Tue Jul 16 20:34:36 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Jul 2013 19:34:36 +0100 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On 16 July 2013 19:18, Vinay Sajip wrote: > > >If I were writing a firm proposal, I'd go for something like entry points > as metadata > > My extended metadata already covers this, though I use the name "exports" > (suggested by PJE) because you can share not just code but data, and "entry > points" generally implies code. The current version of distil creates > wrappers for both gui and console scripts, and adds the appropriate native > executable wrappers (32- or 64-bit, according to the running Python) on > Windows. Nice. So we could say that wheel installers should create exe wrappers based on the 'scripts' key of the 'exports' metadata. Then we process the 'scripts' directory unaltered for backward compatibility. The only other part of the equation is to ensure that wheel *builders* do not put setuptools-generated entry point wrappers into the scripts directory. Before I embarrass myself again (:-)) is this also something you already have implemented in distlib? :-) When you're finished with Guido's time machine, make sure you put the keys back :-) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Jul 16 21:04:56 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 16 Jul 2013 19:04:56 +0000 (UTC) Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: Paul Moore gmail.com> writes: > The only other part of the equation is to ensure that wheel *builders* do > not put setuptools-generated entry point wrappers into the scripts > directory. Before I embarrass myself again () is this also something you > already have implemented in distlib? No, because I didn't want to embarrass you ;-) Seriously - no, because that is policy rather than mechanism. The way distil builds wheels is to perform an installation into a working area, and then call Wheel.build in distlib pointing to the work area. The distlib code just copies what's in the work area to the wheel. Currently the install-scripts part of distil installs all scripts into the work area, including ones defined in exports; it should be a five-minute job to ensure that scripts in exports are excluded from this, when building wheels. A version of distil with these updates should appear soon - I'm waiting for the dust to settle on the most recent changes to PEP 426. Regards, Vinay Sajip From jaraco at jaraco.com Tue Jul 16 21:14:58 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Tue, 16 Jul 2013 19:14:58 +0000 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: Message-ID: <88ead8c673d047baa494a03b46cd05ef@BLUPR06MB003.namprd06.prod.outlook.com> There are two versions of launchers primarily because of my naivet? when addressing the UAC issue. 64-bit launchers were exempt from the UAC restrictions that caused them to launch in a separate window. I believed this to be a proper fix, when in fact those still using 32-bit launchers were still experiencing the problem. See https://bitbucket.org/tarek/distribute/issue/143/easy_install-opens-new-cons ole-cant-read for more detail. So I agree, it would probably be sufficient to only supply 32-bit executables. However, my preference would be to supply architecture-appropriate executables rather than relying on a compatibility layer. Furthermore, I don?t believe the ARM architecture has a compatibility layer (meaning 64-bit executables are required for 64-bit ARM builds), so architecture and word size distinction is necessary. I believe you?re right about leveraging the py launcher. I?d like for setuptools to not have to supply launchers at all but depend on py launcher instead. The py launcher is bundled with Python 3.3 so should become ubiquitously available soon. I believe setuptools can begin to rely on it and not supply a launcher at all. The scripts currently installed by setuptools are suitable for launching by py launcher, so all that will need to happen is to stop supplying its own launcher. At least, that?s how I imagine it happening. From: Paul Moore [mailto:p.f.moore at gmail.com] Sent: Tuesday, 16 July, 2013 09:32 To: Distutils; Jason R. Coombs Subject: Re: Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) On 16 July 2013 14:08, Paul Moore > wrote: PS There is still the proviso that I haven't tested my assumption that the separate 32 and 64 bit wrappers are *needed* (setuptools and distlib use them, so I think it's a valid assumption, but I need to test). I will try to get time to check that ASAP. Hmm. I just did a quick test, and then based on the results checked the setuptools source code. I can see no reason why there needs to be 32 and 64 bit launcher exes. The launchers simply use CreateProcess to launch a separate Python process using the #! line of the script. So there's no DLL loading going on, and no reason that I can see for needing separate 32 and 64 bit builds. Jason - can you shed any light on why there are separate builds for 32 and 64 bits? Actually, the launcher is essentially identical to the "py" launcher for Windows, except that it gets a script name to execute from the name of the launcher. I'm wondering whether the correct approach here would be to enhance the launcher one more time to look for a suitably named script and auto-run it if it's present (i.e. merge the wrapper functionality into the launcher). Then we have a standard wrapper that everyone can use and not reinvent their own. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From vinay_sajip at yahoo.co.uk Tue Jul 16 21:21:27 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 16 Jul 2013 19:21:27 +0000 (UTC) Subject: [Distutils] =?utf-8?q?Wheels_and_console_script_entry_point_wrapp?= =?utf-8?q?ers=09=28Was=3A_Replacing_pip=2Eexe_with_a_Python_script?= =?utf-8?q?=29?= References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: Vinay Sajip yahoo.co.uk> writes: > defined in exports; it should be a five-minute job to ensure that scripts in > exports are excluded from this, when building wheels. It was a quick job, but thinking about it, I should probably update the Wheel.install API to take an optional process_exports=True argument, so that the exported-script processing can be done during installation from wheels in a standardised way. Regards, Vinay Sajip From donald at stufft.io Tue Jul 16 21:34:01 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 16 Jul 2013 15:34:01 -0400 Subject: [Distutils] PEP 426 updated based on last round of discussion In-Reply-To: References: <5E706F7D-0128-4FCB-8C93-7098331A0683@stufft.io> Message-ID: On Jul 16, 2013, at 1:54 PM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > >> So to be clear, this means it's >> >> { >> "requires": [ >> "foo", >> "bar" >> ] >> } >> >> ? >> >> And it means that having multiple combinations of the same >> extra/envs is disallowed so I'm going to have to collapse everything >> back down since it's not stored that way at all? >> > > I posted a working example [1] showing how there's no need to have the same > structure at the RDBMS layer and the JSON layer. I asked for more > information about modelling difficulties you said you had encountered, but > didn't hear anything more about it. AFAICT the code you were talking about > isn't public - at least, I couldn't see it in the branches on your GitHub repo. > > As my example shows, it's possible to have a sensible RDBMS structure which > interoperates with multiple entries in "requires". If I've misunderstood > something, please let me know what it is. > > Regards, > > Vinay Sajip > > [1] https://gist.github.com/vsajip/5929707 The dependency models are located at https://github.com/dstufft/warehouse/blob/f438bdcb17a5ee9de8e209d3eb6c93cc4aee9492/warehouse/packaging/models.py#L280-L380 It's completely possible and if I came across as saying it wasn't then I failed to clarify myself properly. My point was that it was simpler using a single list of dictionaries, not a list of dictionaries itself containing lists because there was less support code required to transform between them. Every additional piece of code comes with an overhead in the form of tests, mental overhead, potential bugs etc. I was trying to advocate for less required code because it makes things simpler :) I was asking for clarification here because my original plan if things were required to be a list was to make single entry lists, again to limit the need to include additional support code. It appears that this plan isn't inline with the current iteration of the PEP but I was making sure :) I have a preference for not introducing more nesting, and making things match the modeling better but I'll make it work either way. I hardly think PEP426 will fail if it's using deeper nesting even if I dislike it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Tue Jul 16 22:13:03 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Jul 2013 21:13:03 +0100 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On 16 July 2013 20:21, Vinay Sajip wrote: > > defined in exports; it should be a five-minute job to ensure that > scripts in > > exports are excluded from this, when building wheels. > > It was a quick job, but thinking about it, I should probably update the > Wheel.install API to take an optional process_exports=True argument, so > that > the exported-script processing can be done during installation from wheels > in a standardised way. I'm not 100% sure what your proposal is here - I'm confused about the precise roles of setup.py/setuptools as "builder" vs distil as "builder" vs distlib as "wheel builder" vs distlib as "wheel installer". I'll try to get some time in the next day or so to review the code and make sure we're not talking at cross purposes here. Bear with me. I understood that distlib simply built wheels from whatever is in the directories supplied. I'm not clear where it would get the "exports" metadata to store in the wheel. Assuming that's sorted, though, then whether or not distlib processes the exports on install is not an issue to me (I think it always should). What is more of an issue is what "the thing that puts stuff into the directories" does. If that's a setuptools-based build, it will process the exports data *itself* and put wrappers into the scripts directory. That is what I think we should be trapping and suppressing. But we have to be careful, as if setuptools is used to install directly, *not* going via a wheel, it has to generate wrappers itself somehow. Using distil to do the build is a whole other route. My main concern here is keeping a reasonable level of backward compatibility/interoperability with all the assorted tools around. And in particular with pip managing (but technically not *doing*) the builds and installs. I'll have a think, and do some code reviews, and try not get sucked into sending more emails until I know the facts a bit more clearly :-) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Jul 16 23:41:40 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 16 Jul 2013 21:41:40 +0000 (UTC) Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: Paul Moore gmail.com> writes: > I'm not 100% sure what your proposal is here - I'm confused about the precise > roles of setup.py/setuptools as "builder" vs distil as "builder" vs distlib > as "wheel builder" vs distlib as "wheel installer". I'll try to get some time > in the next day or so to review the code and make sure we're not talking at > cross purposes here. Bear with me. I'm not sure I've got a concrete proposal yet - I'm still thinking things out, too :-) Currently, the wheel module in distlib doesn't do anything too clever - just the mechanical stuff of archiving and unarchiving. Because the exports stuff wasn't in the PEP, distil wrote the generated scripts and .exe wrappers to the work area, and so they ended up in the wheel. (This is the case for all released versions of distil). > I understood that distlib simply built wheels from whatever is in the > directories supplied. I'm not clear where it would get the "exports" metadata > to store in the wheel. Assuming that's sorted, though, then whether or not Right now it's getting the exports courtesy of distil when you use that to build the wheel - distlib can't rely on that information being available because it's not part of the PEP. If the PEP is updated to include the exports, they should be in the wheel no matter which tool builds it. Then in theory distlib could generate the scripts during installation, but there are a lot of options to consider - did setuptools put them in there already? Do we want native launchers? etc. which is perfectly doable in distlib, but I'm not sure that's the best place for it because I think wheel processing should be uncomplicated. Wheel.install already has quite a few kwargs: dry_run=False: Don't actually do anything executable=None:Custom executable for shebang lines (e.g. to support path searching) warner=None:Used to defer warning behaviour to calling application lib_only=False:Process site-packages contents only (you suggested this) I'd like to not have to add any more, unless it's unavoidable :-) Nevertheless, I will probably try implementing it in distlib as an experiment, too see how it looks. > distlib processes the exports on install is not an issue to me (I think it > always should). What is more of an issue is what "the thing that puts stuff > into the directories" does. If that's a setuptools-based build, it will > process the exports data *itself* and put wrappers into the scripts > directory. That is what I think we should be trapping and suppressing. But > we have to be careful, as if setuptools is used to install directly, *not* > going via a wheel, it has to generate wrappers itself somehow. Exactly why I'm so leery of putting this logic in distlib, until we think it through and add it to the PEP. At the moment distil does it the same way as setuptools/pip only to remain compatible, not for any other reason. > Using distil to do the build is a whole other route. Right. I'm aiming for distil to be able to do just about everything pip can (functionally, the code is pretty much there barring installs from DVCS URLs), but backward compatibility is always a concern and a challenge :-) Another complication for distlib is that I expect it to work in 2.6+, where you can't always rely on the py launcher being present - hence the wrappers in distlib, with flags to disable writing them out. Another area to consider for scripts is which of foo, fooX and foo-X.Y to write to the scripts folder. This is particularly important in user site-packages, where scripts for different Python versions will coexist in the same folder, and the possibility exists of overwriting, which sometimes leads to unexpected behaviour (e.g. if I install foo-dist using 2.x which installs foo and foo-2.x to ~/.local/bin, then install it using 3.x, it would write foo and foo-3.x to ~/.local/bin. Quite apart from headaches with native launchers, it could be that the foo script installed with 3.x doesn't work (e.g. if it tries to process Python code which is 2.x compatible but not 3.x compatible). Fun and games! Regards, Vinay Sajip From p.f.moore at gmail.com Wed Jul 17 00:13:57 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Jul 2013 23:13:57 +0100 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On 16 July 2013 22:41, Vinay Sajip wrote: > If the PEP is updated to include the exports, they should be in the wheel > no > matter which tool builds it. Then in theory distlib could generate the > scripts during installation, but there are a lot of options to consider - > did setuptools put them in there already? Do we want native launchers? etc. > which is perfectly doable in distlib, but I'm not sure that's the best > place > for it because I think wheel processing should be uncomplicated. > Wheel.install already has quite a few kwargs: > I really don't want the wrappers to be present in the wheel, because if they are the wheel becomes architecture-specific. Also, consider that Unix targets should have the actual scripts written with no extension, whereas Windows targets should have foo-script.py and foo.exe. That should be decided at install time, bot at wheel creation time. As regards version-specific scripts, I'd assume it's the project's job to specify precisely what scripts they want. On that one, I'm on the side of providing infrastructure, not setting policy. Although I could be persuaded otherwise if there was a PEP on what commands a distribution should provide. In that case, let the project provide the command names, and let the installer implement the standard versioned-executable policy. Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Jul 17 00:43:11 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 16 Jul 2013 23:43:11 +0100 (BST) Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: <1374014591.78734.YahooMailNeo@web171401.mail.ir2.yahoo.com> >I really don't want the wrappers to be present in the wheel, because if they are the wheel becomes architecture-specific. Also, consider that Unix targets should have the actual scripts written with no extension, whereas Windows targets should have foo-script.py and foo.exe. That should be decided at install time, bot at wheel creation time. I think we're agreed on that as a desirable. I've already changed the distil code to omit writing launchers to the wheel, and that probably won't need to change. But it does generate the scripts for exports - to avoid doing that, we need to get the exports into the wheel in a standardised way (via pydist.json, perhaps, or perhaps a separate file). Iterating over the exports in a distribution path needs to be fast - note that the exports are not just scripts, and it's not yet clear whether the script exports need to be separated out from the rest. >As regards version-specific scripts, I'd assume it's the project's job to specify precisely what scripts they want. On that one, I'm on the side of providing infrastructure, not setting policy. This sounds like you mean that distlib needs to stay basic (mechanism/infrastructure), and the installer needs to do the script generation (policy - perhaps controlled by the user). Currently, distlib allows one to specify foo with optional variants fooX and foo-X.Y (before factoring native launchers into the mix). This is set on the ScriptMaker instance which generates the scripts in the target directory (including shebang rewriting, native launchers etc.) Regards, Vinay Sajip From ncoghlan at gmail.com Wed Jul 17 01:27:30 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 17 Jul 2013 09:27:30 +1000 Subject: [Distutils] PEP 426 updated based on last round of discussion In-Reply-To: References: <5E706F7D-0128-4FCB-8C93-7098331A0683@stufft.io> Message-ID: On 17 Jul 2013 05:34, "Donald Stufft" wrote: > > > On Jul 16, 2013, at 1:54 PM, Vinay Sajip wrote: > > > Donald Stufft stufft.io> writes: > > > >> So to be clear, this means it's > >> > >> { > >> "requires": [ > >> "foo", > >> "bar" > >> ] > >> } > >> > >> ? > >> > >> And it means that having multiple combinations of the same > >> extra/envs is disallowed so I'm going to have to collapse everything > >> back down since it's not stored that way at all? > >> > > > > I posted a working example [1] showing how there's no need to have the same > > structure at the RDBMS layer and the JSON layer. I asked for more > > information about modelling difficulties you said you had encountered, but > > didn't hear anything more about it. AFAICT the code you were talking about > > isn't public - at least, I couldn't see it in the branches on your GitHub repo. > > > > As my example shows, it's possible to have a sensible RDBMS structure which > > interoperates with multiple entries in "requires". If I've misunderstood > > something, please let me know what it is. > > > > Regards, > > > > Vinay Sajip > > > > [1] https://gist.github.com/vsajip/5929707 > > The dependency models are located at https://github.com/dstufft/warehouse/blob/f438bdcb17a5ee9de8e209d3eb6c93cc4aee9492/warehouse/packaging/models.py#L280-L380 > > It's completely possible and if I came across as saying it wasn't then I failed to clarify myself properly. My point was that it was simpler using a single list of dictionaries, not a list of dictionaries itself containing lists because there was less support code required to transform between them. Every additional piece of code comes with an overhead in the form of tests, mental overhead, potential bugs etc. I was trying to advocate for less required code because it makes things simpler :) > > I was asking for clarification here because my original plan if things were required to be a list was to make single entry lists, again to limit the need to include additional support code. It appears that this plan isn't inline with the current iteration of the PEP but I was making sure :) > > I have a preference for not introducing more nesting, and making things match the modeling better but I'll make it work either way. I hardly think PEP426 will fail if it's using deeper nesting even if I dislike it. Yes, in this case I think improving the brevity of the serialisation format will be an aid to debuggability, even though the primary purpose of the format remains communicating between tools. I should add a section discussing this decision in "Rejected Design Ideas", though. Cheers, Nick. > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Jul 17 01:56:57 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 16 Jul 2013 19:56:57 -0400 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: <51E584AA.5060308@ubuntu.com> References: <51E584AA.5060308@ubuntu.com> Message-ID: On Jul 16, 2013, at 1:36 PM, Matthias Klose wrote: > 5. Support cross-compilation of extensions by default. TBH I don't know how much of this has anything to do with pip? As far as compiling goes all pip does is call setup.py install so people are compiling with either setuptools or distutils. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Jul 17 02:00:03 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 17 Jul 2013 10:00:03 +1000 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On 17 Jul 2013 04:19, "Vinay Sajip" wrote: > > > > >If I were writing a firm proposal, I'd go for something like entry points as metadata > > My extended metadata already covers this, though I use the name "exports" (suggested by PJE) because you can share not just code but data, and "entry points" generally implies code. The current version of distil creates wrappers for both gui and console scripts, and adds the appropriate native executable wrappers (32- or 64-bit, according to the running Python) on Windows. Yeah, originally we were going to postpone dealing with entry points to a metadata extension (Daniel even had a proto-PEP kicking around in the pre-JSON days). However, I now think it makes more sense to standardise them as an "exports" field in PEP 426. So run with the assumption that something like that will be part of the standard metadata - either derived from entry_points.txt for existing metadata, or specified directly for next generation metadata. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Jul 17 02:03:14 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 16 Jul 2013 20:03:14 -0400 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: <64062226-5923-47A2-82F3-8999C2C77CC6@stufft.io> On Jul 16, 2013, at 8:00 PM, Nick Coghlan wrote: > > On 17 Jul 2013 04:19, "Vinay Sajip" wrote: > > > > > > > > >If I were writing a firm proposal, I'd go for something like entry points as metadata > > > > My extended metadata already covers this, though I use the name "exports" (suggested by PJE) because you can share not just code but data, and "entry points" generally implies code. The current version of distil creates wrappers for both gui and console scripts, and adds the appropriate native executable wrappers (32- or 64-bit, according to the running Python) on Windows. > > Yeah, originally we were going to postpone dealing with entry points to a metadata extension (Daniel even had a proto-PEP kicking around in the pre-JSON days). > > However, I now think it makes more sense to standardise them as an "exports" field in PEP 426. So run with the assumption that something like that will be part of the standard metadata - either derived from entry_points.txt for existing metadata, or specified directly for next generation metadata. > > Cheers, > Nick. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Are these only the scripts portion of entry points, or the whole kit and caboodle of pluggable entry points? Because I think the first makes sense, the second I'm hesitant on. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Jul 17 02:17:22 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 17 Jul 2013 10:17:22 +1000 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: <64062226-5923-47A2-82F3-8999C2C77CC6@stufft.io> References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> <64062226-5923-47A2-82F3-8999C2C77CC6@stufft.io> Message-ID: On 17 Jul 2013 10:03, "Donald Stufft" wrote: > > > On Jul 16, 2013, at 8:00 PM, Nick Coghlan wrote: > >> >> On 17 Jul 2013 04:19, "Vinay Sajip" wrote: >> > >> > >> > >> > >If I were writing a firm proposal, I'd go for something like entry points as metadata >> > >> > My extended metadata already covers this, though I use the name "exports" (suggested by PJE) because you can share not just code but data, and "entry points" generally implies code. The current version of distil creates wrappers for both gui and console scripts, and adds the appropriate native executable wrappers (32- or 64-bit, according to the running Python) on Windows. >> >> Yeah, originally we were going to postpone dealing with entry points to a metadata extension (Daniel even had a proto-PEP kicking around in the pre-JSON days). >> >> However, I now think it makes more sense to standardise them as an "exports" field in PEP 426. So run with the assumption that something like that will be part of the standard metadata - either derived from entry_points.txt for existing metadata, or specified directly for next generation metadata. >> >> Cheers, >> Nick. >> >> _______________________________________________ >> >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > > Are these only the scripts portion of entry points, or the whole kit and caboodle of pluggable entry points? Because I think the first makes sense, the second I'm hesitant on. Actually, it may be better to have a top level "scripts" field, distinct from a general export mechanism. I'm seeing value in an exports mechanism, though. Yes, *in theory* you can get the same effect with an extension, but extensions can do a lot of other things, too. Python's metaprogramming is built on a model of multiple tools with increasing levels of power, flexibility and complexity, so I'm thinking an exports vs extensions split may be a good approach in a similar vein. No decision on this front yet, but I think it's at least worth my trying it out to see how it looks in the context of the PEP. (After all, we already know entry points are quite a popular feature of the setuptools metadata) A couple of bonus features of standardisation are that we can tie it into the extras system and automatic analysis tools can check the exports can actually be imported without needing to understand arbitrary extensions. Cheers, Nick. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benzolius at yahoo.com Tue Jul 16 10:57:15 2013 From: benzolius at yahoo.com (Benedek Zoltan) Date: Tue, 16 Jul 2013 01:57:15 -0700 (PDT) Subject: [Distutils] buildout doesn't work as expected, please help me Message-ID: <1373965035.93141.YahooMailNeo@web121601.mail.ne1.yahoo.com> Hi, I'm relatively new to buildout, but untill now I could manage to install python packages, supervisord, config files and run system commands by buildout, using the recipes: z3c.recipe.scripts collective.recipe.supervisor collective.recipe.template plone.recipe.command Sometimes I had a timeout error like: "Installing pypackages. While: ? Installing pypackages. ? Getting distribution for 'python-dateutil'. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): ..." and "timeout: timed out" but generally worked. Recently I get nearly always this error or an other one: "? Getting distribution for 'zc.buildout>=1.7.1,<2dev'. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): ..." and "timeout: timed out" I tried: 1. to run even with: bin/buildout -t 30 2.? to put this in buildout.cfg find-links =??????????????????????????????????????????????????????????????????? ??? http://a.pypi.python.org/simple/??????????????????????????????????????????? ??? http://b.pypi.python.org/simple/??????????????????????????????????????????? ??? http://c.pypi.python.org/simple/??????????????????????????????????????????? ??? http://d.pypi.python.org/simple/??????????????????????????????????????????? ??? http://e.pypi.python.org/simple/??????????????????????????????????????????? ??? http://f.pypi.python.org/simple/??????????????????????????????????????????? ??? http://g.pypi.python.org/simple/ 3. I changed z3c.recipe.scripts to zc.recipe.egg Unfortunately the problem is not solved. Any help will be greatly appreciated. Thank you Zoli -------------- next part -------------- An HTML attachment was scrubbed... URL: From liamk at numenet.com Tue Jul 16 19:15:10 2013 From: liamk at numenet.com (Liam Kirsher) Date: Tue, 16 Jul 2013 10:15:10 -0700 Subject: [Distutils] distribute o.7.3 causing installation error? Message-ID: <51E57F9E.9020505@numenet.com> Hi, I ran into an error about a month ago caused by a change in the PyPi version of distribute. Thankfully, someone was able to roll back the change. Unfortunately, I'm getting a similar kind of problem now -- and I notice that 0.7.3 was released on 5 July, so... I'm wondering if it might be related. This is being included in a Chef recipe. I'm attaching the pip.log, which shows it uninstalling distribute (which looks like version 0.6.49), and then failing to find it and attempting to install 0.7.3, and subsequent package installs failing. Anyway, I'm not quite sure what to do here! How can I fix this problem? (And also, how can I prevent it from happening in the future by pegging the version to something that works?) The pip recipe includes the following comments, which may be relevant. > # Ubuntu's python-setuptools, python-pip and py thon-virtualenv packages > # are broken...this feels like Rubygems! > # > http://stackoverflow.com/questions/4324558/whats-the-proper-way-to-install-pip-virtualenv-and-distribute-for-python > # https://bitbucket.org/ianb/pip/issue/104/pip-uninstall-on-ubuntu-linux > remote_file "#{Chef::Config[:file_cache_path]}/distribute_setup.py" do > source node['python']['distribute_script_url'] > mode "0644" > not_if { ::File.exists?(pip_binary) } > end > execute "install-pip" do > cwd Chef::Config[:file_cache_path] > command <<-EOF > #{node['python']['binary']} distribute_setup.py > --download-base=#{node['python']['distribute_option']['download_base']} > #{::File.dirname(pip_binary)}/easy_install pip > EOF > not_if { ::File.exists?(pip_binary) } > end Chef run log: > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Recipe: > python::virtualenv > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com * > python_pip[virtualenv] action install > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com - install package > python_pip[virtualenv] version latest > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Recipe: > supervisor::default > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com * > python_pip[supervisor] action upgrade > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ================================================================================ > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Error executing > action `upgrade` on resource 'python_pip[supervisor]' > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ================================================================================ > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > Mixlib::ShellOut::ShellCommandFailed > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ------------------------------------ > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Expected process to > exit with [0], but received '1' > ec2-54-245-36-62.us-west-2.compute.amazonaws.com ---- Begin output of > pip install --upgrade supervisor ---- > ec2-54-245-36-62.us-west-2.compute.amazonaws.com STDOUT: > Downloading/unpacking supervisor > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Running setup.py > egg_info for package supervisor > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Downloading/unpacking > distribute from > https://pypi.python.org/packages/source/d/distribute/distribute-0.7.3.zip#md5=c6c59594a7b180af57af8a0cc0cf5b4a > (from supervisor) > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Running setup.py > egg_info for package distribute > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Downloading/unpacking > meld3>=0.6.5 (from supervisor) > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Running setup.py > egg_info for package meld3 > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Downloading/unpacking > setuptools>=0.7 (from distribute->supervisor) > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Running setup.py > egg_info for package setuptools > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Installing collected > packages: supervisor, distribute, meld3, setuptools > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Running setup.py > install for supervisor > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Skipping > installation of > /usr/local/lib/python2.7/dist-packages/supervisor/__init__.py > (namespace package) > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Installing > /usr/local/lib/python2.7/dist-packages/supervisor-3.0b2-py2.7-nspkg.pth > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Installing > echo_supervisord_conf script to /usr/local/bin > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Installing > pidproxy script to /usr/local/bin > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Installing > supervisorctl script to /usr/local/bin > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Installing > supervisord script to /usr/local/bin > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Found existing > installation: distribute 0.6.49 > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Uninstalling > distribute: > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Successfully > uninstalled distribute > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Running setup.py > install for distribute > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Running setup.py > install for meld3 > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Traceback (most > recent call last): > ec2-54-245-36-62.us-west-2.compute.amazonaws.com File > "", line 1, in > ec2-54-245-36-62.us-west-2.compute.amazonaws.com ImportError: No > module named setuptools > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Complete output > from command /usr/bin/python -c "import > setuptools;__file__='/tmp/pip-build-root/meld3/setup.py';exec(compile(open(__file__).read().replace('\r\n', > '\n'), __file__, 'exec'))" install --record > /tmp/pip-mDCOBa-record/install-record.txt > --single-version-externally-managed: > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Traceback (most > recent call last): > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com File "", > line 1, in > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com ImportError: No > module named setuptools > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ---------------------------------------- > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Command > /usr/bin/python -c "import setuptools;__file__='/tmp/pip-build > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > -root/meld3/setup.py';exec(compile(open(__file__).read().replace('\r\n', > '\n'), __file__, 'exec'))" install --record > /tmp/pip-mDCOBa-record/install-record.txt > --single-version-externally-managed failed with error code 1 in > /tmp/pip-build-root/meld3 > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Storing complete log > in /home/ubuntu/.pip/pip.log > ec2-54-245-36-62.us-west-2.compute.amazonaws.com STDERR: > ec2-54-245-36-62.us-west-2.compute.amazonaws.com ---- End output of > pip install --upgrade supervisor ---- > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Ran pip install > --upgrade supervisor returned 1 > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Cookbook Trace: > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com --------------- > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > /var/chef/cache/cookbooks/python/providers/pip.rb:155:in `pip_cmd' > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > /var/chef/cache/cookbooks/python/providers/pip.rb:139:in `install_package' > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > /var/chef/cache/cookbooks/python/providers/pip.rb:144:in `upgrade_package' > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > /var/chef/cache/cookbooks/python/providers/pip.rb:60:in `block (2 > levels) in class_from_file' > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > /var/chef/cache/cookbooks/python/providers/pip.rb:58:in `block in > class_from_file' > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Resource Declaration: > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com --------------------- > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com # In > /var/chef/cache/cookbooks/supervisor/recipes/default.rb > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com 29: python_pip > "supervisor" do > ec2-54-245-36-62.us-west-2.compute.amazonaws.com 30: action :upgrade > ec2-54-245-36-62.us-west-2.compute.amazonaws.com 31: version > node['supervisor']['version'] if node['supervisor']['version'] > ec2-54-245-36-62.us-west-2.compute.amazonaws.com 32: end > ec2-54-245-36-62.us-west-2.compute.amazonaws.com 33: > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Compiled Resource: > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com ------------------ > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com # Declared in > /var/chef/cache/cookbooks/supervisor/recipes/default.rb:29:in `from_file' > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > python_pip("supervisor") do > ec2-54-245-36-62.us-west-2.compute.amazonaws.com action [:upgrade] > ec2-54-245-36-62.us-west-2.compute.amazonaws.com retries 0 > ec2-54-245-36-62.us-west-2.compute.amazonaws.com retry_delay 2 > ec2-54-245-36-62.us-west-2.compute.amazonaws.com cookbook_name > "supervisor" > ec2-54-245-36-62.us-west-2.compute.amazonaws.com recipe_name "default" > ec2-54-245-36-62.us-west-2.compute.amazonaws.com package_name > "supervisor" > ec2-54-245-36-62.us-west-2.compute.amazonaws.com timeout 900 > ec2-54-245-36-62.us-west-2.compute.amazonaws.com options " --upgrade" > ec2-54-245-36-62.us-west-2.compute.amazonaws.com end > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Recipe: ntp::default > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com * service[ntp] > action restart > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com - restart service > service[ntp] > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Recipe: rabbitmq::default > ec2-54-245-36-62.us-west-2.compute.amazonaws.com * > service[rabbitmq-server] action restart > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com - restart service > service[rabbitmq-server] > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > [2013-07-16T02:36:50+00:00] ERROR: Running exception handlers > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > [2013-07-16T02:36:51+00:00] FATAL: Saving node information to > /var/chef/cache/failed-run-data.json > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > [2013-07-16T02:36:51+00:00] ERROR: Exception handlers complete > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Chef Client failed. > 35 resources updated > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > [2013-07-16T02:36:51+00:00] FATAL: Stacktrace dumped to > /var/chef/cache/chef-stacktrace.out > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > [2013-07-16T02:36:51+00:00] FATAL: > Mixlib::ShellOut::ShellCommandFailed: python_pip[supervisor] > (supervisor::default line 29) had an error: > Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with > [0], but received '1' > ec2-54-245-36-62.us-west-2.compute.amazonaws.com ---- Begin output of > pip install --upgrade supervisor ---- > ec2-54-245-36-62.us-west-2.compute.amazonaws.com STDOUT: > Downloading/unpacking supervisor > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Running setup.py > egg_info for package supervisor > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Downloading/unpacking > distribute from > https://pypi.python.org/packages/source/d/distribute/distribute-0.7.3.zip#md5=c6c59594a7b180af57af8a0cc0cf5b4a > (from supervisor) > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Running setup.py > egg_info for package distribute > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Downloading/unpacking > meld3>=0.6.5 (from supervisor) > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Running setup.py > egg_info for package meld3 > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Downloading/unpacking > setuptools>=0.7 (from distribute->supervisor) > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Running setup.py > egg_info for package setuptools > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Installing collected > packages: supervisor, distribute, meld3, setuptools > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Running setup.py > install for supervisor > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Skipping > installation of > /usr/local/lib/python2.7/dist-packages/supervisor/__init__.py > (namespace package) > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Installing > /usr/local/lib/python2.7/dist-packages/supervisor-3.0b2-py2.7-nspkg.pth > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Installing > echo_supervisord_conf script to /usr/local/bin > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Installing > pidproxy script to /usr/local/bin > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Installing > supervisorctl script to /usr/local/bin > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Installing > supervisord script to /usr/local/bin > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Found existing > installation: distribute 0.6.49 > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Uninstalling > distribute: > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Successfully > uninstalled distribute > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Running setup.py > install for distribute > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Running setup.py > install for meld3 > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Traceback (most > recent call last): > ec2-54-245-36-62.us-west-2.compute.amazonaws.com File > "", line 1, in > ec2-54-245-36-62.us-west-2.compute.amazonaws.com ImportError: No > module named setuptools > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Complete output > from command /usr/bin/python -c "import > setuptools;__file__='/tmp/pip-build-root/meld3/setup.py';exec(compile(open(__file__).read().replace('\r\n', > '\n'), __file__, 'exec'))" install --record > /tmp/pip-mDCOBa-record/install-record.txt > --single-version-externally-managed: > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Traceback (most > recent call last): > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com File " ec2-54-245-36-62.us-west-2.compute.amazonaws.com g>", line 1, in > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com ImportError: No > module named setuptools > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ec2-54-245-36-62.us-west-2.compute.amazonaws.com > ---------------------------------------- > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Command > /usr/bin/python -c "import > setuptools;__file__='/tmp/pip-build-root/meld3/setup.py';exec(compile(open(__file__).read().replace('\r\n', > '\n'), __file__, 'exec'))" install --record > /tmp/pip-mDCOBa-record/install-record.txt > --single-version-externally-managed failed with error code 1 in > /tmp/pip-build-root/meld3 > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Storing complete log > in /home/ubuntu/.pip/pip.log > ec2-54-245-36-62.us-west-2.compute.amazonaws.com STDERR: > ec2-54-245-36-62.us-west-2.compute.amazonaws.com ---- End output of > pip install --upgrade supervisor ---- > ec2-54-245-36-62.us-west-2.compute.amazonaws.com Ran pip install > --upgrade supervisor returned 1 -- Liam Kirsher PGP: http://liam.numenet.com/pgp/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pip.log Type: text/x-log Size: 149041 bytes Desc: not available URL: From liamk at numenet.com Tue Jul 16 19:48:22 2013 From: liamk at numenet.com (Liam Kirsher) Date: Tue, 16 Jul 2013 10:48:22 -0700 Subject: [Distutils] distribute 0.7.3 causing installation error? follow up Message-ID: <51E58766.2040206@numenet.com> Hi, Also, just noticed that 0.7.3 is available as .zip, but not as .tar.gz like the others are. Liam -- Liam Kirsher PGP: http://liam.numenet.com/pgp/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Wed Jul 17 09:03:27 2013 From: holger at merlinux.eu (holger krekel) Date: Wed, 17 Jul 2013 07:03:27 +0000 Subject: [Distutils] vetting, signing, verification of release files In-Reply-To: <70D36543-935E-4749-9D0F-7B106E2D04E3@stufft.io> References: <20130716091900.GL3125@merlinux.eu> <70D36543-935E-4749-9D0F-7B106E2D04E3@stufft.io> Message-ID: <20130717070327.GN1668@merlinux.eu> On Tue, Jul 16, 2013 at 13:57 -0400, Donald Stufft wrote: > On Jul 16, 2013, at 5:19 AM, holger krekel wrote: > > > > > I am considering implementing gpg-signing and verification of release files > > for devpi. Rather than requiring package authors to sign their release > > files, i am pondering a scheme where anyone can vet for a particular > > published release file by publishing a signature about it. This aims > > to help responsible companies to work together. > > > So I'm not entirely sure what your goals are here. The goal is to facilitate collaboration between individuals and companies in vetting the integrity and, to some degree, authenticity of a published pypi package. > What exactly are you verifying? What is going to verify signatures once you have a (theoretically) trusted set? What is going to keep a malicious actor from poisoning the well? These are typical questions which is why i asked if anyone knows about existing schemes/efforts. I guess most Linux distros do it already so if nothing comes up here PyPI-specific (what is the status of TUF, btw?) i am going to look into the distro's working models. One difference is that i want the vetting/signing to happen after publishing to allow for an incremental approach. cheers, holger -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: Digital signature URL: From vinay_sajip at yahoo.co.uk Wed Jul 17 09:40:13 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 17 Jul 2013 07:40:13 +0000 (UTC) Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> <64062226-5923-47A2-82F3-8999C2C77CC6@stufft.io> Message-ID: Nick Coghlan gmail.com> writes: > Actually, it may be better to have a top level "scripts" field, distinct from a general export mechanism. > I'm seeing value in an exports mechanism, though. The exports functionality is important and used enough to warrant support in the PEP, and not only for the scripts part. There should be some way that this data gets into .dist-info in a standardised way, so that there is an ability to query e.g. implementations of a particular interface. Currently, distlib supports this as an extension to the PEP by reading a file from .dist-info, which distil puts there. At the moment the PEP is silent on the subject, which could lead to fragmentation in the implementations - e.g. whether JSON Or ini-style format is used for the data. I'd like to suggest that the whole of the exports information be included in the PEP 426 metadata, without singling out the scripts part. That way, it ends up in .dist-info via pydist.json. It has been suggested by PJE that the exports information should be in a separate file for speed of searching - though that suggestion was made in a pre-JSON world, where the speed of parsing the metadata wasn't C-assisted. Should performance still be an issue, then the exports dict could still be written out separately in .dist-info as e.g. exports.json by an installer. Regards, Vinay Sajip From donald at stufft.io Wed Jul 17 09:43:19 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 17 Jul 2013 03:43:19 -0400 Subject: [Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) In-Reply-To: References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> <64062226-5923-47A2-82F3-8999C2C77CC6@stufft.io> Message-ID: <3006DC94-C386-4110-A640-8CF98AC27C95@stufft.io> On Jul 17, 2013, at 3:40 AM, Vinay Sajip wrote: > It has been suggested by PJE that the > exports information should be in a separate file for speed of searching - > though that suggestion was made in a pre-JSON world, where the speed of > parsing the metadata wasn't C-assisted. I don't think it was speed of parsing the file that was his concern, rather that if it's a separate file that only exists when there are entry points, then you don't have to open a file for every installed distribution. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Wed Jul 17 09:48:46 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 17 Jul 2013 07:48:46 +0000 (UTC) Subject: [Distutils] vetting, signing, verification of release files References: <20130716091900.GL3125@merlinux.eu> <70D36543-935E-4749-9D0F-7B106E2D04E3@stufft.io> <20130717070327.GN1668@merlinux.eu> Message-ID: holger krekel merlinux.eu> writes: > about existing schemes/efforts. I guess most Linux distros do it already > so if nothing comes up here PyPI-specific (what is the status of TUF, btw?) > i am going to look into the distro's working models. ISTM it works for distros because they're the central authority guaranteeing the provenance of the software in their repos. It's harder with PyPI because it's not a central authority curating the content. Perhaps something like a web of trust would be needed. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Wed Jul 17 09:53:51 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 17 Jul 2013 07:53:51 +0000 (UTC) Subject: [Distutils] =?utf-8?q?Wheels_and_console_script_entry_point_wrapp?= =?utf-8?q?ers=09=28Was=3A_Replacing_pip=2Eexe_with_a_Python_script?= =?utf-8?q?=29?= References: <5891FE61-E0BB-44A1-BA9C-B3AFE9AE2800@mac.com> <1373998704.38172.YahooMailNeo@web171404.mail.ir2.yahoo.com> <64062226-5923-47A2-82F3-8999C2C77CC6@stufft.io> <3006DC94-C386-4110-A640-8CF98AC27C95@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > I don't think it was speed of parsing the file that was his concern, rather that > if it's a separate file that only exists when there are entry points, then you > don't have to open a file for every installed distribution. OK. In that case, exports.json would avoid the need to open pydist.json to look for exports - if exports.json is missing, it would mean that dist has no exports. Regards, Vinay Sajip From reinout at vanrees.org Wed Jul 17 09:58:53 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Wed, 17 Jul 2013 09:58:53 +0200 Subject: [Distutils] distribute o.7.3 causing installation error? In-Reply-To: <51E57F9E.9020505@numenet.com> References: <51E57F9E.9020505@numenet.com> Message-ID: On 16-07-13 19:15, Liam Kirsher wrote: > ec2-54-245-36-62.us-west-2.compute.amazonaws.com ImportError: No module > named setuptools My guess is that there's a left-over distribute somewhere. Probably an egg-link in some dist-packages or site-packages directory. I had a problem like that too. What I did in that case: - Search&destroy any distribute/setuptools anywhere. - Install setuptools from scratch instead of trying to upgrade it. wget https://bitbucket.org/pypa/setuptools/raw/0.8/ez_setup.py sudo /usr/bin/python ez_setup.py Yes, this sucks if you want to maintain a nice clean OS-managed machine. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From holger at merlinux.eu Wed Jul 17 10:16:40 2013 From: holger at merlinux.eu (holger krekel) Date: Wed, 17 Jul 2013 08:16:40 +0000 Subject: [Distutils] vetting, signing, verification of release files In-Reply-To: References: <20130716091900.GL3125@merlinux.eu> <70D36543-935E-4749-9D0F-7B106E2D04E3@stufft.io> <20130717070327.GN1668@merlinux.eu> Message-ID: <20130717081640.GR1668@merlinux.eu> On Wed, Jul 17, 2013 at 07:48 +0000, Vinay Sajip wrote: > holger krekel merlinux.eu> writes: > > > about existing schemes/efforts. I guess most Linux distros do it already > > so if nothing comes up here PyPI-specific (what is the status of TUF, btw?) > > i am going to look into the distro's working models. > > ISTM it works for distros because they're the central authority guaranteeing > the provenance of the software in their repos. It's harder with PyPI because > it's not a central authority curating the content. Perhaps something like a > web of trust would be needed. I am thinking about curating release files _after_ publishing and then configuring install activities to require "signed-off" release files. Basically giving companies and devops the possibility to organise their vetting processes and collaborate, without requiring PyPI to change first. This certainly involves the question of trust but if nothing else an entity can at least trust its own signatures :) best, holger -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: Digital signature URL: From ncoghlan at gmail.com Wed Jul 17 10:50:16 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 17 Jul 2013 18:50:16 +1000 Subject: [Distutils] vetting, signing, verification of release files In-Reply-To: <20130717081640.GR1668@merlinux.eu> References: <20130716091900.GL3125@merlinux.eu> <70D36543-935E-4749-9D0F-7B106E2D04E3@stufft.io> <20130717070327.GN1668@merlinux.eu> <20130717081640.GR1668@merlinux.eu> Message-ID: On 17 Jul 2013 18:17, "holger krekel" wrote: > > On Wed, Jul 17, 2013 at 07:48 +0000, Vinay Sajip wrote: > > holger krekel merlinux.eu> writes: > > > > > about existing schemes/efforts. I guess most Linux distros do it already > > > so if nothing comes up here PyPI-specific (what is the status of TUF, btw?) > > > i am going to look into the distro's working models. > > > > ISTM it works for distros because they're the central authority guaranteeing > > the provenance of the software in their repos. It's harder with PyPI because > > it's not a central authority curating the content. Perhaps something like a > > web of trust would be needed. > > I am thinking about curating release files _after_ publishing and > then configuring install activities to require "signed-off" release files. > Basically giving companies and devops the possibility to organise their > vetting processes and collaborate, without requiring PyPI to change first. > This certainly involves the question of trust but if nothing else an entity > can at least trust its own signatures :) Note that Linux distros don't trust each other's keys and nor do app stores trust other. Secure collaborative vetting of software is an Unsolved Problem. The Update Framework provides a solid technical basis for such collaboration, but even it doesn't solve the fundamental trust issues. Those issues are why we still rely on the CA model for SSL, despite its serious flaws: nobody has come up with anything else that scales effectively. The use of JSON for metadata 2.0 is enough to make it TUF friendly, but there are significant key management issues to be addressed before TUF could be used on PyPI itself. That's no reason to avoid experimenting with private TUF enabled PyPI servers, though - a private server alleviates most of the ugly key management problems. Cheers, Nick. > > best, > holger > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > > iQEcBAEBAgAGBQJR5lLnAAoJEI47A6J5t3LWUkYIAJ1qqyAc185R7NrXqJyEpNo6 > erDSfCMROcMqxtkqLCeoaiKSSBnhyJJpcLJ9a5P2/z8hBsYVTKM54NdOpvJEcgb/ > s/sepYI3vTIXFtUyRTxXPmhUZoxgh+GdvatCWw+7EA8pcAPs3YvrdKPYqHOm3xup > Z1KWAUrPWhVxoUY8laUBaHkHxX3WJ88Hj0buJfzsKEbQvytT8sRO9Nq03VE5EsjL > 85boVh4UIA0KUMtEgzxgRGDjD9Cc47ukFrmN/ViYKdmV6gmIBV1h30dcRXhvof5W > QSuuROqXjQ466Vm5aaE7rfLzIAOtxOvjBuZLygr2bMbZYY8WtHDJD7e0VYFJPCw= > =vZ9n > -----END PGP SIGNATURE----- > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Wed Jul 17 12:00:02 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 17 Jul 2013 11:00:02 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <51E584AA.5060308@ubuntu.com> Message-ID: On 17 July 2013 00:56, Donald Stufft wrote: > > On Jul 16, 2013, at 1:36 PM, Matthias Klose wrote: > > 5. Support cross-compilation of extensions by default. > > TBH I don't know how much of this has anything to do with pip? As far as > compiling goes all pip does is call setup.py install so people are compiling > with either setuptools or distutils. I'm not involved in the current packaging work so someone else can correct me if I'm wrong but: What I'm really looking forward to as a result of all of this work on wheels and packaging metadata is that there will no longer be any need for people to use either setuptools or distutils to compile anything. Once this work is ready it will be possible for developers to build wheels using whatever tools they like and then upload them to pypi. pip will be able to select the appropriate wheel for the end user's OS, CPU, Python version etc. and install the binary wheel without needing any compiler support on the target system. What this means is that it will be possible to do all the compilation on developers machines so that the tools to do so don't need to be in the stdlib any more. Then someone will be able to release the "fancycompiler" package on pypi that will support features like cross-compilation, other developers can use it, and the end user doesn't need to care how the wheels were made. Oscar From p.f.moore at gmail.com Wed Jul 17 12:17:38 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 17 Jul 2013 11:17:38 +0100 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <51E584AA.5060308@ubuntu.com> Message-ID: On 17 July 2013 00:56, Donald Stufft wrote: > On Jul 16, 2013, at 1:36 PM, Matthias Klose wrote: > > 5. Support cross-compilation of extensions by default. > > > TBH I don't know how much of this has anything to do with pip? As far as > compiling goes all pip does is call setup.py install so people are > compiling with either setuptools or distutils. > Nothing at all. Cross-compilation is not handled by pip, nor is it relevant to the discussions about bundling pip with Python. The much longer-term goal of the packaging discussions around decoupling "builders" and "installers" might allow for the development of better 3rd party build tools with cross-compilation support (things in the space that tools like bento occupy) but it will never be something that pip needs to concern itself with. I doubt that the standard library (i.e. distutils or any successor) will be involved in this either - it's specialised enough that 3rd party tools are the correct way to handle it. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Wed Jul 17 13:01:12 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 17 Jul 2013 12:01:12 +0100 Subject: [Distutils] PEP 426 updated based on last round of discussion In-Reply-To: References: Message-ID: On 16 July 2013 14:40, Nick Coghlan wrote: > > The latest version of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/ Just looking at the "Build requires" section I found myself wondering: is there any way to say that e.g. a C compiler is required for building, or a Fortran compiler or any other piece of software that isn't a "Python distribution"? The example shows Cython which is commonly built and used with MinGW on Windows. I guess that it would be possible to create a pypi distribution that would install MinGW and set it up as part of a Python installation so that a project such as Cython could depend on it with e.g.: "name": "Cython", "build_requires": [ { "requires": ["pymingw"], "environment": "sys.platform == 'win32'" } ] "run_requires": [ { "requires": ["pymingw"], "environment": "sys.platform == 'win32'" } ] But it would be unfortunate to depend on MinGW in the event that the user actually has the appropriate MSVC version. Or perhaps there could be a meta-distribution called "CCompiler" that installs MinGW only if the the appropriate MSVC version is not available. Or could there be an environment marker to indicate the presence of particularly common requirements such as having a C compiler? Oscar From p.f.moore at gmail.com Wed Jul 17 13:10:42 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 17 Jul 2013 12:10:42 +0100 Subject: [Distutils] PEP 426 updated based on last round of discussion In-Reply-To: References: Message-ID: On 17 July 2013 12:01, Oscar Benjamin wrote: > On 16 July 2013 14:40, Nick Coghlan wrote: > > > > The latest version of PEP 426 is up at > http://www.python.org/dev/peps/pep-0426/ > > Just looking at the "Build requires" section I found myself wondering: > is there any way to say that e.g. a C compiler is required for > building, or a Fortran compiler or any other piece of software that > isn't a "Python distribution"? > [...] > Or perhaps there could be a meta-distribution called "CCompiler" that > installs MinGW only if the the appropriate MSVC version is not > available. Or could there be an environment marker to indicate the > presence of particularly common requirements such as having a C > compiler? I can't imagine it's practical to auto-install a C compiler - or even to check for one before building. But I can see it being useful for introspection purposes to know about this type of requirement. (A C compiler could be necessary, or optional for speedups, a particular external library could be needed, etc) The data would likely only be as good as what project developers provide, but nevertheless having standard places to record the data could encourage doing so... OTOH, maybe this is metadata 3.0 stuff - I feel like at the moment we need to get what we have now out of the door rather than continually adding extra capabilities. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Wed Jul 17 13:45:37 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 17 Jul 2013 12:45:37 +0100 Subject: [Distutils] PEP 426 updated based on last round of discussion In-Reply-To: References: Message-ID: On 17 July 2013 12:10, Paul Moore wrote: > > I can't imagine it's practical to auto-install a C compiler Why not? > - or even to check for one before building. > > But I can see it being useful for > introspection purposes to know about this type of requirement. (A C compiler > could be necessary, or optional for speedups, a particular external library > could be needed, etc) Perhaps instead the installer tool would give you a way to clarify that you do have a C compiler and to warn if not. Alternatively a meta-package could be used to indicate (when installed) that a compatible C-compiler is available and then other distributions could depend on it for building. > The data would likely only be as good as what project developers provide, > but nevertheless having standard places to record the data could encourage > doing so... > > OTOH, maybe this is metadata 3.0 stuff - I feel like at the moment we need > to get what we have now out of the door rather than continually adding extra > capabilities. I wasn't proposing to hold anything up or add new capabilities. I'm just trying to see how far these changes go towards making non-pure Python software automatically installable. Everything I would want to build "build requires" software that is not on pypi. It would be great if e.g. the instructions for installing Cython on Windows could just be "pip install cython" instead of this: http://wiki.cython.org/InstallingOnWindows Oscar From ncoghlan at gmail.com Wed Jul 17 13:56:18 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 17 Jul 2013 21:56:18 +1000 Subject: [Distutils] Expectations on how pip needs to change for Python 3.4 In-Reply-To: References: <51E584AA.5060308@ubuntu.com> Message-ID: On 17 July 2013 20:00, Oscar Benjamin wrote: > On 17 July 2013 00:56, Donald Stufft wrote: >> >> On Jul 16, 2013, at 1:36 PM, Matthias Klose wrote: >> >> 5. Support cross-compilation of extensions by default. >> >> TBH I don't know how much of this has anything to do with pip? As far as >> compiling goes all pip does is call setup.py install so people are compiling >> with either setuptools or distutils. > > I'm not involved in the current packaging work so someone else can > correct me if I'm wrong but: > > What I'm really looking forward to as a result of all of this work on > wheels and packaging metadata is that there will no longer be any need > for people to use either setuptools or distutils to compile anything. > Once this work is ready it will be possible for developers to build > wheels using whatever tools they like and then upload them to pypi. > pip will be able to select the appropriate wheel for the end user's > OS, CPU, Python version etc. and install the binary wheel without > needing any compiler support on the target system. > > What this means is that it will be possible to do all the compilation > on developers machines so that the tools to do so don't need to be in > the stdlib any more. Then someone will be able to release the > "fancycompiler" package on pypi that will support features like > cross-compilation, other developers can use it, and the end user > doesn't need to care how the wheels were made. Yep, one of the key goals of current package efforts is to decouple builders from installers so that the developer's choice of build system has no impact on the end user's choice of installer. However, as a concession to practical reality, we're continuing with setuptools as the "official" build system for now, as devising a more tool neutral metabuild system to replace it will be a significant challenge in its own right, and we can only sensibly work on so many things at once. Cross compilation is one of the things the metabuild system would need to take into account, which is one of the reasons it has been deferred (see http://www.python.org/dev/peps/pep-0426/#metabuild-system - that draft design *doesn't* handle cross compilation at all) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Jul 17 14:17:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 17 Jul 2013 22:17:24 +1000 Subject: [Distutils] PEP 426 updated based on last round of discussion In-Reply-To: References: Message-ID: On 17 July 2013 21:45, Oscar Benjamin wrote: > On 17 July 2013 12:10, Paul Moore wrote: >> >> I can't imagine it's practical to auto-install a C compiler > > Why not? > >> - or even to check for one before building. >> >> But I can see it being useful for >> introspection purposes to know about this type of requirement. (A C compiler >> could be necessary, or optional for speedups, a particular external library >> could be needed, etc) > > Perhaps instead the installer tool would give you a way to clarify > that you do have a C compiler and to warn if not. > > Alternatively a meta-package could be used to indicate (when > installed) that a compatible C-compiler is available and then other > distributions could depend on it for building. > >> The data would likely only be as good as what project developers provide, >> but nevertheless having standard places to record the data could encourage >> doing so... >> >> OTOH, maybe this is metadata 3.0 stuff - I feel like at the moment we need >> to get what we have now out of the door rather than continually adding extra >> capabilities. > > I wasn't proposing to hold anything up or add new capabilities. I'm > just trying to see how far these changes go towards making non-pure > Python software automatically installable. Everything I would want to > build "build requires" software that is not on pypi. > > It would be great if e.g. the instructions for installing Cython on > Windows could just be "pip install cython" instead of this: > http://wiki.cython.org/InstallingOnWindows External dependencies are platform specific, so the task of defining them has been deliberately left to metadata extensions. Depending on a Windows exe or MSI, an OS X app bundle, a Fedora/RHEL/EPEL/Suse RPM or a Debian/Ubuntu DEB are all very different things not easily accommodated in a cross platform standard. That said, the new metadata standard does deliberately include a few pieces intended to make such things easier to define: 1. The extensions concept - using a structured data format like JSON makes it much easier for platform specific tools (or even pip itself) to say "declare this metadata, and we will run these commands automatically" 2. The "direct reference" format as a tool for linking a name with a specific URL 3. The "install hooks" system for running arbitrary additional commands post-install (if the installation tool is instructed to allow that) 4. A more robust mechanism for defining supported platforms 5. The extras system extended across all the different kinds of dependency (so if you don't want to build optional C accelerators, you can express that clearly and skip all the associated dependencies) Essentially, the standard tries to provide a better toolkit for solving these kinds of problems, without trying to solve them directly. It's still going to be up to other projects like zc.buildout and conda to provide a way to package that kind of thing up at least somewhat neatly. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From holger at merlinux.eu Wed Jul 17 14:18:51 2013 From: holger at merlinux.eu (holger krekel) Date: Wed, 17 Jul 2013 12:18:51 +0000 Subject: [Distutils] devpi-0.9.4: bug fixes for upload/install activities In-Reply-To: <20130716090810.GK3125@merlinux.eu> References: <20130716090810.GK3125@merlinux.eu> Message-ID: <20130717121851.GU1668@merlinux.eu> I just released 0.9.4 packages of the devpi system, providing a self-updating pypi caching and index server and an optional ``devpi`` command line tool to help with common upload/test/release activities. For general docs see: http://doc.devpi.net For the 0.9.4 bug fixes see below. best, holger krekel CHANGES 0.9.4 --------------------------- server: - fix issue where lookups into subpages of the simple index (simple/NAME/VER) would not trigger a 404 as they should. client: - fix uploading by adding setup.py's dir to sys.path: setup.py files that import modules/packages for obtaining versions etc. now work. Thanks jbasko. - fix automatic devpi-server startup on python26/windows -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: Digital signature URL: From benzolius at yahoo.com Wed Jul 17 11:43:04 2013 From: benzolius at yahoo.com (Benedek Zoltan) Date: Wed, 17 Jul 2013 02:43:04 -0700 (PDT) Subject: [Distutils] buildout doesn't work as expected, please help me Message-ID: <1374054184.35101.YahooMailNeo@web121603.mail.ne1.yahoo.com> Hi, I suspect the problem is with the changing infrastructure of setuptools-distribute. I'm happy to hear of this merge, thank you for all of you, who contributed to these tools. https://python-packaging-user-guide.readthedocs.org/en/latest/current.html I tried to clean everyting buildout made: rm -rf bin/ eggs/ parts/ rm bootstrap.py.installed.cfg wget http://downloads.buildout.org/1/bootstrap.py python bootstrap.py and after several tries finally worked. Could be helpful too: http://reinout.vanrees.org/weblog/2013/07/08/new-setuptools-buildout.html http://www.coresoftwaregroup.com/blog/distribute-setuptools-and-a-failing-bootstrap Thanks Zoli -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Wed Jul 17 14:40:02 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 17 Jul 2013 13:40:02 +0100 Subject: [Distutils] PEP 426 updated based on last round of discussion In-Reply-To: References: Message-ID: On 17 July 2013 13:17, Nick Coghlan wrote: > That said, the new metadata standard does deliberately include a few > pieces intended to make such things easier to define: > > 1. The extensions concept - using a structured data format like JSON > makes it much easier for platform specific tools (or even pip itself) > to say "declare this metadata, and we will run these commands > automatically" Okay, so this is where you can put the "I need a [specific] C-compiler" information. Then a pip alternative (or a future pip) that knew more about C compilation could respond appropriately. The PEP doesn't explicitly say anything about how a tool should handle unrecognised metadata extensions; it seems fairly obvious to me that they are supposed to be ignored but perhaps this should be explicitly stated. On the other hand it would be useful to be able to say: if you don't understand my "fortran" metadata extension then you don't know how to install/build this distribution. Is there a way e.g. to indicate a build/install dependency on the tool understanding some section in the extension metadata, or that an extension is compulsory somehow? Then a user could do: $ pip install autocont Error installing "autocont": required extension "fortran" not understood. See http://pypa.org/list_of_known_extensions.htm for more information. Oscar From ncoghlan at gmail.com Wed Jul 17 15:05:44 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 17 Jul 2013 23:05:44 +1000 Subject: [Distutils] buildout doesn't work as expected, please help me In-Reply-To: <1374054184.35101.YahooMailNeo@web121603.mail.ne1.yahoo.com> References: <1374054184.35101.YahooMailNeo@web121603.mail.ne1.yahoo.com> Message-ID: On 17 July 2013 19:43, Benedek Zoltan wrote: > Hi, > > I suspect the problem is with the changing infrastructure of > setuptools-distribute. > I'm happy to hear of this merge, thank you for all of you, who contributed > to these tools. > > https://python-packaging-user-guide.readthedocs.org/en/latest/current.html > > > I tried to clean everyting buildout made: > > rm -rf bin/ eggs/ parts/ > rm bootstrap.py .installed.cfg > > wget http://downloads.buildout.org/1/bootstrap.py > > python bootstrap.py > > and after several tries finally worked. Thanks for the update! Glad you were able to get it working, and sorry for the last few remnants of the setuptools/distribute fork process causing you trouble :( Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From theller at ctypes.org Wed Jul 17 15:13:16 2013 From: theller at ctypes.org (Thomas Heller) Date: Wed, 17 Jul 2013 15:13:16 +0200 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> References: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> Message-ID: Am 15.07.2013 19:26, schrieb Donald Stufft: > Maybe this is a crazy idea, but would a windows only extension work? > .pye(executable) Then just associate .pye with the launcher. Python > won't see .pye as importable so there's no shadow issues. pip.bat? Thomas From ncoghlan at gmail.com Wed Jul 17 15:09:58 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 17 Jul 2013 23:09:58 +1000 Subject: [Distutils] PEP 426 updated based on last round of discussion In-Reply-To: References: Message-ID: On 17 July 2013 22:40, Oscar Benjamin wrote: > On 17 July 2013 13:17, Nick Coghlan wrote: >> That said, the new metadata standard does deliberately include a few >> pieces intended to make such things easier to define: >> >> 1. The extensions concept - using a structured data format like JSON >> makes it much easier for platform specific tools (or even pip itself) >> to say "declare this metadata, and we will run these commands >> automatically" > > Okay, so this is where you can put the "I need a [specific] > C-compiler" information. Then a pip alternative (or a future pip) that > knew more about C compilation could respond appropriately. > > The PEP doesn't explicitly say anything about how a tool should handle > unrecognised metadata extensions; it seems fairly obvious to me that > they are supposed to be ignored but perhaps this should be explicitly > stated. > > On the other hand it would be useful to be able to say: if you don't > understand my "fortran" metadata extension then you don't know how to > install/build this distribution. Is there a way e.g. to indicate a > build/install dependency on the tool understanding some section in the > extension metadata, or that an extension is compulsory somehow? Yes, you can include a build_requires on something that understands the extension and make setup.py a shim that invokes that build system rather than setuptools/distutils (similar to the way d2to1 allows setup.cfg based projects work with tools that are expecting a setup.py file) It's not as elegant as a proper metabuild system (since there's a fair bit of legacy cruft in the setup.py CLI), but it works :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Wed Jul 17 15:17:37 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 17 Jul 2013 14:17:37 +0100 Subject: [Distutils] PEP 426 updated based on last round of discussion In-Reply-To: References: Message-ID: On 17 July 2013 12:45, Oscar Benjamin wrote: > > OTOH, maybe this is metadata 3.0 stuff - I feel like at the moment we > need > > to get what we have now out of the door rather than continually adding > extra > > capabilities. > > I wasn't proposing to hold anything up or add new capabilities. I'm > just trying to see how far these changes go towards making non-pure > Python software automatically installable. Everything I would want to > build "build requires" software that is not on pypi. > Sorry - my comment was more directed at myself suggesting extra capabilities in the metadata, than at you for asking the question. The more questions we ask about "how would this work in practice, for such-and-such use case" the better... Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Jul 17 15:33:15 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 17 Jul 2013 14:33:15 +0100 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> Message-ID: On 17 July 2013 14:13, Thomas Heller wrote: > Am 15.07.2013 19:26, schrieb Donald Stufft: > > Maybe this is a crazy idea, but would a windows only extension work? >> .pye(executable) Then just associate .pye with the launcher. Python >> won't see .pye as importable so there's no shadow issues. >> > > pip.bat? > That's my cue to cry :-) If you missed my earlier comments about bat files, then no - bat files have a significant number of failings that make them unsuitable for this sort of thing. The simplest example I can give is that bat files don't nest. So if the "pip" command is implemented as a bat file, and you have a script to build your virtualenv that looks like this: @echo off virtualenv foo foo\scripts\activate pip install wibble echo Complete! then the final message would never be executed as the pip bat file would not return to its caller. (Actually, the activate command is also a bat file, so pip would never be executed either, but you get my point). That's the worst sort of silent failure, and I have spent significant time in the past debugging scripts that fell foul of this behaviour. To fix this you have to use "call pip install wibble". And once you admit the possibility that certain commands could be implemented as bat files you have to either check everything, or use "call" everywhere, even when not necessary (e.g. "call python") just to be safe. If you don't think this problem is sufficient, I can offer more :-( I'm afraid exe files as wrappers are probably the only viable option. The basic reason is that the OS recognises exe files in all context, with no special configuration needed. This is not true of any other file type. So anything else will have corner cases that will give unexpected results. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From theller at ctypes.org Wed Jul 17 15:49:38 2013 From: theller at ctypes.org (Thomas Heller) Date: Wed, 17 Jul 2013 15:49:38 +0200 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> Message-ID: Am 17.07.2013 15:33, schrieb Paul Moore: > On 17 July 2013 14:13, Thomas Heller > wrote: > > Am 15.07.2013 19:26, schrieb Donald Stufft: > > Maybe this is a crazy idea, but would a windows only extension work? > .pye(executable) Then just associate .pye with the launcher. Python > won't see .pye as importable so there's no shadow issues. > > > pip.bat? > > > That's my cue to cry :-) > > If you missed my earlier comments about bat files, then no - bat files > have a significant number of failings that make them unsuitable for this > sort of thing. Yes, missed them. Sorry that I did not read enough of the discussion, I just stumbled over some messages in this thread and .bat popped up in my head. I still use them for quite some things although I have been bitten by their problems myself often enough. > I'm afraid exe files as wrappers are probably the only viable option. > The basic reason is that the OS recognises exe files in all context, > with no special configuration needed. This is not true of any other file > type. So anything else will have corner cases that will give unexpected > results. Thomas From p.f.moore at gmail.com Wed Jul 17 15:50:50 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 17 Jul 2013 14:50:50 +0100 Subject: [Distutils] Replacing pip.exe with a Python script In-Reply-To: References: <565EE2A4-AFC2-4DC5-8E71-90FDBD2A021A@stufft.io> Message-ID: On 17 July 2013 14:49, Thomas Heller wrote: > Yes, missed them. Sorry that I did not read enough of the discussion, > I just stumbled over some messages in this thread and .bat popped up in > my head. > No problem. It's probably useful to have a summary of the issues with bat files (more accurately, the reasons we need exe wrappers) written down here in any case. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Jul 17 17:12:01 2013 From: brett at python.org (Brett Cannon) Date: Wed, 17 Jul 2013 11:12:01 -0400 Subject: [Distutils] Q about best practices now (or near future) Message-ID: I'm going to be pushing an update to one of my projects to PyPI this week and so I figured I could use this opportunity to help with patches to the User Guide's packaging tutorial. But to do that I wanted to ask what the current best practices are. * Are we even close to suggesting wheels for source distributions? * Are we promoting (weakly, strongly?) the signing of distributions yet? * Are we saying "use setuptools" for everyone, or still only if you need it (asking since there is a stub about installing setuptools but the simple example doesn't have a direct need for it ATM, but could use find_packages() and such)? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Jul 17 17:46:05 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 17 Jul 2013 11:46:05 -0400 Subject: [Distutils] Q about best practices now (or near future) In-Reply-To: References: Message-ID: On Wed, Jul 17, 2013 at 11:12 AM, Brett Cannon wrote: > I'm going to be pushing an update to one of my projects to PyPI this week > and so I figured I could use this opportunity to help with patches to the > User Guide's packaging tutorial. > > But to do that I wanted to ask what the current best practices are. > > * Are we even close to suggesting wheels for source distributions? No, wheels don't replace source distributions at all. They just let you install something without having to have whatever built the wheel from its sdist. It is currently nice to have them available. I'd like to see an ambitious person begin uploading wheels that have no traditional sdist. > * Are we promoting (weakly, strongly?) the signing of distributions yet? No change. > * Are we saying "use setuptools" for everyone, or still only if you need it > (asking since there is a stub about installing setuptools but the simple > example doesn't have a direct need for it ATM, but could use find_packages() > and such)? Setuptools is the preferred distutils-derived system. distutils should no longer be considered morally superior. The MEBS idea, or a simple setup.py emulator and a contract with the installer on which commands it will actually call, will eventually let you do a proper job of choosing build systems. From ronaldoussoren at mac.com Wed Jul 17 17:49:45 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 17 Jul 2013 17:49:45 +0200 Subject: [Distutils] Q about best practices now (or near future) In-Reply-To: References: Message-ID: <16EB0FEF-6EBA-476C-A451-59FC3C1D0B5F@mac.com> On 17 Jul, 2013, at 17:46, Daniel Holth wrote: > On Wed, Jul 17, 2013 at 11:12 AM, Brett Cannon wrote: >> I'm going to be pushing an update to one of my projects to PyPI this week >> and so I figured I could use this opportunity to help with patches to the >> User Guide's packaging tutorial. >> >> But to do that I wanted to ask what the current best practices are. >> >> * Are we even close to suggesting wheels for source distributions? > > No, wheels don't replace source distributions at all. They just let > you install something without having to have whatever built the wheel > from its sdist. It is currently nice to have them available. > > I'd like to see an ambitious person begin uploading wheels that have > no traditional sdist. Do you mean an sdist without a setup.py? That will likely take some time, for the time being projects will still need a setup.py that just prints information on how to build them (or bootstraps the actual wheel building tool). Ronald From p.f.moore at gmail.com Wed Jul 17 17:55:45 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 17 Jul 2013 16:55:45 +0100 Subject: [Distutils] Q about best practices now (or near future) In-Reply-To: References: Message-ID: On 17 July 2013 16:46, Daniel Holth wrote: > > * Are we saying "use setuptools" for everyone, or still only if you need > it > > (asking since there is a stub about installing setuptools but the simple > > example doesn't have a direct need for it ATM, but could use > find_packages() > > and such)? > > Setuptools is the preferred distutils-derived system. distutils should > no longer be considered morally superior. > Personally, I still reserve judgement on setuptools. But that's mainly if you actually use its features (you should carefully consider and understand the implications if you use its script wrapper functionality, for example). I see no reason to knee-jerk use it if you don't use any of its functionality, though. I may be in a minority on that, though :-) > The MEBS idea, or a simple setup.py emulator and a contract with the > installer on which commands it will actually call, will eventually let > you do a proper job of choosing build systems. By the way, what *does* MEBS mean? I've seen a few people use the term, but never found an explanation... Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Jul 17 17:59:43 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 17 Jul 2013 11:59:43 -0400 Subject: [Distutils] Q about best practices now (or near future) In-Reply-To: References: Message-ID: On Wed, Jul 17, 2013 at 11:55 AM, Paul Moore wrote: > > On 17 July 2013 16:46, Daniel Holth wrote: >> >> > * Are we saying "use setuptools" for everyone, or still only if you need >> > it >> > (asking since there is a stub about installing setuptools but the simple >> > example doesn't have a direct need for it ATM, but could use >> > find_packages() >> > and such)? >> >> Setuptools is the preferred distutils-derived system. distutils should >> no longer be considered morally superior. > > > Personally, I still reserve judgement on setuptools. But that's mainly if > you actually use its features (you should carefull