Mailman 3 September 2013 - Distutils-SIG

mini-pip (Was: PEP 453 Round 2)
by anatoly techtonik 16 Sep '13

16 Sep '13

Because my proposal back in 2010 [1] and Richard Jones's PEP-439 [2] from 2013 were about the same thing - bootstrap script to get latest version of package management tool, and were rejected, I'd like to discuss the concept of mini-pip separately, as I believe not all points were addressed. Let's define what a mini-pip is. When pip is not installed: $ python -m pip install django pip package management tool is not installed on this system. Would you like to download and install pip first? [Yn] Y ... pip version x.x.x is installed successfully. Removing pip bootstrap script. Done installing pip. Proceeding with 'install django'. ... mini-pip is the human name for pip.py module that is bundled with Python. It is packaged as pip.py to allow users do 'python -m pip'. This script does two things: 1. informs users that pip is not installed 2. proposes to install pip and does this stuff if user agrees Algorithm is the following: 1. Start 2. Propose 3 Download or 4 Exit 3.1 Check server certificate 3.2 Check package signature 3.3 Unpack 3.4 Run python setup.py install 3.5 Remove itself 3.6 Continue to run user command * When pip.py is removed, "python -m pip" will start serving from pip/__main__.py * PEP 439 was rejected, because bootstrap script didn't ask user for permission, this proposal includes that More issues are below: On Sun, Sep 15, 2013 at 11:02 PM, Donald Stufft <donald(a)stufft.io> wrote: > On Sep 15, 2013, at 3:56 PM, Alex Burke <alexjeffburke(a)gmail.com> wrote: > > Instead of a copy of pip, could there not be minimal code to simply fetch > the wheel and install that or perhaps even use it directly (a la what was > possible with eggs)? > > Basically three reasons: > - There's a lot of code to handle a variety of situations in pip, it > would > be much harder to extract this code and keep it up to date in a way > that the "mini pip" could use it. It would also not be as battle > worn as > the pip code. > mini-pip doesn't need share or import anything with pip - it is a standalone bootstrap script, the basic parts of which - certificate checking, downloading, unpacking - should already be available from stdlib, so it is no different from shell script or any other system script except that it is cross-platform. There is no "variety of situations" in mini-pip - it doesn't do any argument processing, postponing it to the phase when main pip library can take the flag. > - On top of the one time cost there is the ongoing cost. As the > packaging > ecosystem progresses the stdlib implementation will need to be kept > up to date as well as the pip code. > pip code, yes, but not mini-pip bootstrap script. All it ever needs is a location of main pip package and it's signature. - We need to include a copy of pip in order to support offline installs > either > way. Since we have the private copy there to support offline we > might as > well use it to handle everything. > Offline installation is not an option - bootstrap script is for bootstrapping and it requires "external media" with the rest of "operating system" to be present. 1. https://mail.python.org/pipermail/distutils-sig/2010-March/015894.html 2. http://www.python.org/dev/peps/pep-0439/ -- anatoly t.

1 0

bdist_wheel (wheel) 0.22.0 released
by Daniel Holth 15 Sep '13

15 Sep '13

wheel 0.22.0 released Includes entry_points.txt -> "exports", "commands" conversion in experimental pydist.json, and a few Python 2.6 fixes and "wheel version" courtesy pombredanne. Daniel Holth

1 0

Rough road map posted
by Nick Coghlan 15 Sep '13

15 Sep '13

With some prompting from Marcus, I finally tidied up the tentative road map and posted it to the user guide (replacing my old essay): https://python-packaging-user-guide.readthedocs.org/en/latest/future.html I linked my old essay, along with the PyCon US Q&A panel and my PyCon AU talk from the resources section. If there's anything else people think is worth linking to, feel free to add it. If anything seems particularly off with the timeline, either let me know, send a pull request or just fix it directly if you have PyPA dev access on BitBucket (since I'm not completely across the current status of pip development). The source for the guide is at https://bitbucket.org/pypa/python-packaging-user-guide Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Packaging panel in Montreal next year?
by Nick Coghlan 12 Sep '13

12 Sep '13

The PyCon submission deadline is just a few days away, and I'm planning to submit a packaging Q&A panel again (same concept as the one this year, but with a further 12 months of progress to report). Before I do that, though, I just wanted to check there would be at least a few other distutils-sig denizens around that I could dragoon into participating. While it seems like a pretty safe bet, I don't see any reason to rely on assumptions when I don't need to :) Cheers, Nick.

2 1

pip does not work as expected from a zip
by Vinay Sajip 12 Sep '13

12 Sep '13

When run from a .zip file, pip does not seem to work as expected - it seems to need to be conventionally installed. I zipped up pip 1.4.1, setuptools (1.1.4) and wheel (0.21.0) into a .pyz. I have pip 1.0 and setuptools 0.6 installed in system site-packages. If I run "pip.pyz --version", it prints "1.0" rather than "1.4.1". If I run "pip.pyz wheel", it tells me that it needs setuptools >= 0.8. The problem seems to be that in at least these two places, pip runs code looking like dist = pkg_resources.get_distribution('XXX') which of course will not find the setuptools or pip in the .pyz. Is this behaviour by design? If so, it seems somewhat sub-optimal. If not, should I create an issue on the pip tracker? Regards, Vinay Sajip

4 8

PEP449 - Removal of the PyPI Mirror Auto Discovery and Naming Scheme
by Donald Stufft 12 Sep '13

12 Sep '13

Ok round two of this PEP. I've made a number of modifications to it: * Changed the title to be more relevant to the actual proposal. * Cleared up language to make it obvious that this was explicitly the naming and auto discovery being removed and not the mirrors in general. * Removed pointless distinctions between "Official" and "Unofficial" mirrors * Removed comments about staleness and points about the CDN which were not directly related to this PEP as the PEP doesn't attempt to solve issues with the mirroring protocol, just the discovery and naming scheme. * Incorporate feedback from the thread and simplify the proposal as well as extend the time frame. * Respond to some of the feedback from the thread that wasn't incorporated. I've explicitly CC'd Christian to this email so he can hopefully see it easier. Abstract ======== This PEP provides a path to deprecate and ultimately remove the auto discovery of PyPI mirrors as well as the hard coded naming scheme which requires delegating a domain name under pypi.python.org to a third party. Rationale ========= The PyPI mirroring infrastructure (defined in `PEP381`_) provides a means to mirror the content of PyPI used by the automatic installers. It also provides a method for auto discovery of mirrors and a consistent naming scheme. There are a number of problems with the auto discovery protocol and the naming scheme: * They give control over a \*.python.org domain name to a third party, allowing that third party to set or read cookies on the pypi.python.org and python.org domain name. * The use of a sub domain of pypi.python.org means that the mirror operators will never be able to get a SSL certificate of their own, and giving them one for a python.org domain name is unlikely to happen. * The auto discovery uses an unauthenticated protocol (DNS). * The lack of a TLS certificate on these domains means that clients can not be sure that they have not been a victim of DNS poisoning or a MITM attack. * The auto discovery protocol was designed to enable a client to automatically select a mirror for use. This is no longer a requirement because the CDN that PyPI is now using a globally distributed network of servers which will automatically select one close to the client without any effort on the clients part. * The auto discovery protocol and use of the consistent naming scheme has only ever been implemented by one installer (pip), and its implementation, besides being insecure, has serious issues with performance and is slated for removal with it's next release (1.5). * While there are provisions in `PEP381`_ that would solve *some* of these issues for a dedicated client it would not solve the issues that affect a users browser. Additionally these provisions have not been implemented by any installer to date. Due to the number of issues, some of them very serious, and the CDN which provides most of the benefit of the auto discovery and consistent naming scheme this PEP proposes to first deprecate and then remove the [a..z].pypi.python.org names for mirrors and the last.pypi.python.org name for the auto discovery protocol. The ability to mirror and the method of mirror will not be affected and will continue to exist as written in `PEP381`_. Operators of existing mirrors are encouraged to acquire their own domains and certificates to use for their mirrors if they wish to continue hosting them. Plan for Deprecation & Removal ============================== Immediately upon acceptance of this PEP documentation on PyPI will be updated to reflect the deprecated nature of the official public mirrors and will direct users to external resources like http://www.pypi-mirrors.org/ to discover unofficial public mirrors if they wish to use one. Mirror operators, if they wish to continue operating their mirror, should acquire a domain name to represent their mirror and, if they are able, a TLS certificate. Once they have acquired a domain they should redirect their assigned N.pypi.python.org domain name to their new domain. On Feb 15th, 2014 the DNS entries for [a..z].pypi.python.org and last.pypi.python.org will be removed. At any time prior to Feb 15th, 2014 a mirror operator may request that their domain name be reclaimed by PyPI and pointed back at the master. Why Feb 15th, 2014 ------------------ The most critical decision of this PEP is the final cut off date. If the date is too soon then it needlessly punishes people by forcing them to drop everything to update their deployment scripts. If the date is too far away then the extended period of time does not help with the migration effort and merely puts off the migration until a later date. The date of Feb 15th, 2014 has been chosen because it is roughly 6 months from the date of the PEP. This should ensure a lengthy period of time to enable people to update their deployment procedures to point to the new domains names without merely padding the cut off date. Why the DNS entries must be removed ----------------------------------- While it would be possible to simply reclaim the domain names used in mirror and direct them back at PyPI in order to prevent users from needing to update configurations to point away from those domains this has a number of issues. * Anyone who currently has these names hard coded in their configuration has them hard coded as HTTP. This means that by allowing these names to continue resolving we make it simple for a MITM operator to attack users by rewriting the redirect to HTTPS prior to giving it to the client. * The overhead of maintaining several domains pointing at PyPI has proved troublesome for the small number of N.pypi.python.org domains that have already been reclaimed. They often times get mis-configured when things change on the service which often leaves them broken for months at a time until somebody notices. By leaving them in we leave users of these domains open to random breakages which are less likely to get caught or noticed. * People using these domains have explicitly chosen to use them for one reason or another. One such reason may be because they do not wish to deploy from a host located in a particular country. If these domains continue to resolve but do not point at their existing locations we have silently removed this choice from the existing users of those domains. That being said, removing the entries *will* require users who have modified their configuration to either point back at the master (PyPI) or select a new mirror name to point at. This is regarded as a regrettable requirement to protect PyPI itself and the users of the mirrors from the attacks outlined above or, at the very least, require them to make an informed decision about the insecurity. Public or Private Mirrors ========================= The mirroring protocol will continue to exist as defined in `PEP381`_ and people are encouraged to to host public and private mirrors if they so desire. The recommended mirroring client is `Bandersnatch`_. .. _PyPI: https://pypi.python.org/ .. _PEP381: http://www.python.org/dev/peps/pep-0381/ .. _Bandersnatch: https://pypi.python.org/pypi/bandersnatch ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

5 18

OpenID login -> Login failed:NotAuthenticated('Replay attack detected', 9)
by Antoine Pitrou 11 Sep '13

11 Sep '13

Hi, I'm getting the following error when I try to login using OpenID on PyPI: Login failed:NotAuthenticated('Replay attack detected', 9) Is there any known cause for it? Regards Antoine.

3 3

bootstrap 1.5.2, python2.6 and DistributionNotFound
by Alessandro Dentella 11 Sep '13

11 Sep '13

Hi, I'm having problems using buildout on squeeze (that uses python 2.6). On ubuntu 12.10 (that uses python2.7) I have no problems. I'm still usig buildout 1.5.2 as I need a recipe for virtual env that is not yet available for buildout 2.+ (tl.buildout_virtual_python). When executing "python bootstrap" I'm getting: pkg_resources.DistributionNotFound: distribute This problem occurs also with a very simple conf that I found on the net when I started lunar.tar.gz, I uploaded the version I'm using here [1] after substituting bootstrap.py with a recent one. Here what I get: root@fw1-vpn-saa:/tmp/lunar/lunar# python bootstrap.py Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg Creating directory '/tmp/lunar/lunar/bin'. Creating directory '/tmp/lunar/lunar/parts'. Creating directory '/tmp/lunar/lunar/eggs'. Creating directory '/tmp/lunar/lunar/develop-eggs'. Getting distribution for 'setuptools'. /usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'src_root' warnings.warn(msg) Got setuptools 1.1.4. Generated script '/tmp/lunar/lunar/bin/buildout'. root@fw1-vpn-saa:/tmp/lunar/lunar# bin/buildout Traceback (most recent call last): File "bin/buildout", line 17, in <module> import zc.buildout.buildout File "/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py", line 40, in <module> import zc.buildout.download File "/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/download.py", line 20, in <module> from zc.buildout.easy_install import realpath File "/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/easy_install.py", line 31, in <module> import setuptools.package_index File "/usr/local/lib/python2.6/dist-packages/distribute-0.6.24-py2.6.egg/setuptools/package_index.py", line 157, in <module> sys.version[:3], require('distribute')[0].version File "build/bdist.linux-i686/egg/pkg_resources.py", line 673, in require File "build/bdist.linux-i686/egg/pkg_resources.py", line 576, in resolve dist = best[req.key] = env.best_match(req, self, installer) pkg_resources.DistributionNotFound: distribute If I use 'python bootstrap -d' that uses distribute, it will raise this error even on ubuntu 12.10/python2.7: root@fw1-vpn-saa:/tmp/lunar/lunar# python bootstrap.py -d Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.49.tar.gz Extracting in /tmp/tmpIsQPGq Now working in /tmp/tmpIsQPGq/distribute-0.6.49 Building a Distribute egg in /tmp/tmpkqSNaa /tmp/tmpkqSNaa/distribute-0.6.49-py2.6.egg Getting distribution for 'distribute'. warning: install_lib: 'build/lib.linux-i686-2.6' does not exist -- no Python modules to install Got distribute 0.7.3. While: Bootstrapping. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "/tmp/tmpkqSNaa/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py", line 1866, in main getattr(buildout, command)(args) File "/tmp/tmpkqSNaa/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py", line 399, in bootstrap ws.require('zc.buildout') File "build/bdist.linux-i686/egg/pkg_resources.py", line 698, in require needed = self.resolve(parse_requirements(requirements)) File "build/bdist.linux-i686/egg/pkg_resources.py", line 596, in resolve raise DistributionNotFound(req) DistributionNotFound: setuptools>=0.7 Any help is appreciated Thanks sandro [1] http://www.e-den.it/misc/lunar-sd.tar.gz

2 2

error in windows 7 installation
by Yuanyu Tseng 10 Sep '13

10 Sep '13

I use this "CLI-32.exe". It seems work. https://bitbucket.org/tarek/distribute/src/3bf82d0cbc6e0a6d383f177c4def1222…

1 0

Specification for a local PyPI simple index
by Paul Moore 09 Sep '13

09 Sep '13

Is the spec at http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api still the definitive version of what must be provided for a local PyPI index (for use by something like pip)? Or is there a more up to date version anywhere? For example, are MD5 signatures still the only supported version? I thought we were moving away from MD5. And while I haven't really followed the offsite hosting changes, are the rel="homepage"/rel="download" links still as stated? (I think I'd want rel="download" on everything as I only expect to provide URLs for actual package content). Also, how definitive is item 7, which states that the root URL must result in a page containing all projects, but it can be omitted if case-insensitive safe_name() matching of projects is implemented? The reason I ask is that providing a full listing will be somewhat costly in my application, but providing case-insensitive matching should be doable (actually, I'm not sure yet what's feasible, but I want to know whether it's worth my time even investigating). I'm still thinking about design at the moment, so what I need is far from decided, but I want to be sure that I'm actually implementing the correct spec as a starting point! Thanks, Paul

3 5

Distutils-SIG September 2013