Mailman 3 July 2013 - Distutils-SIG

Call for PEP author/champion: Bundling pip with CPython installers
by Nick Coghlan July 15, 2013

July 15, 2013

Based on the recent discussions, I now plan to reject the pip bootstrapping-on-first-invocation approach described in PEP 439 in favour of a new PEP that proposes: * bundling the latest version of pip with the CPython binary installers for Mac OS X and Windows for all future CPython releases (including maintenance releases) * aligns the proposal with the Python 3.4 release schedule by noting that CPython changes must be completed by the first 3.4 beta, while pip changes must be completed by the first 3.4 release candidate. * ensuring that, for Python 3.4, "python3" and "python3.4" are available for command line invocation of Python, even on Windows * ensuring that the bundled pip, for Python 3.4, ensures "pip", "pip3" and "pip3.4" are available for command line invocation of Python, even on Windows * ensuring that the bundled pip is able to upgrade/downgrade itself independent of the CPython release cycle * ensuring that pip is automatically available in virtual environments created with pyvenv * adding a "get-pip.py" script to Tools/scripts in the CPython repo for bootstrapping the latest pip release from PyPI into custom CPython source builds Note that there are still open questions to be resolved, which is why an author/champion is needed: * what guidance, if any, should we provide to Linux distro packagers? * how should maintenance updates handle the presence of an existing pip installation? Automatically upgrade older versions to the bundled version, while leaving newer versions alone? Force installation of the bundled version? Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 6

[issue155] 0.7 breaks existing packages with hyphens in 'name'
by Naftuli Tzvi Kay July 15, 2013

July 15, 2013

New submission from Naftuli Tzvi Kay: Please see this on StackOverflow for a full explanation: http://stackoverflow.com/questions/17659561/hyphens-in-project-names-in-set… I have a nice little script library called [buildout-starter](https://github.com/rfkrocktk/buildout-starter) which makes creating Buildout projects really easy. Then, the latest Buildout declared a dependency on `setuptools>=0.7`, so I had to upgrade my `setuptools` here to be compliant. After the upgrade, `setuptools` now fails along with my Buildout. Whereas the following would work before 0.7, it now fails: from setuptools import setup, find_packages setup( name = "tornado-chat-example", version = "0.0.1-SNAPSHOT", packages = find_packages('src'), package_dir = { '': 'src'}, install_requires = [ 'setuptools', ], ) My `src` directory looks like this: src ├── tornadochatexample └── tornado_chat_example.egg-info Here's the error I get: Develop: '/home/naftuli/tornado-chat-example/.' Installing python_section. Couldn't find index page for 'tornadochatexample' (maybe misspelled?) Getting distribution for 'tornadochatexample'. Couldn't find index page for 'tornadochatexample' (maybe misspelled?) While: Installing python_section. Getting distribution for 'tornadochatexample'. Error: Couldn't find a distribution for 'tornadochatexample'. Like I mentioned before, this example would seemingly run on `setuptools` 0.6, but now fails in the latest `setuptools` 0.7. How can I get this to work? I'd like my project to be named `tornado-chat-example` but have a package of `tornadochatexample`. Is this a bug, as it used to work before? ---------- messages: 739 nosy: rfkrocktk priority: bug status: unread title: 0.7 breaks existing packages with hyphens in 'name' _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue155> _______________________________________________

1 0

Current status of PEP 439 (pip boostrapping)
by Nick Coghlan July 15, 2013

July 15, 2013

In addition to the long thread based on Richard's latest set of updates, I've also received a few off-list comments on the current state of the proposal. So, I figured I'd start a new thread summarising my current point of view and see where we want to go from there. 1. However we end up solving the bootstrapping problem, I'm *definitely* a fan of us updating pyvenv in 3.4 to ensure that pip is available by default in new virtual environments created with that tool. I also have an idea for a related import system feature that I'll be sending to import-sig this afternoon (it's a variant on *.pth and *.egg-link files that should be able to address a variety of existing problems, including the one of *selectively* making system and user packages available in a virtual environment in a cross-platform way without needing to copy them) 2. While I was originally a fan of the "implicit bootstrapping on demand" design, I no longer like that notion. While Richard's bootstrap script is a very nice piece of work, the edge cases and "neat tricks" have built up to the point where they trip my "if the implementation is hard to explain, it's a bad idea" filter. Accordingly, I no longer think the implicit bootstrapping is a viable option. 3. That means there are two main options available to us that I still consider viable alternatives (the installer bundling idea was suggested in one of the off list comments I mentioned): * an explicit bootstrapping script * bundling a *full* copy of pip with the Python installers for Windows and Mac OS X, but installing it to site-packages rather than to the standard library directory. That way pip can be used to upgrade itself as normal, rather than making it part of the standard library per se. This is then closer to the "bundled application" model adopted for IDLE in PEP 434 (we could, in fact, move to distributing idle the same way). I'm currently leaning towards offering both, as we're going to need a tool for bootstrapping source builds, but the simplest way to bootstrap pip for Windows and Mac OS X users is to just *bundle a copy with the binary installers*. So long as the bundled copy looks *exactly* the way it would if installed later (so it can update itself), then we avoid the problem of coupling the pip update cycles to the standard library feature release cycle. The bundled version can be updated to the latest available versions when we do a Python maintenance release. For Linux, if you're using the system Python on a Debian or Fedora derivative, then "sudo apt-get python-pip" and "sudo yum install python-pip" are both straightforward, and if you're using something else, then it's unlikely getting pip bootstrapped using the bootstrap script is a task that will bother you :) The "python -m getpip" command is still something we will want to provide, as it is useful to people that build their own copy of Python from source. The bundling idea will obviously need to be discussed with the installer builders, and on python-dev in general, but that was always going to be the case for this PEP anyway (since it *does* touch CPython directly, rather than just being related to the packaging ecosystem). It achieves the aim of allowing people to assume some version of pip will be present on Python 3.4+ installations (or readily available in the case of Linux), while avoiding the problem of coupling pip updates to major Python version updates. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

10 26

Another conversation starter - pip documentation in the Python docs
by Paul Moore July 14, 2013

July 14, 2013

I don't think there is any doubt that we need to document to at least some extent how to use pip, in the Python documentation for 3.4. The obvious place is "Installing Python Modules" as a chapter to itself. What is less clear to me is how much to document - just basic "pip install XXX", or the whole pip command set, or somewhere in between? Should there be some type of "refer to www.pip-installer.org for the complete documentation" reference? Paul.

4 18

Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
by Paul Moore July 14, 2013

July 14, 2013

On 13 July 2013 10:05, Paul Moore <p.f.moore(a)gmail.com> wrote: > How robust is the process of upgrading pip using itself? Specifically on > Windows, where these things typically seem less reliable. OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it. The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but "a little bit broken" is still broken :-( On the other hand, "python -m pip install -U pip" works fine because it avoids the exe wrappers. There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend "don't do that" - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython. Does anyone have any suggestions? Paul. PS In better news, apart from this issue, pip upgrades of pip and setuptools seem fine.

6 13

buildout/setuptools/distribute unhelpful error message (0.7.x issue?)
by Chris Withers July 14, 2013

July 14, 2013

Hi All, What is this exception trying to tell me? Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-0.7.2.tar.gz Extracting in /tmp/tmpJNVsOY Now working in /tmp/tmpJNVsOY/setuptools-0.7.2 Building a Setuptools egg in /tmp/tmpBLZGeg /tmp/tmpBLZGeg/setuptools-0.7.2-py2.6.egg Traceback (most recent call last): File "bootstrap.py", line 91, in <module> pkg_resources.working_set.add_entry(path) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 451, in add_entry self.add(dist, entry, False) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 542, in add self._added_new(dist) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 705, in _added_new callback(dist) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2727, in <lambda> add_activation_listener(lambda dist: dist.activate()) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2227, in activate self.insert_on(path) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2328, in insert_on "with distribute. Found one at %s" % str(self.location)) ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /tmp/tmpBLZGeg/setuptools-0.7.2-py2.6.egg I don't see any distribute in there, and I don't know where it might be... $ python2.6 Python 2.6.8 (unknown, Jan 29 2013, 10:05:44) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import setuptools Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named setuptools cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

3 3

[issue154] file protection group/world writeable on egg-info files since 0.7x
by richard July 14, 2013

July 14, 2013

New submission from richard: since 0.7.x, there are some issues in the tarball setuptools.egg-info directory...namely, 1. the file protection of the files is group/world writeable (should be group/world read-only) 2. there are additional .orig files 3. there is an additional directory EGG-INFO containing an obsolete copy of PKG-INFO. This should be trivial to fix in the source tree. extract below from the 0.8 tarball: drwxrwxrwx 0/0 0 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/ -rw-rw-rw- 0/0 654 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/dependency_links.txt drwxrwxrwx 0/0 0 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/EGG-INFO/ -rw-rw-rw- 0/0 153 2013-07-02 17:30 setuptools-0.8/setuptools.egg-info/EGG-INFO/PKG-INFO -rw-rw-rw- 0/0 2773 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/entry_points.txt -rw-rw-rw- 0/0 2773 2013-07-03 14:12 setuptools-0.8/setuptools.egg-info/entry_points.txt.orig -rw-rw-rw- 0/0 45823 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/PKG-INFO -rw-rw-rw- 0/0 186 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/requires.txt -rw-rw-rw- 0/0 186 2013-07-03 14:12 setuptools-0.8/setuptools.egg-info/requires.txt.orig -rw-rw-rw- 0/0 3663 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/SOURCES.txt -rw-rw-rw- 0/0 49 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/top_level.txt -rw-rw-rw- 0/0 2 2013-07-02 17:48 setuptools-0.8/setuptools.egg-info/zip-safe ---------- messages: 737 nosy: richard priority: bug status: unread title: file protection group/world writeable on egg-info files since 0.7x _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue154> _______________________________________________

1 0

Best practices for optional C extensions
by Ben Darnell July 14, 2013

July 14, 2013

I'd like to add a C extension to speed up a small bit of code in a package (Tornado), but make it optional both for compatibility with non-cpython implementations and for ease of installation on systems without a C compiler available. Ideally any user who runs "pip install tornado" on a system capable of compiling extensions would get the extensions; if this capability cannot be detected automatically I'd prefer the opt-out case to be the one that requires non-default arguments. Are there any packages that provide a good example to follow for this? PEP 426 uses "c-accelerators" as an example of an "extra", but it's unclear how this would work (based on the equivalent setuptools feature). There doesn't appear to be a way to know what extras are requested at build time. If the extra required a package like cython then you could build the extension whenever that package is present, but what about hand-written extensions? Extras are also opt-in instead of opt-out, so I'd have to recommend that most people use "pip install tornado[fast]" instead of "pip install tornado" (with "tornado[slow]" available as an option for limited environments). Thanks, -Ben

3 2

flip the pip dependencies (was Current status of PEP 439)
by Marcus Smith July 13, 2013

July 13, 2013

> I think we need to flip the dependencies so that pip as the installer has > all the essential code for installation from PyPI and then setuptools and > distlib depend on that pip infrastructure. No need to add anything to the > standard library prematurely when we can add it to pip instead. > not sure about the flip, but let me break some things down a bit for those who don't know: what pip has internally already (i.e. literally in it's package namespace): - pypi crawling/downloading - wheel installing (does not require the pypi wheel project; only building wheels requires that) what pip has "bundled' already: - distlib (in 'pip.vendor'; currently only used for some --pre version logic) what pip still needs to be self-sufficient to do wheel installs: - something bundled or internal that does what pkg_resources does theoretical options: 1) bundle setuptools/pkg_resources 2) use the bundled distlib to replace our use of pkg_resources 3) internalize pkg_resources as pip.pkg_resources (i.e. fork off pkg_resources) Marcus

3 6

Best practices for optional C extensions
by Ben Darnell July 13, 2013

July 13, 2013

I'd like to add a C extension to speed up a small bit of code in a package (Tornado), but make it optional both for compatibility with non-cpython implementations and for ease of installation on systems without a C compiler available. Ideally any user who runs "pip install tornado" on a system capable of compiling extensions would get the extensions; if this capability cannot be detected automatically I'd prefer the opt-out case to be the one that requires non-default arguments. Are there any packages that provide a good example to follow for this? PEP 426 uses "c-accelerators" as an example of an "extra", but it's unclear how this would work (based on the equivalent setuptools feature). There doesn't appear to be a way to know what extras are requested at build time. If the extra required a package like cython then you could build the extension whenever that package is present, but what about hand-written extensions? Extras are also opt-in instead of opt-out, so I'd have to recommend that most people use "pip install tornado[fast]" instead of "pip install tornado" (with "tornado[slow]" available as an option for limited environments). Thanks, -Ben

1 0