Mailman 3 March 2013 - Distutils-SIG

self.introduce(distutils-sig)
by Steve Dower 23 Mar '13

23 Mar '13

Hi all I just joined up after the various discussions at PyCon and wanted to say hi. (If you were also there and want to put a face/voice to the name, I did the Visual Studio demo at one of the lightning talks.) The main reason I want to get involved is the openly acknowledged lack of Windows expertise that's available. I work at Microsoft and part of my job description is to contribute code/testing/time/documentation/help/etc. to CPython. (I can also do testing/time/help for other projects, but copyrightable artifacts are more complicated and, for now, not okay with our lawyers.) I expect I'll mainly be lurking until I can be useful, which is why I wanted to start with this post. I'm pretty good with Windows, and I have direct access to all the experts and internal mailing lists. So just shout out when something comes up and I'll be happy to clarify or research an answer. Cheers, Steve

12 20

distribute does not install with python3 on Ubuntu machine
by Giulio Genovese 23 Mar '13

23 Mar '13

If I run (on an Ubuntu machine) the command: sudo pip install --upgrade distribute Everything goes well and the package installs. But if I run the same with Python3 with: sudo pip-3.2 install --upgrade distribute I get this: "File "setuptools/dist.py", line 103 except ValueError, e: ^ SyntaxError: invalid syntax" I think the problem is that with someone recently forgot to put the parentheses (i.e. "except ValueError, e:" should be "except (ValueError, e):") and therefore this does not work anymore with python3. It should be easy to fix. Furthermore, this problem seems to be widespread within the package. Giulio

3 3

Parallel installation of incompatible versions
by Nick Coghlan 21 Mar '13

21 Mar '13

pkg_resources.requires() is our only current solution for parallel installation of incompatible versions. This can be made to work and is a lot better than the nothing we had before it was created, but also has quite a few issues (and it can be a nightmare to debug when it goes wrong). Based on the exchanges with Mark McLoughlin the other week, and chatting to Matthias Klose here at the PyCon US sprints, I think I have a design that will let us support parallel installs in a way that builds on existing standards, while behaving more consistently in edge cases and without making sys.path ridiculously long even in systems with large numbers of potentially incompatible dependencies. The core of this proposal is to create an updated version of the installation database format that defines semantics for *.pth files inside .dist-info directories. Specifically, whereas *.pth files directly in site-packages are processed automatically when Python starts up, those inside dist-info directories would be processed only when explicitly requested (probably through a new distlib API). The processing of the *.pth file would insert it into the path immediately before the path entry containing the .dist-info directory (this is to avoid an issue with the pkg_resources insert-at-the-front-of-sys.path behaviour where system packages can end up shadowing those from a local source checkout, without running into the issue with append-to-the-end-of-sys.path where a specifically requested version is shadowed by a globally installed version) To use CherryPy2 and CherryPy3 on Fedora as an example, what this would allow is for CherryPy3 to be installed normally (i.e. directly in site-packages), while CherryPy2 would be installed as a split install, with the .dist-info going into site-packages and the actual package going somewhere else (more on that below). A cherrypy2.pth file inside the dist-info directory would reference the external location where cherrypy 2.x can be found. To use this at runtime, you would do something like: distlib.some_new_requires_api("CherryPy (2.2)") import cherrypy The other part of this question is how to avoid the potential explosion of one sys.path entry per dependency. The first part of that is that for cases where there is no incompatible version installed, there won't be a *.pth file, and hence no extra sys.path entry (the module/package will just be installed directly into site-packages as usual). The second part has to do with a possible way to organise the versioned installs: group them by the initial fragment of the version number according to semantic versioning. For example, define a "versioned-packages" directory that sits adjacent to "site-packages". When doing the parallel install of CherryPy2 the actual *code* would be installed into "versioned-packages/2/", with the cherrypy2.pth file pointing to that directory. For 0.x releases, there would be a directory per minor version, while for higher releases, there would only be a directory per major version. The nice thing though is that Python wouldn't actually care about the actual layout of the installed versions, so long as the *.pth files in the dist-info directories described the mapping correctly. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

5 15

Setuptools-Distribute merge announcement
by PJ Eby 21 Mar '13

21 Mar '13

Jason Coombs (head of the Distribute project) and I are working on merging the bulk of the improvements distribute made into the setuptools code base. He has volunteered to take over maintenance of setuptools, and I welcome his assistance. I appreciate the contributions made by the distribute maintainers over the years, and am glad to have Jason's help in getting those contributions into setuptools as well. Continuing to keep the code bases separate isn't helping anybody, and as setuptools moves once again into active development to deal with the upcoming shifts in the Python-wide packaging infrastructure (the new PEPs, formats, SSL, TUF, etc.), it makes sense to combine efforts. Aside from the problems experienced by people with one package that are fixed in the other, the biggest difficulties with the fork right now are faced by the maintainers of setuptools-driven projects like pip, virtualenv, and buildout, who have to either take sides in a conflict, or spend additional time and effort testing and integrating with both setuptools and distribute. We'd like to end that pain and simplify matters for end users by bringing distribute enhancements to setuptools and phasing out the distribute fork as soon as is practical. In the short term, our goal is to consolidate the projects to prevent duplication, wasted effort, and incompatibility, so that we can start moving forward. This merge will allow us to combine resources and teams, so that we may focus on a stable but actively-maintained toolset. In the longer term, the goal is for setuptools as a concept to become obsolete. For the first time, the Python packaging world has gotten to a point where there are PEPs *and implementations* for key parts of the packaging infrastructure that offer the potential to get rid of setuptools entirely. (Vinay Sajip's work on distlib, Daniel Holth's work on the "wheel" format, and Nick Coghlan's taking up the reins of the packaging PEPs and providing a clear vision for a new way of doing things -- these are just a few of the developments in recent play.) "Obsolete", however, doesn't mean unmaintained or undeveloped. In fact, for the "new way of doing things" to succeed, setuptools will need a lot of new features -- some small, some large -- to provide a migration path. At the moment, the merge is not yet complete. We are working on a common repository where the two projects' history has been spliced together, and are cleaning up the branch heads to facilitate re-merging them. We'd hoped to have this done by PyCon, but there have been a host of personal, health, and community issues consuming much of our available work time. But we decided to go ahead and make an announcement *now*, because with the big shifts taking place in the packaging world, there are people who need to know about the upcoming merge in order to make the best decisions about their own projects (e.g. pip, buildout, etc.) and to better support their own users. Thank you once again to all the distribute contributors, for the many fine improvements you've made to the setuptools package over the years, and I hope that you'll continue to make them in the future. (Especially as I begin to phase myself out of an active role in the project!) I now want to turn the floor over to Jason, who's put together a Roadmap/FAQ for what's going to be happening with the project going forward. We'll then both be here in the thread to address any questions or concerns you might have.

8 9

Building wheels - project metadata, and specifying compatibility
by Paul Moore 20 Mar '13

20 Mar '13

When building wheels, it is necessary to know details of the compatibility requirements of the code. The most common case is for pure Python code, where the code could in theory be valid for a single Python version, but in reality is more likely to be valid either for all Pythons, or sometimes for just Python 2 or Python 3 (where separate code bases or 2to3 are involved). The wheel project supports a "universal" flag in setup.cfg, which sets the compatibility flags to 'py2.py3', but that is only one case. Ultimately, we need a means (probably in metadata) for (pure Python) projects to specify any of the following: 1. The built code works on any version of Python (that the project supports) 2. The built code is specific to the major version of Python that it was built with 3. The built code is only usable for the precise Python version it was built with The default is currently (3), but this is arguably the least common case. Nearly all code will support at least (2) and more and more is supporting (1). Note that this is separate from the question of what versions the project supports. It's about how the code is written. Specifically, there's no point in marking code that uses new features in Python 3.3 as .py33 - it's still .py3 as it will work with Python 3.4. The fact that it won't work on Python 3.2 is just because the project doesn't support Python 3.2. Installing a .py3 wheel into Python 3.2 is no different from installing a sdist there. So overspecifying the wheel compatibility so that a sdist gets picked up for earlier versions isn't helpful. In addition to a means for projects to specify this themselves, tools (bdist_wheel, pip wheel) should probably have a means to override the default at the command line, as it will be some time before projects specify this information, even once it is standard. There's always the option to rename the generated file, but that feels like a hack... Where C extensions are involved, there are other questions. Mostly, compiled code is implementation, architecture, and minor version specific, so there's little to do here. The stable ABI is relevant, but I have no real experience of using it to know how that would work. There is also the case of projects with C accelerators - it would be good to be able to easily build both the accelerated version and a fallback pure-python wheel. I don't believe this is easy as things stand - distutils uses a compiler if it's present, so forcing a pure-python build when you have a compiler is harder work than it needs to be when building binary distributions. Comments? Should the default in bdist_wheel and pip wheel be changed or should it remain "as safe as possible" (practicality vs purity)? If the latter, should override flags be added, or is renaming the wheel in the absence of project metadata the recommended approach? And does anyone have any experience of how this might all work with C extensions? Paul

2 3

pip merges wheel
by Daniel Holth 20 Mar '13

20 Mar '13

Earlier today we merged the existing wheel branch into mainline pip. This adds opt-in wheel install support (built into pip, "pip install --use-wheel ...") and the convenient "pip wheel ..." command for creating the wheels you need. "pip wheel ..." uses the wheel reference implementation ("pip install wheel") to compile a dependency tree as .whl archives. Used together with "pip install --use-wheel ..." it provides a powerful way to speed up repeated installs and reap other good packaging benefits. I've been using this code in production for months and it works well. I am now a pip maintainer. We are committed to offering excellent wheel support in pip including a good way to produce and consume the format. In the future we will likely refactor the code to offer the same features with more distlib and less setuptools but this change will be mostly transparent to the end user. Once everyone is comfortable with the format we will move towards installing wheels by default when they are available instead of requiring the --use-wheel flag. Enjoy! Please share your experiences with the new feature. The most common issue is that you must install distribute >= 0.6.34 for everything to work well. Daniel Holth

9 21

PEP DRAFT - Inclusion of pip bootstrap in Python installation
by Richard Jones 20 Mar '13

20 Mar '13

Hi all, I present for your deliberation a draft PEP for the inclusion of a pip bootstrap program in Python 3.4. Discussion of this PEP should remain here on the distutils SIG. The PEP is revision controlled in my bitbucket account https://bitbucket.org/r1chardj0n3s/pypi-pep (this is also where I'm intending to develop the implementation.) Richard PEP: XXXX Title: Inclusion of pip bootstrap in Python installation Version: Last-Modified: Author: Richard Jones <richard(a)python.org> BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com> Discussions-To: <distutils-sig(a)python.org> Status: Draft Type: Standards Track Created: 18-Mar-2013 Python-Version: 3.4 Post-History: 19-Mar-2013 Abstract ======== This PEP proposes the inclusion of a pip boostrap executable in the Python installation to simplify the use of 3rd-party modules by Python users. This PEP does not propose to include the pip implementation in the Python standard library. Nor does it propose to implement any package management or installation mechanisms beyond those provided by PEPs 470 ("The Wheel Binary Package Format 1.0") and TODO distlib PEP. Rationale ========= Currently the user story for installing 3rd-party Python modules is not as simple as it could be. It requires that all 3rd-party modules inform the user of how to install the installer, typically via a link to the installer. That link may be out of date or the steps required to perform the install of the installer may be enough of a roadblock to prevent the user from further progress. Large Python projects which emphasise a low barrier to entry have shied away from depending on third party packages because of the introduction of this potential stumbling block for new users. With the inclusion of the package installer command in the standard Python installation the barrier to installing additional software is considerably reduced. It is hoped that this will therefore increase the likelihood that Python projects will reuse third party software. It is also hoped that this is reduces the number of proposals to include more and more software in the Python standard library, and therefore that more popular Python software is more easily upgradeable beyond requiring Python installation upgrades. Proposal ======== Python install includes an executable called "pip" that attempts to import pip machinery. If it can then the pip command proceeds as normal. If it cannot it will bootstrap pip by downloading the pip implementation wheel file. Once installed, the pip command proceeds as normal. A boostrap is used in the place of a the full pip code so that we don't have to bundle pip and also the install tool is upgradeable outside of the regular Python upgrade timeframe and processes. To avoid issues with sudo we will have the bootstrap default to installing the pip implementation to the per-user site-packages directory defined in PEP 370 and implemented in Python 2.6/3.0. Since we avoid installing to the system Python we also avoid conflicting with any other packaging system (on Linux systems, for example.) If the user is inside a virtual environment (TODO PEP ref) then the pip implementation will be installed into that virtual environment. The bootstrapping process will proceed as follows: 1. The user system has Python (3.4+) installed. In the "scripts" directory of the Python installation there is the bootstrap script called "pip". 2. The user will invoke a pip command, typically "pip install <package>", for example "pip install Django". 3. The boostrap script will attempt to import the pip implementation. If this succeeds, the pip command is processed normally. 4. On failing to import the pip implementation the bootstrap notifies the user that it is "upgrading pip" and contacts PyPI to obtain the latest download wheel file (see PEP 427.) 5. Upon downloading the file it is installed using the distlib installation machinery for wheel packages. Upon completing the installation the user is notified that "pip has been upgraded." TODO how is it verified? 6. The pip tool may now import the pip implementation and continues to process the requested user command normally. Users may be running in an environment which cannot access the public Internet and are relying solely on a local package repository. They would use the "-i" (Base URL of Python Package Index) argument to the "pip install" command. This use case will be handled by: 1. Recognising the command-line arguments that specify alternative or additional locations to discover packages and attempting to download the package from those locations. 2. If the package is not found there then we attempt to donwload it using the standard "https://pypi.python.org/pypi/simple/pip" index. 3. If that also fails, for any reason, we indicate to the user the operation we were attempting, the reason for failure (if we know it) and display further instructions for downloading and installing the file manually. Manual installation of the pip implementation will be supported through the manual download of the wheel file and "pip install <downloaded wheel file>". This installation will not perform standard pip installation steps of saving the file to a cache directory or updating any local database of installed files. The download of the pip implementation install file should be performed securely. The transport from pypi.python.org will be done over HTTPS but the CA certificate check will most likely not be performed. Therefore we will utilise the embedded signature support in the wheel format to validate the downloaded file. Beyond those arguments controlling index location and download options, the "pip" boostrap command may support further standard pip options for verbosity, quietness and logging. The "--no-install" option to the "pip" command will not affect the bootstrapping process. An additional new Python package will be proposed, "pypublish", which will be a tool for publishing packages to PyPI. It would replace the current "python setup.py register" and "python setup.py upload" distutils commands. Again because of the measured Python release cycle and extensive existing Python installations these commands are difficult to bugfix and extend. Additionally it is desired that the "register" and "upload" commands be able to be performed over HTTPS with certificate validation. Since shipping CA certificate keychains with Python is not really feasible (updating the keychain is quite difficult to manage) it is desirable that those commands, and the accompanying keychain, be made installable and upgradeable outside of Python itself. Implementation ============== TBD Risks ===== The Fedora variant of Linux has had a separate program called "pip" (a Perl package installer) available for install for some time. The current Python "pip" program is installed as "pip-python". It is hoped that the Fedora community will resolve this issue by renaming the Perl installer. Currently pip depends upon setuptools functionality. It is intended that before Python 3.4 is shipped that the required functionlity will be present in Python's standard library as the distlib module, and that pip would be modified to use that functionality when present. TODO PEP reference for distlib References ========== None, so far, beyond the PEPs. Acknowledgments =============== Nick Coghlan for his thoughts on the proposal and dealing with the Red Hat issue. Jannis Leidel and Carl Meyer for their thoughts. Copyright ========= This document has been placed in the public domain.

5 7

Setuptools-Distribute merge announcement
by Jason R. Coombs 19 Mar '13

19 Mar '13

As PJE mentioned in his e-mail, he and I have been working on a merge of the code lines of Setuptools and Distribute. I'm excited about this transition and I hope you are too. In this message, I will provide some answers based on questions that he and I encountered in our discussions and subsequent merge activity. If you have further questions, please direct them to both of us and we intend to answer promptly and also update the FAQ at the wiki (https://bitbucket.org/jaraco/setuptools/wiki/Setuptools%20and%20Distribute% 20Merge%20FAQ). - Jason R. Coombs Where does the merge occur? The merge is occurring between the heads of the default branch of Distribute and the setuptools-0.6 branch of Setuptools. The Setuptools SVN repo has been converted to a Mercurial repo hosted on Bitbucket. The work is still underway, so the exact changesets included may change, although the anticipated merge targets are Setuptools at 0.6c12 and Distribute at 0.6.35. What happens to other branches? Distribute 0.7 was abandoned long ago and won't be included in the resulting code tree, but may be retained for posterity in the original repo. Setuptools default branch (also 0.7 development) may also be abandoned or may be incorporated into the new merged line if desirable (and as resources allow). What history is lost/changed? As setuptools was not on Mercurial when the fork occurred and as Distribute did not include the full setuptools history (prior to the creation of the setuptools-0.6 branch), the two source trees were not compatible. In order to most effectively communicate the code history, the Distribute code was grafted onto the (originally private) setuptools Mercurial repo. Although this grafting maintained the full code history with names, dates, and changes, it did lose the original hashes of those changes. Therefore, references to changes by hash (including tags) are lost. Additionally, any heads that were not actively merged into the Distribute 0.6.35 release were also omitted. As a result, the changesets included in the merge repo are those from the original setuptools repo and all changesets ancestral to the Distribute 0.6.35 release. What features will be in the merged code base? In general, all "features" added in distribute will be included in setuptools. Where there exist conflicts or undesirable features, we will be explicit about what these limitations are. Changes that are backward-incompatible from setuptools 0.6 to distribute will likely be removed, and these also will be well documented. Bootstrapping scripts (ez_setup/distribute_setup) and docs, as with distribute, will be maintained in the repository and built as part of the release process. Documentation and bootstrapping scripts will be hosted at python.org, as they are with distribute now. Documentation at telecommunity will be updated to refer or redirect to the new, merged docs. On the whole, the merged setuptools should be largely compatible with the latest releases of both setuptools and distribute and will be an easy transition for users of either library. Who is invited to contribute? Who is excluded? While we've worked privately to initiate this merge due to the potential sensitivity of the topic, no one is excluded from this effort. We invite all members of the community, especially those most familiar with Python packaging and its challenges to join us in the effort. We have lots of ideas for how we'd like to improve the codebase, release process, everything. Like distribute, the post-merge setuptools will have its source hosted on bitbucket. (So if you're currently a distribute contributor, about the only thing that's going to change is the URL of the repository you follow.) Also like distribute, it'll support Python 3, and hopefully we'll soon merge Vinay Sajip's patches to make it run on Python 3 without needing 2to3 to be run on the code first. Why Setuptools and not Distribute or another name? We do understand that this announcement might be unsettling for some. The setuptools name has been subjected to a lot of deprecation in recent years, so the idea that it will now be the preferred name instead of distribute might be somewhat difficult or disorienting for some. We considered use of another name (Distribute or an entirely new name), but that would serve to only complicate matters further. Instead, our goal is to simplify the packaging landscape but without losing any hard-won advancements. We hope that the people who worked to spread the first message will be equally enthusiastic about spreading the new one, and we especially look forward to seeing the new posters and slogans celebrating the new setuptools. What is the timeframe of release? There are no hard timeframes for any of this effort, although progress is underway and a draft merge is underway and being tested privately. As an unfunded volunteer effort, our time to put in on it is limited, and we've both had some recent health and other challenges that have made working on this difficult, which in part explains why we haven't met our original deadline of a completed merge before PyCon. What version number can I expect for the new release? The new release will roughly follow the previous trend for setuptools and release the new release as 0.7. This number is somewhat arbitrary, but we wanted something other than 0.6 to distinguish it from its ancestor forks but not 1.0 to avoid putting too much emphasis on the release itself and to focus on merging the functionality. In the future, the project will likely adopt a versioning scheme similar to semver to convey semantic meaning about the release in the version number.

7 9

python-meta-packaging resource hub
by Kevin Horn 18 Mar '13

18 Mar '13

Howdy! I've been lurking on the list for a while, but have been pretty quiet so far. I was watching the live stream of the PyCon packaging panel today and the pytjon-meta-packaging resource hub idea was mentioned, which I hadn't heard of before, and was spurred to action. So I created a skeleton Sphinx project in a fork of this project here: https://bitbucket.org/khorn/python-meta-packaging Sadly, bitbucket won't let me do a pull request for some reason. If anyone knows why that is, feel free to let me know. (FYI I think it's because the main repo has no commits in it, which makes the "Create a pull request not load properly.) At any rate, somebody somewhere should feel free pull it into the main repo, so we can get things moving on that front. I'm happy to help out where I can. -- Kevin Horn

2 7

Packaging & Distribution Mini-Summit at PyCon US
by Nick Coghlan 14 Mar '13

14 Mar '13

As folks may be aware, I am moderating a panel called "Directions in Packaging" on the Saturday afternoon at PyCon US. Before that though, I am also organising what I am calling a "Packaging & Distribution Mini-Summit" as an open space on the Friday night (we have one of the larger open space rooms reserved, so we should have a fair bit of space if a decent crowd turns up). An overview of what I'm hoping we can achieve at the session is at https://us.pycon.org/2013/community/openspaces/packaginganddistributionmini… (that page should be editable by anyone that has registered for PyCon US). We're certainly not going to resolve all our disagreements and come up with a grand plan to "fix Python packaging" in a couple of hours on a Friday night. My hopes are far more modest: that we can get an idea of the different things people are worried about and trying to resolve, and start pondering ways we can work towards an cooperative ecosystem of interoperable tools, rather than the "one tool to rule them all" model of distutils. I fully expect that discussions on the Friday night will continue as hallway track discussions during the conference, development efforts at the sprints, and, of course, requests for feedback on the mailing lists (since there will likely be quite a few interested people that won't be present at PyCon US). This is not a new conversation - it is one that has been going on for years, and while some improvements have been made, we still have a long way to go. For those that are able to make it, I look forward to meeting you in person in March :) Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 4