Mailman 3 May 2015 - Distutils-SIG

Embedding python (and added goodies) in a fundamentally relocatable product distribution
by Domen Kožar May 28, 2015

May 28, 2015

You could achieve that using Nix package manager[0] (I won't go into a long story about Conda vs Nix). Once you build python interpreter including all dependencies using buildPythonPackage.buildEnv[1], you'll get a closure (runtime dependency graph[2]) that can be copied to any system with the same architecture that it was built on. The binary doesn't require anything to be installed separately (even not Nix). We have a number of Python packages[6], but there is an on-going effort (lot's of work to be done) to package the whole PyPI (or at least have the capability to do so) with system dependencies. Currently a plain Python interpreter with all runtime dependencies is about 250MB, but this could be reduced down to 50MB using multiple outputs[3] - meaning each package is split into /lib and everything else and only /lib is used for runtime dependency graph. There is a Pull Request open[4] to address the closure size, but it's not yet ready to be merged (I expect that to happen in 2015). If you have more questions, you're welcome to join nix-dev[5] mailing list or come to #nixos channel on Freenode IRC network. Domen [0] http://nixos.org/nix [1] http://nixos.org/nixpkgs/manual/#python-build-env [2] http://static.domenkozar.com/slides/pycon-us-2015/#/5 [3] http://static.domenkozar.com/slides/pycon-us-2015/#/24 [4] https://github.com/NixOS/nixpkgs/pull/7701 [5] http://lists.science.uu.nl/mailman/listinfo/nix-dev [6] http://nixos.org/nixos/packages.html

1 0

Embedding python (and added goodies) in a fundamentally relocatable product distribution
by jrm May 27, 2015

May 27, 2015

I work for a company that ships software products containing ready-to-run python The products contain both a command line python and embedded python used to script complex applications. Our distributions are *fundamentally relocatable*. What I mean by this, is that our install is nothing but an archive extraction. Location information is not modified or patched in anywhere in the tree of files representing our distribution. Our products can be extracted on an NFS or SMB share - mounted differently on various client systems - and run fine everywhere. To distinguish our approach from static/install-time relocation - I sometimes call it a just in time install. We would like to improve the capability of our installed python, to include some of the typical scientific and engineering extensions (numpy, scipy, and so on). Growing weary (and wary) of doing all of our own builds for various third party software - I've begun looking at alternatives that collect a compatible set of python and extensions together. The most capable and easy to get collection of python - with scientific support - appears to be Conda. It's extremely good at pulling together a collection of extensions that play nicely together - and has an appropriately open license. Unfortunately, a Conda distribution must be statically located before use. Other distribution methods I've seen are similarly limited. The Conda distribution is also a bit of a space hog - using hard links that would expand to duplicate copies of files if contained within a distribution such as ours. I can imagine ways to patch up the Conda distribution with symbolic links and smarter start scripts - but I was wondering if any fundamentally better stuff was out there. Any ideas? -jrm James Mason Exa Corp Burlington, MA

2 1

Released: pip 7.0 and virtualenv 13.0
by Donald Stufft May 26, 2015

May 26, 2015

Hey, I'm happy to say that I've just cut the releases of pip 7.0 and virtualenv 13.0 and I have uploaded them to PyPI. For the full list of changes go visit the respective changelogs, however the biggest change here is that in pip 7.0 when pip finds and downloads a sdist, instead of installing that sdist directly it will instead build a wheel of that and cache it locally. From then on out it will use that cached wheel to install instead of downloading and building the sdist each time. This can have a profound impact upon installation speed. For instance, taking a look at the popular lxml library: # Without a locally cached wheel $ time pip install lxml ... pip install lxml 36.00s user 1.40s system 98% cpu 38.117 total # The next time, with a primed cache. $ time pip install lxml ... pip install lxml 0.61s user 0.10s system 94% cpu 0.750 total Some important notes about this new feature: * If the wheel project is not installed, then this feature will be disabled, however get-pip.py and virtualenv both will now install wheel by default. * If attempting to actually *build* the wheel fails for any reason, it will fall back to the older method of simply installing the sdist directly. * If a project cannot be installed correctly from a wheel, but it can successfully build a wheel, you can disable using wheels for that project by adding the flag --no-binary project1,project2,project3 to tell pip not to use binaries for those projects. You can use the :all: psuedo identifier to disable all wheels. I'm pretty excited about this release, caching built wheels is going to result in a tremendous speedup for a lot of common cases. As with any big change there is a pretty good change that this will cause breakages for some percentage of projects as well as have bugs within the system itself. As always, if you find a bug please feel free to open an issue up on the pip issue tracker at https://github.com/pypa/pip/issues. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

7 7

Beyond wheels 1.0: helping downstream, FHS and more
by David Cournapeau May 22, 2015

May 22, 2015

Hi there, During pycon, Nick mentioned there was interest in updating the wheel format to support downstream distributions. Nick mentioned Linux distributions, but I would like to express interest for other kind of downstream distributors like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work for Enthought). Right now, wheels have the following limitations for us: 1. lack of post/pre install/removing 2. more fine-grained installation scheme 3. lack of clarify on which tags vendors should use for custom wheels: some packages we provide would not be installable on "normal" python, and it would be nice to have a scheme to avoid confusion there as well. At least 1. and 2. are of interest not just for us. Regarding 2., it looks like anything in the <wheel_name>.data/data directory will be placed as is in sys.prefix by pip. This is how distutils scheme is defined ATM, but I am not sure whether that's by design or accident ? I would suggest to use something close to autotools, with some tweaks to work well on windows. I implemented something like this in my project bento ( https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfi…), but we could of course tweak that. For 1., I believe it was a conscious decision not to include them in wheel 1.0 ? Would it make sense to start a discussion to add it to wheel ? I will be at the pycon sprints until wednesday evening, so that we can flesh some concrete proposal first, if there is enough interest. As a background: at Enthought, we have been using eggs to distribute binaries of python packages and other packages (e.g. C libraries, compiled binaries, etc...) for a very long time. We had our own extensions to the egg format to support this, but I want to get out of eggs so as to make our own software more compatible with where the community is going. I would also like to avoid making ad-hoc extensions to wheels for our own purposes. thanks, David

18 109

Making pip and PyPI work with conda packages
by David Mertz May 21, 2015

May 21, 2015

I've just started monitoring this SIG to get a sense of the issues and status of things. I've also just started working for Continuum Analytics. Continuum has a great desire to make 'pip' work with conda packages. Obviously, we love for users to choose the Anaconda Python distribution but many will not for a variety of reasons (many good reasons). However, we would like for users of other distros still to be able to benefit from our creation of binary packages for many platforms in the conda format. As has been discussed in recent threads on dependency solving, the way conda provides metadata apart from entire packages makes much of that work easier. But even aside from that, there are simply a large number of well-tested packages (not only for Python, it is true, so that's possibly a wrinkle in the task) we have generated in conda format. It is true that right now, a user can in principle type: % pip install conda % conda install some_conda_package But that creates two separate systems for tracking what's installed and what dependencies are resolved; and many users will not want to convert completely to conda after that step. What would be better as a user experience would be to let users do this: % pip install --upgrade pip % pip install some_conda_package Whether that second command ultimately downloads code from pyip.python.org or from repo.continuum.io is probably less important for a user experience perspective. Continuum is very happy to upload all of our conda packages to PyPI if this would improve this user experience. Obviously, the idea here would be that the user would be able to type 'pip list' and friends afterward, and have knowledge of what was installed, even as conda packages. I'm hoping members of the SIG can help me understand both the technical and social obstacles that need to be overcome before this can happen. Yours, David... -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.

11 48

PyPI and Uploading Documentation
by Donald Stufft May 21, 2015

May 21, 2015

Hey! First, for anyone who isn't aware we recently migrated PyPI and TestPyPI so that instead of storing files and documentation locally (really in a glusterfs cluster) it will store them inside of S3. This will reduce maintenance overhead of running PyPI by two servers since we'll no longer need to run our own glusterfs cluster as well as improve the reliaiblity and scalability of the PyPI service as a whole since we've had nothing but problems from glusterfs in this regard. One of the things that this brought to light was that the documentation upload ability in PyPI is something that is not often used* however it represents something which is one of our slowest routes. It's not a well supported feature and I feel that it's going outside of the core competancy for PyPI itself and instead PyPI should be focused on the files themselves. In addition since the time this was added to PyPI a number of free services or cheap services have came about that allow people to sanely upload raw document without a reliance on any particular documentation system and we've also had the rise of ReadTheDocs for when someone is using Sphinx as their documentation system. I think that it's time to retire this aspect of PyPI which has never been well supported and instead focus on just the things that are core to PyPI. I don't have a fully concrete proposal for doing this, but I wanted to reach out here and figure out if anyone had any ideas. The rough idea I have currently is to simply disable new documentation uploads and add two new small features. One will allow users to delete their existing documentation from PyPI and the other would allow them to register a redirect which would take them from the current location to wherever they move their documentation too. In order to prevent breaking documentation for projects which are defunct or not actively maintained we would maintain the archived documentation (sans what anyone has deleted) indefinetely. Ideally I hope people start to use ReadTheDocs instead of PyPI itself. I think that ReadTheDocs is a great service with heavy ties to the Python community. They will do a better job at hosting documentation than PyPI ever could since that is their core goal. In addition there is a dialog between ReadTheDocs and PyPI where there is an opportunity to add integration between the two sites as well as features to ReadTheDocs that it currently lacks that people feel are a requirement before we move PyPI's documentation to read-only. Thoughts? * Out of ~60k projects only ~2.8k have ever uploaded documentation. It's not easy to tell if all of them are still using it as their primary source of documentation though or if it's old documentation that they just can't delete. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

11 14

Help required for setup.py
by salil GK May 20, 2015

May 20, 2015

Hello I was trying to create my package for distribution. I have a requirement that I need to check if one particular command is available in the system ( this command is not installed through a package - but a bundle is installed to get the command in the system ). I am using Ubuntu 14.04 Thanks in advance Salil

3 3

PyPI is a sick sick hoarder
by Robert Collins May 16, 2015

May 16, 2015

So, I am working on pip issue 988: pip doesn't resolve packages at all. This is O(packages^alternatives_per_package): if you are resolving 10 packages with 10 versions each, there are approximately 10^10 or 10G combinations. 10 packages with 100 versions each - 10^100. So - its going to depend pretty heavily on some good heuristics in whatever final algorithm makes its way in, but the problem is exacerbated by PyPI's nature. Most Linux (all that i'm aware of) distributions have at most 5 versions of a package to consider at any time - installed(might be None), current release, current release security updates, new release being upgraded to, new release being upgraded to's security updates. And their common worst case is actually 2 versions: installed==current release and one new release present. They map alternatives out into separate packages (e.g. when an older soname is deliberately kept across an ABI incompatibility, you end up with 2 packages, not 2 versions of one package). To when comparing pip's challenge to apt's: apt has ~20-30K packages, with altnernatives ~= 2, or pip has ~60K packages, with alternatives ~= 5.7 (I asked dstufft) Scaling the number of packages is relatively easy; scaling the number of alternatives is harder. Even 300 packages (the dependency tree for openstack) is ~2.4T combinations to probe. I wonder if it makes sense to give some back-pressure to people, or at the very least encourage them to remove distributions that: - they don't support anymore - have security holes If folk consider PyPI a sort of historical archive then perhaps we could have a feature to select 'supported' versions by the author, and allow a query parameter to ask for all the versions. -Rob -- Robert Collins <rbtcollins(a)hp.com> Distinguished Technologist HP Converged Cloud

10 23

new devpi-server/web/client/common: wheel testing, SHA256, ...
by holger krekel May 13, 2015

May 13, 2015

devpi-{server-2.2,web-2.3,client-2.2}: plugins, wheel support, pypi compat ============================================================================ With devpi-server-2.2, devpi-web-2.3, devpi-client-2.2 you'll get a host of fixes and improvements as well as some major new features for the private pypi system: - refined devpi-server plugin architecture based on "pluggy", the plugin and hook mechanism factored out from pytest. - devpi-client "test" now support testing of universal wheels IFF you have have an sdist alongside your wheel. python-specific wheels are not supported at this point. - devpi client "refresh NAME" allows to force a refresh of a pypi package from the command line. - internal refactorings to allow future pypi compatibility (for when it changes to SHA256 checksums). Also we use SHA256 checksumming ourselves now. For many more changes and fixes, please see the respective CHANGELOG of the respective packages. UPGRADE note: devpi-server-2.2 requires to ``--export`` your 2.1 server state and then using ``--import`` with the new version before you can serve your private packages through devpi-server-2.2. UPGRADE note: if you use ``updatetrigger_jenkins``, you have to install the devpi-jenkins plugin. You will get an error during import if you forget it. For docs and quickstart tutorials see http://doc.devpi.net many thanks to Florian Schulze who co-implemented many of the new features. And special thanks go to the two companies who funded major parts of the above work. have fun, holger krekel, merlinux GmbH

1 0

Beyond wheel 1.0: more fine-grained installation scheme
by David Cournapeau May 13, 2015

May 13, 2015

Hi, I am splitting up the previous thread into one thread / proposal to focus the discussion. Assuming the basis of this proposal does not sound too horrible, I would make a proof of concept in a pip branch, so that we can flush out the details and then write an actual spec (I guess an updated wheel format would need a new PEP ?). The goal of this thread is to flush out a more fine-grained installation scheme, so that wheels can install files anywhere they want (at least within sys.prefix/sys.exec_prefix). I see two issues: 1. defining what the scheme should be 2. how should it be implemented in wheel: there are different trade-offs depending on whether we want this feature to be part of wheel format 1.* or 2.0. First, my understanding of the current situation: * per the wheel PEP 427, anything in the wheel top directory and not in distribution-1.0.data is installed in site-package * every top directory in distribution-1.0.data/ needs to be mapped to the scheme as defined in distutils install command. * pip rejects any directory in distribution-1.0.data/ which is not in the scheme from 2. My suggestion for a better scheme would be to use an extended version of the various default directories defined by autotools. The extension would handle windows-specifics. More concretely: # Suggested variables The goal of supporting those variables is to take something that is flexible enough to support almost any installation scheme, without putting additional burden on the developer. People who do not want/need the flexibility will not need to do anything more than what they do today. The variables I would suggest are every variable defined in https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfi…, except for destdir which is not relevant here. On unix, the defaults follow autotools, and on windows, I mapped almost everything relative to sys.exec_prefix, except for the bindir/sbindir/libexecdir which map to "$prefix\Scripts". The $sitedir variable would need to be updated to use the value from distutils instead of the hardcoded value I put in that file as well. # How to handle the augmented scheme Right now, if I wanted to install something in say $prefix/share/doc, I would need to put it in distribution-1.0.data/data/share/doc, but this prevents use from handling different platforms differently. OTOH, this is the only way I can see to make the new scheme backward compatible with pip versions who would not understand the new scheme. I don't have a good sense of what we should do there, the core pip team may have a better sense. For now, I would be happy to just make a proof of concept not caring about backward compatibility in a pip branch. Does that sound like a workable basis to flush out an actual proposal ? thanks, David

4 5