Mailman 3 November 2016 - Distutils-SIG

Packaging multiple wheels in the same archive
by Nir Cohen 24 Nov '16

24 Nov '16

Hey all, As part of working on Cloudify (http://github.com/cloudify-cosmo) we've had to provide a way for customers to install our plugins in an environment where PyPI isn't accessible. These plugins are sets of Python packages which necessarily depend on one another (i.e. a regular python package with dependencies). We decided that we want to package sets of wheels together created or downloaded by `pip wheel`, add relevant metadata, package them together into a single archive (tar.gz or zip) and use the same tool which packs them up to install them later on, on the destination hosts. We came up with a tool (http://github.com/cloudify-cosmo/wagon) to do just that and that's what we currently use to create and install our plugins. While wheel solves the problem of generating wheels, there is no single, standard method for taking an entire set of dependencies packaged in a single location and installing them in a different location. We thought it would be a good idea to propose a PEP for that and wanted to get your feedback before we start writing the proposal. Our proposed implementation is not the issue here of course, but rather if you think there should be a PEP describing the way multiple wheels should be packaged together to create a standalone installable package. We would greatly appreciate your feedback on this. Thanks!

4 6

Fwd: Packaging multiple wheels in the same archive
by Wayne Werner 22 Nov '16

22 Nov '16

This should've gone to the list but apparently the google groups doesn't have me signed up? If you get a double post, sorry about that ---------- Forwarded message ---------- From: Wayne Werner <waynejwerner(a)gmail.com> Date: Tue, Nov 22, 2016 at 2:59 PM Subject: Re: [Distutils] Packaging multiple wheels in the same archive To: Nir Cohen <nir36g(a)gmail.com> Cc: distutils-sig <distutils-sig(a)googlegroups.com> FWIW you can already specify the `--find-links` option[1] to specify that you want to install wheels from a given directory. Add `--no-index` if you want to install from *only* that dir. An approach that certainly might be interesting is if pip supported a zipfile of wheels vs a directory of wheels, then this becomes as easy as zipping up all your wheels. -Wayne [1]: https://pip.pypa.io/en/stable/reference/pip_wheel/#cmdoption-f =========================================================== I welcome VSRE emails. Learn more at http://vsre.info/ =========================================================== On Tue, Nov 22, 2016 at 11:44 AM, Nir Cohen <nir36g(a)gmail.com> wrote: > Hey all, > > As part of working on Cloudify (http://github.com/cloudify-cosmo) we've > had to provide a way for customers to install our plugins in an environment > where PyPI isn't accessible. These plugins are sets of Python packages > which necessarily depend on one another (i.e. a regular python package with > dependencies). > > We decided that we want to package sets of wheels together created or > downloaded by `pip wheel`, add relevant metadata, package them together > into a single archive (tar.gz or zip) and use the same tool which packs > them up to install them later on, on the destination hosts. > > We came up with a tool (http://github.com/cloudify-cosmo/wagon) to do > just that and that's what we currently use to create and install our > plugins. > While wheel solves the problem of generating wheels, there is no single, > standard method for taking an entire set of dependencies packaged in a > single location and installing them in a different location. > > We thought it would be a good idea to propose a PEP for that and wanted to > get your feedback before we start writing the proposal. Our proposed > implementation is not the issue here of course, but rather if you think > there should be a PEP describing the way multiple wheels should be packaged > together to create a standalone installable package. > > We would greatly appreciate your feedback on this. > > Thanks! > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > https://mail.python.org/mailman/listinfo/distutils-sig > >

1 0

Role of distutils.cfg
by Christoph Groth 11 Nov '16

11 Nov '16

Hi, I have a question on how to best handle parameters to the distribution given that they can be shadowed by the global configuration file, distutils.cfg. Our project [1] contains C-extensions that include NumPy’s headers. To this end, our setup.py [2] sets the include_dirs parameter of setup() to NumPy’s include path. We have chosen this way, since it allows to add a common include path to all the extensions in one go. One advantage of this approach is that when the include_dirs parameters of the individual extensions are reconfigured (for example with a build configuration file), this does not interfere with the numpy include path. This has been working well for most of other users, but recently we got a bug report by someone for whom it doesn’t. It turns out that his system has a distutils.cfg that precedes over the include_dirs parameter to setup() [3]. My question is now: is there a policy on the correct use of distutils.cfg? After all, it can override any parameter to any distutils command. As such, is passing options like include_dirs to setup() a bad idea in the first place, or should rather the use of distutils.cfg be reserved to cases like choosing an alternative compiler? Thanks for any clarification, Christoph

3 5

Current Python packaging status (from my point of view)
by Nick Coghlan 09 Nov '16

09 Nov '16

Hi folks, Prompted by a few posts I read recently about the current state of the Python packaging ecosystem, I figured it made sense to put together an article summarising my own perspective on the current state of things: http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.h… It's pretty long, so the short version tailored specifically for the distutils-sig audience would be: * restating the point that pip & conda solve different problems, so while there's some overlap in their core capabilities, neither is a substitute for the other * we've actually managed to put some pretty hard problems behind us in the last few years, so my thanks to everyone that's played a part in that process * some of the thorniest problems that still remain really do require proper funding of the core ecosystem infrastructure (most notably PyPI), which means that either commercial redistributors need to step up and handle the problem on behalf of their customers, or else the PSF needs to figure out alternative sources of funding (with "let's do both!" really being my preferred outcome on that front) Cheers, Nick. P.S. For the Twitter users amongst you, feel free to pass along my link to the article: https://twitter.com/ncoghlan_dev/status/777087670837088256 -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

15 68

Released: pip v9.0.0
by Donald Stufft 02 Nov '16

02 Nov '16

I’d like to announce the release of pip v9.0. This release features: * The 9.x series will be the last pip versions to support Python 2.6. * Support for Requires-Python (will require additional support in setuptools/PyPI) to allow releasing sdists that will be ignored by specific versions of Python (e.g. foobar 5.x doesn’t get downloaded on 2.6). * Ability to pass platform, Python version, implementation, etc into ``pip download`` to download wheels for other platforms. * Add a ``pick check`` command to check the state of installed dependencies. * Add new formats for ``pip list``, including a new columnar layout and a JSON format for ease of scripting. For a full list of changes, see https://pip.pypa.io/en/stable/news/ <https://pip.pypa.io/en/stable/news/>. — Donald Stufft

1 0

Improving compiler (GCC) detection?
by Aleksandr Mezin 01 Nov '16

01 Nov '16

I have problems with distutils not detecting when compiler is GCC. It passes "-R" option on command line, which isn't supported by GCC without "-Wl,". In my build environment (I build Python from source), "CC" environment variable is set to "/usr/bin/cc", which actually points to GCC. But distutils simply checks executable name. Also I'd like to use ccache... So, I think, distutils should somehow check __GNUC__ definition instead. Are there any reasons why such detection can't/shouldn't be implemented? I'd like to work on the patch, but at first I'd like to know if it's possible/will be accepted. Also, clang on linux doesn't accept "-R" too: warning: unknown remark option '-R/usr/lib'; did you mean '-Rpass'? [-Wunknown-warning-option]

1 0