Mailman 3 December 2013 - Distutils-SIG

Clarifying the meaning of the sysconfig "data" directory
by Nick Coghlan 08 Dec '13

08 Dec '13

In the binary dependency management threads, Daniel pointed out that in all currently defined sysconfig schemes, the "data" directory will end up pointing to the target installation directory. The "data" name in the scheme doesn't actually mean "this is a data file", it means "this has no defined semantic meaning at the sysconfig level, so just put it exactly where the relative path says to put it". That means that you can ship relatively arbitrary software in a wheel file by dumping it in "{distribution}-{version}.data/data/<path>", and then installing it to the appropriate target location (e.g. if you use FHS paths inside the data directory, then you would just need to install the wheel as "pip install --root /" to get things into the desired location). However, you'd be forgiven for not realising that from the sysconfig docs - I certainly didn't realise it until Daniel pointed it out. From the sysconfig docs, I had expected "data" to mean "application data files" not "arbitrary, potentially executable, file with no assigned semantics". So, I think we should fix the docs to reflect the way the schemes are defined in practice. I'd like to clarify them to say that "data" will always point to the installation target directory for the software being installed (on POSIX systems, typically /, /usr, /usr/local, ~/.local or a virtual environment directory), and thus the "data" path is intended for any file which doesn't fit one of the predefined categories - they're arbitrary data files from the *packaging system's* point of view, but not necessarily from the *application's* point of view. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 11

More control over uploading an sdist and egg to PyPI.
by Kannan Goundan 07 Dec '13

07 Dec '13

We use setuptools for our library. I'm trying to figure out a way to upload an sdist and an egg to PyPI without having to put my PyPI credentials in a file. Please let me know if this is not the right mailing list for this question. Our setup is that the entire team has access to the source code repository, but only the release managers have access to the credentials needed to make releases to PyPI (they're in a "fake" home directory). To do a release, you run: cd project-dir HOME="/release-manager-secrets/fake-pypi-home-dir" \ python setup.py bdist_egg sdist upload This works ok, but we would prefer to not have our PyPI credentials on the filesystem (unencrypted) at all. I'd like to run "setup.py bdist_egg sdist" first and then run a separate command that uploaded the egg and sdist. This command would ask for my credentials on stdin. How can I write such a program? Is there a PyPI HTTP API I can use? Does setuptools have a public interface that I can call? Thanks! - Kannan

2 2

Install a script to <prefix>/sbin instead of <prefix>/bin
by Michael Jansen 07 Dec '13

07 Dec '13

Hi I am currently working on the cobbler (http://cobblerd.org) setup.py and trying to improve it. It currently is only capable of installing to /usr for several reasons. Since i would like to have virtualenv support when running it i am trying to change that. While doing that i managed to meet one of the standard problems with distutils/setuptools - a search returns results of 2004 or older - which still seems to be unsolved. I would be willing to change that but my recent research leaves me a bit confused with all that setuptools is not dead, distlibs, tuf and wheels stuff. So here is the question. If i am willing to do the work is it possible to add something like sbin_scripts (or admin_scripts?) to distutils? I locally solved the problem like that http://pastebin.kde.org/pqrwrud1p (or attached). The implementation expands INSTALL_SCHEMA and reuses install_scripts to install those sbin_scripts too. That could be debatable. Perhaps a dedicated build|install_xxx_scripts command. I want to add that fhs also has a libexec folder (usually /usr/lib/<project>/libexec these days) destined for scripts that are supposed to be only called by other scripts/programs not manually. And cobbler for installs python scripts for modwsgi into /srv/www/cobbler/ . So i guess just adding sbin_scripts is not really the solution. Perhaps something more flexible is needed. Or should i wait for nextgen distlib based swiss knife python build tool and for now keep my local additions (there are more). -- Michael Jansen http://michael-jansen.biz

6 25

Binary dependency management, round 2 :)
by Nick Coghlan 06 Dec '13

06 Dec '13

There was some really good feedback in the binary dependency thread, but it ended up going off on a few different tangents. Rather than expecting people to read the whole thing, I figured I'd try to come up with a summary of where it has gone so far, and where we might want to take it from here. Given the diversity of topics, this should arguably be multiple new threads, but that approach has its own problems, since some of these topics are at least somewhat interrelated. In several cases, some of the discussion could possible move to the tracker issues I created :) == Regarding documentation == One of the things I got from the thread is that we don't currently have a clear overview published anywhere of *why* people use binary extensions. The distutils docs and both dated and focus solely on the mechanics, without discussing either use cases, or the benefits and limitations of using them. We also don't have a good location to describe the approach of statically linking or bundling additional libraries into "platlib" to avoid issues with external dependencies and incompatible ABI changes when distributing wheel files. This seems like something that would be suitably covered as an "Advanced Topic" in the packaging user's guide, so I filed an issue with some more specific ideas: https://bitbucket.org/pypa/python-packaging-user-guide/issue/36/add-a-secti… == Regarding conda == In terms of providing an answer to the question "Where does conda fit in the scheme of packaging tools?", my conclusion from the thread is that once a couple of security related issues are fixed (think PyPI before the rubygems.org compromise for the current state of conda's security model), and once the Python 3.3 compatibility issue is addressed on Windows, it would be reasonable to recommend it as one of the current options for getting hold of pre-built versions of the scientific Python stack. I think this is important enough to warrant a "NumPy and the Scientific Python stack" section in the user guide (with Linux distro packages, Windows installers and conda all discussed as options): https://bitbucket.org/pypa/python-packaging-user-guide/issue/37/add-a-dedic… == Regarding alternative index servers == Something I believe conda supports is the idea of installer configuration settings that are specific to a particular virtual environment (I can't find specific docs confirming that capability, but I certainly got that impression somewhere along the line). At the moment, it isn't straightforward to tell pip "when in this virtual environment, use these additional settings", but I believe such a feature could prove useful in dealing with some of the thornier binary compatibility problems. In particular, it would be good to be able to lock an environment to a particular custom index server that it will use instead of defaulting to PyPI. I've posted this idea to the pip issue tracker: https://github.com/pypa/pip/issues/1362 == Regarding NumPy, build variants and API/ABI consistency == My current reading of the NumPy situation is that it boils down to needing two additional features: - a richer platform tagging mechanism (which we need for *nix systems anyway) - a way to ensure internal consistency of the installed *builds* in an environment, not just the abstract dependencies I've opened a wheel format definition issue for the first problem: https://bitbucket.org/pypa/pypi-metadata-formats/issue/15/enhance-the-platf… I've posted a metadata 2.0 standard extension proposal for the latter: https://bitbucket.org/pypa/pypi-metadata-formats/issue/14/add-a-metadata-ba… (the alternative index server idea is also relevant, since PyPI wouldn't support hosting multiple variants with the same wheel level compatibility tags) == Regarding custom installation directories == This technically came up in the cobblerd thread (regarding installing scripts to /usr/sbin instead of /usr/bin), but I believe it may also be relevant to the problem of shipping external libraries inside wheels, static data files for applications, etc. It's a little underspecified in PEP 427, but the way the wheel format currently handles installation to paths other than purelib and platlib (or to install to both of those as part of the same wheel) is to use the sysconfig scheme names as subdirectories within the wheel's .data directory. This approach is great for making it easy to build well-behaved cross platform wheels that play nice with virtual environments, but allowing a "just put it here" escape clause could potentially be a useful approach for platform specific wheels (especially on *nix systems that use the Filesystem Hierarchy Standard). I've posted this idea to the metadata format issue tracker: https://bitbucket.org/pypa/pypi-metadata-formats/issue/13/add-a-new-subdire… Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

5 9

Handling the binary dependency management problem
by Nick Coghlan 06 Dec '13

06 Dec '13

I've had a couple of conversations recently that suggest it's high time I posted this particular idea publicly, so here's my suggestion for dealing with (or, more accurately, avoiding) the binary dependency management problem for upstream distribution in the near term. = Defining the problem = The core packaging standards and tools need to deal with a couple of different use cases. We want them to be usable for beginners and folks that aren't professional developers running Python locally, but we also want them to be suitable for professionally administered server environments, and to integrate reasonably well with downstream redistributors (most notably Linux distributions, but also other redistributors like ActiveState, Enthought and Continuum Analytics). For pure Python modules, these two objectives are easy to reconcile. For simple, self-contained, C extension modules, it gets harder, which is why PyPI only allows wheel files for Mac OS X and Windows, and expects them to be compatible with the binary installers provided on python.org. Resolving this to allow platform specific wheel files on PyPI for Linux distributions, Cygwin, MacPorts, homebrew, etc is going to require some work on better defining the "platform" tag in PEP 425. For arbitrary binary dependencies, however, I contend that reconciling the two different use cases is simply infeasible, as pip and venv have to abide by the following two restrictions: 1. Allow low impact security updates to components low in the stack (e.g. deploying a CPython security update shouldn't require rebuilding any other components) 2. Allow interoperability with a variety of packaging systems, including supporting access to Python modules provided solely through the system package manager (e.g. yum/rpm related modules on Fedora and derived distros) At this level, it's often impractical to offer prebuilt binary components with external dependencies, as you *also* need a mechanism to deliver the external binary dependencies themselves, that platform integrators can override if they want. = Proposed approach = Since supporting both cross-platform external binaries *and* allowing the substition of platform specific binaries in the same tool is difficult, I propose that we don't even try (at least not until we're much further down the packaging improvement road, and have solved all the *other* issues facing the packaging infrastructure) Instead, I suggest that the Python Packaging User Guide *explicitly* recommend a two level solution: 1. For simple cases without complex binary dependencies, and for cases where platform integration and/or low impact security updates are needed, use the core pip/virtualenv toolchain. This may sometimes mean needing to build components with external dependencies from source, and that's OK (it's the price we pay at this level for supporting platform integration). 2. For cross-platform handling of external binary dependencies, we recommend boostrapping the open source conda toolchain, and using that to install pre-built binaries (currently administered by the Continuum Analytics folks). Specifically, commands like the following should work on POSIX systems without needing any local build machinery, and without needing all the projects in the chain to publish wheels: "pip install conda && conda init && conda install ipython" For many end users just running things locally (especially beginners and non-developers), using conda will be the quickest and easiest way to get up and running. For professional developers and administrators, option 1 will provide the finer control and platform interoperability that they often need. conda already has good coverage of the scientific stack, but may need additional contributions to cover other notoriously hard to build components (such as crypto libraries and game development libraries). = What would this mean for the wheel format? = conda has its own binary distribution format, using hash based dependencies. It's this mechanism which allows it to provide reliable cross platform binary dependency management, but it's also the same mechanism that prevents low impact security updates and interoperability with platform provided packages. So wheels would remain the core binary format, and we'd continue to recommend publishing them whenever feasible. However, we'd postpone indefinitely the question of attempting to deal with arbitrary external dependency support. = Blocking issues = Even if we agree this a good way forward in the near term, there are a couple of technical issues that would need to be resolved before making it official in the user guide. There is currently at least one key blocking issue on the conda side, where it breaks if run inside a symlinked virtual environment (which both virtualenv and pyvenv create by default in POSIX environments): https://github.com/ContinuumIO/conda/issues/360 I believe conda also currently conflicts with virtualenvwrapper regarding the meaning of activate/deactivate at the command line. There aren't any blockers I am aware on the pip/virtualenv side, but the fact --always-copy is currently broken in virtualenv prevents a possible workaround for the conda issue above: https://github.com/pypa/virtualenv/issues/495 (pyvenv doesn't offer an --always-copy option, just the option to use symlinks on Windows where they're not used by default due to the associated permissions restrictions, so the primary resolution still needs to be for conda to correctly handle that situation and convert the venv Python to a copy rather than failing) Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

14 113

Please accept Python 3.4 selector
by Jesus Cea 05 Dec '13

05 Dec '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Just publishing a new release of Berkeley DB I got "'Programming Language :: Python :: 3.4'" selector is rejected as invalid. - -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUqBwEplgi5GaxT1NAQKVpgP/VHXatbfPRE0Wu9XF6psC9Ci+U/OFgHIr keD6FF+YRzpoT/hHtNWRnD+/eLfT7AigO5OXiyAhhgvwMjpv+9Q1MrtDNolpRKFl HJhyB5Po9fH6k9B/TTyngdJzRADtvfdDugWCUaxWjPwJaVH8K6ajhXo1kcE0PLig w6e6obS+Mc8= =pHa+ -----END PGP SIGNATURE-----

2 1

Re: [Distutils] Binary dependency management, round 2 :)
by Marcus Smith 05 Dec '13

05 Dec '13

> > >> ok, but could/would the pip/wheel toolchain ever expand itself to handle >> delivery of external dependencies (like qt, tk, and numpy's "fortran >> stuff"). >> > > "fortran stuff" is pretty poorly defined -- I'm not sure we'd ever want > pip to install a fortran compiler for you.... > to be very literal, I'm talking about this anaconda "system" package http://repo.continuum.io/pkgs/free/linux-64/system-5.8-1.tar.bz2 e.g., numpy's full requirement list in anaconda is like so (specifically for numpy-1.7.1-py27_0) openssl-1.0.1c-0 python-2.7.4-0 // not re-installed when using "conda init" readline-6.2-0 sqlite-3.7.13-0 system-5.8-1 // fortran "stuff" tk-8.5.13-0 zlib-1.2.7-0 > but Anoconda does some a nifty thing: it make s conda package that holds > the shared lib, then other packages that depend on it depend on that > package, so it will both get auto--installed > But I don't see why you couldn't do that with wheels. > exactly, that's what I'm really proposing/asking, is that maybe wheels should formally go in that direction. i.e. not just packaging python projects, but packaging non-python dependencies that python projects need (but have those dependencies be optional, for those who want to fulfill those deps using the OS package mgr)

2 1

Buildout - redo_pyc function too slow
by Kamal Mustafa 05 Dec '13

05 Dec '13

Installing large package such as Django on EC2 micro instance took a very long time, 8-9 minutes with 99% cpu usage. Initially, I taught it caused by setuptools analyzing the packages to figure out it zip_safe or not [1]. But after looking at this closely, that's not the case. Analyzing the egg only took few seconds and can be negligible to the total time it took to install the whole package. I have also test by adding zip_safe=False to django's setup.py and didn't see any drastic improvement to the time taken to install it. I test by using easy_install directly and it took around 1-2 minutes to finish so it mean the other 8 minutes being spent in buildout itself rather than in setuptools/easy_install. The install process basically went like this:- ... Writing /tmp/easy_install-GwjQPW/django-master/setup.cfg Running django-master/setup.py -q bdist_egg --dist-dir /tmp/easy_install-GwjQPW/django-mas ter/egg-dist-tmp-Yk_MYR warning: no previously-included files matching '__pycache__' found under directory '*' warning: no previously-included files matching '*.py[co]' found under directory '*' ... ... <LONG GAP HERE ...> Got Django 1.7. Picked: Django = 1.7 Generated script '/home/kamal/test_buildout/bin/django-admin.py'. Generated interpreter '/home/kamal/test_buildout/bin/python'. Stepping through the code, I figure out the LONG GAP starting after:- dists = self._call_easy_install( dist.location, ws, self._dest, dist) in line 531 of zc/buildout/easy_install.py. Next after this line is:- for dist in dists: redo_pyc(dist.location) Commenting this function call I manage to cut down the installation time to 2m30s. So what the purpose of this function ? Skipping it seem to be fine, I can import the package without any error. My buildout.cfg:- [buildout] parts = base [base] recipe = zc.recipe.egg eggs = Django interpreter = python The only reference to redo_pyc I found is http://www.makina-corpus.org/blog/minitage-10-out which just saying redo_pyc to be somehow slow. [1]:https://github.com/buildout/buildout/issues/116

1 0

Does pypi's simple index not set last modification time?
by Paul Moore 04 Dec '13

04 Dec '13

I'm trying to mirror the simple index (just the index, not the referenced files) in order to do some queries on package availability. I was planning on doing a simple "wget --timestamping --recursive --level=1" but when I do, everything seems to be downloaded each time. Does PyPI not set the last modified date on the index files? If I try wget -S it looks like the last modified date is always now. Is this caused by the CDN, or is it how PyPI works? Assuming last-modified *is* inaccurate, is there any other way of doing an incremental mirror of the simple index? Paul

1 0

Package a project
by Imran M Yousuf 02 Dec '13

02 Dec '13

Hi, I am new to setuptools. I am using it to build and package a project of mine. Currently if I execute `python setup.py bdist` it generates a tarball with all files located in paths './abs/path/to/project/bin/[entry points]' and './abs/path/to/project/lib/python-2.7/site-packages/[rest of the sources]'. This does not seem to be logical :(, I would rather want the binary distribution to be structure - './project-name/bin/' and './project-name/lib/'. Can some please advise me how to achieve it? I am using VirtualEnv for development of this project and its setup.py looks like - from setuptools import setup, find_packages setup(name='project-name', version='1.0', description='Description', author='Imran M Yousuf', author_email='imran(a)smitsol.com', url='http://www.smitsol.com', install_requires = ['setuptools', 'pycrypto==2.6'], packages=find_packages('src', ["tests"]), package_dir={'': 'src'}, test_suite="tests", entry_points={ 'console_scripts': ['manager=client.manager:main'] } ) Thank you, -- Imran M Yousuf Entrepreneur & CEO Smart IT Solution http://smitsol.com 25/5B, Block F, Haji Chinu Miah Road Bylane Joint Quarter, Mohammadpur Dhaka - 1207, Bangladesh Email: imran(a)smitsol.com Twitter: @imyousuf - http://twitter.com/imyousuf Skype: imyousuf Blog: http://imyousuf-tech.blogs.smartitengineering.com/ Mobile: +880-1711402557 +880-1746119494

3 5