Mailman 3 January 2014 - Distutils-SIG

Please do not remove dependency_links
by Hannes Schmidt 18 Jan '14

18 Jan '14

I read through the "Removing dependency_links" thread [1] and I beg you not follow through with the deprecation and removal of dependency_links and to rethink your approach. The mentioned thread indicates that research was done to gauge the popularity of the dependency_links in publicly hosted Python projects. That approach is fundamentally flawed: Publicly hosted projects are much more likely to also be available on PyPI than private, closed-source projects. Consequently, their dependencies are also more likely to be hosted on PyPI as well. Because of that, they are much less likely to rely on the dependency_links feature. Another misconception seem to be that dependency_links is predominantly used for installing patched or customized versions of dependencies hosted on PyPI. I'm pretty sure the predominant use case for dependency_links is with projects that are hosted privately, e.g. for an organization's internal use. I represent such an organization and removing dependency_links would impact us negatively. We host a set of internal projects and their dependencies on Bitbucket and we rely on dependency_links to install them directly from there. I understand the motivation for this change – security – but there must be smarter way to handle it. Could we fallback to dependency_links if a PyPI lookup isn't successful? Could we restrict dependency_links to links that share a prefix with the link from which the package is currently being installed? A combination of the two? [1]: https://mail.python.org/pipermail/distutils-sig/2013-October/022937.html -- Hannes Schmidt Software Application Developer Data Migration Engineer Cancer Genomics Hub University of California, Santa Cruz (206) 696-2316 (cell) hannes(a)ucsc.edu

6 17

pip 1.5.1rc1 and virtualenv 1.11.1rc2 released
by Donald Stufft 17 Jan '14

17 Jan '14

I’ve just released virtualenv 1.11.1rc2 and pip 1.5.1rc1, it fixes a number of issues with the latest releases of each one, but most importantly: * virtualenvs created with —system-site-packages not getting pip or setuptools installed into it * Errors about setuptools being too old * Bug fixes that affected certain third party servers hosting packages with Content-Encoding headers * Have —allow-unverified imply —allow-external * Bug fixes that affected cases when a DNS name failed to resolve * Errors with older versions of Python 2.6, most notably from OSX system Pythons In order to try this out in an isolated manner simply: $ curl -L -O https://github.com/pypa/virtualenv/archive/1.11.1rc2.tar.gz $ echo "b23fd95ef06effa2fde9f06fdb86ec38 1.11.1rc2.tar.gz" | md5sum -c 1.11.1rc2.tar.gz: OK $ tar zxf 1.11.1rc2.tar.gz $ python virtualenv-1.11.1rc2/virtualenv.py myVE $ myVE/bin/pip install SomePackage Report any bugs to the pip or virtualenv issues trackers located at: https://github.com/pypa/pip/issues https://github.com/pypa/virtualenv/issues Please do try these out and let us know of any issues! Thanks, ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 0

distlib 0.1.7 released on PyPI
by Vinay Sajip 16 Jan '14

16 Jan '14

I've just released version 0.1.7 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: Fixed issue #39: Changed to work in a situation where the user's home directory is unavailable. Fixed issue #40: Resources code manipulates paths differently for file system and zip resources. Script generation now uses correct logic to determine if running in a venv. Native strings are used in path computations to avoid errors on Windows with Unicode paths. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.1.7 [2] http://pythonhosted.org/distlib/overview.html#change-log-for-distlib [3] https://bitbucket.org/pypa/distlib/issues/new

1 0

"Python Packaging User Guide" has moved
by Marcus Smith 16 Jan '14

16 Jan '14

Fyi, the "Python Packaging User Guide" has moved from bitbucket to github. The new project home is here: https://github.com/pypa/python-packaging-user-guide and the built site is still here: https://python-packaging-user-guide.readthedocs.org/en/latest/ For those who are interested in helping, a quick survey of the text will show many sections marked up with "FIXME" comments. Although general editing is helpful, I think we mostly need "packaging experts" at this point who are willing to invest the time in filling out and correcting the sections they feel strong on. As stated in the README, "The guide is part of a larger effort to improve all of the packaging and installation docs, including pip, setuptools, virtualenv, and wheel. Ultimately, users need more than a "guide" to feel confident about the current tools. They need complete, accurate and inter-consistent documentation across all the projects" As such, a lot of my effort will be going into improving and normalizing the pip/setuptools/virtualenv/wheel docs directly so the Packaging User Guide doesn't have to do all the work on top of those docs. I have issues open for each of the four projects: https://github.com/pypa/python-packaging-user-guide/issues My goal is to have something that's sufficient for a public announcement by PyCon. Marcus

10 30

Metadata 2.0 and single-component version numbers
by Paul Moore 16 Jan '14

16 Jan '14

Metadata 2.0 doesn't support versions that are a single number. This has recently been flagged as a pip issue (https://github.com/pypa/pip/pull/1445) because pip won't handle wheels with a single-component version. We can obviously accept the patch for pip as a backward-compatibility measure, but I'm not sure if that's a good idea. wheels, and much of the new pip functionality, is based around Metadata 2.0 and the new packaging PEPs, at least in principle, so I'd rather get a better view on why Metadata 2.0 doesn't accept these versions. Based on that, we can decide what to do in pip. As a data point, pywin32 uses a single-digit version and can't be built with the free MS compilers, so it would be an ideal candidate for distribution as a wheel. But I've seen nothing from the project indicating that they are even aware of wheels yet, and there may be good reasons why wheels won't work for them (registration of WIndows COM components, for example) So - should Metadata 2.0 allow single-number versions? Should we be advocating a version format change to projects like pywin32? If neither, what will be the impact (and do people have views on whether it's a serious enough issue to warrant pip special-casing this)? Thanks, Paul

4 5

Re: [Distutils] setup.py issue - some files are included as intended, but one is not
by Ethan Furman 14 Jan '14

14 Jan '14

On 01/14/2014 01:26 PM, Dan Stromberg wrote: > On Sat, Jan 11, 2014 at 2:04 PM, Dan Stromberg <drsalists(a)gmail.com> wrote: >> Hi folks. >> >> I have a setup.py problem that's driving me nuts. > > Anyone? I've received 0 responses. I have no answer, but forwarding to Distutils (hopefully it's an appropriate topic ;) >> I have a treap.py file that tries to "import * from pyx_treap.so", and >> failing that, it'll "import * from py_treap.py" (sans extensions of >> course). Naturally, all 3 of these should be included - although in >> the case of pyx_treap.so, it probably should be a .c that's included. >> >> It is also intended to include nest.py, which has a simple form. >> >> Well, treap.py, nest.py and pyx_treap.c are included fine, but I can't >> seem to get py_treap.py to be included for some reason. >> >> I get no errors during "python setup.py sdist upload", but upon >> "python $(which pip) install treap", I get: >> $ sudo /usr/bin/python $(which pip) install treap >> Downloading/unpacking treap >> Running setup.py egg_info for package treap >> >> file py_treap.py (for module py_treap) not found >> Installing collected packages: treap >> Running setup.py install for treap >> file py_treap.py (for module py_treap) not found >> file py_treap.py (for module py_treap) not found >> >> file py_treap.py (for module py_treap) not found >> file py_treap.py (for module py_treap) not found >> Successfully installed treap >> Cleaning up... >> >> And it's not successfully installed - py_treap.py is missing. The pyx >> code does its job, so the problem is masked, other than the messages >> above, and the absence of py_treap.py from >> /usr/local/lib/python2.7/dist-packages >> >> I can clearly see py_treap.py in ./dist/treap-1.35.tar.gz - it's at >> least getting packaged up that much. It's not listed in >> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/SOURCES.txt >> , but I can see it in >> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/top_level.txt >> . >> >> My setup.py is at: >> http://stromberg.dnsalias.org/svn/treap/trunk/setup.py >> >> I've tried that setup.py and several variations on that theme, but >> none seem to include py_treap.py . >> >> Please make some suggestions? How can I get py_treap.py included in >> the pip install? >> >> Thanks!

2 1

PyPI download issues from Rackspace Cloud
by Marius Gedminas 14 Jan '14

14 Jan '14

I recently spin up a Windows VM on Rackspace Cloud. I'm seeing a very weird problem: downloads from PyPI fail with checksum errors, nondeterministically. Sometimes it's a md5 hash mismatch error from pip[1]. Sometimes the error is a CRC error deep in the gzip module. It's not only pip -- I've seen this error from virtualenv[2], I've seen it from get-pip.py. [1] https://jenkins.gedmin.as/job/restview-on-windows/1/console https://jenkins.gedmin.as/job/restview-on-windows/2/console https://jenkins.gedmin.as/job/restview-on-windows/3/console https://jenkins.gedmin.as/job/check-manifest-on-windows/11/console [2] https://jenkins.gedmin.as/job/check-manifest-on-windows/7/console https://jenkins.gedmin.as/job/check-manifest-on-windows/6/console The error is sporadic and tends to go away (and come back) on retries (look at the restview-on-windows jobs: first pip install mock fails, then succeeds, then fails again on next retry). I haven't encountered any download troubles with Firefox. In fact that's how I got 'pip install tox' to work in the end -- pypi downloads of virtualenv/py libraries kept failing, so I downloaded the tarballs with Firefox and pip installed them from local files. I've ran curl https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.10.1.tar.… | md5sum - like 10 times and got the same (correct) md5 hash each time. Meanwhile unset PIP_DOWNLOAD_CACHE rm -f virtualenv-1.10.1.tar.gz && pip install -d . virtualenv fails with a bad md5hash error 9 times out of 10. Curiously, it's the same bad hash every time: 20cf17baaf2126f99d6f652831f82d75. I used http://www.python.org/ftp/python/2.7.6/python-2.7.6.msi to install Python on this Windows 2012 server (64-bit OS, 32-bit Python). Any ideas? Marius Gedminas -- IBM motto: "We found five vowels hiding in a corner, and we used them _all_ for the 'eieio' instruction so that we wouldn't have to use them anywhere else" -- Linus Torvalds

3 5

A couple of questions on the PyPI API
by Paul Moore 12 Jan '14

12 Jan '14

I'm writing an application to maintain a mirror of PyPI package metadata, and I'm trying to understand the changelog data. In particular, I've not been able to find anything that lists the valid values for the "action" element of a change record. Looking at what's available if I just query the database, I get: pkg rename from XXX pkg, version remove pkg remove Owner XXX pkg remove Maintainer XXX pkg update hosting_mode pkg, version update list,of,fields pkg, version new release pkg add Owner XXX pkg add Maintainer XXX pkg, version add url XXX pkg, version add XXX file YYY But I don't think that can be all (for a start, there's no "add a new package" - unless the "new release" action can have a version of None, which I haven't seen) Is there a full list anywhere? I had a brief look at the Warehouse code, but short of tracking down all the places journal entries are stored, I couldn't see a way of finding out. Actually, given that the XMLRPC API only lets you get the releases of a package, the release data for a given release or a list of the url data for a given release, there's not really much need for me to worry beyond that level (so, for example, add url and add file both just mean "grab the url data and replace what's there", and "update" just means "replace the package data but don't bother refreshing the urls"). Does that sound reasonable? Thanks, Paul

1 0

Packaging today
by Steve Spicklemire 11 Jan '14

11 Jan '14

Hi Distutils folks, I'm kind of a lurker here, due primarily to the fact that I'm too swamped with various other things to materially contribute to the effort, but I've been lurking for some time hoping to learn enough to avoid troubling anyone with pesky questions. In that respect I've apparently failed, because here comes the question! Background: I'm helping out with a python package: vpython <http://vpython.org> and I'm also teaching an intro scientific computing class this spring. I'm mostly a Mac/Linux user, but my students are often windows users. I would love to permit my students to use enthought/canopy and/or continuum analytics (C.A.) along with vpython. At the moment we're creating binary releases of vpython for windows and mac and posting them on sourceforge <https://sourceforge.net/projects/vpythonwx/>. Bruce has been building the windows binary using VC (no setup.py) in a way that's compatible withpython.org python for windows. I've been building the mac version using a setup.py script I cobbled together that works on MacOSX and Linux. I've noticed that the anaconda system that C.A. installs uses MinGW on windows to build extensions. I'd love to figure out how to build vpython under this system so that my windows users could use them together transparently. (BTW is this the same 'anaconda' that has been discussed on this list recently, or is that something different?) I'm pretty sure I could work out how to build vpython with continuum analytics on the mac (which means building boost + wxPython using the C.A. python). Questions: Is there any way, *today*, to incorporate dependencies on external libraries (e.g., boost) in setup.py? (I've noticed the recent conversation about binary dependencies, but from the discussion it seemed to be mostly about the future...) Where should I go to get the 'latest' advise/documentation on distribution for a package that I want to distribute today (rather than pestering you folks)? thanks! -steve (P.S., I tried to send this earlier today, but it appears to have failed. Apologies if you've gotten this twice) Steve Spicklemire University of Indianapolis Dept. of Physics and Earth Space Sciences spicklemire(a)uindy.edu (317) 788-3313

4 12

Wheel "build tag"
by Bohuslav Kabrda 09 Jan '14

09 Jan '14

Hi all, the wheel package format, as defined in pep 427 specifies, that wheel name can contain an optional "build tag" [1]. This build tag is supposed to be a tie-breaker if two wheels have the same version. I decided that I would try to send a patch to upstream wheel project [2] to support adding this tag. I have a few questions and maybe someone here (Nick?) can help me with this: - Where is the "build tag" supposed to be stored in a built distribution? The pep doesn't seem to specify this. Should it be somewhere in METADATA or pydist.json? - How should the bdist_wheel command be told to use a "build tag"? By using some commandline option, e.g. "--build-tag 3"? My motivation for doing this is: For Fedora's Python 3.4 packaging, I'll have to unbundle the bundled setuptools and pip wheels and make ensurepip work differently. The best approach seems to be to make python3 require setuptools and pip (in other words, they'll always install along), so if user runs "python3 -m ensurepip", ensurepip won't need to do anything. This gets tricky if the same command is run in a virtualenv, since I don't have the bundled wheels. I've discussed this with Nick and we've agreed that the best solution would be to recreate the wheels from the system installed RPMs dynamically and then installing these wheels into virtualenv (I've already done some progress on this front with a small project that I call rewheel [3]). I'm now working on a downstream patch of ensurepip, that would use rewheel to do this. Now let me explain a bit about how we package setuptools/pip in Fedora. When there is a security issue or just a bug in a package, we don't upgrade it to latest upstream version in stable Fedora releases, since as a distribution we try to maintain API/ABI stability. We rather bump the "release" (which is sort of downstream version of the package). E.g. if there is a bug in package pip-%{version}-%{release}, we fix it and bump it to pip-%{version}-%{release+1}. But that means that if we fix a package and someone runs "python -m ensurepip --upgrade", he won't get the fix into virtualenv because the version is the same. That's why I'd like to use Fedora's release as the "build tag", so that this use case would work. I hope this all makes some sense :) Thanks! -- Regards, Bohuslav "Slavek" Kabrda. [1] http://www.python.org/dev/peps/pep-0427/#file-name-convention [2] https://bitbucket.org/dholth/wheel/ [3] https://github.com/bkabrda/rewheel

4 3