Mailman 3 January 2014 - Distutils-SIG

Explicitly constructing SOABI for Python 2.7
by Nick Coghlan 29 Jan '14

29 Jan '14

Currently, bdist_wheel doesn't set the ABI tag properly on Python 2.x - the SOABI symbol it expects to find isn't present. PEP 425 also disclaims dealing with the problem. However, PEP 3149 (which defined SOABI) explains how to define that based on 3 other syconfig flags: Py_DEBUG, Py_UNICODE_SIZE and WITH_PYMALLOC >>> sysconfig.get_config_var("SOABI") >>> sysconfig.get_config_var("Py_DEBUG") 0 >>> sysconfig.get_config_var("Py_UNICODE_SIZE") 4 >>> sysconfig.get_config_var("WITH_PYMALLOC") 1 With those settings and the platform module (to get the current implementation), it is possible to figure out that the wheel ABI tag should be: cp27mu If Py_DEBUG was true, then it would be "cp27dmu" If WITH_PYMALLOC was false for some reason: "cp27u" And for a narrow Unicode build: "cp27m" That way, the ABI tagging would work properly in 2.x as well, rather than people getting the impression that wheels can't mark the ABI compatibility requirements properly. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 2

PyPI Migrated to New Infrastructure with some Breakage
by Donald Stufft 27 Jan '14

27 Jan '14

Today (Sat Jan 25, 2014) the Infrastructure team has migrated PyPI to new infrastructure. The old infrastructure was: - a single database server managed by OSUOSL - a pair of load balancers shared by all of the python.org services hosted on OSUOSL - a single backend VM that served as everything else for PyPI The VM that was acting as the backend server from PyPI was partially hand configured and partially setup to be managed by chef. Additionally it had an issue that caused it to kernel panic every so often which had been the cause of a number of downtimes in the last few months. Because it was primarily configured and administered by hand and because the way it was set up it was not feasible to have any sort of failover or spare. The new infrastructure is: - 2 Web VMs - 2 Database servers in Master/Slave Configuration - 2 PgPool Servers pooling connections to the database servers and load balancing reads across them. - 2 GlusterFS servers backed by Cloud Block Storage acting as the file storage for package and package docs - 1 metrics server to handle updating the download counts as they come in from Fastly All of the VMs are hosted on Rackspace’s Public Cloud and have their configuration completely controlled and managed using Salt. Going forward this will allow us to easily scale out as required or kill malfunctioning servers and spin up new ones easily. Additionally the setup has been setup so that where possible there is two servers performing the same role, ideally in an Active/Active configuration but at least in a Master/Slave configuration. This should allow PyPI to be far more stable moving forward and make downtimes much easier to recover from. The services are still fronted by Fasty’s CDN and in the new infrastructure we’ve removed our load balancer and have replaced it by having Fastly handle the load balancing for us. Additionally we’ve recently setup a static mirror of PyPI that is updated once every minute. This is hosted on Rackspace cloud as well but in a separate data center from the rest of PyPI. Fastly is configured to fall back to this static mirror in the case that neither of the two web heads are functioning. This should ensure that even in the event of a catastrophic failure of the PyPI service that the bulk of package installations should hopefully remain working. The bad news, (and the “Breakage” from the subject) is that while the new infrastructure was being planned out, built, and migrated to the “pypissh” package was forgotten. The pypissh package is an alternative way to upload packages to PyPI however it is very difficult, because of the way it works, to provide HA support for it as we’ve set up for everything else. We don’t have any numbers for how many people are actively using this package but looking at a roughly 2 week chunk of time in PyPI’s download history, the pypissh package was downloaded 7 times by a browser, and 7 times by pip. All other downloads were caused by the mirroring system. As of right now pypissh is non functional and due to the difficulty in HAing and monitoring the current setup and because it is apparently has a very small set of users we would like to effectively kill off this particular service. Additionally the benefits of pypissh have been reduced now that PyPI is available over a TLS connection with a well trusted certificate. My question to you is, is this something that distutils-sig is willing to have happen? If we are to re-enable pypissh we’ll need to write a new solution to doing it that can be properly HA’d and we’d prefer to put our efforts into improving things for a much larger set of people. So yea, PyPI should be loads more stable and more reliable now. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

9 18

pip 1.5.2 and virtualenv 1.11.2 released
by Donald Stufft 26 Jan '14

26 Jan '14

I’ve just released pip 1.5.2 and virtualenv 1.11.2 1.5.2 (2014-01-26) ------------------ * Upgraded the vendored ``pkg_resources`` and ``_markerlib`` to setuptools 2.1. * Fixed an error that prevented accessing PyPI when pyopenssl, ndg-httpsclient, and pyasn1 are installed * Fixed an issue that caused trailing comments to be incorrectly included as part of the URL in a requirements file 1.11.2 (2014-01-26) ~~~~~~~~~~~~~~~~~~~ * Fixed easy_install installed virtualenvs by updated pip to 1.5.2 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 0

error message from testpypi.python.org
by Thomas Heller 23 Jan '14

23 Jan '14

I'm trying out to submit a wheel to testpypi.python.org and then install it from there with pip. It doesn't work; what does the following error message mean? c:\>py -3.4 -m pip install py2exe -r https://testpypi.python.org/pypi/ Exception: Traceback (most recent call last): File "C:\Python34-64\lib\site-packages\pip\basecommand.py", line 122, in main status = self.run(options, args) File "C:\Python34-64\lib\site-packages\pip\commands\install.py", line 258, in run for req in parse_requirements(filename, finder=finder, options=options, session=session): File "C:\Python34-64\lib\site-packages\pip\req.py", line 1572, in parse_requirements req = InstallRequirement.from_line(line, comes_from, prereleases=getattr(options, "pre", None)) File "C:\Python34-64\lib\site-packages\pip\req.py", line 172, in from_line return cls(req, comes_from, url=url, prereleases=prereleases) File "C:\Python34-64\lib\site-packages\pip\req.py", line 70, in __init__ req = pkg_resources.Requirement.parse(req) File "C:\Python34-64\lib\site-packages\pip\_vendor\pkg_resources.py", line 2612, in parse reqs = list(parse_requirements(s)) File "C:\Python34-64\lib\site-packages\pip\_vendor\pkg_resources.py", line 2538, in parse_requirements raise ValueError("Missing distribution spec", line) ValueError: ('Missing distribution spec', '<?xml version="1.0" encoding="utf-8"?>') Storing debug log for failure in C:\Users\thomas\AppData\Roaming\pip\pip.log Thanks, Thomas

3 3

Re: [Distutils] setup.py issue - some files are included as intended, but one is not
by Dan Stromberg 23 Jan '14

23 Jan '14

Hi all. Yes, it does currently require make. I'm very comfortable with make, but I'm open to redoing the make rules in setup.py if that'll help get this problem solved. It'd probably be less puzzling to Python people anyway. I kept increasing the version number as I tried different ways of getting setup.py to do what I wanted. That's why the version numbers don't appear to match up. If there's a better way of testing setup.py's, please let me know! Thanks! >hello, I checked out your project from svn. >looks like it requires some make commands, before it can be packaged? >I'm getting tripped up on that. >also confused in that your src tree is at 1.30, but the tar you mention is >1.35, and then your install dir was 1.31. >maybe get your src tree up to date with where your working copy, and post >any make commands that need to be run. > >On Tue, Jan 14, 2014 at 1:32 PM, Ethan Furman <ethan at stoneleaf.us> wrote: > >> On 01/14/2014 01:26 PM, Dan Stromberg wrote: >> >>> On Sat, Jan 11, 2014 at 2:04 PM, Dan Stromberg <drsalists at gmail.com> >>> wrote: >>> >>>> Hi folks. >>>> >>>> I have a setup.py problem that's driving me nuts. >>>> >>> >>> Anyone? I've received 0 responses. >>> >> >> I have no answer, but forwarding to Distutils (hopefully it's an >> appropriate topic ;) >> >> >> I have a treap.py file that tries to "import * from pyx_treap.so", and >>>> failing that, it'll "import * from py_treap.py" (sans extensions of >>>> course). Naturally, all 3 of these should be included - although in >>>> the case of pyx_treap.so, it probably should be a .c that's included. >>>> >>>> It is also intended to include nest.py, which has a simple form. >>>> >>>> Well, treap.py, nest.py and pyx_treap.c are included fine, but I can't >>>> seem to get py_treap.py to be included for some reason. >>>> >>>> I get no errors during "python setup.py sdist upload", but upon >>>> "python $(which pip) install treap", I get: >>>> $ sudo /usr/bin/python $(which pip) install treap >>>> Downloading/unpacking treap >>>> Running setup.py egg_info for package treap >>>> >>>> file py_treap.py (for module py_treap) not found >>>> Installing collected packages: treap >>>> Running setup.py install for treap >>>> file py_treap.py (for module py_treap) not found >>>> file py_treap.py (for module py_treap) not found >>>> >>>> file py_treap.py (for module py_treap) not found >>>> file py_treap.py (for module py_treap) not found >>>> Successfully installed treap >>>> Cleaning up... >>>> >>>> And it's not successfully installed - py_treap.py is missing. The pyx >>>> code does its job, so the problem is masked, other than the messages >>>> above, and the absence of py_treap.py from >>>> /usr/local/lib/python2.7/dist-packages >>>> >>>> I can clearly see py_treap.py in ./dist/treap-1.35.tar.gz - it's at >>>> least getting packaged up that much. It's not listed in >>>> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/SOURCES.txt >>>> , but I can see it in >>>> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/top_level.txt >>>> . >>>> >>>> My setup.py is at: >>>> http://stromberg.dnsalias.org/svn/treap/trunk/setup.py >>>> >>>> I've tried that setup.py and several variations on that theme, but >>>> none seem to include py_treap.py . >>>> >>>> Please make some suggestions? How can I get py_treap.py included in >>>> the pip install? >>>> >>>> Thanks! >>>> >>> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >

1 1

(no subject)
by Warren Postma 23 Jan '14

23 Jan '14

Today I ran across something that made me go "wat?". There is a package named wx in PyPI and pip happily installs it. It's completely useless, the module doc string is not in english, and it's definitely not wxPython. This has been noticed by others: https://bitbucket.org/pypa/pypi/issue/62/please-delete-or-rename-the-wx-pac… What is the proper channel for asking for the renaming or removal of a package? If this package is indeed useless, then it should be removed. If it is useful, but completely undocumented, and appears that import wx followed by dir(wx) shows absolutely nothing useful inside it. This is the entire content of the unfortunately named wx.py module: # -*- coding:utf-8 -*- """ 模拟 PHP 函数 file_get_content 我的Python测试模块 """ def file_get_contents(file): file = open(file) data = '' for line in file: data = data + line file.close() return data The last part translated from Translate.google.com is "My Python test module". How did this get into PyPI? It's pypi page url is: https://pypi.python.org/pypi/wx/1.0.0 Does anyone have to approve these things before they get into PyPI or can any post their hello world programs to PyPi? Can I write three lines, call it pyQt, and upload it? that particular four letter name is unclaimed, yet I bet most pip users would like it if it gave them an error rather than installing some package which is not PyQt, it is just called that. Warren

5 4

inconsistency between PEP425 and wheel tags
by Brian Wickman 22 Jan '14

22 Jan '14

For interpreter-specific tags PEP425 says: "The version is py_version_nodot. CPython gets away with no dot, but if one is needed the underscore _ is used instead. PyPy should probably use its own versions here pp18, pp19." But in practice if you build with an alternate interpreter you just get the equivalent CPython version level, e.g. pycrypto-2.6.1-pp27-none-macosx_10_7_x86_64.whl for PyPy 2.2.1, when the PEP indicates it should be pp22. As an implementer, should I follow the spec or follow reality? If the latter, should PEP425 be amended? thanks! brian twitter.com/wickman

4 5

Local version identifiers and wheels
by Paul Moore 21 Jan '14

21 Jan '14

PEP 440 allows for local version identifiers, to mark local patches of a distribution. As this stands, local versions cannot be used in wheels, because the identifier is delimited with a dash, which clashes with the separator for the different parts in a wheel filename. What needs to be done to address this? Should we use a different delimiter in PEP 440? Or should we specify a translation for the dash to be used in wheel filenames? This is directly relevant at the moment, because there is a discussion on the distlib tracker of an API to allow locally modifying wheels, and I'm advocating that if this is done, the modified wheel should be forced to have a local version identifier, to avoid confusion with "official" versions. Any thoughts? Thanks, Paul

1 0

Pip 1.5.1 and Virtualenv 1.11.1 Released
by Donald Stufft 21 Jan '14

21 Jan '14

I’ve just released pip 1.5.1 and virtualenv 1.11.1 which fixes a number of issues with 1.5 and 1.11 respectively. It is recommend that all users upgrade to these releases. 1.5.1 (2014-01-20) ------------------ * pip now only requires setuptools (any setuptools, not a certain version) when installing distributions from src (i.e. not from wheel). (Pull #1434). * `get-pip.py` now installs setuptools, when it's not already installed (Pull #1475) * Don't decode downloaded files that have a ``Content-Encoding`` header. (Pull #1435) * Fix to correctly parse wheel filenames with single digit versions. (Pull #1445) * If `--allow-unverified` is used assume it also means `--allow-external`. (Pull #1457) 1.11.1 (2014-01-20) ~~~~~~~~~~~~~~~~~~~ * Fixed an issue where pip and setuptools were not getting installed when using the ``--system-site-packages`` flag. * Updated setuptools to fix an issue when installed with easy_install * Fixed an issue with Python 3.4 and sys.stdout encoding being set to ascii * Upgraded pip to v1.5.1 * Upgraded setuptools to v2.1 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 0

[ANN] pypiserver 1.1.5 - minimal private pypi server
by Ralf Schmitt 21 Jan '14

21 Jan '14

Hi, I've just uploaded pypiserver 1.1.5 to the python package index. pypiserver is a minimal PyPI compatible server. It can be used to serve a set of packages and eggs to easy_install or pip. pypiserver is easy to install (i.e. just 'pip install pypiserver'). It doesn't have any external dependencies. https://pypi.python.org/pypi/pypiserver/ should contain enough information to easily get you started running your own PyPI server in a few minutes. The code is available on github: https://github.com/schmir/pypiserver I didn't announce the previous version, so here's the changelog for the last two versions: 1.1.5 (2014-01-20) ------------------ - only list devpi-server and proxypypi as alternatives - fix wheel file handling for certain wheels - serve wheel files as application/octet-stream - make pypiserver executable from wheel file - build universal wheel - remove scripts subdirectory - add --index-url cli parameter 1.1.4 (2014-01-03) ------------------ - make pypiserver compatible with pip 1.5 (https://github.com/schmir/pypiserver/pull/42) -- Cheers Ralf

1 0