Mailman 3 December 2014 - Distutils-SIG

Local version identifiers from PEP 440 in practice
by Maurits van Rees Dec. 17, 2014

Dec. 17, 2014

I wanted to try out local version identifiers for packages. Or actually, I already used them, but with a wrong notation, and ran into trouble, also after fixing the notation. So I did some testing and made notes on what does or does not work. See conclusions at the bottom. Local version identifiers: general idea --------------------------------------- PEP440 is here, specifically the local version identifiers: http://legacy.python.org/dev/peps/pep-0440/#local-version-identifiers Use case in short, for those who do not know what local version identifiers are. Upstream community package largetrout has version 1.0. As integrator you notice a problem, fix it in the code, and create a pull request. While waiting for a new official release you already want to use the new code in production. If you release a version 1.1 yourself, this will conflict with a later official version. This is where local version identifiers come in. You create largetrout version 1.0+local.1 that you put in a directory on a webserver. You add the url of the package as find-link for pip/zc.buildout and can now use your local version. That is the general idea. How does it work in practice? Two things need to work: creating a distribution, and installing the distribution with pip or buildout. Creating a distribution ----------------------- I have created a very basic python project called 'myproject'. It does nothing. I have released a few versions here: http://pypi.zestsoftware.nl/public/packagingtest/ The various versions only differ in their version numbers and in the way in which the distributions were created. All were created with Python 2.7.8. on Mac OSX 10.9.5, with ``python setup.py sdist --formats=zip``. - setup.py version = 1.0 * Created with: setuptools 2.2 * Distribution file: myproject-1.0.zip * PKG-INFO version: 1.0 * works everywhere - setup.py version = 1.1+maurits.1 * Created with: setuptools 2.2 * Distribution file: myproject-1.1-maurits.1.zip * PKG-INFO version: 1.1-maurits.1 * NOT installable with zc.buildout 2.3 and setuptools 8 (see below) - setup.py version = 1.1+maurits.2 * Created with: setuptools 7.0 * Distribution file: myproject-1.1-maurits.2.zip * PKG-INFO version: 1.1-maurits.2 * NOT installable with zc.buildout 2.3 and setuptools 8 (see below) - setup.py version = 1.1+maurits.3 * Created with: setuptools 8.0.4 * Distribution file: myproject-1.1+maurits.3.zip * PKG-INFO version: 1.1+maurits.3 * NOT installable with current pip (see below) * hard to install with development pip (BUG, see below) * NOT installable with zc.buildout 2.2 and setuptools<8 So: with any setuptools version you can specify a local version identifier (``+something``) in the setup.py, but you need setuptools 8.0+ to correctly handle it. In earlier versions, the plus sign is replaced by a dash. This may lead to problems while installing; we will see that below. Note: ``python2.7 setup.py --version`` always correctly returns the version number as specified in setup.py. Installing with pip ------------------- Now let's see how installing goes. Again we are using Python 2.7. Below, some warnings or errors are expected (possibly with a workaround, marked with TIP), some may be surprising (marked with DANGER), others may be bugs (marked with BUG). I picked two combinations of pip and setuptools versions. Other combinations may give other results. current pip, setuptools 7.0 --------------------------- A basic virtualenv: $ pip list pip (1.5.6) setuptools (7.0) wsgiref (0.1.2) $ pip install -U -f http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.0 ... $ python -c "import myproject" && echo "importing works" importing works We try ``pip install`` for the available versions. - myproject==1.0 * importing works - myproject=1.1-maurits.1 * importing works - myproject==1.1-maurits.2 * importing works - myproject==1.1-maurits.3 * pip says: Could not find a version that satisfies the requirement myproject==1.1-maurits.3 (from versions: 1.0, 1.1, 1.1-maurits.1, 1.1-maurits.2) - myproject==1.1+maurits.3 * pip says: Exception: Traceback (most recent call last): ... File ".../pip/_vendor/pkg_resources.py", line 2583, in scan_list "Expected ',' or end-of-list in",line,"at",line[p:] ValueError: ("Expected ',' or end-of-list in", 'myproject==1.1+maurits.3', 'at', '+maurits.3') * DANGER: older pip versions cannot install packages with local version identifiers. - pip install http://pypi.zestsoftware.nl/public/packagingtest/myproject-1.1+maurits.3.zip * importing works development pip, setuptools 8.0 ------------------------------- A basic virtualenv: $ pip list pip (6.0.dev1) setuptools (8.0.4) $ pip install -U --trusted-host pypi.zestsoftware.nl -f http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.0 ... $ python -c "import myproject" && echo "importing works" importing works We try ``pip install`` for the available versions. - myproject==1.0 * importing works - myproject==1.1-maurits.1 * pip warns: Requested myproject==1.1-maurits.1, but installing version 1.1+maurits.1 * importing works - myproject==1.1-maurits.2 * pip warns: Requested myproject==1.1-maurits.2, but installing version 1.1+maurits.2 * importing works - myproject==1.1-maurits.3 * pip says: Collecting myproject==1.1+maurits.3 Could not find a version that satisfies the requirement myproject==1.1-maurits.3 (from versions: ) No distributions matching the version for myproject==1.1-maurits.3 - myproject==1.1+maurits.3 * pip says: Collecting myproject==1.1+maurits.3 Could not find a version that satisfies the requirement myproject==1.1+maurits.3 (from versions: ) No distributions matching the version for myproject==1.1+maurits.3 * BUG: this should work, right? - pip install http://pypi.zestsoftware.nl/public/packagingtest/myproject-1.1+maurits.3.zip * importing works Installing with buildout ------------------------ We try the same with buildout. I use this basic buildout config: [buildout] find-links = http://pypi.zestsoftware.nl/public/packagingtest/ eggs-directory = eggs download-cache = downloads parts = mypy test show-picked-versions = true [mypy] recipe = zc.recipe.egg eggs = myproject interpreter = mypy [test] recipe = plone.recipe.command command = grep myproject bin/mypy && bin/mypy -c "import myproject" && echo "importing works" update-command = ${:command} [versions] myproject = 1.0 plone.recipe.command = 1.1 #setuptools = ... #zc.buildout = ... zc.recipe.egg = 2.0.1 This buildout creates a bin/mypy python interpreter with the myproject package installed. And it greps bin/mypy for 'myproject' so I can easily see whether this has the version I expect. And it checks to see it importing the module works. Using previous zc.buildout and setuptools ----------------------------------------- zc.buildout = 2.2.5 setuptools = 7.0 - myproject = 1.0 * all is well - myproject = 1.1-maurits.1 * all is well - myproject = 1.1-maurits.2 * all is well - myproject = 1.1-maurits.3 * all is well - myproject = 1.1+maurits.3 * buildout fails: Traceback (most recent call last): ... File ".../setuptools-7.0-py2.7.egg/pkg_resources.py", line 2690, in scan_list raise ValueError(msg, line, "at", line[p:]) ValueError: ("Expected ',' or end-of-list in", 'myproject[]==1.1+maurits.3', 'at', '+maurits.3') * TIP: you can use a dash instead of a plus in the version pin. Using latest zc.buildout and setuptools --------------------------------------- zc.buildout = 2.3.1 setuptools = 8.0.2 - myproject = 1.0 * all is well - myproject = 1.1-maurits.1 * buildout fails: Installing myproject 1.1-maurits.1 Caused installation of a distribution: myproject 1.1+maurits.1 with a different version. Got None. While: Updating mypy. Error: There is a version conflict. We already have: myproject 1.1+maurits.1 We require myproject==1.1-maurits.1 * when I do not specify a version in the buildout config, but the latest on server is 1.1-maurits.1, buildout succeeds and importing works. buildout reports this (note the plus-sign): Versions had to be automatically picked. The following part definition lists the versions picked: [versions] myproject = 1.1+maurits.1 * BUG? pip only warns, buildout may want to do the same. Maybe somehow normalize versions before comparison. - myproject = 1.1-maurits.2 * buildout fails: Installing myproject 1.1-maurits.2 Caused installation of a distribution: myproject 1.1+maurits.2 with a different version. ... * So: same remarks as for 1.1-maurits.1 apply. - myproject = 1.1-maurits.3 * buildout fails: Error: Couldn't find a distribution for 'myproject==1.1-maurits.3'. - myproject = 1.1+maurits.3 * importing works Conclusions ----------- - BUG in pip: development version of pip cannot find local version identifiers (version numbers with a plus sign). - TIP: do NOT use setuptools 7.0 or older to create a distribution with a local version identifier. Installation of this may fail, because the version according to the filename differs from the actual version. - BUG in buildout? Where pip only warns about non-matching versions from the above tip, buildout actually quits with an error. Maybe somehow normalize versions before comparison. - TIP: only use local version identifiers when you use setuptools 8 or higher both when creating and when using the distribution. Then installing with zc.buildout will work. Installing with pip can be done by pointing to the exact file url. -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl

6 18

setuptools 8 changes are great, but ...
by Jim Fulton Dec. 17, 2014

Dec. 17, 2014

I think the changes in version management in setuptools 8 are a great step forward, but I think the transition is going to hurt a lot. For buildout, I'm thinking of of releasing 2.3.1 that reverts the changes in 2.3 and adds a requirement for setuptools <8 to give more time to respond to these changes. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

9 22

Changing the status of PEP 440 to Provisional
by Nick Coghlan Dec. 17, 2014

Dec. 17, 2014

After an offline discussion with Donald regarding feedback on the setuptools 8.0 release, I'm proposing we change the status of PEP 440 to be Provisional (in the PEP 411 sense) until we sort out the additional issues that were revealed through actual adoption. I can't actually make that change to the PEP text itself yet (as PEP 1, and hence the PEP index generator, don't currently include the notion of provisional acceptance), but this is the basis on which we're proposing to make changes to the previously accepted PEP text (rather than creating a new PEP). I'll also be proposing the relevant changes to PEP 1 and the index generator to make this an official feature of the PEP process to python-dev. Cheers, Nick.

2 2

Support for UNC paths on windows in pip 6?
by David Genest Dec. 16, 2014

Dec. 16, 2014

Hi, Is there by any chance an opening to include UNC path support in pip 6? I made a Pull request (https://github.com/pypa/pip/pull/1834) some time ago, which received positive attention from Steve Dower, but nobody else commented on the patch. This is linked to (https://github.com/pypa/pip/issues/1846) Thanks, D.

2 1

Buildout 2.3.1 released
by Jim Fulton Dec. 16, 2014

Dec. 16, 2014

This: - Fixes the picked-version bug - Suppresses spurious (and not spurious) legacy version warnings. (When that setuptools/packaging problem is fixed. I'll release 2.3.2 that undoes this.) If we're still in significant pain tomorrow, then I'll release a 2.4 that requires setuptools <8. This isn't a long-term, because old setuptools won't handle new-style (better) versions. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

2 1

change in setuptools 8.0
by Robin Becker Dec. 16, 2014

Dec. 16, 2014

A bitbucket user informs me angrily that he cannot use the version of reportlab that's latest on pypi because it has a dependency pillow==2.0.0,>=2.4.0 which is now treated as an 'and' condition by setuptools 8.0 so can not be satisfied. In our latest code we have removed the '==2.0.0,', but presumably there's nothing I can do to make the situation less broken for older versions other than remove those from pypi. Unfortunately we had this as part of the install_requires as install_requires=['pillow ==2.0.0, >=2.4.0','pip>=1.4.1', 'setuptools>=2.2'] so it's our fault for being too lax in describing the requirement. Presumably the , in the list was always an 'and' and now the ',' in the elements is also :( -- Robin Becker

5 12

Cookie-related PyPI 503 errors
by Marius Gedminas Dec. 15, 2014

Dec. 15, 2014

I keep getting 503 errors from PyPI that go away after I clear my cookies. The entire cycle is as follows: 1. Visit PyPI, log in (because I need to do some maintenance or something of my packages). 2. Things continue to work for about a week. 3. Visit PyPI, get a 503 error from Varnish after a noticeable timeout: Error 503 backend read error backend read error Guru Mediation: Details: cache-iad2132-IAD 1418630124 4121731453 Varnish cache server 4. Clear the PyPI cookie, reload page, things work again. 5. Go back to step 1. Having to clear my cookies about once a week is getting annoying. Can this be fixed please? Marius Gedminas -- Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald Knuth

2 1

Distribution metadata: build timestamp of the distribution
by Ben Finney Dec. 14, 2014

Dec. 14, 2014

Howdy all, Where can I find the build timestamp when a distribution was built by ‘distutils.setup’? The application I'm writing has a need to get a distribution and use the timestamp when that distribution was built:: import pkg_resources distribution_name = "foo" distribution = pkg_resources.get_distribution(distribution_name) distribution_timestamp = distribution.build_timestamp That doesn't work because the attribute doesn't exist. Where can I find the timestamp the specified distribution was built? -- \ “The right to search for truth implies also a duty; one must | `\ not conceal any part of what one has recognized to be true.” | _o__) —Albert Einstein | Ben Finney

1 0

zc.buildout 2.3.0 released
by Jim Fulton Dec. 14, 2014

Dec. 14, 2014

This fixes some compatibility issues with (and requires) setuptools 8. The bootstrap file has been fixed and also moved to a more secure location: https://bootstrap.pypa.io/bootstrap-buildout.py Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

1 0

PYD/SO platform tags (#22980)
by Steve Dower Dec. 12, 2014

Dec. 12, 2014

Hi distutils-sig, There's a bit of discussion going on at http://bugs.python.org/issue22980 about extending SO tags to include bitness values, and I took the opportunity to look into adding platform tags for .pyd files on Windows. There's a patch up there, but I'm interested in any thoughts or concerns people may have that I haven't considered. Distribution is likely to be most affected, and I figured distutils-sig is most likely to have people with strong opinions on the subject anyway, so I'm posting here :) Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll go from [".pyd"] to [".cp35-win32.pyd", ".cp3-win32.pyd", ".pyd"] (where "win32" may also be "win_amd64", "win_ia64" or "win_arm", which are the platforms that have #ifdefs in pyconfig.h). By default, distutils' build_ext will use the first extension, and so builds will get the ".cp35-win32.pyd" tag. You can only get the ".cp3-win32.pyd" tag (for stable ABI) by customizing the build process - I don't think that's an unreasonable expectation, especially on Windows. I can see potential for making more generic installable packages with this change, though I'm not pushing for that. The big win is when you're building and testing 32-bit and 64-bit pyds through an IDE - it greatly simplifies how the output directories and search paths are organized. Thoughts/concerns? Cheers, Steve

3 5