Mailman 3 April 2015 - Distutils-SIG

version for VCS/local builds between alpha and release
by Robert Collins 15 Apr '15

15 Apr '15

Tl;dr: I woud like to change PEP-440 to remove the stigmata around dev versions that come between pre-releases and releases. The basic scenario here is developers and CD deployers building versions from VCS of arbitrary commits. So we need to be able to deliver strictly increasing version numbers, automatically, without interfering with actual publishing of pre-release and release versions to PyPI. Today, this is fairly sane as a sort order and mechanism to deliver this: >>> v = ['1.0', '1.2.3.a1.dev1', '1.2.3.a1', '1.2.3.a2.dev1', '1.2.3.b1', '1.2.3.b2.dev1', '1.2.3.rc1', '1.2.3.rc2.dev1', '1.2.3', '1.2.3.dev1'] >>> sorted(v, key=packaging.version.parse) ['1.0', '1.2.3.dev1', '1.2.3.a1.dev1', '1.2.3.a1', '1.2.3.a2.dev1', '1.2.3.b1', '1.2.3.b2.dev1', '1.2.3.rc1', '1.2.3.rc2.dev1', '1.2.3'] But it doesn't feel good using a 'strongly recommended against' order. I think the recommendation is overly strong: If we touch up the language we can make the spec clearer, and a simple example like above will speak a thousand words :). -Rob -- Robert Collins <rbtcollins(a)hp.com> Distinguished Technologist HP Converged Cloud

3 5

buildout/setuptools slow as it scans the whole project dir
by Reinout van Rees 15 Apr '15

15 Apr '15

Hi, In some of my projects, buildout takes a looooooong time to complete. Sometimes the same problem occurs on the server or on another developer's laptop, sometimes not. The difference is something like "15 seconds if there's no problem" and "5 minutes if the problem occurs". It is terribly hard to pinpoint the exact problem and/or cause. An error in the setup.py or or MANIFEST.in is unlikely, as the very same project might take 5 minutes locally and 15 seconds on the server or vice versa... Anyway, if I run "bin/buildout" as "strace -f bin/buildout", I see a lot of "stat" calls. Setuptools walks through all the files inside my project dir. Including parts/omelette/* and, if available, a bower_compontents/ directory full of thousands of javascript files. Removing some of these directories (which running the buildout re-creates) fixes the speed issue for one run. I modified my local buildout copy to run "setup.py develop" with a -v instead of a -q option. This way I found out where it approximately happens: /usr/bin/python /tmp/tmp6UdsMl -v develop -mxN -d /vagrant/sso/develop-eggs/tmpfioc1Ibuild running develop running egg_info writing requirements to sso.egg-info/requires.txt writing sso.egg-info/PKG-INFO writing top-level names to sso.egg-info/top_level.txt writing dependency_links to sso.egg-info/dependency_links.txt writing entry points to sso.egg-info/entry_points.txt ### This is where the process seems to stop a couple of minutes to scan all the files. reading manifest file 'sso.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'sso.egg-info/SOURCES.txt' running build_ext Creating /vagrant/sso/develop-eggs/tmpfioc1Ibuild/sso.egg-link (link to .) The MANIFEST.in loooks like this: # Include docs in the root. include *.rst # Include everything in our project directory (sso/views.py, sso/static/some.js, etc) graft sso It is a git project. The setup.py looks like this: from setuptools import setup version = '1.1.dev0' long_description = '\n\n'.join([ open('README.rst').read(), open('CREDITS.rst').read(), open('CHANGES.rst').read(), ]) install_requires = [ 'Django >= 1.4.2, < 1.7', 'django-nose', 'lizard-auth-server', 'gunicorn', 'raven', 'werkzeug', 'south', 'django-auth-ldap', 'django-mama-cas', ], setup(name='sso', version=version, description="Single sign on server (and more) for lizard", long_description=long_description, # Get strings from http://www.python.org/pypi?%3Aaction=list_classifiers classifiers=['Programming Language :: Python', 'Framework :: Django', ], keywords=[], author='Do not blame Reinout', author_email='reinout.vanrees(a)nelen-schuurmans.nl', url='', license='GPL', packages=['sso'], zip_safe=False, install_requires=install_requires, entry_points={ 'console_scripts': [ ]}, ) Conclusion for me: something somewhere in setuptools is reading my whole project folder. It does it after "writing entry points to sso.egg-info/entry_points.txt" and before "reading manifest file 'sso.egg-info/SOURCES.txt'". The SOURCES.txt itself is small: CHANGES.rst CREDITS.rst LICENSE.rst MANIFEST.in README.rst setup.cfg setup.py sso/__init__.py sso/__init__.pyc sso/admin.py sso/developmentsettings.py sso/developmentsettings.pyc sso/models.py sso/models.pyc sso/settings.py sso/settings.pyc sso/stagingsettings.py sso/tests.py sso/urls.py sso/views.py sso.egg-info/PKG-INFO sso.egg-info/SOURCES.txt sso.egg-info/dependency_links.txt sso.egg-info/entry_points.txt sso.egg-info/not-zip-safe sso.egg-info/requires.txt Hm. I see some .pyc files in there. Something that needs fixing, but not the cause, I think. Is there something I'm missing? Where should I look? I cannot find "writing entry points to..." in the setuptools source code right away. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout(a)vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"

3 6

Make PEP 426 less boring
by Thomas Güttler 14 Apr '15

14 Apr '15

Hi, somehow I feel bored if I read PEP 426. https://www.python.org/dev/peps/pep-0426/ One concrete improvement would be to remove this paragraph: {{{ The design draws on the Python community's 15 years of experience with distutils based software distribution, and incorporates ideas and concepts from other distribution systems, including Python's setuptools, pip and other projects, Ruby's gems, Perl's CPAN, Node.js's npm, PHP's composer and Linux packaging systems such as RPM and APT. }}} Because something like this was already saied some lines before {{{ Metadata 2.0 represents a major upgrade to the Python packaging ecosystem, and attempts to incorporate experience gained over the 15 years(!) since distutils was first added to the standard library. Some of that is just incorporating existing practices from setuptools/pip/etc, some of it is copying from other distribution systems (like Linux distros or other development language communities) and some of it is attempting to solve problems which haven't yet been well solved by anyone (like supporting clean conversion of Python source packages to distro policy compliant source packages for at least Debian and Fedora, and perhaps other platform specific distribution systems). }}} **And** I would move the historic background (the second of the above quotes) at the end. Meta: are you interested in feedback like this? Regards, Thomas Güttler

2 1

Re: [Distutils] Installing a file into sitepackages
by Stuart Axon 12 Apr '15

12 Apr '15

I had a further look - and on windows the file ends up inside the .egg file, on linux it ends up inside the site packages as intended. At a guess it seems like there might be a bug in the path handling on windows. .. I wonder if it's something like this http://stackoverflow.com/questions/4579908/cross-platform-splitting-of-path… which seems an easy way to get an off-by-one error in a path ?

2 11

[pycon] Packaging-related discussions
by David Cournapeau 10 Apr '15

10 Apr '15

Hi there, I would like to know what are the events/discussions related to packaging happening this year at PyCon ? My employer is sponsoring my trip to the conference this year, and I would like to make sure I am not missing an important event. I will also be there for most of the sprints, cheers, David

5 11

Method for calculating .so file names
by Randy Syring 08 Apr '15

08 Apr '15

As a follow-up to my earlier question, is there a way to pragmatically determine the naming scheme of installed shared object files? I've seen a couple different naming formats: > # python 2.7.6 - default ubuntu version > $ ls /usr/lib/python2.7/dist-packages/_dbus_* > /usr/lib/python2.7/dist-packages/_dbus_bindings.so > /usr/lib/python2.7/dist-packages/_dbus_glib_bindings.so > > # python 3.4.0 - default ubuntu version > $ ls /usr/lib/python3/dist-packages/_dbus_* > /usr/lib/python3/dist-packages/_dbus_bindings.cpython-34m-x86_64-linux-gnu.so > /usr/lib/python3/dist-packages/_dbus_glib_bindings.cpython-34m-x86_64-linux-gnu.so > > # python 3.4.3 - manual install on ubuntu > $ ls /opt/python34/lib/python3.4/site-packages/_dbus_* > /opt/python34/lib/python3.4/site-packages/_dbus_bindings.so > /opt/python34/lib/python3.4/site-packages/_dbus_glib_bindings.so I can piece together the file structure for 3.4.0 from information in sysconfig, but I'm wondering if there is a better way. Thanks. *Randy Syring* Husband | Father | Redeemed Sinner /"For what does it profit a man to gain the whole world and forfeit his soul?" (Mark 8:36 ESV)/

3 3

installing package - problems
by Leung, Edward 08 Apr '15

08 Apr '15

Dear Sir/Madam, I was trying to install a python package pysftp using the following at command prompt: python -m pip install pysftp and I got the following error. Could you tell me what is not working?? Thanks. copying lib\Crypto\Random\Fortuna\FortunaGenerator.py -> build\lib.win-amd64 -3.4\Crypto\Random\Fortuna copying lib\Crypto\Random\Fortuna\SHAd256.py -> build\lib.win-amd64-3.4\Cryp to\Random\Fortuna copying lib\Crypto\Random\Fortuna\__init__.py -> build\lib.win-amd64-3.4\Cry pto\Random\Fortuna creating build\lib.win-amd64-3.4\Crypto\Random\OSRNG copying lib\Crypto\Random\OSRNG\fallback.py -> build\lib.win-amd64-3.4\Crypt o\Random\OSRNG copying lib\Crypto\Random\OSRNG\nt.py -> build\lib.win-amd64-3.4\Crypto\Rand om\OSRNG copying lib\Crypto\Random\OSRNG\posix.py -> build\lib.win-amd64-3.4\Crypto\R andom\OSRNG copying lib\Crypto\Random\OSRNG\rng_base.py -> build\lib.win-amd64-3.4\Crypt o\Random\OSRNG copying lib\Crypto\Random\OSRNG\__init__.py -> build\lib.win-amd64-3.4\Crypt o\Random\OSRNG creating build\lib.win-amd64-3.4\Crypto\SelfTest copying lib\Crypto\SelfTest\st_common.py -> build\lib.win-amd64-3.4\Crypto\S elfTest copying lib\Crypto\SelfTest\__init__.py -> build\lib.win-amd64-3.4\Crypto\Se lfTest creating build\lib.win-amd64-3.4\Crypto\SelfTest\Cipher copying lib\Crypto\SelfTest\Cipher\common.py -> build\lib.win-amd64-3.4\Cryp to\SelfTest\Cipher copying lib\Crypto\SelfTest\Cipher\test_AES.py -> build\lib.win-amd64-3.4\Cr ypto\SelfTest\Cipher copying lib\Crypto\SelfTest\Cipher\test_ARC2.py -> build\lib.win-amd64-3.4\C rypto\SelfTest\Cipher copying lib\Crypto\SelfTest\Cipher\test_ARC4.py -> build\lib.win-amd64-3.4\C rypto\SelfTest\Cipher copying lib\Crypto\SelfTest\Cipher\test_Blowfish.py -> build\lib.win-amd64-3 .4\Crypto\SelfTest\Cipher copying lib\Crypto\SelfTest\Cipher\test_CAST.py -> build\lib.win-amd64-3.4\C rypto\SelfTest\Cipher copying lib\Crypto\SelfTest\Cipher\test_DES.py -> build\lib.win-amd64-3.4\Cr ypto\SelfTest\Cipher copying lib\Crypto\SelfTest\Cipher\test_DES3.py -> build\lib.win-amd64-3.4\C rypto\SelfTest\Cipher copying lib\Crypto\SelfTest\Cipher\test_pkcs1_15.py -> build\lib.win-amd64-3 .4\Crypto\SelfTest\Cipher copying lib\Crypto\SelfTest\Cipher\test_pkcs1_oaep.py -> build\lib.win-amd64 -3.4\Crypto\SelfTest\Cipher copying lib\Crypto\SelfTest\Cipher\test_XOR.py -> build\lib.win-amd64-3.4\Cr ypto\SelfTest\Cipher copying lib\Crypto\SelfTest\Cipher\__init__.py -> build\lib.win-amd64-3.4\Cr ypto\SelfTest\Cipher creating build\lib.win-amd64-3.4\Crypto\SelfTest\Hash copying lib\Crypto\SelfTest\Hash\common.py -> build\lib.win-amd64-3.4\Crypto \SelfTest\Hash copying lib\Crypto\SelfTest\Hash\test_HMAC.py -> build\lib.win-amd64-3.4\Cry pto\SelfTest\Hash copying lib\Crypto\SelfTest\Hash\test_MD2.py -> build\lib.win-amd64-3.4\Cryp to\SelfTest\Hash copying lib\Crypto\SelfTest\Hash\test_MD4.py -> build\lib.win-amd64-3.4\Cryp to\SelfTest\Hash copying lib\Crypto\SelfTest\Hash\test_MD5.py -> build\lib.win-amd64-3.4\Cryp to\SelfTest\Hash copying lib\Crypto\SelfTest\Hash\test_RIPEMD.py -> build\lib.win-amd64-3.4\C rypto\SelfTest\Hash copying lib\Crypto\SelfTest\Hash\test_SHA.py -> build\lib.win-amd64-3.4\Cryp to\SelfTest\Hash copying lib\Crypto\SelfTest\Hash\test_SHA224.py -> build\lib.win-amd64-3.4\C rypto\SelfTest\Hash copying lib\Crypto\SelfTest\Hash\test_SHA256.py -> build\lib.win-amd64-3.4\C rypto\SelfTest\Hash copying lib\Crypto\SelfTest\Hash\test_SHA384.py -> build\lib.win-amd64-3.4\C rypto\SelfTest\Hash copying lib\Crypto\SelfTest\Hash\test_SHA512.py -> build\lib.win-amd64-3.4\C rypto\SelfTest\Hash copying lib\Crypto\SelfTest\Hash\__init__.py -> build\lib.win-amd64-3.4\Cryp to\SelfTest\Hash creating build\lib.win-amd64-3.4\Crypto\SelfTest\Protocol copying lib\Crypto\SelfTest\Protocol\test_AllOrNothing.py -> build\lib.win-a md64-3.4\Crypto\SelfTest\Protocol copying lib\Crypto\SelfTest\Protocol\test_chaffing.py -> build\lib.win-amd64 -3.4\Crypto\SelfTest\Protocol copying lib\Crypto\SelfTest\Protocol\test_KDF.py -> build\lib.win-amd64-3.4\ Crypto\SelfTest\Protocol copying lib\Crypto\SelfTest\Protocol\test_rfc1751.py -> build\lib.win-amd64- 3.4\Crypto\SelfTest\Protocol copying lib\Crypto\SelfTest\Protocol\__init__.py -> build\lib.win-amd64-3.4\ Crypto\SelfTest\Protocol creating build\lib.win-amd64-3.4\Crypto\SelfTest\PublicKey copying lib\Crypto\SelfTest\PublicKey\test_DSA.py -> build\lib.win-amd64-3.4 \Crypto\SelfTest\PublicKey copying lib\Crypto\SelfTest\PublicKey\test_ElGamal.py -> build\lib.win-amd64 -3.4\Crypto\SelfTest\PublicKey copying lib\Crypto\SelfTest\PublicKey\test_importKey.py -> build\lib.win-amd 64-3.4\Crypto\SelfTest\PublicKey copying lib\Crypto\SelfTest\PublicKey\test_RSA.py -> build\lib.win-amd64-3.4 \Crypto\SelfTest\PublicKey copying lib\Crypto\SelfTest\PublicKey\__init__.py -> build\lib.win-amd64-3.4 \Crypto\SelfTest\PublicKey creating build\lib.win-amd64-3.4\Crypto\SelfTest\Random copying lib\Crypto\SelfTest\Random\test_random.py -> build\lib.win-amd64-3.4 \Crypto\SelfTest\Random copying lib\Crypto\SelfTest\Random\test_rpoolcompat.py -> build\lib.win-amd6 4-3.4\Crypto\SelfTest\Random copying lib\Crypto\SelfTest\Random\test__UserFriendlyRNG.py -> build\lib.win -amd64-3.4\Crypto\SelfTest\Random copying lib\Crypto\SelfTest\Random\__init__.py -> build\lib.win-amd64-3.4\Cr ypto\SelfTest\Random creating build\lib.win-amd64-3.4\Crypto\SelfTest\Random\Fortuna copying lib\Crypto\SelfTest\Random\Fortuna\test_FortunaAccumulator.py -> bui ld\lib.win-amd64-3.4\Crypto\SelfTest\Random\Fortuna copying lib\Crypto\SelfTest\Random\Fortuna\test_FortunaGenerator.py -> build \lib.win-amd64-3.4\Crypto\SelfTest\Random\Fortuna copying lib\Crypto\SelfTest\Random\Fortuna\test_SHAd256.py -> build\lib.win- amd64-3.4\Crypto\SelfTest\Random\Fortuna copying lib\Crypto\SelfTest\Random\Fortuna\__init__.py -> build\lib.win-amd6 4-3.4\Crypto\SelfTest\Random\Fortuna creating build\lib.win-amd64-3.4\Crypto\SelfTest\Random\OSRNG copying lib\Crypto\SelfTest\Random\OSRNG\test_fallback.py -> build\lib.win-a md64-3.4\Crypto\SelfTest\Random\OSRNG copying lib\Crypto\SelfTest\Random\OSRNG\test_generic.py -> build\lib.win-am d64-3.4\Crypto\SelfTest\Random\OSRNG copying lib\Crypto\SelfTest\Random\OSRNG\test_nt.py -> build\lib.win-amd64-3 .4\Crypto\SelfTest\Random\OSRNG copying lib\Crypto\SelfTest\Random\OSRNG\test_posix.py -> build\lib.win-amd6 4-3.4\Crypto\SelfTest\Random\OSRNG copying lib\Crypto\SelfTest\Random\OSRNG\test_winrandom.py -> build\lib.win- amd64-3.4\Crypto\SelfTest\Random\OSRNG copying lib\Crypto\SelfTest\Random\OSRNG\__init__.py -> build\lib.win-amd64- 3.4\Crypto\SelfTest\Random\OSRNG creating build\lib.win-amd64-3.4\Crypto\SelfTest\Util copying lib\Crypto\SelfTest\Util\test_asn1.py -> build\lib.win-amd64-3.4\Cry pto\SelfTest\Util copying lib\Crypto\SelfTest\Util\test_Counter.py -> build\lib.win-amd64-3.4\ Crypto\SelfTest\Util copying lib\Crypto\SelfTest\Util\test_number.py -> build\lib.win-amd64-3.4\C rypto\SelfTest\Util copying lib\Crypto\SelfTest\Util\test_winrandom.py -> build\lib.win-amd64-3. 4\Crypto\SelfTest\Util copying lib\Crypto\SelfTest\Util\__init__.py -> build\lib.win-amd64-3.4\Cryp to\SelfTest\Util creating build\lib.win-amd64-3.4\Crypto\SelfTest\Signature copying lib\Crypto\SelfTest\Signature\test_pkcs1_15.py -> build\lib.win-amd6 4-3.4\Crypto\SelfTest\Signature copying lib\Crypto\SelfTest\Signature\test_pkcs1_pss.py -> build\lib.win-amd 64-3.4\Crypto\SelfTest\Signature copying lib\Crypto\SelfTest\Signature\__init__.py -> build\lib.win-amd64-3.4 \Crypto\SelfTest\Signature creating build\lib.win-amd64-3.4\Crypto\Protocol copying lib\Crypto\Protocol\AllOrNothing.py -> build\lib.win-amd64-3.4\Crypt o\Protocol copying lib\Crypto\Protocol\Chaffing.py -> build\lib.win-amd64-3.4\Crypto\Pr otocol copying lib\Crypto\Protocol\KDF.py -> build\lib.win-amd64-3.4\Crypto\Protoco l copying lib\Crypto\Protocol\__init__.py -> build\lib.win-amd64-3.4\Crypto\Pr otocol creating build\lib.win-amd64-3.4\Crypto\PublicKey copying lib\Crypto\PublicKey\DSA.py -> build\lib.win-amd64-3.4\Crypto\Public Key copying lib\Crypto\PublicKey\ElGamal.py -> build\lib.win-amd64-3.4\Crypto\Pu blicKey copying lib\Crypto\PublicKey\pubkey.py -> build\lib.win-amd64-3.4\Crypto\Pub licKey copying lib\Crypto\PublicKey\RSA.py -> build\lib.win-amd64-3.4\Crypto\Public Key copying lib\Crypto\PublicKey\_DSA.py -> build\lib.win-amd64-3.4\Crypto\Publi cKey copying lib\Crypto\PublicKey\_RSA.py -> build\lib.win-amd64-3.4\Crypto\Publi cKey copying lib\Crypto\PublicKey\_slowmath.py -> build\lib.win-amd64-3.4\Crypto\ PublicKey copying lib\Crypto\PublicKey\__init__.py -> build\lib.win-amd64-3.4\Crypto\P ublicKey creating build\lib.win-amd64-3.4\Crypto\Signature copying lib\Crypto\Signature\PKCS1_PSS.py -> build\lib.win-amd64-3.4\Crypto\ Signature copying lib\Crypto\Signature\PKCS1_v1_5.py -> build\lib.win-amd64-3.4\Crypto \Signature copying lib\Crypto\Signature\__init__.py -> build\lib.win-amd64-3.4\Crypto\S ignature Skipping implicit fixer: buffer Skipping implicit fixer: idioms Skipping implicit fixer: set_literal Skipping implicit fixer: ws_comma running build_ext building 'Crypto.Random.OSRNG.winrandom' extension warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastm ath. error: Microsoft Visual C++ 10.0 is required (Unable to find vcvarsall.bat). ---------------------------------------- Command "C:\Python34\python.exe -c "import setuptools, tokenize;__file__='C: \\Users\\leunge\\AppData\\Local\\Temp\\pip-build-qwwdy8a4\\pycrypto\\setup.py';e xec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n '), __file__, 'exec'))" install --record C:\Users\leunge\AppData\Local\Temp\pip- tc1xxkzk-record\install-record.txt --single-version-externally-managed --compile " failed with error code 1 in C:\Users\leunge\AppData\Local\Temp\pip-build-qwwdy /***************************/ Edward Leung, Ph.D. Quantitative Research Analyst Invesco Quantitative Strategies 1166 Ave of the Americas, 27th Floor New York, NY 10036 212-278-9744 (w) 646-236-1453 (c) Edward_Leung(a)invesco.com **************************************************************** Confidentiality Note: The information contained in this message, and any attachments, may contain confidential and/or privileged material. It is intended solely for the person(s) or entity to which it is addressed. Any review, retransmission, dissemination, or taking of any action in reliance upon this information by persons or entities other than the intended recipient(s) is prohibited. If you received this in error, please contact the sender and delete the material from any computer. ****************************************************************

2 3

Method for calculating virtualenv site-packages directory
by Randy Syring 07 Apr '15

07 Apr '15

If I'm running a python script in a virtualenv, what is the best method for calculating the path of that virtualenv's site-packages directory? I'm only running this command for development environments and as a shortcut for developers, so something that works most of the time is fine. This code isn't going to end up in a production system anywhere. *Randy Syring* Husband | Father | Redeemed Sinner /"For what does it profit a man to gain the whole world and forfeit his soul?" (Mark 8:36 ESV)/

2 1

Question about wheels and python-dbg 2.7
by Ionel Cristian Mărieș 03 Apr '15

03 Apr '15

Hey, It appears that using a debug build of python 2.7 doesn't mark the wheels built using it in any special way. Pip would install them on a regular python 2.7 (if they would be on an package index) and then later on imports for C extensions would fail (not sure why, tho I suspect the different "_d.so" naming scheme). Is there something more complicated preventing this be fixed or there's just hasn't been enough interest? Why does the import fail anyway? Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro

2 1

Get dependencies of a package without full download
by Thomas Güttler 02 Apr '15

02 Apr '15

Hi, just out of curiosity: Is it possible to get the dependencies of a package without full download from pypi? If you want to build a graph of dependencies this would be nice, since it would reduce the network traffic a lot. Regards, Thomas Güttler

6 8