Mailman 3 March 2011 - Distutils-SIG

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

[buildout] branches languishing? (site-packages and distutils scripts)
by Chris Withers 03 Apr '12

03 Apr '12

Hi All, I'm wondering what the state of play is with the following branches: reinout-scripts gary-4-include-site-packages What more needs to happen for these to get merged to trunk and a release made? cheers, Chris

10 28

DRAFT PEP 396 - module version number
by Barry Warsaw 09 Jun '11

09 Jun '11

Hi distutilizers! At Pycon, I signed up to write an informational PEP on module version numbers. There have been some email discussion since then, and while I haven't completely worked my way through all those messages yet, I think I have covered the important top-level issues. I'm posting my current draft (as yet uncommitted to the new hg repository) here before taking it to python-dev. I'd love to get your feedback about any aspect of the PEP, but especially on deriving module version numbers from setup.cfg files, and on including them in metadata at build time (as opposed to installation time). Also, I'd appreciate any corrections based on my misunderstanding of distutils2/packaging and the relevant PEPs. Current draft attached below. Remember, this is an informational PEP, so its purpose is to codify best practices. Module authors will continue to have the freedom to do whatever they want. Cheers, -Barry PEP: 396 Title: Module Version Numbers Version: $Revision: 65628 $ Last-Modified: $Date: 2008-08-10 09:59:20 -0400 (Sun, 10 Aug 2008) $ Author: Barry Warsaw <barry(a)python.org> Status: Draft Type: Informational Content-Type: text/x-rst Created: 2011-03-16 Post-History: Abstract ======== Given that it is useful and common to specify version numbers for Python modules, and given that different ways of doing this have grown organically within the Python community, it is useful to establish standard conventions for module authors to adhere to and reference. This informational PEP describes best practices for Python module authors who want to define the version number of their Python module. Conformance with this PEP is optional, however other Python tools (such as `distutils2` [1]_) may be adapted to use the conventions defined here. User Stories ============ Alice is writing a new module, called ``alice.py``, which she wants to share with other Python developers. ``alice.py`` is a simple module and lives in one file. Alice wants to specify a version number so that her users can tell which version they are using. Because her module lives entirely in one file, she wants to add the version number to that file. Bob has written a module called ``bob.py`` which he has shared with many users. ``bob.py`` contains a version number for the convenience of his users. Bob learns about the Cheeseshop [2]_, and adds some simple packaging using classic distutils so that he can upload *The Bob Package* to the Cheeseshop. Because ``bob.py`` already specifies a version number which his users can access programmatically, he wants the same API to continue to work even though his users now get it from the Cheeseshop. Carole maintains several namespace packages, each of which are independently developed and distributed. In order for her users to properly specify dependencies on the right versions of her packages, she specifies the version numbers in the namespace package's ``setup.py`` file. Because Carol wants to have to update one version number per package, she specifies the version number in her module and has the ``setup.py`` extract the module version number at *sdist* tarball build time. David maintains a package in the standard library, and also produces standalone versions for other versions of Python. The standard library copy defines the version number in the module, and this same version number is used for the standalone distributions as well. Rationale ========= Python modules, both in the standard library and available from third parties, have long included version numbers. There are established de-facto standards for describing version numbers, and many ad-hoc ways have grown organically over the years. Often, version numbers can be retrieved from a module programmatically, by importing the module and inspecting an attribute. Classic Python distutils ``setup()`` functions [3]_ describe a ``version`` argument where the package's version number can be specified. PEP 8 [4]_ describes the use of a module attribute called ``__version__`` for recording "Subversion, CVS, or RCS" version strings using keyword expansion. In the PEP author's own email archives, the earliest example of the use of an ``__version__`` module attribute by independent module developers dates back to 1995. Another example of version information is the sqlite3 [5]_ library with its ``sqlite_version_info``, ``version``, and ``version_info`` attributes. It may not be immediately obvious which attribute contains a version number for the module, and which contains a version number for the underlying SQLite3 library. This informational PEP codifies established practice, and recommends standard ways of describing module version numbers, along with some use cases for when -- and when *not* -- to include them. Its adoption is purely voluntary, but other tools in the Python universe may support the standards defined herein. Specification ============= #. In general, modules in the standard library SHOULD NOT have version numbers. They implicitly carry the version number of the Python release they are included in. #. On a case-by-case basis, standard library modules which are also distributed as standalone packages for other Python versions MAY include a module version number when included in the standard library, and SHOULD include a version number when packaged separately. #. When a module includes a version number, it SHOULD be available in the ``__version__`` attribute on that module. #. For modules which are also packages, the top level module namespace SHOULD include the ``__version__`` attribute. #. For modules which live inside a namespace package, the sub-package name SHOULD include the ``__version__`` attribute. The namespace module itself SHOULD NOT include its own ``__version__`` attribute. #. The ``__version__`` attribute's value SHOULD be a simple string. #. Module version numbers SHOULD conform to the normalized version format specified in PEP 386 [6]_. #. Module version numbers SHOULD NOT contain version control system supplied revision numbers, or any other semantically different version numbers (e.g. underlying library version number). #. Wherever a ``__version__`` attribute exists, a module MAY also include a ``__version_info__`` attribute, containing a tuple representation of the module version number, for easy comparisons. #. ``__version_info__`` SHOULD be of the format returned by PEP 386's ``parse_version()`` function. #. The ``version`` attribute in a classic distutils ``setup.py`` file, or the ``Version`` metadata field in a PEP 345 [7]_ SHOULD be derived from the ``__version__`` field, or vice versa. Examples ======== Retrieving the version number from a third party package:: >>> import bzrlib >>> bzrlib.__version__ '2.3.0' Retrieving the version number from a standard library package that is also distributed as a standalone module:: >>> import email >>> email.__version__ '5.1.0' Version numbers for namespace packages:: >>> import flufl.i18n >>> import flufl.enum >>> import flufl.lock >>> print flufl.i18n.__version__ 1.0.4 >>> print flufl.enum.__version__ 3.1 >>> print flufl.lock.__version__ 2.1 >>> import flufl >>> flufl.__version__ Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'module' object has no attribute '__version__' >>> Deriving ======== Module version numbers can appear in at least two places, and sometimes more. For example, in accordance with this PEP, they are available programmatically on the module's ``__version__`` attribute. They also need to be specified in either the classic distutils ``setup()`` function, as the ``version`` attribute in a ``setup.py``, or in the distutils2 ``setup.cfg`` file's ``version`` keyword, and they need to get into the PEP 345 package metadata, preferably at *sdist* package build-time. It's also desirable for module authors to only have to specify the version number once, and have all the other uses derive from this single definition. While there are any number of ways this could be done, this section describes one possible approach, for each scenario. Let's say Elle adds this attribute to her module, ``elle.py``:: __version__ = '3.1.1' Classic distutils ----------------- In classic distutils, the simplest way to add this version to the ``setup()`` function in ``setup.py`` is to do something like this:: from elle import __version__ setup(name='elle', version=__version__) In the PEP author's experience however, this can fail in some cases, such as when the module uses automatic Python 3 conversion via the ``2to3`` program. In that case, it's not much more difficult to write a little code to parse the ``__version__`` from the file rather than importing it:: import re DEFAULT_VERSION_RE = re.compile(r'(?P<version>\d+\.\d(?:\.\d+)?)') def get_version(filename, pattern=None): if pattern is None: cre = DEFAULT_VERSION_RE else: cre = re.compile(pattern) with open(filename) as fp: for line in fp: if line.startswith('__version__'): mo = cre.search(line) assert mo, 'No valid __version__ string found' return mo.group('version') raise AssertionError('No __version__ assignment found') setup(name='elle', version=get_version('elle.py')) Distutils2 ---------- Because the distutils2 style ``setup.cfg`` is declarative, we can't run any code to extract the ``__version__`` attribute, either via import or via parsing. This PEP suggests a special key be allowed for the ``version`` field in a ``setup.cfg`` file to indicate "get the version from this file". Something like this might work:: [metadata] version: parse:elle.py where ``parse`` means to use a parsing method similar to the above, on the file named after the colon. The exact recipe for doing this will be discussed in the appropriate distutils2 development forum. An alternative is to only define the version number in setup.cfg and use the ``pkgutil`` module [8]_ to make it available programmatically. E.g. in ``elle.py``:: from distutils2._backport import pkgutil __version__ = pkgutil.get_distribution('elle').metadata['version'] Static egg info =============== For distutils2, it is highly desirable to place the version information into the egg metadata at build-time rather than install-time. This way, the metadata will be available for introspection even when the code is not installed. For example, a package's metadata can be made available via a simple HTTP GET operation on a URL within the package's presence on the Cheeseshop. References ========== .. [1] Distutils2 documentation (http://distutils2.notmyidea.org/) .. [2] The Cheeseshop (Python Package Index) (http://pypi.python.org) .. [3] http://docs.python.org/distutils/setupscript.html .. [4] PEP 8, Style Guide for Python Code (http://www.python.org/dev/peps/pep-0008) .. [5] sqlite3 module documentation (http://docs.python.org/library/sqlite3.html) .. [6] PEP 386, Changing the version comparison module in Distutils (http://www.python.org/dev/peps/pep-0386/) .. [7] PEP 345, Metadata for Python Software Packages 1.2 (http://www.python.org/dev/peps/pep-0345/#version) .. [8] pkgutil - Package utilities (http://distutils2.notmyidea.org/library/pkgutil.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

7 23

[PEP376] RECORD line separator
by Atsushi Odagiri 09 Jun '11

09 Jun '11

Hi I read pep276, and I have a question. This PEP says that line separator of RECORD file is `os.separator`. and 'bdist-*' command will also create RECORD file. Does it even work with the package which doesn't depend on the OS like bdist_egg ? I think that file format should not depend on os environment and good choice is CRLF. It's standard line separator as used in email headers. by aodag -- /* Atsushi Odagiri http://blog.aodag.jp mailto:aodagx@gmail.com */

2 1

Re: [Distutils] [buildout] "private" releases
by Chris Withers 27 Apr '11

27 Apr '11

Hi Christian, On 25/03/2011 16:49, Christian Theune wrote: > the German speaking Zope Users Group (DZUG e.V.) organizes a series of 4 > sprints this year to support feature development within the proximity of > the ZTK and solve problems encountered by Zope, Plone and Python developers. > <snip> > Topics > ====== > > * Discussing how to deal with "private" releases FWIW, I've had no problems with this, here's a sample buildout.cfg: [buildout] extensions = lovely.buildouthttp find-links = https://example.com/password/protected/folder ...and just dump the .tgz sdists in that folder. Of course, if you don't need password protection such as when you have your "egg server" on a private network, you just need the find-links. I'm not really sure why people have written a complicated array of "egg servers" and the like when a simple http or file system served folder is just fine ;-) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

4 6

Including symlinked data files in an egg
by Wichert Akkerman 22 Apr '11

22 Apr '11

I have a package with a source that looks like this: setup.py MANIFEST.in Prototype/libraries/jquery.js Prototype/style/main.css my/ my/package/ my/package/templates/libraries -> ../../../Prototype/libraries my/package/templates/style -> ../../../Prototype/style what I would like to accomplish is that when a bdist is made either Prototype/ is installed somewhere and the symlinks point to it, or the symlinks are replaced with a copy of the data they point to. For a package maintained in subverion this works: when the sdist is created the symlink is replaced with a copy of the thing it points to, which is then included in the bdist. For a package maintained in git this does not work: the symlink always appears to be ignored. Is there a way to accomplish this? Wichert. -- Wichert Akkerman <wichert(a)wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.

3 11

passing command-line options to an additional python interpreter?
by Mark Visser 05 Apr '11

05 Apr '11

I'm using buildout for managing a project that runs on several different python interpreters, some of which are embedded in other applications. As such, I'm at the whim of those packaging python for those applications, and sometimes they don't set things up... well, optimally. The problem I've run into is that one of the exposed non-standard python interpreters (mayapy) is actually a shell script. Thus, you can't use its direct path with a shebang. Normally, I get around this by using #!/usr/bin/env mayapy. For the life of me, I can't figure out a way to pass this to buildout. I've tried this: [mayapy] executable = /usr/bin/env mayapy [mayatests] python = mayapy ... etc But buildout doesn't like the space. Is there a way to specify executable options? I tried this to no avail: [mayapy] executable = /usr/bin/env options = mayapy Can anyone suggest a solution? thanks, -Mark

2 1

window 64bit madness
by Adam GROSZER 05 Apr '11

05 Apr '11

Hello, Having problems here installing http://pypi.python.org/packages/2.6/r/reportlab/reportlab-2.5.win-amd64-py2… . Neither setuptools 0.6c11 nor distribute 0.6.14 gets it right. We don't have and don't want to have a compiler on the server, so we need rely on binary packages. - setuptools just fails miserably even with easy_install: http://paste.lisp.org/+2KZD - distribute installs it (tho in a ...-win32.egg folder): http://paste.lisp.org/+2KZE - zc.buildout cannot install it with distribute, because it wants to use the source tarball (and we don't have a compiler) or when I make it not to be able to see the tarball, it does not find the reportlab-2.5.win-amd64-py2.6.exe. I guess it does not get the platform right. Any hints? -- Best regards, Adam GROSZER -- Quote of the day: Age before beauty and pearls before swine. -- Dorothy Parker

5 8