Hi all,
I've been aware that the distutils sig has been simmerring away, but
until recently it has not been directly relevant to what I do.
I like the look of the proposed api, but have one question. Will this
support an installed system that has multiple versions of the same
package installed simultaneously? If not, then this would seem to be a
significant limitation, especially when dependencies between packages
are considered.
Assuming it does, then how will this be achieved? I am presently
managing this with a messy arrangement of symlinks. A package is
installed with its version number in it's name, and a separate
directory is created for an application with links from the
unversioned package name to the versioned one. Then I just set the
pythonpath to this directory.
A sample of what the directory looks like is shown below.
I'm sure there is a better solution that this, and I'm not sure that
this would work under windows anyway (does windows have symlinks?).
So, has this SIG considered such versioning issues yet?
Cheers,
Tim
--------------------------------------------------------------
Tim Docker timd(a)macquarie.com.au
Quantative Applications Division
Macquarie Bank
--------------------------------------------------------------
qad16:qad $ ls -l lib/python/
total 110
drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1
-r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so
drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1
drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1
dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8
drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1
dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1
dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3
drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7
dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1
qad16:qad $ ls -l lib/python/1.1
total 30
lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so
lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so
lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so
lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so
lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so
lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so
lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1
lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8
lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1
lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1
lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so
lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so
lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3
lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7
lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1
Hi All,
I'm wondering what the state of play is with the following branches:
reinout-scripts
gary-4-include-site-packages
What more needs to happen for these to get merged to trunk and a release
made?
cheers,
Chris
Hi distutilizers!
At Pycon, I signed up to write an informational PEP on module version numbers.
There have been some email discussion since then, and while I haven't
completely worked my way through all those messages yet, I think I have
covered the important top-level issues.
I'm posting my current draft (as yet uncommitted to the new hg repository)
here before taking it to python-dev. I'd love to get your feedback about any
aspect of the PEP, but especially on deriving module version numbers from
setup.cfg files, and on including them in metadata at build time (as opposed
to installation time). Also, I'd appreciate any corrections based on my
misunderstanding of distutils2/packaging and the relevant PEPs.
Current draft attached below. Remember, this is an informational PEP, so its
purpose is to codify best practices. Module authors will continue to have the
freedom to do whatever they want.
Cheers,
-Barry
PEP: 396
Title: Module Version Numbers
Version: $Revision: 65628 $
Last-Modified: $Date: 2008-08-10 09:59:20 -0400 (Sun, 10 Aug 2008) $
Author: Barry Warsaw <barry(a)python.org>
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 2011-03-16
Post-History:
Abstract
========
Given that it is useful and common to specify version numbers for
Python modules, and given that different ways of doing this have grown
organically within the Python community, it is useful to establish
standard conventions for module authors to adhere to and reference.
This informational PEP describes best practices for Python module
authors who want to define the version number of their Python module.
Conformance with this PEP is optional, however other Python tools (such as
`distutils2` [1]_) may be adapted to use the conventions defined here.
User Stories
============
Alice is writing a new module, called ``alice.py``, which she wants to
share with other Python developers. ``alice.py`` is a simple module
and lives in one file. Alice wants to specify a version number so
that her users can tell which version they are using. Because her
module lives entirely in one file, she wants to add the version number
to that file.
Bob has written a module called ``bob.py`` which he has shared with
many users. ``bob.py`` contains a version number for the convenience
of his users. Bob learns about the Cheeseshop [2]_, and adds some
simple packaging using classic distutils so that he can upload *The
Bob Package* to the Cheeseshop. Because ``bob.py`` already specifies
a version number which his users can access programmatically, he wants
the same API to continue to work even though his users now get it from
the Cheeseshop.
Carole maintains several namespace packages, each of which are
independently developed and distributed. In order for her users to
properly specify dependencies on the right versions of her packages,
she specifies the version numbers in the namespace package's
``setup.py`` file. Because Carol wants to have to update one version
number per package, she specifies the version number in her module and
has the ``setup.py`` extract the module version number at *sdist*
tarball build time.
David maintains a package in the standard library, and also produces
standalone versions for other versions of Python. The standard
library copy defines the version number in the module, and this same
version number is used for the standalone distributions as well.
Rationale
=========
Python modules, both in the standard library and available from third
parties, have long included version numbers. There are established
de-facto standards for describing version numbers, and many ad-hoc
ways have grown organically over the years. Often, version numbers
can be retrieved from a module programmatically, by importing the
module and inspecting an attribute. Classic Python distutils
``setup()`` functions [3]_ describe a ``version`` argument where the
package's version number can be specified. PEP 8 [4]_ describes the
use of a module attribute called ``__version__`` for recording
"Subversion, CVS, or RCS" version strings using keyword expansion. In
the PEP author's own email archives, the earliest example of the use
of an ``__version__`` module attribute by independent module
developers dates back to 1995.
Another example of version information is the sqlite3 [5]_ library
with its ``sqlite_version_info``, ``version``, and ``version_info``
attributes. It may not be immediately obvious which attribute
contains a version number for the module, and which contains a version
number for the underlying SQLite3 library.
This informational PEP codifies established practice, and recommends
standard ways of describing module version numbers, along with some
use cases for when -- and when *not* -- to include them. Its adoption
is purely voluntary, but other tools in the Python universe may
support the standards defined herein.
Specification
=============
#. In general, modules in the standard library SHOULD NOT have version
numbers. They implicitly carry the version number of the Python
release they are included in.
#. On a case-by-case basis, standard library modules which are also
distributed as standalone packages for other Python versions MAY
include a module version number when included in the standard
library, and SHOULD include a version number when packaged
separately.
#. When a module includes a version number, it SHOULD be available in
the ``__version__`` attribute on that module.
#. For modules which are also packages, the top level module namespace
SHOULD include the ``__version__`` attribute.
#. For modules which live inside a namespace package, the sub-package
name SHOULD include the ``__version__`` attribute. The namespace
module itself SHOULD NOT include its own ``__version__`` attribute.
#. The ``__version__`` attribute's value SHOULD be a simple string.
#. Module version numbers SHOULD conform to the normalized version
format specified in PEP 386 [6]_.
#. Module version numbers SHOULD NOT contain version control system
supplied revision numbers, or any other semantically different
version numbers (e.g. underlying library version number).
#. Wherever a ``__version__`` attribute exists, a module MAY also
include a ``__version_info__`` attribute, containing a tuple
representation of the module version number, for easy comparisons.
#. ``__version_info__`` SHOULD be of the format returned by PEP 386's
``parse_version()`` function.
#. The ``version`` attribute in a classic distutils ``setup.py`` file,
or the ``Version`` metadata field in a PEP 345 [7]_ SHOULD be
derived from the ``__version__`` field, or vice versa.
Examples
========
Retrieving the version number from a third party package::
>>> import bzrlib
>>> bzrlib.__version__
'2.3.0'
Retrieving the version number from a standard library package that is
also distributed as a standalone module::
>>> import email
>>> email.__version__
'5.1.0'
Version numbers for namespace packages::
>>> import flufl.i18n
>>> import flufl.enum
>>> import flufl.lock
>>> print flufl.i18n.__version__
1.0.4
>>> print flufl.enum.__version__
3.1
>>> print flufl.lock.__version__
2.1
>>> import flufl
>>> flufl.__version__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute '__version__'
>>>
Deriving
========
Module version numbers can appear in at least two places, and
sometimes more. For example, in accordance with this PEP, they are
available programmatically on the module's ``__version__`` attribute.
They also need to be specified in either the classic distutils
``setup()`` function, as the ``version`` attribute in a ``setup.py``,
or in the distutils2 ``setup.cfg`` file's ``version`` keyword, and
they need to get into the PEP 345 package metadata, preferably at
*sdist* package build-time. It's also desirable for module authors to
only have to specify the version number once, and have all the other
uses derive from this single definition.
While there are any number of ways this could be done, this section
describes one possible approach, for each scenario.
Let's say Elle adds this attribute to her module, ``elle.py``::
__version__ = '3.1.1'
Classic distutils
-----------------
In classic distutils, the simplest way to add this version to the
``setup()`` function in ``setup.py`` is to do something like this::
from elle import __version__
setup(name='elle', version=__version__)
In the PEP author's experience however, this can fail in some cases,
such as when the module uses automatic Python 3 conversion via the
``2to3`` program. In that case, it's not much more difficult to write
a little code to parse the ``__version__`` from the file rather than
importing it::
import re
DEFAULT_VERSION_RE = re.compile(r'(?P<version>\d+\.\d(?:\.\d+)?)')
def get_version(filename, pattern=None):
if pattern is None:
cre = DEFAULT_VERSION_RE
else:
cre = re.compile(pattern)
with open(filename) as fp:
for line in fp:
if line.startswith('__version__'):
mo = cre.search(line)
assert mo, 'No valid __version__ string found'
return mo.group('version')
raise AssertionError('No __version__ assignment found')
setup(name='elle', version=get_version('elle.py'))
Distutils2
----------
Because the distutils2 style ``setup.cfg`` is declarative, we can't
run any code to extract the ``__version__`` attribute, either via
import or via parsing. This PEP suggests a special key be allowed for
the ``version`` field in a ``setup.cfg`` file to indicate "get the
version from this file". Something like this might work::
[metadata]
version: parse:elle.py
where ``parse`` means to use a parsing method similar to the above, on
the file named after the colon. The exact recipe for doing this will
be discussed in the appropriate distutils2 development forum.
An alternative is to only define the version number in setup.cfg and
use the ``pkgutil`` module [8]_ to make it available
programmatically. E.g. in ``elle.py``::
from distutils2._backport import pkgutil
__version__ = pkgutil.get_distribution('elle').metadata['version']
Static egg info
===============
For distutils2, it is highly desirable to place the version
information into the egg metadata at build-time rather than
install-time. This way, the metadata will be available for
introspection even when the code is not installed. For example, a
package's metadata can be made available via a simple HTTP GET
operation on a URL within the package's presence on the Cheeseshop.
References
==========
.. [1] Distutils2 documentation
(http://distutils2.notmyidea.org/)
.. [2] The Cheeseshop (Python Package Index)
(http://pypi.python.org)
.. [3] http://docs.python.org/distutils/setupscript.html
.. [4] PEP 8, Style Guide for Python Code
(http://www.python.org/dev/peps/pep-0008)
.. [5] sqlite3 module documentation
(http://docs.python.org/library/sqlite3.html)
.. [6] PEP 386, Changing the version comparison module in Distutils
(http://www.python.org/dev/peps/pep-0386/)
.. [7] PEP 345, Metadata for Python Software Packages 1.2
(http://www.python.org/dev/peps/pep-0345/#version)
.. [8] pkgutil - Package utilities
(http://distutils2.notmyidea.org/library/pkgutil.html)
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
Hi
I read pep276, and I have a question.
This PEP says that line separator of RECORD file is `os.separator`.
and 'bdist-*' command will also create RECORD file.
Does it even work with the package which doesn't depend on the OS like
bdist_egg ?
I think that file format should not depend on os environment and good
choice is CRLF.
It's standard line separator as used in email headers.
by aodag
--
/*
Atsushi Odagiri
http://blog.aodag.jp
mailto:aodagx@gmail.com
*/
I am trying to figure out the intended behaviour of dealing with
symlinks in package sources. My use case is that I have a number of
package implementation web things, and they tend to have an interactive
(static) prototype parts of which are reused directly in the
application. The filesystem structure for those packages is one of two
options:
1. Prototype is at the toplevel of the source. For example:
Prototype/index.html
Prototyle/style/main.css
setup.py
MANIFEST.in
src/namespace/__init__.py
src/namespace/templates/index.html
src/namespace/templates/style -> ../../../Prototype/style
2. Prototype is at outside the source. For example:
Prototype/index.html
Prototyle/style/main.css
buildout/bootstrap.py
buildout/buildout.cfg
buildout/src/mypackage/setup.py
buildout/src/mypackage/MANIFEST.in
buildout/src/mypackage/src/namespace/__init__.py
buildout/src/mypackage/src/namespace/templates/index.html
buildout/src/mypackage/src/namespace/templates/style ->
../../../../../../Prototype/style
My goal is to be able to create an sdist for my package and upload that
to pypi, and when people install that namespace/templates/style/main.css
must exist. Currently this does not appear to be possible. The problems
I see are:
a) symlinks are skipped by distutils
b) if you use a file finder plugin which traverses the symlink and
reports the files they end up twice in the sdist: once for the
Prototype and once inside the package. They second one is stored
as a link inside the sdist, but that link is not unpacked when
installing from the sdist and the files go missing.
Trying to analyse things a bit I see a number of possible scenarios for
symlinks:
i) symlink pointing to file inside the packgage
ii) symlink pointing to directory inside the packgage
iii) symlink pointing to file outside the packgage, but in SCM
iv) symlink pointing to directory outside the packgage, but in SCM
v) symlink pointing to something outside package and not managed by SCM
My expectation is that i and ii work out of the box, iii and iv should
work if you have a file finder plugin for setuptools that traverses the
symlink, and v to not work. Is that expectation reasonable?
Wichert.
Hi I have a couple of packages that I don't want to upload to PyPI
just yet but they are hosted some other place, and one (project2)
depends on the other (project1). I've packaged them using the standard
distutils setup.
For project1, setup.py contains:
setup(name='project1', ..., requires=[], provides=['project1'], ...)
and for project2, setup.py contains:
setup(name='project2', ..., requires=['project1'], provides=['project2'], ...)
I put them in some host https://my.host.org/repository, but when I run
pip install -f https://my.host.org/repository project2
installs project2 correctly, but it does not install project1.
So my first question is: is the 'requires' keyword in setup.py
actually used when installing packages? or it only works for projects
hosted on PyPI?
Thanks
--
Ernesto Posse
Modelling and Analysis in Software Engineering
School of Computing
Queen's University - Kingston, Ontario, Canada
Hi Christian,
On 25/03/2011 16:49, Christian Theune wrote:
> the German speaking Zope Users Group (DZUG e.V.) organizes a series of 4
> sprints this year to support feature development within the proximity of
> the ZTK and solve problems encountered by Zope, Plone and Python developers.
>
<snip>
> Topics
> ======
>
> * Discussing how to deal with "private" releases
FWIW, I've had no problems with this, here's a sample buildout.cfg:
[buildout]
extensions = lovely.buildouthttp
find-links = https://example.com/password/protected/folder
...and just dump the .tgz sdists in that folder.
Of course, if you don't need password protection such as when you have
your "egg server" on a private network, you just need the find-links.
I'm not really sure why people have written a complicated array of "egg
servers" and the like when a simple http or file system served folder is
just fine ;-)
cheers,
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
