Mailman 3 January 2006 - Distutils-SIG

New features in setuptools SVN
by Phillip J. Eby 10 Jan '06

10 Jan '06

I just checked in a couple of new features for EasyInstall that some of you will be interested in. First, '.' is now allowed in project names, and '_' and '-' in names should no longer confuse EasyInstall when it searches PyPI. It will use whatever characters you originally entered. The bad news is that this could break any existing dependencies to projects with a '.' in their names, because the '.' was being converted to '-' before, and isn't now. The good news is that there are only 5 PyPI projects with a '.' in the name (rsync.py, spider.py, _sre.py, zope.interface, and zope.testbrowser), and of them only '_sre.py' had a prayer of working with EasyInstall before. Now, 'spider.py' also works due to the next new feature, which is that installation of bare '.py' files (no accompanying setup.py) is now supported. If a PyPI "Download URL" points to a file with a .py extension, EasyInstall will automatically create a proper setup.py alongside it. It can also do this for any link to a .py file that includes an '#egg=name-version' fragment identifier, so that you can use a link like this: http://bellsouthpwp.net/m/e/mefjr75/python/PySourceColor.py#egg=PySourceCol… to point to somebody's .py file and assign it a project name and version. Note that the '#egg' tag must be *unambiguous*, which means you cannot have a '-' in the project name or version. You must escape them as '_' characters, using the '-' only to delimit the name from the version. Anyway, if you do this on *your* project's PyPI page to link to your project's unpackaged dependencies, you can fill in for the dependency's lack of a PyPI page or setup script, while still using the normal setuptools way of specifying project dependencies. Please let me know if you have any questions or problems using these new features. Thanks!

1 0

Dynamic linking - another idea
by Phillip J. Eby 07 Jan '06

07 Jan '06

Okay, so here's a crazy idea to make dynamic libraries in the same package work. Suppose the linker includes an rpath of '.' (aka os.curdir), and renames each extension module that's using a library such that it's not directly importable by Python. Setuptools already generates a stub .py file loader for extension modules anyway during bdist_egg, so that zipped extensions can be extracted. So, this stub could just change directory to the directory of the extension before doing the import, with setdlopenflags(RTLD_NOW). If I understand correctly, this should force all symbols in the extension to be resolved immediately, so the '.' rpath would let the extension load the nearby library. After this was done, the stub code would restore the original current directory and dlopen flags. Since extensions built this way would not be directly importable, it would ensure that the stub is always used. And since they do not require a hardcoded real path, they would be relocatable and manually extractable from the zipfile. Does any see any holes in this, aside from the sheer craziness of doing it? :) That is, are there any platforms where a '.' rpath is forbidden? Where RTLD_NOW won't let it resolve all the symbols? Where the '.' rpath is interpreted as the executable's startup directory rather than as the load-time current directory?

1 0

Re: [Distutils] Dynamic linking on Linux, Mac OS, et al
by Phillip J. Eby 07 Jan '06

07 Jan '06

At 12:29 PM 1/6/2006 +0800, Jeff Pitman wrote: >Are these just extensions that are imported? Because if they are, then you >don't need to mess with LD_LIBRARY_PATH and you can put them anywhere on >Linux as long as PYTHONPATH is correct. -- -jeff Nope. They're shared libraries being built by the distribution itself, as in the case of PyICU, and certain Windows C extensions that need to include the target library (e.g. sqlite) because it's not a "system" library there. It could perhaps be questioned whether it's a good idea to include any other kind of shared library in an extension project, and in fact the common practice to link between Python extensions is to use a Python CObject to wrap an API function pointer table. This neatly bypasses all of the issues... *if* you control the library in question. So far I've done a few experiments with trying to hack LD_LIBRARY_PATH at runtime, use dl.open() with RTLD_GLOBAL, etc. to force finding the right library, but so far it seems like there's no way to get the extensions to consider the opened library to be "the same" as the one they're looking for. (Although it may just be that I'm missing some trick on the build side of the equation.)

3 7

Automatically going for a pre-req dev
by Ben Bangert 07 Jan '06

07 Jan '06

When installing a package that requires the dev version of another package, it fails to to and download a dev version of the package. This appears to be because it either searches the filesystem for the prereq version, or asks CheeseShop. Given that the prereq indicates it wants a dev version, it would seem reasonable to search CheeseShop with ==dev equivalent active. As an example, if you install: easy_install -U hellahella==dev It installs fine from svn, then goes back searching for Pylons, and gets: Processing dependencies for hellahella==0.1dev-r576 Searching for Pylons>=0.1dev-r289 Reading http://www.python.org/pypi/Pylons/ Reading http://pylons.groovie.org/ No local packages or download links found for Pylons>=0.1dev-r289 error: Could not find distribution for Requirement.parse ('Pylons>=0.1dev-r289') Yet, if you run: easy_install -U Pylons==dev It has no problem fetching the latest Pylons and the Paste prereqs (which would have the same problem if I hadn't linked to them explicitly on the Pylons page). Since you already indicate you want ==dev, shouldn't it keep that during the prereq if necessary? - Ben

3 5

[setuptools] When .py files are "data", strange things happen
by David Handy 06 Jan '06

06 Jan '06

I want setuptools to install a bunch of sample scripts in a package-relative directory. Each end user will later run a script that copies these to a directory relative to his or her home directory. However, strange things happen to data files that have a .py extension. 1) Saying "include_package_data = True" was not enough to get them included in the egg. The .txt files were included, but the .py files were magically excluded. 2) I overcame the above problem with the following setup parameter: package_data = { # Include all .py files in the data directories, they # are excluded by default. 'cpif.data': ['lander/*.py', 'samples/*.py'], } But now I get a bogus .pyc file generated for each of these sample scripts. These bogus .pyc files get included in the egg. 3) I tried to overcome problem #2 above with the following setup parameter: exclude_package_data = { # The example programs should not be shipped with .pyc files # XXX This is not working, the .pyc files are being created and # included anyway. 'cpif.data': ['lander/*.pyc', 'samples/*.pyc'], } But this didn't work. Putting the following lines in my MANIFEST.in file also didn't work: exclude cpif/data/lander/*.pyc exclude cpif/data/samples/*.pyc I suppose I could turn off .pyc file generation, but that's not what I want. I just want to treat certain .py files as data files. The "data" .py files are in a subdirectory of a python package, but the directory in which they directly reside does not contain an __init__.py file, so they are not directly importable. So I don't think .pyc files should be generated for them in any case. Thanks in advance for ideas/sympathy/fixes, etc. -- David Handy Computer Programming is Fun! Beginning Computer Programming with Python http://www.handysoftware.com/cpif/

3 3

Re: [Distutils] [setuptools] When .py files are "data", strange things happen
by Phillip J. Eby 05 Jan '06

05 Jan '06

At 01:48 PM 1/5/2006 -0800, Ben Bangert wrote: >On Jan 5, 2006, at 9:07 AM, Phillip J. Eby wrote: > >>If I understand your use case correctly, I don't see how that hurts >>anything. > >It hurts a ton if you want to use Paster templates, because you'll >have files ending in .py that aren't meant to be processed by paster, >and _tmpl, all of which are inside non-package data directories, thus >won't get included any other way. You lost me. How does having the .py included and compiled to .pyc hurt anything? .pyc files include a date/time stamp of the .py they were built from, so if you accidentally copy them somewhere and change the corresponding .py file, the .pyc will be ignored and overwritten in the import case, and ignored altogether in an execfile or script-running case. In no scenario that I understand, does the presence of a .pyc file hurt anything. Could you please explain what problem you think it will cause?

1 0

setuptools 0.6a9 is released
by Phillip J. Eby 05 Jan '06

05 Jan '06

This version contains mostly bug fixes and robustness improvements in a variety of areas, ranging from improved web spidering to better handling of spaces in the Windows directory names for generated .exe files. The biggest new feature included, however, is system packager support including improved bdist_* command handling, and support for installing projects in a "single version, externally managed" format. You can now also build RPMs and Windows installers of setuptools-baseds project without any special steps, even if the project uses egg metadata. The other new features are for developers of setuptools-based projects, and mostly have to do with improved handling of source distribution manifest generation and specifying package data to be included in a project. You can now simply specify "include_package_data=True" and any file under revision control or listed in your source manifest (or MANIFEST.in) will be installed as part of the package. There is also an 'exclude_package_data' option that can be used to trim back the list, and the 'package_data' option can still be used to add files that aren't in revision control or the source manifest. Finally, there have been many documentation updates, including an overhaul of the installation instructions and a new "What your users should know" section of the setuptools manual for project developers. The next release of setuptools will be 0.6a10, for which I plan to add shared library building support (it's needed for an OSAF project, PyICU). There are also some minor features (like dependency_links) that didn't make it into 0.6a9 but which aren't big enough to need waiting till 0.7. Special thanks to Kevin Dangoor, Ian Bicking, and everyone else who reported bugs or helped track them down. Thanks to your participation, 0.6a9 is in much better shape than 0.6a8, and I look forward to a great 0.6a10!

2 4

[setuptools] setuptools and fat binaries
by Ronald Oussoren 04 Jan '06

04 Jan '06

Hi, Apple supports fat binaries on Mac OS X (they call them universal binaries), that is binaries that contain executable code for multiple architectures. In released version of the os this can be used to build binaries and libraries that support both PPC and PPC64 architectures, which isn't used very much, and in the future this will be the mechanism to support both the current PPC based systems and the to-be-introduced intel systems with a single binary. I'm currently playing around with Python on an Intel developer system (my goal is a patch that will make it easy to build a fat/universal build of Python on Mac OS X), and I wondering how setuptools (and specifically eggs) can support universal binaries. The reason I ask is because of eggs that contain extensions, whose name contains the architecture, such as aem-0.10.0-py2.4-macosx-10.4-ppc.egg. Just using two eggs, one for each architecture, would work but seems wasteful and would make it harder to create self fat application bundles. Ronald

2 4

Re: [Distutils] Setuptools' goal in rpm/deb-based systems
by Phillip J. Eby 02 Jan '06

02 Jan '06

At 11:05 PM 12/29/2005 +0800, Jeff Pitman wrote: >I see that --single-version-externally-managed has been added as an >option so that .deb/.rpm packaging can take advantage of that relying >upon apt/yum/smart to take care of business. Hopefully, this will be >pushed out soon. FYI, unless there's some policy preventing it, you can use the current SVN snapshot as a basis for a Debian package of setuptools. If built from a source checkout, it will correctly include its Subversion revision number as part of its name, e.g. "0.6a9dev-r41780". In setuptools' version numbering, this is an *older* release than "0.6a9", so when the final 0.6a9 comes out it will replace the "dev" versions. I realize that this isn't necessarily the same thing as how Debian package versions work, but I assume it must have some facility for pre-release tags. If all else fails, you could treat it as "0.6a8-r41780" instead, which might be a more compatible numbering approach. At this point, the only thing left on my to-do list for 0.6a9 is more documentation for EasyInstall, so don't let that stop you from packaging it. The only reason to wait would be to see if anybody reports bugs with the current version, which probably won't happen until *after* 0.6a9 is released. :)

2 1

Distutils-SIG January 2006