Mailman 3 September 2005 - Distutils-SIG

setuptools 0.6a3 released
by Phillip J. Eby 24 Sep '05

24 Sep '05

This is mostly a bugfix release for various problems that people reported in the last week or two, but I also added improved version conflict handling for scripts (see my post re: sys.path and __requires__), and added the ability to generate GUI scripts with standard wrappers on non-Windows platforms, and .pyw/.exe wrappers on Windows.

1 0

Putting eggs first on sys.path
by Phillip J. Eby 24 Sep '05

24 Sep '05

One of the things that occasionally creates problems for installing applications with setuptools, and for certain non-root package installs on Unix, is the fact that eggs are normally added to the *end* of sys.path, rather than the beginning. I did this because I needed to maintain various invariants in pkg_resources, such as namespace packages' __path__ items needing to match sys.path order. Also, .pth files add entries to the end of sys.path, so changing this isn't really an option for EasyInstall-supplied default eggs. The problems with this are: 1. If you install an application but then later set an incompatible version of one of its requirements as the default version of that project, then the application will stop working 2. If you are using a simplistic non-root installation, system-installed eggs can override or conflict with your personal eggs, and prevent the use of entry points from them. So, after some thought, I think I have a way to adjust the existing policy that will deal with these problems, while still allowing most invariants to remain intact. It's a little kludgy, so I'm hoping somebody has a better idea. For EasyInstall-generated wrapper scripts, it's no big deal and is invisible to the user. For manual use, it seems a little clumsy, though. The idea is this: when pkg_resources is imported, it will check the __main__ module for a __requires__ variable. If found, it will do the equivalent of require()-ing that value, but with sys.path set to an empty list. It will then restore the old value of sys.path, adding it *after* the entries added by the require() process. Thus, the very first require() will insert entries at the start of sys.path, but in a consistent order. Thus, a script can effectively require package versions that are not the default. (If you try this currently, you get VersionConflict errors.) You might now ask, "Why __requires__? Why not just do this for the first require() call?" Unfortunately, it's not that simple. pkg_resources needs to export a global "working_set" object that lists the active eggs and their entry points. Once you've imported pkg_resources, then, this list needs to be in a consistent state. So it's a bit of a chicken and egg problem, in that you need pkg_resources imported to do require(), but if pkg_resources is imported then you need to have already done any require()s that override existing sys.path entries. Thus, putting a variable in a common module (e.g., by putting it in the script before importing pkg_resources) allows us to pass a parameter to something that hasn't been imported yet. For use in the interactive interpreter, things are a bit more complex, because it's possible that you could import something further down on sys.path before importing pkg_resources, possibly leading to an implicit conflict of some kind. You can still set __requires__ and import pkg_resources, but it looks weird to do that, and it's certainly not the usual way to do a require(), so it seems potentially confusing as well. Of course, I suppose I could just make it an "undocumented internal feature" of pkg_resources and setuptools, reserved for EasyInstall-generated scripts. This probably makes sense in that the default versions of packages are the ones that you'll nominally be using in the interactive interpreter. On the other hand, a simple non-root install won't work if you want to override site-wide EasyInstalled defaults, unless you do some fancy footwork in a sitecustomize.py or ~/ Maybe I'm just expecting too much, though. Perhaps it's unrealistic to expect to add new features, be 100% backward compatible, support everybody's personal quirky directory layout on Unix, AND still not have any kludgy bits. :) Thoughts, anyone?

1 0

python eggs and lxml
by Martijn Faassen 24 Sep '05

24 Sep '05

Hi there, I've been looking at Python eggs, easy install and setuptools with a lot of interest -- very impressive work. I've been thinking of packaging lxml with it, and, on the larger scale, look into packaging Zope 3 with it. Concerning lxml I run into a few questions however. lxml depends on large external C libraries (libxml2 and libxslt). a) Is there a way to require versions of C libraries to be available in the Python eggs dependencies? I can't seem to find a reference to this scenario, but perhaps I didn't look carefully enough. The goal here would be to give users trying to install lxml (or something that depends on lxml) useful feedback about what in their system they're missing (or have the wrong version of). b) Going further, it'd be nice for some scenarios to actually be able to include private versions of libxml2/libxslt in a Python eggs. This is especially interesting on Windows deployments (where you'd include a binary of these libraries). Has something like this considered? I saw references to Pyrex support, but in lxml's case, the Pyrex code depends on a large underlying library. c) It's also interesting though for deployment on linux. It'd be nice to be able to include the source versions of specific versions of libxml2 and libxslt with lxml and to be able to build/install them such that they are only used for lxml. This way the system libraries (which may be out of date or have otherwise a wrong version) would not be in play and wouldn't be affected. If something like this were arranged, it'd be much easier to make lxml a requirement of a large package like for instance Zope 3 (which is being considered). I realize that any or all of these might be out of scope for easy install -- in the Linux case, it might be deferred to a Linux package management system, for instance. Still, I imagine the case where a Python library has a dependency on a potentially large non-Python codebase could be fairly common, and it'd be nice if such libraries could be "first class" easy_install citizens so that other Python libraries can safely depend on them. What are people's thoughts were about supporting such scenarios? Regards, Martijn

3 5

setuptools or PyPI problem...?
by Ian Bicking 24 Sep '05

24 Sep '05

I don't know which side this belongs to, but I had a problem when I tried to create a package with a "-" in it ("Paste-Deploy"). setup.py register worked fine, and created a "Paste-Deploy" project; however, when I did an upload it created a "Paste_Deploy-0.1.tar.gz" file, and PyPI wouldn't accept it, I believe because it thought it belonged to the (nonexistant) Paste_Deploy project. In the end I just smushed the words together, but I figure this really should work. I don't know if it's setuptools (or distutils) that is uploading to PyPI incorrectly, or PyPI that is mismatching projects. -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

5 11

FW: Distributing Python Modules - Error using foo.py in section 2.1 - Corrected
by Michael Delaney 23 Sep '05

23 Sep '05

Greg We do not have admin rights for our workstations here. After I was able to temporarily obtain admin rights and add python to the Path, your example works. Thanks. > -----Original Message----- > From: Michael Delaney > Sent: Friday, September 23, 2005 1:20 PM > To: 'distutils-sig(a)python.org' > Subject: Distributing Python Modules - Error using foo.py in > section 2.1 > > Greg Ward, > > I am reading the documentation you wrote titled "Distributing > Python Modules". In section 2.1 (A Simple Example), I made a simple > script named foo.py but I am not able to get it running following the > steps you provided. The error I receive is listed below and I have > attached the files. I am executing this in a console prompt using > Windows XP as the Operating System. Could you please tell me what I > am doing incorrectly? > > > 2.1 A Simple Example > > foo.py > > # A setup script. > # > > from distutils.core import setup > setup(name="foo", > version="1.0", > py_modules=["foo"]) > > > If all you want to do is distribute a module called foo, contained in > a file foo.py, > then your setup script can be as simple as this: > > from distutils.core import setup > setup(name="foo", > version="1.0", > py_modules=["foo"]) > > > To create a source distribution for this module, you would create a > setup script, > Setup.py, containing the above code, and run: > > python setup.py sdist > > > > The following error is displayed: > > C:\Program Files\Python24>python setup.py > > Traceback (most recent call last): > File "setup.py", line 7, in ? > py_modules=["foo"]) > File "C:\Program Files\Python24\distutils\core.py", line 101, > in setup _setup_distribution=dist=Klass(attrs) > File "C:\Program Files\Python24\distutils\dist.py", line 130, > in __init__settattr(self, method_name, getattr(self.metadata, > method_name)) > Attribute Error: Distribution Metadata instance has no attribute > 'get___doc___' > > > > <<foo.py>> > > <<setup.py>> > > > Thank you very much for your time. > > Michael Delaney > Application Programmer > Checkout Direct(r) > 200 Carillon Parkway > St. Petersburg, FL 33716 > Ph: 727-579-5144 > Michael.Delaney(a)CatalinaMarketing.com >

1 0

Distributing Python Modules - Error using foo.py in section 2.1
by Michael Delaney 23 Sep '05

23 Sep '05

Greg Ward, I am reading the documentation you wrote titled "Distributing Python Modules". In section 2.1 (A Simple Example), I made a simple script named foo.py but I am not able to get it running following the steps you provided. The error I receive is listed below and I have attached the files. I am executing this in a console prompt using Windows XP as the Operating System. Could you please tell me what I am doing incorrectly? 2.1 A Simple Example foo.py # A setup script. # from distutils.core import setup setup(name="foo", version="1.0", py_modules=["foo"]) If all you want to do is distribute a module called foo, contained in a file foo.py, then your setup script can be as simple as this: from distutils.core import setup setup(name="foo", version="1.0", py_modules=["foo"]) To create a source distribution for this module, you would create a setup script, Setup.py, containing the above code, and run: python setup.py sdist The following error is displayed: C:\Program Files\Python24>python setup.py Traceback (most recent call last): File "setup.py", line 7, in ? py_modules=["foo"]) File "C:\Program Files\Python24\distutils\core.py", line 101, in setup _setup_distribution=dist=Klass(attrs) File "C:\Program Files\Python24\distutils\dist.py", line 130, in __init__settattr(self, method_name, getattr(self.metadata, method_name)) Attribute Error: Distribution Metadata instance has no attribute 'get___doc___' <<foo.py>> <<setup.py>> Thank you very much for your time. Michael Delaney Application Programmer Checkout Direct(r) 200 Carillon Parkway St. Petersburg, FL 33716 Ph: 727-579-5144 Michael.Delaney(a)CatalinaMarketing.com

1 0

Working around .pth/site problems
by Phillip J. Eby 23 Sep '05

23 Sep '05

The recent TurboGears release has been really interesting, in that it's given me a much better perspective on the kinds of problems Unix users have been having with EasyInstall. Basically, the fundamental problem in Unix-land is that the system-packaged Python expects to control all packages, and the default setup of Python's sys.path supports that theory. In essence, the problem is that Python adds all .pth-specified paths to the *end* of sys.path, meaning that system-installed packages in the stdlib or site-packages take precedence over anything else you can install, except for directories specified in PYTHONPATH or that are in the startup script's directory. Effectively, this means that easy-install.pth cannot be used to resolve the issue without using 'import' hackery such as importing pkg_resources or some other bootstrap module, in order to fudge the contents of sys.path. However, even such hackery isn't exactly ideal. The normal precedence for sys.path is: 1. the script directory, or current directory if no script 2. PYTHONPATH directories 3. system-defined directories 4. "site" directories hardcoded by site.py 5. directories specified by .pth files What we need to do is to insert certain directories between #2 and #3. Luckily, for Python 2.3 and 2.4, the first directory in category 3 always ends with "python2X.zip", so at least we can determine the correct insertion point. (By the way, in case you're wondering why eggs should go before system-defined directories as well as site directories, it's because it allows us to distribute e.g. back-ported standard library modules. For example, I'd love to make an egg for Python 2.4's doctest module and use it with Python 2.3. 2.3 also has a doctest module, but it lacks many useful unittest integration features.) Okay, so I could rework the .pth file processing to do this, such that it always chucks in a bootstrap module and the .pth file itself contains hackery to call it. That would allow you to override system packages if you had write access to an existing site-packages directory. But that's not really very useful, in terms of solving the big picture problem, which is how to use eggs (including setuptools itself) without affecting your main installation at *all*. And ideally, you should still be able to have default versions of packages selected, or at minimum there needs to be a way to get pkg_resources to be present, since pkg_resources can't be used to find its own egg. Okay, so suppose we just ditch easy-install.pth and setuptools.pth. Whenever we install setuptools in a target directory, we go ahead and install pkg_resources the "old-fashioned" way, because it's the only package that can't ever be multi-versioned anyway. (Ironically, setuptools itself could be multi-versioned, but setuptools really depends quite heavily on the pkg_resources version, so multi-versioning it isn't really practical.) Anyway, easy-install.pth could now be reserved for site-packages installs and other "site" directories, with all the current caveats applying. If you have control of your site-packages directory, the current scheme works just fine. But, if you don't have control, you can just set up an arbitrary "bin" directory for a given project, and install whatever you like there, as long as it's multi-version. If pkg_resources inserts require()'d eggs at the *beginning* of sys.path, instead of the end, then this would be perfect. However, there's a possible race condition here. Suppose the "foobar" package is installed in site-packages, and in an egg, and you import it before require()-ing it. Your require() would succeed, and add the egg to the front of sys.path. You're actually using the wrong version of "foobar", but have no way to know. Or worse, if there are multiple top-level modules or packages in "foobar", you might import some of them from one version and some from another! This problem is why I originally made pkg_resources add eggs to the end of sys.path. But, I now have a lot more data at my disposal. For example, eggs now carry a "top_level.txt" file listing their top-level module names, so in principle I could use that to check whether any of those modules had already been imported, and raise a VersionConflict error if so. That would probably address that issue. So, to recap my conclusions thus far: * Always install and compile pkg_resources the old-fashioned way, in addition to its egg form. * pkg_resources should add new eggs to the start of sys.path, and check for already-imported, top-level, non-namespace modules or packages With just these two changes, it should be possible to create usable runtime environments for tools that understand eggs, and any code that's packaged as a distutils or setuptools project. (Because you can then use setup.py develop or easy_install to generate script wrappers that ensure the right versions of things are require()'d onto sys.path.) This does not address default versions using easy-install.pth or setuptools.pth - in effect those are limited to installations with site-packages access, or for people using the "non-root installation" instructions for EasyInstall. However, it's fairly reasonable, I think, to say that you can't make something the default version if you already have a system-defined default version of something. The fact that you still have an option to explicitly require() what you want is a nice bonus, and it helps move us forward to the day when --multi-version will be the normal way to install things. The downside to these changes, of course, is that they may introduce new problems. In particular, I'll need to do some rework on ez_setup to allow for the possibility that pkg_resources can be on sys.path, but not setuptools. Thoughts, anyone?

2 1

api documentation wrong
by Wolfgang Langner 22 Sep '05

22 Sep '05

Hello, with my experiments I found an API documentation bug on: http://peak.telecommunity.com/DevCenter/PythonEggs install_on(path=sys.path) is obsolete now this is activate Version 0.6a2 of setuptools bye by Wolfgang

2 1

buildutils
by Ian Bicking 22 Sep '05

22 Sep '05

I thought I'd note that I've been doing a little bit of work on Ryan Tomayko's buildutils lately (http://buildutils.lesscode.org/). It now has distutils.command entry points, so any setuptools-based package will get the new commands. And I've moved the svntag command from PasteDeploy to buildutils. I guess next up I'm going to re-look at how to enable some of these commands conditionally, along the lines discussed in this thread: http://mail.python.org/pipermail/distutils-sig/2005-August/005011.html Maybe it will actually mean removing some of the current entry points, and requiring developers to specifically configure them (e.g., pudge, or publish). -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

2 2

setuptools: Inconsistent quoting
by Ian Bicking 22 Sep '05

22 Sep '05

I have a setup file, that looks roughly like: setup(... extras_require={ 'WSGI-Utils': ['WSGI-Utils'], }, entry_points=""" [paste.server_runner] wsgiutils=paste.script.wsgiutils_server:run_server [WSGI-Utils] """) When I try to use the wsgiutils entry point I get: pkg_resources.UnknownExtra: PasteScript 0.1dev-r3227 has no such extra feature 'wsgi_utils' If I rename it to WSGIUtils is all works. So apparently quoting is being applied differently depending on context. You can get the full setup.py script with: svn cat -r 3231 http://svn.pythonpaste.org/Paste/Script/trunk/setup.py -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

1 0