Mailman 3 July 2005 - Distutils-SIG

Re: [Distutils] distutils install probs on suse 9.0
by Mats Wichmann April 29, 2018

April 29, 2018

At 05:43 AM 5/18/2004, David Handy wrote: >On Tue, 18 May 2004, Floris van der Tak wrote: > >> my attempt to install the distutils on suse 9.0 (which provides python >> 2.3) fails: > >Why are you installing distutils on a system that already has Python 2.3 >installed? Python 2.3 normally comes with distutils as part of the Python >standard library. > >Did suse distribute a version of Python that had distutils removed? It should come from python-devel

2 1

Sample setup.py for Numeric Python
by Greg Ward April 29, 2018

April 29, 2018

Hi again -- [cc'd to Paul Dubois: you said you weren't following the distutils sig anymore, but this directly concerns NumPy and I'd like to get your input!] here's that sample setup.py for NumPy. See below for discussion (and questions!). ------------------------------------------------------------------------ #!/usr/bin/env python # Setup script example for building the Numeric extension to Python. # This does sucessfully compile all the .dlls. Nothing happens # with the .py files currently. # Move this file to the Numerical directory of the LLNL numpy # distribution and run as: # python numpysetup.py --verbose build_ext # # created 1999/08 Perry Stoll __rcsid__ = "$Id: numpysetup.py,v 1.1 1999/09/12 20:42:48 gward Exp $" from distutils.core import setup setup (name = "numerical", version = "0.01", description = "Numerical Extension to Python", url = "http://www.python.org/sigs/matrix-sig/", ext_modules = [ ( '_numpy', { 'sources' : [ 'Src/_numpymodule.c', 'Src/arrayobject.c', 'Src/ufuncobject.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/numpy.def' } ), ( 'multiarray', { 'sources' : ['Src/multiarraymodule.c'], 'include_dirs' : ['./Include'], 'def_file': 'Src/multiarray.def' } ), ( 'umath', { 'sources': ['Src/umathmodule.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/umath.def' } ), ( 'fftpack', { 'sources': ['Src/fftpackmodule.c', 'Src/fftpack.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/fftpack.def' } ), ( 'lapack_lite', { 'sources' : [ 'Src/lapack_litemodule.c', 'Src/dlapack_lite.c', 'Src/zlapack_lite.c', 'Src/blas_lite.c', 'Src/f2c_lite.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/lapack_lite.def' } ), ( 'ranlib', { 'sources': ['Src/ranlibmodule.c', 'Src/ranlib.c', 'Src/com.c', 'Src/linpack.c', ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/ranlib.def' } ), ] ) ------------------------------------------------------------------------ First, what d'you think? Too clunky and verbose? Too much information for each extension? I kind of think so, but I'm not sure how to reduce it elegantly. Right now, the internal data structures needed to compile a module are pretty obviously exposed: is this a good thing? Or should there be some more compact form for setup.py that will be expanded later into the full glory we see above? I've already made one small step towards reducing the amount of cruft by factoring 'include_dirs' out and supplying it directly as a parameter to 'setup()'. (But that needs code not in the CVS archive yet, so I've left the sample setup.py the same for now.) The next thing I'd like to do is get that damn "def_file" out of there. To support it in MSVCCompiler, there's already an ugly hack that unnecessarily affects both the UnixCCompiler and CCompiler classes, and I want to get rid of that. (I refer to passing the 'build_info' dictionary into the compiler classes, if you're familiar with the code -- that dictionary is part of the Distutils extension-building system, and should not propagate into the more general compiler classes.) But I don't want to give these weird "def file" things standing on the order of source files, object files, libraries, etc., because they seem to me to be a bizarre artifact of one particular compiler, rather than something present in a wide range of C/C++ compilers. Based on the NumPy model, it seems like there's a not-too-kludgy way to handle this problem. Namely: if building extension "foo": if file "foo.def" found in same directory as "foo.c" add "/def:foo.def" to MSVC command line this will of course require some platform-specific code in the build_ext command class, but I figured that was coming eventually, so why put it off? ;-) To make this hack work with NumPy, one change would be necessary: rename Src/numpy.def to Src/_numpy.def to match Src/_numpy.c, which implements the _numpy module. Would this be too much to ask of NumPy? (Paul?) What about other module distributions that support MSVC++ and thus ship with "def" files? Could they be made to accomodate this scheme? Thanks for your feedback -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

4 5

Can now compile simple extensions
by Greg Ward April 29, 2018

April 29, 2018

Hi all -- at long last, I found the time to hack in the ability to compile extension modules to the Distutils. Mainly, this meant adding a 'build_ext' command which uses a CCompiler instance for all its dirty work. I also had to add a few methods to CCompiler (and, of course, UnixCCompiler) to make this work. And I added a new module, 'spawn', which takes care of running sub-programs more efficiently and robustly (no shell involved) than os.system. That's needed, obviously, so we can run the compiler! If you're in the mood for grubbing over raw source code, then get the latest from CVS or download a current snapshot. See http://www.python.org/sigs/distutils-sig/implementation.html for a link to the code snapshot. I'm still waiting for more subclasses of CCompiler to appear. At the very least, we're going to need MSVCCompiler to build extensions on Windows. Any takers? Also, someone who knows the Mac, and how to run compilers programmatically there, will have to figure out how to write a Mac-specific concrete CCompiler subclass. The spawn module also needs a bit of work to be portable. I suspect that _win32_spawn() (the intended analog to my _posix_spawn()) will be easy to implement, if it even needs to go in a separate function at all. It looks from the Python Library documentation for 1.5.2 that the os.spawnv() function is all we need, but it's a bit hard to figure out just what's needed. Windows wizards, please take a look at the 'spawn()' function and see if you can make it work on Windows. As for actually compiling extensions: well, if you can figure out the build_ext command, go ahead and give it a whirl. It's a bit cryptic right now, since there's no documentation and no example setup.py. (I have a working example at home, but it's not available online.) If you feel up to it, though, see if you can read the code and figure out what's going on. I'm just hoping *I'll* be able to figure out what's going on when I get back from the O'Reilly conference next week... ;-) Enjoy -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Two fixes (at last)
by Greg Ward April 29, 2018

April 29, 2018

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker April 29, 2018

April 29, 2018

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller April 29, 2018

April 29, 2018

3 2

Setuptools CVS broken for python2.3
by Vincenzo Di Massa Aug. 1, 2005

Aug. 1, 2005

Is setuptools CVS going to work using python 2.3? After 25 of July it does not work anymore. I need it to build packages for both 2.4 and 2.3 . Should I stay on the 25 of July version? Thank you very much for this nice sftware. It is very usefull to use and to read its code to learn pythonic programming. Vincenzo

2 4

EasyInstall 0.4a1: now with PyPI autodownload
by Phillip J. Eby Aug. 1, 2005

Aug. 1, 2005

I've just released a new version of EasyInstall, that supports automatic package location via PyPI; installing a package can now be as simple as: easy_install "SQLObject>=0.6" There are many new options and features related to this; see the EasyInstall home page for more information: http://peak.telecommunity.com/DevCenter/EasyInstall For the benefit of the PyPI maintainers, here is a summary of EasyInstall's fairly minimal assumptions about PyPI's current behavior, assuming that "base-url" is the root URL of the package index: * Going to "base-url/SomePackage" produces an HTML page that either has a title containing "Index of Packages" and links to zero or more pages for specific versions, or else it is a single-version package page. * Single-version package pages may have a home page and download URL link, each of which occurs after '<th>Home Page' and '<th>Download URL' respectively, if present. * Going to "base-url/" (note trailing '/') produces an HTML page containing links to all active versions of registered packages * Links to package pages always have URLs of the form "base-url/SomePackage/itsVersion" - i.e., exactly two path parts following the base URL, with no query strings, parameters, fragments, etc. EasyInstall should continue to work with PyPI if these assumptions continue to hold. However, I'd also like to suggest that PyPI deprecate the use of spaces and other non-alphanumeric characters (other than '-') in package names, and move to a case-insensitive matching mechanism for package names. (Currently, if a user types a package name in the wrong case, EasyInstall downloads the full package list in order to do its own case-insensitive search.) By the way, EasyInstall does not rely solely on the download URL of a PyPI entry, nor does it assume that the download URL is in fact the URL where the package's source distribution is found. Instead, EasyInstall inspects the URLs for whether the extension suggests an egg or source distribution. If not, it retrieves the listed URL, and if it contains HTML, it scans the HTML for links to egg or source distributions (again identified by extension). It does this for both the home page and the download URL, in case there is a usable download link on the package's home page. This approach was chosen to maximise the odds of successful downloading, given the current contents of PyPI.

1 1

pkg_resources: .egg-info files
by Ian Bicking July 29, 2005

July 29, 2005

What's the correct way to get access to files in the .egg-info directory (where I'm putting some custom configuration files)? I got this far: req = pkg_resources.Requirement('ProjectName') pkg_resources.resource_string(req, 'filename') But that gives me access to the base dir of the project, not the .egg-info dir.

2 1

bugs in 0.6a0
by Ashley Walsh July 29, 2005

July 29, 2005

Hi, I've been using some of the latest setuptools features, but had to make a couple of trivial patches for win32/python2.3: Index: pkg_resources.py =================================================================== RCS file: /cvsroot/python/python/nondist/sandbox/setuptools/ pkg_resources.py,v retrieving revision 1.54 diff -r1.54 pkg_resources.py 1040c1040 < return fspath[len(self.zip_pre):] --- > return fspath[len(self.zip_pre):].replace(os.sep, '/') 1446c1446 < parent = '.'.join(package.split('.')[:-1]) --- > parent = '.'.join(packageName.split('.')[:-1]) Index: setup.py =================================================================== RCS file: /cvsroot/python/python/nondist/sandbox/setuptools/setup.py,v retrieving revision 1.33 diff -r1.33 setup.py 47c47 < for cmd in SETUP_COMMANDS if cmd!="build_py" or sys.version>="2.4" --- > for cmd in SETUP_COMMANDS if not (cmd=="build_py" and sys.version>="2.4") Hope this helps. Cheers, Ashley Walsh

2 1

Not repeating requirements for eggs
by Kevin Dangoor July 28, 2005

July 28, 2005

setuptools lets you declare install_requires with a set of packages. This is handy and is nicer than the "old way". In one of my __init__.py files, I am redeclaring my requirements using pkg_resources.require, to ensure that they're all still available and to load them properly. Does it make sense to run pkg_resources.require at runtime? (I think so...) If so, it seems like a function could be added to pkg_resources that goes through this egg's required dependencies and does a require() on them. Something like pkg_resources.requireAll("MyProjectName"). What do you think? Kevin

3 3

Returned mail: see transcript for details
by gallwey＠sun.com July 26, 2005

July 26, 2005

The original message was received at Tue, 26 Jul 2005 08:11:25 +0200 from sun.com [69.151.169.189] ----- The following addresses had permanent fatal errors ----- <distutils-sig(a)python.org>

1 0

Re: [Distutils] [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
by Phillip J. Eby July 25, 2005

July 25, 2005

[cc:'d to distutils-sig because this is mostly about cool uses for the new EntryPoint facility of setuptools/pkg_resources] At 12:01 AM 7/26/2005 +0100, James Gardner wrote: >Hi Phillip, > >>There's one other advantage: this format will hopefully become as >>successful as WSGI itself in adoption by servers and applications. >>Hopefully within a year or so, *the* normal way to deploy a Python web >>app will be using a .wsgi file. >> >>Beyond that, we can hopefully begin to see "Python" rather than >>"framework X" as being what people write their web apps with. > >Well that would be absolutely wonderful but also looking fairly likely >which is great news. I've got to say a massive thank you for the eggs >format and easy install as well.. Python was really crying out for it and >it will be phenomenally useful. I've split all my code up as a result >because there is no need to worry about people having to install lots of >packages if it is all done automatically. > >One thought: I'd ideally like to be able to backup a WSGI deployment to >allow it to be easily redeployed on another server with a different >configuration or just restored in the event of data loss. This would >probably just involve making a zip file of all data files (including an >SQL dump) and then redistributing it with the .wsgi file. Have you had any >thoughts on how that could be achieved or is that something you wouldn't >want the .wsgi file to be used for? Whatever software installed the >dependencies of the .wsgi file would need to be aware of the data file and >what to do with it, perhaps simply by calling an install handler >somewhere? Umm, all getting a bit complicated but I just wondered if you >had had any thoughts of that nature? Well, you could define another set of entry point groups, like "wsgi.middleware.backup_handlers", which would contain entry points with the same names as in middleware, that would get called with the same configuration arguments as the application factories, but would then do some kind of backing up. Similarly, you could have an entry point group for restoration functions. These would have to defined by the same egg as the one with the factory, of course, although perhaps we could make the entry point names be the entry point targets instead of using the original entry point names. That additional level of indirection would let one egg define backup and restore services for another's factories. Perhaps the backup functions would return the name of a directory tree to back up, and the restore functions would receive some kind of zipfile or archive. Obviously that's a totally unbaked idea that would need a fair amount of thought, but there's nothing stopping anybody from fleshing it out as a tool and publishing a spec for the entry points it uses. >Oh sorry, another quick question: Is there any work underway auto-document >eggs using some of the docutils code if an appropriate specifier is made >in the egg_info file saying the egg is in restructured text or similar? >Would that be something you would be willing to include as a feature of >easy_install or is it a bit too messy? I'd love to be able to distribute a >.wsgi file and have all the documentation for the downloaded modules auto >created. If only some of the modules supported it it would still be quite >handy. I'm having a little trouble envisioning what you mean exactly. All that's really coming across is the idea that "there's some way to generate documentation from eggs". I'd certainly like to be able to see tools like epydoc or pydoc support generating documentation for an egg. However, there's a fair amount of balkanization in how you specify inputs for Python documentation tools, not unlike the previous balkanization of web servers and web apps/frameworks. Maybe somebody will come up with a similar lingua franca for documentation tools. With respect to adding more cool features to setup(), I plan to add a couple of entry point groups to setuptools that would support what you have in mind, though. There's already a distutils.commands group that allows you to register setup commands, but I also plan to add egg_info.writers and distutils.setup_args. The setup_args entry points would have the name of a parameter you'd like setup() to have, and would be a function that would get called on the Distribution object during its finalize_options() (so you can validate the argument). The egg_info.writers group will define entry points for writing metadata files as part of the egg_info command. Last, but not least, I'll add a 'setup_requires' argument to setup() that will specify eggs that need to be present for the setup script to run. With these three things in place, tools like the build_utils or py2exe and py2app won't have to monkeypatch the distutils in order to install themselves; they can instead just define entry points for setup() args and the new commands they add. And for your documentation concept, this could include document-specification arguments and an egg_info.writers entry point to put it in the EGG-INFO. Packages using the arguments would have to use 'setup_requires', however, to list the eggs needed to process those arguments. My idea for stuff like this was mainly to support frameworks; for example if an application needs plugin metadata other than entry points, it can define an egg that extends setuptools with the necessary setup arguments and metadata writers. Then, when you're building a plugin for the tool, you just setup_requires=["AppSetup"], where "AppSetup" is the egg with the setuptools extensions for "App". (Most apps will want their setuptools extensions in a separate egg, because the app itself may need the same extensions in order to be built, which would lead to a hairy chicken-and-egg problem. setuptools itself was a little tricky to bootstrap since it finds its own commands via entry points now!) >Thanks for the answers anyway, the whole thing looks tremendously exciting! That's because it is. :)

1 0

Re: [Distutils] setuptools weird install issue
by Phillip J. Eby July 25, 2005

July 25, 2005

At 10:34 PM 7/24/2005 -0500, Ian Bicking wrote: >So, I just got an updated version of setuptools (running behind due to SF >anon CVS stupidness). Now it's like setuptools isn't using itself... Make sure that you get rid of your old setuptools.egg-info directory, because I just added one to CVS, and it *must* get checked out or setuptools won't run. The issue is that setuptools now uses "entry points" to discover its own commands. That means it has to have an entry_points.txt file in the .egg-info directory in order to run. But, if you have a local, non-CVS-controlled copy of setuptools.egg-info (or you update without using -d) CVS won't check out the entry_points.txt file. Try: rm -rf setuptools.egg-info cvs upd -d This should get you the entry_points.txt file, and it should work again.

1 0

[Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
by Phillip J. Eby July 24, 2005

July 24, 2005

[cc:ed to distutils-sig because much of the below is about a new egg feature; follow-ups about the web stuff should stay on web-sig] At 04:04 AM 7/24/2005 -0500, Ian Bicking wrote: >So maybe here's a deployment spec we can start with. It looks like: > > [feature1] > someapplication.somemodule.some_function > > [feature2] > someapplication.somemodule.some_function2 > >You can't get dumber than that! There should also be a "no-feature" >section; maybe one without a section identifier, or some special section >identifier. > >It goes in the .egg-info directory. This way elsewhere you can say: > > application = SomeApplication[feature1] I like this a lot, although for a different purpose than the format Chris and I were talking about. I see this fitting into that format as maybe: [feature1 from SomeApplication] # configuration here >And it's quite unambiguous. Note that there is *no* "configuration" in >the egg-info file, because you can't put any configuration related to a >deployment in an .egg-info directory, because it's not specific to any >deployment. Obviously we still need a way to get configuration in >there, but lets say that's a different matter. Easily fixed via what I've been thinking of as the "deployment descriptor"; I would call your proposal here the "import map". Basically, an import map describes a mapping from some sort of feature name to qualified names in the code. I have an extension that I would make, though. Instead of using sections for features, I would use name/value pairs inside of sections named for the kind of import map. E.g.: [wsgi.app_factories] feature1 = somemodule:somefunction feature2 = another.module:SomeClass ... [mime.parsers] application/atom+xml = something:atom_parser ... In other words, feature maps could be a generic mechanism offered by setuptools, with a 'Distribution.load_entry_point(kind,name)' API to retrieve the desired object. That way, we don't end up reinventing this idea for dozens of frameworks or pluggable applications that just need a way to find a few simple entry points into the code. In addition to specifying the entry point, each entry in the import map could optionally list the "extras" that are required if that entry point is used. It could also issue a 'require()' for the corresponding feature if it has any additional requirements listed in the extras_require dictionary. So, I'm thinking that this would be implemented with an entry_points.txt file in .egg-info, but supplied in setup.py like this: setup( ... entry_points = { "wsgi.app_factories": dict( feature1 = "somemodule:somefunction", feature2 = "another.module:SomeClass [extra1,extra2]", ), "mime.parsers": { "application/atom+xml": "something:atom_parser [feedparser]" } }, extras_require = dict( feedparser = [...], extra1 = [...], extra2 = [...], ) ) Anyway, this would make the most common use case for eggs-as-plugins very easy: an application or framework would simply define entry points, and plugin projects would declare the ones they offer in their setup script. I think this is a fantastic idea and I'm about to leap into implementing it. :) >This puts complex middleware construction into the function that is >referenced. This function might be, in turn, an import from a >framework. Or it might be some complex setup specific to the >application. Whatever. > >The API would look like: > > wsgiapp = wsgiref.get_egg_application('SomeApplication[feature1]') > >Which ultimately resolves to: > > wsgiapp = some_function() > >get_egg_application could also take a pkg_resources.Distribution object. Yeah, I'm thinking that this could be implemented as something like: import pkg_resources def get_wsgi_app(project_name, app_name, *args, **kw): dist = pkg_resources.require(project_name)[0] return dist.load_entry_point('wsgi.app_factories', app_name)(*args,**kw) with all the heavy lifting happening in the pkg_resources.Distribution class, along with maybe a new EntryPoint class (to handle parsing entry point specifiers and to do the loading of them. >Open issues? Yep, there's a bunch. This requires the rest of the >configuration to be done quite lazily. Not sure I follow you; the deployment descriptor could contain all the configuration; see the Web-SIG post I made just previous to this one. > But I can fit this into source >control; it is about *all* I can fit into source control (I can't have >any filenames, I can't have any installation-specific pipelines, I can't >have any other apps), but it is also enough that the deployment-specific >parts can avoid many complexities of pipelining and factories and all >that -- presumably the factory functions handle that. +1. > I don't think >this is useful without the other pieces (both in front of this >configuration file and behind it) but maybe we can think about what >those other pieces could look like. I'm particularly open to >suggestions that some_function() take some arguments, but I don't know >what arguments. At this point, I think this "entry points" concept weighs in favor of having the deployment descriptor configuration values be Python expressions, meaning that a WSGI application factory would accept keyword arguments that can be whatever you like in order to configure it. However, after more thought, I think that the "next application" argument should be a keyword argument too, like 'wsgi_next' or some such. This would allow a factory to have required arguments in its signature, e.g.: def some_factory(required_arg_x, required_arg_y, optional_arg="foo", ....): ... The problem with my original idea to have the "next app" be a positional argument is that it would prevent non-middleware applications from having any required arguments. Anyway, I think we're now very close to being able to define a useful deployment descriptor format for establishing pipelines and setting options, that leaves open the possibility to do some very sophisticated things. Hm. Interesting thought... we could have a function to read a deployment descriptor (from a string, stream, or filename) and then return the WSGI application object. You could then wrap this in a simple WSGI app that does filesystem-based URL routing to serve up *.wsgi files from a directory. This would let you bootstrap a deployment capability into existing WSGI servers, without them having to add their own support for it! Web servers and frameworks that have some kind of file extension mapping mechanism could do this directly, of course. I can envision putting *.wsgi files in my web directories and then configuring Apache to run them using either mod_python or FastCGI or even as a CGI, just by tweaking local .htaccess files. However, once you have Apache tweaked the way you want, .wsgi files can be just dropped in and edited. Of course, there are still some open design issues, like caching of .wsgi files (e.g. should the file be checked for changes on each hit? I guess that could be a setting under "WSGI options", and would only work if the descriptor parser was given an actual filename to load from.)

2 6

--develop => --editable + --build-directory
by Phillip J. Eby July 23, 2005

July 23, 2005

I tried to implement a --develop option for EasyInstall, but ran into a few snags that forced me to rethink the option a bit. Here's what I've settled on, that can actually be made to work. I'm adding an --editable/-e option, which checks out or extracts a package to a project-named subdirectory of the --build-directory. Unlike the original concept of a "--develop" option, this does *not* run the "develop" command on the checked out or extracted package. It simply fetches and extracts it for you. The problem I had was that it's very difficult to integrate the target package's setup script with the parent easy_install command in order to maintain state properly between the two. In particular, there's no way to tell if a setup script uses setuptools or not, and no way to ensure that the setup script isn't going to just modify stuff on its own, as some packages' setup scripts do. Using the sandbox facility doesn't really help, since the "develop" command *needs* to be able to write to the install directory. So, in the end, the simplest thing was just to make it "--editable" and then you can go to the directory and run "setup.py develop" yourself, which will also take care of that package's dependencies. (Or of course you can run "easy_install ." to install it normally, again with its dependencies.) In order to use --editable, you must also specify a --build-directory. Checked-out or extracted packages are copied into that directory. If you specify --build-directory but not --editable, then if there is already a package with the desired name in the build directory, you'll just get a warning. But if you specify --editable and a package of that name exists, you'll get an error message and an abort. To use --editable, you must also specify only project names and versions on the command line. If you want to download from a URL or filename, you must specify it as a --find-links/-f value instead. (Note that this means you need to tag checkout URLs with a "#egg=" suffix.) Finally, if you want to force EasyInstall to check out a subversion URL instead of a binary distribution, you'll need to ensure that you specify a version that matches the checkout link. I recommend using #egg=projectname-dev to tag links to development checkouts, so that people can do:: easy_install -eb ~/projects projectname==dev to specifically check out the development version in editable form to their ~/projects/projectname directory. The --editable flag causes easy_install to only look for source distributions or checkouts, but a source distribution will take precedence over a checkout if both match the specified version. All in all, there some caveats to using --editable, but that's mainly because there aren't many #egg links out there yet, nor are conventions for using them established as yet. Anyway, this is all now in the CVS HEAD of setuptools; see EasyInstall.txt for more docs and examples.

1 0

Easy Install source download
by Vincenzo Di Massa July 23, 2005

July 23, 2005

Hi, I'm developing an easy_install based packager. It will create distribution specific packages containing eggs. (deb, rpm...) I would like to generate source packages from which binary egg-packages are built. I spent one day looking at setuptools code, but I did not find how to force easy_install to download a source distribution even if an egg is available. I think package_index has code I can call to download just the tar.gz or zip source... but how? thanks in advance Vincenzo

2 1

Re: [Distutils] Mail Delivery (failure ricardo.vieira@fmglobal.com) (FMGMSWRET-02EX)
by BounceReplies＠fmglobal.com July 22, 2005

July 22, 2005

Your message has not been delivered to its intended recipient at FM Global. The e-mail's content and/or its attached files may not comply with FM Global's Electronic Communications Policy. If this is a legitimate business e-mail, please forward this e-mail to ReturnedMail(a)fmglobal.com. You will receive an acknowledgement once it has been received. We will then review the message content and determine if it should be released to the intended recipient. You may also contact the FM Global Corporate Help Desk in Johnston, R.I., USA at +1 (1)401 275 3000 x 1299.

1 0

easy_install: application description files
by Ian Bicking July 20, 2005

July 20, 2005

Are there currently any thoughts about application descriptions? For reasons actually unrelated to easy_install, I'm going to be adding files describing applications at work. These files contain all sorts of company-specific metadata. I'd like to include descriptions of dependencies, for use by various tools (or maybe just by easy_install, should it do everything I want). Right now all the metadata goes in setup(), which is perhaps okay, but requires distutils/setuptools to implement all the features, because the file itself isn't (realistically) parseable without running it. But in this context an ini file would probably be better. Phillip, I know you like the ini files, so you must have something in mind for this. I'm not seeing anything particular in the documentation, though. Or do I put everything in setup() and use some of the files produced by that? I see .egg-info/requires.txt contains the requirement information (BTW, it's missing a trailing newline). Should I just use that directory as my application-metadata storage area? Would I add that directory to my repository? If so, is it recommended I ignore top_level.txt and requires.txt as derivative? Completely unrelated, easy_install installs executable .pyc files. Is that intended? They actually don't seem to be .pyc files either, just copies of the .py files. -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

2 1

Returned mail: see transcript for details
by xemacs-beta＠xemacs.org July 20, 2005

July 20, 2005

This message was undeliverable due to the following reason(s): Your message could not be delivered because the destination computer was unreachable within the allowed queue period. The amount of time a message is queued before it is returned depends on local configura- tion parameters. Most likely there is a network problem that prevented delivery, but it is also possible that the computer is turned off, or does not have a mail system running right now. Your message was not delivered within 8 days: Host 105.68.58.128 is not responding. The following recipients did not receive this message: <distutils-sig(a)python.org> Please reply to postmaster(a)python.org if you feel this message to be in error.

1 0

setuptools.pth being installed with wrong permissions
by Nicholas Riley July 19, 2005

July 19, 2005

Hi, I was installing setuptools on another machine so I could use the egg-based plugin support in the upcoming Trac 0.9 (projects.edgewall.com/trac). It's looking really neat - you'll be able to upload an egg via the Web and have it installed into the currently active Trac, or just drop an egg into a directory. Previously I had no version of setuptools installed on this machine, so I downloaded ez_setup.py and ran it. It didn't work the first time, because setuptools.pth got installed with the wrong permissions (my umask is 077). The egg itself had the correct permissions, however. -- Nicholas Riley <njriley(a)uiuc.edu> | <http://www.uiuc.edu/ph/www/njriley>

2 2

setuptools and Mac OS X minor updates
by Kevin Dangoor July 19, 2005

July 19, 2005

This morning, I updated to Mac OS 10.4.2 and then discovered that one of my eggs could no longer be found. Rebuilding the egg gave me the answer: Darwin had bumped from 8.1 to 8.2. My guess is that an egg built under Mac OS 10.4.2 will work just fine on a 10.4.1 Mac. It would be a drag to have a separate egg for each minor revision of the OS. Should this be done differently? I seem to remember an option for setting the platform, but I wouldn't want to do something that would break how easy_install does things. Kevin

2 2

Wanted: ideas for using distutils with preprocessors/dependencies
by Jack Jansen July 18, 2005

July 18, 2005

I'm currently looking at integrating bgen with distutils. Bgen is a little-known part of the Python core distribution: it is similar to swig, in that it generates C extension modules. In some respects it is more powerful than swig, the main one being that it reads standard C .h files in stead of adorned .i files. (Incidentally, this is also the reason it's part of core python, it is used to generate the MacOS API modules, so these are (almost) automatically updated when Apple adds new functionality. At least, that was true under MacOS9 and will be again when I get my act together:-). But bgen has a lot of disadvantages when compared to swig, the main one being that it is a rather fearsome tool to try and master. Integration with distutils is one of the things I want to do to lower that barrier. But now I'm at a loss as to how to proceed. I had a look at how swig is integrated into distutils, and I don't really like it, it smells like a hack. And, according to the comments in the source and the manual, the author agrees with me:-) Swig support is basically done in the build_ext command, by filtering out all ".i" files in the source file list very early in the process, running swig on them, and replacing them by the .c or .cpp equivalents. I can see various ways of adding bgen support, but I'm not sure which one is the best one, and/or whether there are other options. So I'd be interested in hearing what other people think, and how other packages have added a preprocessor to distutils. There's a fair amount of Python code needed to drive bgen, at least for interfaces to complex APIs (bridging C types to Python, handling callbacks, how to parse the specific .h files for this API, etc). Currently that code is in two .py files but it will be put in a class, probably modeled somewhat after Extension (but having C/C++ source files as output in stead of dynamic extension modules). What I don't know is how I'd connect this to the Extension object that will create the extension module. Ideally I'd like the bgen process to be optional. In other words, the distribution packager has three options: (a) include the bgen C output in the distribution and don't run bgen unless the end users specifically asks for it; (b) include the bgen C output but only run bgen if the normal timestamp dependencies require it; or (c) always run bgen. But it doesn't seem the Extension object currently has any support for such make-like chaining, and I'm not sure how to add it. One way would be to allow non-strings in the sources argument, and do something smart there. A similar mod could be used for libraries and extra_objects to allow chaining there too. Another way would be to add a "dependencies" argument, where those dependencies are objects that get run early, and can add their results to sources, libraries and extra_objects. I think this latter solution is probably better, as such a dependency object could modify multiple arguments of Extension in one fell swoop. As a somewhat contrived example, an "OptionalJPEGSupport" dependency could check whether the relevant libraries and include files are available to enable JPEG support in an imaging package, and then add the right source files, libraries, defines, library paths and include paths to the relevant Extension arguments. But all of this is made quite a bit more difficult (I think) by the fact the Extension doesn't really do anything, it's only a container and all the logic is in build_ext. Maybe I should follow the paradigm set by "build_clib", and add a "build_bgen" command with build_ext picking up the results? And maybe there are better solutions that I haven't thought of yet? -- Jack Jansen, <Jack.Jansen(a)cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman

4 3

Heads up: major setuptools update in CVS
by Phillip J. Eby July 17, 2005

July 17, 2005

FYI, I've just checked in 0.6a0, which is a pretty major refactoring of certain aspects of the pkg_resources implementation and API. The original purpose was to make the API conform to the design spec I posted a few weeks ago, but in the process I ended up finding lots of latent bugs that I was able to stomp. For example, much of the setuptools/EasyInstall internals were vulnerable to being confused by case-insensitive filesytems, and to being fooled by symlinks into thinking that a pair of paths were different even though they referred to the same file. There are also a lot of new API features, mainly to do with the new WorkingSet class, which embeds the logic needed to manage a "working set" of active (importable) distributions. This allows you to do things like: # List all distributions that can be imported without another # require() call # for dist in pkg_resources.working_set: print dist You can also use the 'subscribe' method of a working set to add a callback that will be invoked for every active distribution (those already present in the working set, plus any that are added later). This is so that frameworks and extensible applications can register to scan plugins for new and useful metadata. Anyway, because of the breadth of the changes, I'd like assistance with testing on different platforms, Python versions, etc. before I turn this into an official 0.6a1 release. Note that if you have any scripts that use any pkg_resources APIs for Distribution, Requirement, or AvailableDistributions objects, they may need to be updated; see the changelog in setuptools.txt (in the CVS version of setuptools) for details of the changes. P.S. I've also implemented a stricter version of Ian Bicking's #egg syntax, and included docs on how to use it in setuptools.txt under the heading, "Making your package available for EasyInstall". I have not, however, implemented a --develop option yet.

1 0

Req: easy_install.py --site-dirs
by Robert Kern July 17, 2005

July 17, 2005

I tried doing a bit of EasyInstall evangelism, and apparently, the Kool-Aid isn't quite sweet enough yet. ;-) I got feedback from someone trying the very first example with SQLObject in the documentation. He is using Debian Linux and thus does not install anything that's not from a .deb into /usr/lib. User-installed Python packages need to go into /usr/local/lib/python2.x/site-packages. To encourage this, Debian's site.py is patched to add that to the list of .pth-enabled directories. I imagine that similarly conscientious/anal distributions do likewise. My friend is a bit more conservative than that, even. He manages /usr/local using GNU Stow so he can make installations as non-root. This is very important to him. He's even willing to use tricky .pth hacks[1] to permit this. Fortunately, all of his technical concerns can be addressed by adding --site-dirs. [1] As documented here: http://article.gmane.org/gmane.comp.python.distutils.devel/1895 -- Robert Kern rkern(a)ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter

4 9

easy_install: fragment support
by Ian Bicking July 17, 2005

July 17, 2005

I've attached a patch to detect fragments. There was a couple ideas bandied about; this seems the simplest. You use a link do: url#egg=package You can include a version, otherwise it's unversioned. The only package I've tested it on is "flup", listed on http://pythonpaste.org/package_index.html -- in that case there's no competing or versioned listings in PyPI; if there was the entry would simply be ignored and would be useless. PJE aso mentioned calling the packages things like flup_devel, so that it's a completely different package from flup. The problem, then, is that requirements can't be satisfied by the development package. It seems better to have some magic version string for development, that is neither more or less than other versions (or at least it depends on context -- it's the highest version number when using --develop [should that be implemented], and the lowest otherwise). But that's only useful given a --develop option to easy_install.py. -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org Index: setuptools/package_index.py =================================================================== RCS file: /cvsroot/python/python/nondist/sandbox/setuptools/setuptools/package_index.py,v retrieving revision 1.11 diff -r1.11 package_index.py 8a9,10 > # make emacs happy: ' > EGG_FRAGMENT = re.compile(r'egg=([^=&]*)') 44,45c46,47 < < path = urlparse.urlparse(url)[2] --- > scheme, server, path, parameters, query, fragment = \ > urlparse.urlparse(url) 46a49,53 > if fragment: > match = EGG_FRAGMENT.search(fragment) > if match: > return interpret_distro_name( > url, match.group(1), metadata) 421a429,432 > # Remove any fragments from the URL > # > url = url.split('#', 1)[0] >

3 5

Building rpms for different interpreters?
by Phil Schwartz July 16, 2005

July 16, 2005

I scanned the distutils mailing list archive to no avail, so hopefully this question hasn't been asked and answered before. Basically, my ReleaseForge application uses distutils to generate zip, tar and rpm packages. Everything has worked perfectly up until now because I now want to generate a python2.4 specific rpm in addition to a python2.3 rpm. My development system is FC3 and has rpms of Python2.3 (default system wide) and Python2.4 installed: $ rpm -q python python2.4 python-2.3.4-13.1 python2.4-2.4-1pydotorg When I issue the following command: $ python setup.py bdist_rpm the generated rpm files have references to /usr/lib/python2.3/site-packages. This is ofcourse expected since python2.3 is the default. However, what is unexepected is that when I invoke python2.4 directly to generate the rpm: $ python2.4 setup.py bdist_rpm the generated rpm still has references to /usr/lib/python2.3/site-packages rather than /usr/lib/python2.4/site-packages. $ python2.4 Python 2.4 (#1, Nov 30 2004, 11:25:14) [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2 >> import sys >>> sys.path ['', '/usr/lib/python24.zip', '/usr/lib/python2.4', '/usr/lib/python2.4/plat-linux2', '/usr/lib/python2.4/lib-tk', '/usr/lib/python2.4/lib-dynload', '/usr/lib/python2.4/site-packages'] What I'd like to see is that when I invoke "python2.4 setup.py bdist_rpm" are for the files in the generated rpm to reference /usr/lib/python2.4/site-packages rather than the respective 2.3 directory. FWIW, the 2.3 and 2.4 generated rpms will be essentially the same (w/ the only difference being the location of site-packages) What painfully obvious point am I missing? Thanks for any help, Phil Schwartz http://releaseforge.sourceforge.net

2 1

Adding egg support (Resource API) to existing packages
by Paul Moore July 14, 2005

July 14, 2005

I have been looking at how I can use eggs to manage the packages I currently use in my Python installation. Mostly, everything has gone extremely smoothly, and I can build eggs either from bdist_wininst executables, or from source packages, without any problems (and without needing to modify the package sources!). With one package (Python Dateutils) I noticed that the resulting egg was "not zip safe", and I thought I'd take a look at why. It turns out to be a relatively simple issue, as the package contains a data file which isn't accessed via the resource manager protocol. Patching the source isn't hard, but it left me with a dilemma. I'd like to submit the change back to the package author, but I'm not sure how receptive he'd be to converting to setuptools (ie, adding a dependency on setuptools, changing setup.py, etc). I'd like to be able to offer a minimal change, which just switched to using the resource management protocol - people who wanted to build eggs could change setup.py, or just use the --command-packages option in Python 2.4. The problem is, how to depend on *just* the pkg_resources module. I can bundle it with the application, which is simplest, but produces a maintenance burden. I can just say that users must install setuptools, but this seems ugly, as there is no obvious reason why a date utility package should depend on a setup utility package. Having pkg_resources in the standard library is clearly the best long-term solution, but in the shorter term, shouldn't pkg_resources be unbundled from setuptools? In many ways, isn't that the *point* of setuptools/eggs, to ease unbundling of packages - and so, shouldn't setuptools "practice what it preaches", and unbundle itself...? I realise that this could cause chicken and egg problems, where setuptools needs pkg_resources to ensure that it can support not having pkg_resources bundled, but even so... Any comments/suggestions? Paul.

4 9

setuptools for people behind a firewall
by Paul Moore July 14, 2005

July 14, 2005

I've just had a look at the new documentation for setuptools. I've not read it all in detail yet, but one thing struck me regarding the "automatically download dependencies" feature. It isn't going to work for people (like me) stuck behind a firewall that Python doesn't support (Windows NTLM based firewall). Obviously, setuptools is never going to be able to resolve a situation like this, nor would I expect it to. But can I suggest two possible changes to make it easier for people with limited internet access? 1. A "manual download" mode, where setuptools lists the files which it wants you to obtain, and then leaves it to you how you get them. I'm not sure how plausible this would be, given the necessarily iterative process involved in resolving dependencies, but even a little help would be useful (a report of unresolved dependencies when run with a --no-download flag would be the most basic help). 2. A way of specifying an external command to use to download files over HTTP. This would (for example) allow me to use curl, which does support HTLM proxies, rather than relying on Python's built-in HTTP support, which doesn't. Regards, Paul.

4 16

setuptools: extras_require
by Ian Bicking July 14, 2005

July 14, 2005

I can't see any way to get extras_require to be installed, unless some other package indicates an extra feature in the package that needs to be supported. For example, if you want HTTP support in Paste, you can install WSGIUtils. But this isn't required. How can I get WSGIUtils installed? Or, more specifically, there's a bunch of examples distributed in the Paste source distribution; these examples require lots of extra software, even if Paste doesn't. How can I easily get all the examples working? -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

2 7

py2app and eggs part deux
by Kevin Dangoor July 13, 2005

July 13, 2005

I really want to use eggs, but I *have* to use py2app. Basically, the issues that I see are: 1) modulegraph needs to know about eggs so that it can track down the dependencies of things in the eggs 2) the eggs should either be put into the site-packages.zip file in the bundle in "basket" form, or should be tossed directly into the site-packages directory. For the first point, I'm curious if modulegraph already knows about zip packages. I didn't see anything in the code that immediately jumped out at me saying "hey! this package is zipped up". What I had seen was that the contents of eggs were just being ignored. If module graph does have some knowledge about zip libraries, that would probably translate well. As for #2, the ideal would probably be to put everything in basket form in site-packages.zip. I figure that by opening up the eggs and only pulling out the referenced parts, the overall app can be smaller. Any package resources need to get in there, though. For expedience, though, I'm quite find with a little bit larger app that has the .egg files stored directly in site-packages with a .pth file. Any commentary on the difficulty involved here? (Bob?) Thanks, Kevin

3 11

easy_install - some thoughts
by Paul Moore July 13, 2005

July 13, 2005

I just built a couple of eggs, for interest, experimentation, and generally "having a go" with setuptools. A couple of things struck me about the end-user experience with easy_install: 1. It would be much better if easy_install was usable via python -m, either as well as or instead of the existing easy_install.py script. On Windows, C:\Python24\Scripts is not normally on %PATH%, so easy_install isn't directly available after installing setuptools, without an additional step. 2. It would be useful if the -f option worked with a file: URL (or better still, a simple directory name), to handle local repositories. 3. I still dislike the fact that I can't get a listing of installed eggs. And uninstalling by deletion isn't good, IMHO. I am *strongly* averse to using raw file operations (delete) on files in Python\lib\site-packages. Elsewhere, fine, but I view the Python standard library as a "managed" area, and management tools should exist for it. That's the key benefit to me of bdist_wininst, and I don't see eggs providing it. 4. --dry-run doesn't work as expected: >\Apps\Python24\Scripts\easy_install.py --dry-run dist/alarm-1.0-py2.4-win32.egg Processing alarm-1.0-py2.4-win32.egg Copying alarm-1.0-py2.4-win32.egg to C:\Apps\Python24\Lib\site-packages Traceback (most recent call last): File "C:\Apps\Python24\Scripts\easy_install.py", line 18, in ? main(sys.argv[1:]) File "C:\Apps\Python24\Lib\site-packages\setuptools\command\easy_install.py", line 987, in main setup(script_args = ['-q','easy_install', '-v']+argv, **kw) File "C:\Apps\Python24\Lib\site-packages\setuptools\__init__.py", line 51, in setup return distutils.core.setup(**attrs) File "C:\Apps\Python24\lib\distutils\core.py", line 149, in setup dist.run_commands() File "C:\Apps\Python24\Lib\site-packages\setuptools\dist.py", line 419, in run _commands self.run_command(cmd) File "C:\Apps\Python24\lib\distutils\dist.py", line 966, in run_command cmd_obj.run() File "C:\Apps\Python24\Lib\site-packages\setuptools\command\easy_install.py", line 222, in run self.easy_install(spec) File "C:\Apps\Python24\Lib\site-packages\setuptools\command\easy_install.py", line 260, in easy_install return self.install_item(None, spec, tmpdir, True) File "C:\Apps\Python24\Lib\site-packages\setuptools\command\easy_install.py", line 294, in install_item dists = self.install_eggs(download, tmpdir) File "C:\Apps\Python24\Lib\site-packages\setuptools\command\easy_install.py", line 414, in install_eggs return [self.install_egg(dist_filename, tmpdir)] File "C:\Apps\Python24\Lib\site-packages\setuptools\command\easy_install.py", line 491, in install_egg return self.egg_distribution(destination) File "C:\Apps\Python24\Lib\site-packages\setuptools\command\easy_install.py", line 456, in egg_distribution metadata = EggMetadata(zipimport.zipimporter(egg_path)) zipimport.ZipImportError: not a Zip file Don't get me wrong - eggs are definitely going in the right direction. But there's still a way to go (for me, at least) yet. Some more unfocused and rambling thoughts: A nice to have feature would be the ability for easy_install to pick apart a bdist_wininst installer and build an egg from it. This would enormously help Windows users who don't have a compiler, but do want to use eggs - in my view, it's fairly "all or nothing" choice. You *don't* want to have part of your installation managed via bdist_wininst installers (more accurately, by the OS native package management), and another part by EasyInstall. On a related note, had you thought of having EasyInstall put eggs in a specific subdirectory? Something like site-packages/eggs? As you're using .pth files, this wouldn't be any harder to manage, would it? I can see a use case for a "combination" egg. One which packages together a group of packages. For example, if I want a standard set of packages installed on all machines, so that users can assume they are present when writing one-off scripts. If I could package *existing* eggs up into a combination site-std.egg (this is a pure aggregation operation, and should not require the ability to build extensions) then I could install the one egg across the site (and version that independently of the individual packages). There's a similar facility available in TCL's "starkits" (see http://www.equi4.com/starkit.html) - which are well worth a look for ideas, as they are fairly similar in concept to eggs. Paul. -- Once the game is over, the King and the pawn go back in the same box. -- Italian Proverb

4 13

A quick question on pkg_resources.find_distributions
by Paul Moore July 13, 2005

July 13, 2005

I have only one egg installed in my site-packages, namely setuptools. However, with the following script: import sys import pkg_resources def list_packages(): dists = [] for p in sys.path: for dist in pkg_resources.find_distributions(p): dists.append((p, dist)) for p, dist in dists: print p, dist.name, dist.version if __name__ == '__main__': list_packages() I get the output: C:\Apps\Python24\lib\site-packages setuptools 0.5a12 C:\Apps\Python24\Lib\site-packages\setuptools-0.5a12-py2.4.egg setuptools 0.5a12 How come I see setuptools twice? It clearly doesn't matter for purposes of importing it, but it makes a listing of installed packages look messy. A complementary question - how can I detect when two Distribution instances refer to the "same" egg. Is comparing the path attribute sufficient? Thanks, Paul.

2 2

setuptools 0.5a12
by Phillip J. Eby July 13, 2005

July 13, 2005

Hopefully, this is the last of the current spew of bugfix releases, although in this case I also threw in some revisions to pkg_resources' handling of resource extraction. You can now control the cache explicitly using a PYTHON_EGGS_CACHE variable, and on Windows the default if you don't specify it is now a "Python-Eggs" directory under the current user's "application data" directory. Also, extraction takes place to a temporary filename that is then renamed to the final name, in order to prevent race conditions with other threads or processes. So, assuming this change works well, I will start making the zip safety checks a little less stringent, so that mere use of C extensions or data files won't disqualify an egg from being installed in zipped form. However, that will be in the 0.6 series of releases, which will be focused mainly on getting both pkg_resources and the entire easy_install family of code under a solid test regime. I'm currently releasing as many new bugs as I am new features, so that's got to stop right away. My current plan is to see if I can jury-rig doctest to do tests on the output of various easy_install commands, running them in various temporary environments. The result of this process may be that EasyInstall ends up with additional options to control testing behavior. (For example, I can't test easy-install.pth manipulation by installing to a temporary directory!) As part of the move to full test automation, I'll also be refactoring pkg_resources to match the design I posted here a while back, and implementing core engine features like Ryan's "legacy.py" option, my "add_activation_listener()", etc. The PythonEggs page could really use some updating, too, to catch up even to the current API, let alone the future one. It also has some bits that should be removed since they document manual ways to do things that setuptools does automatically. (e.g. creating a 'depends.txt' file when setuptools now just uses the 'install_requires' and 'extras_require' keywords.

2 2

setuptools: MANIFEST generation
by Ian Bicking July 13, 2005

July 13, 2005

I noticed in the documentation it said that setuptools can find files based on version control when creating source distributions. This almost never applies to my packages (at least the well made ones ;) because I have html files generated from txt sources, and I like to include the html. Is there any way to tell setuptools to add certain files, but still automatically detect everything else? It seemed to do an okay job when I created a MANIFEST.in with everything but "recursive-include . *.html" commented out, but there might be something I'm missing about it. Additionally, it would be nice to be able to exclude files, like files I use for internal package administration, or large svn externals which should be separate packages. -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

2 1

ez_setup doesn't finish partial install
by Nicholas Riley July 12, 2005

July 12, 2005

I was trying to install setuptools, and I ended up running into a permissions error: % MACOSX_DEPLOYMENT_TARGET=10.4 python ez_setup.py Downloading http://www.python.org/packages/source/s/setuptools/setuptools-0.5a11-py2.3.… Processing setuptools-0.5a11-py2.3.egg Copying setuptools-0.5a11-py2.3.egg to /System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages Adding setuptools 0.5a11 to easy-install.pth file Installing easy_install.py script to /System/Library/Frameworks/Python.framework/Versions/2.3/bin error: /System/Library/Frameworks/Python.framework/Versions/2.3/bin/easy_install.py: Permission denied So I fixed permissions and tried again: % MACOSX_DEPLOYMENT_TARGET=10.4 python ez_setup.py Setuptools successfully installed or upgraded. But after the above, I still don't have an easy_install.py script. I had to delete the easy-install and setuptools .egg and .pth files from /Library/Python/2.3/site-packages to get it to install easy_install.py. -- Nicholas Riley <njriley(a)uiuc.edu> | <http://www.uiuc.edu/ph/www/njriley>

3 3

Problem with setuptools >=0.5a9 and develop command
by Michael Twomey July 12, 2005

July 12, 2005

Hi, I've been busily converting some of my smaller servers to eggs, and I've hit a problem with the develop command. From setuptools 0.5a9 onwards the develop command barfs with the following traceback: $ python2.4 setup.py develop --help Traceback (most recent call last): File "setup.py", line 33, in ? install_requires = [ File "/usr/lib/python2.4/site-packages/setuptools-0.5a10-py2.4.egg/setuptools/__init__.py", line 51, in setup File "/usr/lib/python2.4/distutils/core.py", line 135, in setup ok = dist.parse_command_line() File "/usr/lib/python2.4/site-packages/setuptools-0.5a10-py2.4.egg/setuptools/dist.py", line 107, in parse_command_line File "/usr/lib/python2.4/distutils/dist.py", line 432, in parse_command_line args = self._parse_command_opts(parser, args) File "/usr/lib/python2.4/site-packages/setuptools-0.5a10-py2.4.egg/setuptools/dist.py", line 385, in _parse_command_opts File "/usr/lib/python2.4/distutils/dist.py", line 530, in _parse_command_opts parser.set_negative_aliases(negative_opt) File "/usr/lib/python2.4/distutils/fancy_getopt.py", line 143, in set_negative_aliases self._check_alias_dict(negative_alias, "negative alias") File "/usr/lib/python2.4/distutils/fancy_getopt.py", line 125, in _check_alias_dict raise DistutilsGetoptError, \ distutils.errors.DistutilsGetoptError: invalid negative alias 'always-unzip': option 'always-unzip' not defined Looking at commands/develop.py I notice that it's the only command which sets 'command_consumes_arguments = False', and it also doesn't override the negative_opt dict. I tried setting 'negative_opt = {}' and it started working again. So this patch "fixes" the issue though I don't know the internals of distutils/setuptools to tell you exactly why: --- /tmp/setuptools-0.5a10/setuptools/command/develop.py 2005-07-06 19:15:52.000000000 +0100 +++ setuptools/command/develop.py 2005-07-12 11:50:04.000000000 +0100 @@ -21,6 +21,8 @@ boolean_options = [ 'multi-version', 'exclude-scripts', 'always-copy', 'uninstall' ] + + negative_opt = {} command_consumes_arguments = False # override base Is anyone else seeing this problem? I've reproduced it on a debian linux box and my apple laptop. cheers, Michael

2 2

setuptools 0.5a10 released with your bugs/features
by Phillip J. Eby July 12, 2005

July 12, 2005

* auto-import when requested resource's module isn't imported (Kevin Dangoor) * fix detection of "baskets" in sys.path directories (Paul Moore) * allow directories as arguments to --find-links (Paul Moore) * fix python -m easy_install not working (Paul Moore) Enjoy. (P.S. this also fixes some breakage w/buildutils 0.1.1 trying to import stuff from the 'easy_install' module. But note that buildutils doesn't seem to be smart enough about installing dependencies yet anyway, because trying to install pudge at runtime complains about it not being found in PyPI. I'm guessing that if Ryan's not going to put these modules up on PyPI, he needs to add some --find-links to the easy_install subcommand that buildutils runs to install a command's dependencies.) (P.P.S. That reminds me... Ryan, don't bother putting http://peak.telecommunity.com/dist/ in your --find-links list; now that I've got automatic PyPI uploading figured out, I plan to put future "stable" releases on PyPI, and snapshot releases (of projects other than setuptools itself) will go to http://peak.telecommunity.com/snapshots/ in the future.)

2 1

pkg_resources: getting to resources in other packages
by Kevin Dangoor July 11, 2005

July 11, 2005

I've found a scenario that isn't working, and I figured I'd toss this out there to see if a change is needed in setuptools or in my scenario... Picture something like this: toppackage __init__.py somemod.py lowerpackage __init__.py someresource.txt In somemod.py: import pkg_resources s = pkg_resource.resource_string("toppackage.lowerpackage", "someresource.txt") This will fail with a KeyError in get_provider(moduleName). If there is an import toppackage.lowerpackage before the resource_string call, then it works fine. In this particular instance, I'm going to go ahead and implement logic to ensure that the package has been imported. But, I was figuring that it might be nice if pkg_resource handled this... What say you all? Kevin

2 1

Setuptools and non-system installs
by Michael Hoffman July 11, 2005

July 11, 2005

I was trying to install setuptools in my home directory on a system I don't have root access to using the command "python setup.py install --home=~". Of course, since it installs as an egg, I can't use it unless I *specifically* add it to my PYTHONPATH. This would be a pain should it be necessary to add for every egg I install. I've added a workaround to my ~/lib/python/sitecustomize.py (~/lib/python is in PYTHONPATH) which uses the existing machinery in site.py to process .pth files in all of the entries of sys.path, not just the default sitedirs. I've noticed, however, that Guido has rejected a patch <http://www.python.org/sf/1174614> to have the default site.py search .pth files in other directories, mainly because all the extra path searching at each startup might slow down startup on some systems. The best solution I can think of is to think of is to have some directory that is explicitly searched for .pth files (or even .egg/.zip files) and has them added to the path. This would still require some changes to sitecustomize.py (and hopefully eventually site.py), so it keeps eggs from working OOTB on this type of system. What do you think is the best way to have eggs loaded from non-system installs? -- Michael Hoffman <hoffman(a)ebi.ac.uk> European Bioinformatics Institute

2 1

bdist_nsi
by Jean-Christophe G. July 10, 2005

July 10, 2005

Just a short announce concerning the bdist_nsi mod: http://bdist-nsi.sourceforge.net/ It's an alternative way to build win32 package installer using Nsis. This adds : Silent install, Modern UI, internationnalization, uninstall Have a look at the screenshots! It's still in early stage development but works quite well on all package I tested without any modification as long as your setup.py is not esotheric

1 0

ANN: buildutils-0.1.0 - Distutils extensions for developing Python libraries and applications.
by Ryan Tomayko July 8, 2005

July 8, 2005

I'm happy to announce that an initial 0.1.0 version of Python Buildutils is available. The `buildutils` package contains extensions to Python's standard distribution utilities (`distutils`) that are often useful during the development of Python projects. Buildutils was created to scratch an itch: removing ``make`` from the Python development process and partially to gain a better understanding of how `distutils` works. The following extension commands are included: announce send a release announcement to mailing lists like python-announce-list(a)python.org checksum generate MD5 and SHA-1 checksum files for distributables. etags generate an TAGS file over all packages and module (for use in Emacs). flakes find lint using the pyflakes utility. info dumps information about the project. publish push distributables and documentation up to a project site using ssh/scp/sftp. pudge build Python documentation from restructured text documents and Python doc strings. pytest run py.test unit tests. stats dump statistics on the number of lines, files, modules, packages, etc. use bring in a working version of a dependency (uses setuptools egg stuff). For more information, including User's Guide, Command Reference, and installation options, visit the Buildutils project site: <http://buildutils.lesscode.org/> There's also a bit more information on the release at the lesscode.org blog: <http://lesscode.org/> Ryan Tomayko <rtomayko(a)lesscode.org>

3 10

Out of Office AutoReply: [CNET NETWORKS AV: VIRUS REMOVED] Delivery reports about your e-mail
by Jon Drukman July 8, 2005

July 8, 2005

I am out of the office until Monday July 11. Please contact Noel Morrison or Scott Bedard if your matter is urgent.

1 0

Something that doesn't run from an egg
by Kevin Dangoor July 7, 2005

July 7, 2005

This is not wholly surprising, but py.test doesn't appear to run from an egg. (I say it's not surprising, given that py is doing its own import trickery...) The way it fails is interesting. I can import py from a python prompt, but not from the py.test script. /usr/local/bin/python Python 2.4.1 (#2, Mar 31 2005, 00:05:10) [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import py.test >>> py.test py lib is at /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/py-0.8.0_alpha1-py2.4.egg/py/__init__.pyc Traceback (most recent call last): File "/usr/local/bin/py.test", line 4, in ? pkg_resources.run_main('py==0.8.0-alpha1', 'py.test') File "build\bdist.win32\egg\pkg_resources.py", line 110, in run_main File "build\bdist.win32\egg\pkg_resources.py", line 599, in run_script File "/Library/Frameworks/Python.framework/Versions/2.4/bin/py.test", line 3, in ? import pkg_resources ImportError: cannot import name py Another of the pylib scripts, py.countloc, works fine. That one may not depend on the rest of the package, though. I'd thought I'd toss this out there in case anyone cares, but I'm not sure if anyone will care about this particular case. Kevin

4 11

py2app and eggs
by Kevin Dangoor July 7, 2005

July 7, 2005

I've done some quick experimentation with py2app and packages installed as eggs (stopping short of monkeying with py2app itself). As it currently stands, py2app's dependency finder misses the dependencies of modules within eggs. I was easily able to get my setup script to copy the eggs I needed into the app's site-packages and generate a .pth file to use the eggs, but found that the app wouldn't run because of standard library modules that hadn't been included. In some respects, eggsploding the egg's packages into a directory and letting py2app pick the modules up from there is beneficial, because that would probably include just a subset of the egg in the final app. A smaller final app is a good thing. py2app also includes just the .pyc files this way, whereas the eggs I have include the source. I haven't tried py2exe yet, but I will probably end up with a similar approach. Kevin

1 0

eggs and py2app/py2exe
by Kevin Dangoor July 6, 2005

July 6, 2005

Let's say you've got some libraries installed as eggs. If you run py2app or py2exe on the application, it seems like you'd want the egg (or at least it's contents) to come along with the application. Or, at least, I'd prefer it for the eggs to come along with the app, rather than being downloaded via require. (For curiousity's sake, I just tried it and found that the contents of the eggs were left behind when I ran py2app. Not at all surprising, given that I don't think py2app or py2exe know about eggs yet...) Anyone have opinions on how a standalone app should look with eggs? Kevin

2 2

setuptools 0.5a5: "develop" and "test" commands
by Phillip J. Eby July 6, 2005

July 6, 2005

I've just released setuptools 0.5a5, which updates the old "test" command to do in-place build-and-test instead of the previous install-and-test behavior. It also automatically require()s the package's dependencies, if any, and updates the .egg-info metadata. I also added a new "develop" command, which is like an install that doesn't actually install the package. It runs "build_ext --inplace" (so that any C extensions get built in the source directory) and then installs a .egg-link pointing to the source tree, optionally updating easy-install.pth and installing any dependencies for the package. (It only takes a subset of the easy_install command options, however, so it's currently better if you install your dependencies before using the "develop" command, especially if you want to "develop" any of those as well.) The "develop" command basically adds your source checkout to the eggs that will be searched at runtime to resolve a 'require()' request, and if you're installing to site-packages, then it will be added to sys.path as well (unless you use --multi-version or -m). This lets you edit your source checkout and do development without having to actually install the package. Wrapper scripts are installed to the --script-dir; the wrappers do a require() and then execfile() your in-development script source files, so you don't have to re-run "develop" just because you've changed script contents. The only time you need to re-run the "develop" command is when your metadata (e.g. version or requirements) changes, or if you add a new script, or need to rebuild changed C extensions. When you're done with development, "setup.py develop --uninstall" will remove the .egg-link and any easy-install.pth entry for your development tree, but you will need to update or replace any installed scripts by hand. All of these commands respect the standard distutils .cfg files, so you can e.g. use setup.cfg to set options for developing and testing. There were a couple of other important changes in this release; easy_install the *module* is now found under setuptools.command; easy_install the *script* no longer contains the actual command code. Also, a new "egg_info" command was split out of "bdist_egg", so that it could be used by "test" and "develop" as well as "bdist_egg". As a result of this, "bdist_egg" no longer supports the various "tag" options that it did before; you must now do something like this:: setup.py egg_info --tag-svn-revision bdist_egg if you want to set tagging options. You can also use this with the "develop" and "test" commands, e.g.:: setup.py egg_info --tag-svn-revision develop in order to change the version info that will be used for the "virtual egg". At this point, I think setuptools now has sufficient features to support development of packages with "cheap" external dependencies. That is, a package using setuptools to create its setup script can add external dependencies with almost no effort on the part of the package developer, and no effort at all for the package installer. And, with further effort by persons familiar with the appropriate packaging systems, the dependency information included in setup.py could be used to generate native packages with dependency information (assuming there is some uniform mapping from PyPI project names to platform package names). In other words, I think we've now achieved an interim state of packaging nirvana. There are many additional stages of enlightenment possible, but I think we are now in a position to begin the journey. Of course, having some docs would help quite a lot, as right now there's no formal documentation for stuff like version/requirement syntax, how to specify requirements in the setup() call, etc. I'd like to start upgrading my existing packages to use the new features before I try and write any, but if any brave souls happen to get to it first, well, the more the merrier where docs are concerned. The one area where docs might be a problem for a while is the actual pkg_resources infrastructure, since I intend to do a major refactoring of it in the 0.6 releases. In addition to restructuring based on the cleaner concepts of distributions, environments, working sets, etc., it will also be test-driven, such that nearly all the pkg_resources code will have unit tests for it, and as much of the setuptools command code as possible will too. I'll also be trying to clean up the older attempts at a dependency management system that lurk deeper in setuptools; these never quite worked out before. Finally, there are a few older features that are now of questionable utility given the ease of adding external dependencies. But, I'm not sure if I can remove any/all of these things yet, because I don't know if anybody (besides Fred Drake and myself) has based anything interesting on those features. So, those changes might have to wait for the 0.7 release series, depending.

2 3

RETURNED MAIL: DATA FORMAT ERROR
by dparson＠csc.com July 5, 2005

July 5, 2005

The original message was received at Tue, 5 Jul 2005 18:00:36 -0400 from csc.com [208.137.142.50] ----- The following addresses had permanent fatal errors ----- distutils-sig(a)python.org ----- Transcript of session follows ----- ... while talking to python.org.: >>> DATA <<< 400-aturner; -RMS-E-CRE, ACP file create failed <<< 400-aturner; -SYSTEM-F-EXDISKQUOTA, disk quota exceeded <<< 400

1 0

eggs: Python version independence
by Ian Bicking July 4, 2005

July 4, 2005

Most (90%+) of the libraries I work with don't depend on a Python version. But unfortunately when using easy_install/eggs they always have an explicit version, and multiple installations are necessary otherwise. Generally speaking, all pure-Python packages are version independent. At least on the source level. I suppose .pyc files aren't always (though they haven't changed for a long time, have they?) -- would it be reasonable to have version-independent packages? -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

5 5

easy_install: setup.py-less packages
by Ian Bicking July 4, 2005

July 4, 2005

Hi; trying to get back into this again... So, there's a package at http://svn.saddi.com/flup/trunk/ that I want to install, but it has no setup.py file. I think that file should look something like: from setuptools import setup setup(name="flup", packages=['flup', 'flup.middleware', 'flup.resolver', 'flup.server'] version='0.0-devel-r???') And I'd probably add other things, but I think that's all that's required. So... how should I make this happen? Should I hardcode 'http://svn.saddi.com/flup/trunk' into the setup.py file? Can I make the location overrideable with an option? How do I download the checkout? What was in a function before is now in setuptools.package_index.PackageIndex, and involves all sorts of data that doesn't seem to imply (because it's not a package without the setup.py file, and there's no index to get it from). Should I calculate r??? on my own, and if so where? Right before I call setup()? I need the repository location to do so, so if the repository location was overrideable then I'd need to get the real location. Once the files are checkout out, how do I put it in place? Is there a way to set the "base path" for the packages, so I could say "install these packages, found in /tmp/wherever-it-was-downloaded-to"? -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

3 5

buildutils
by Ryan Tomayko July 3, 2005

July 3, 2005

I'm getting ready to announce/release a new project I've been working on over the past week or so called "buildutils". It's a set of extensions to distutils that are oriented toward the build process. For example, there are commands for generating MD5/SHA-1 checksum files for distributables, running py.test based unit tests, pushing doc/distributables to another site via ssh/scp, etc. My goal is to remove make from my python development process. The package consists almost entirely of additional distutils commands; there's very little additional apparatus. It uses setuptools, taking advantage of its dependency resolution features to install command dependencies when commands are executed, which lets us put all commands in the box but not require dependencies until commands are actually used. Everything is still a little rough but there's more information here: http://buildutils.lesscode.org/ I was wondering if there would be any interest in directing mailing list discussion to the distutils-sig mailing list. Build commands may not be part of the SIG's charter but I figure most of the discussion will be relevant. The expertise of the members of this list would be a huge help. Perhaps I can try to point command development related discussion here while pushing support and usage related discussion to a different list? I apologize if I'm being dense. I'm not familiar with SIG process but would like to involve as much of the community as possible. Thanks, Ryan Tomayko rtomayko(a)gmail.com http://naeblis.cx/rtomayko/

1 0

easy_install: package index
by Ian Bicking July 3, 2005

July 3, 2005

I notice there is an option for the index URL. But it doesn't look like a multi-option, i.e., you can't give several URLs, to be checked in turn. That would be useful. It'd also be nice if you could configure this; for instance, there's several libraries I'm using that haven't had proper releases yet, so they aren't in PyPI. If I could tell someone to just add a particular line to ~/.setup.cfg or somesuch, then that would make it easier. Maybe that's already possible? But that'd be no good if that disabled the use of PyPI. -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

2 1