Mailman 3 July 2005 - Distutils-SIG

Not repeating requirements for eggs
by Kevin Dangoor 28 Jul '05

28 Jul '05

setuptools lets you declare install_requires with a set of packages. This is handy and is nicer than the "old way". In one of my __init__.py files, I am redeclaring my requirements using pkg_resources.require, to ensure that they're all still available and to load them properly. Does it make sense to run pkg_resources.require at runtime? (I think so...) If so, it seems like a function could be added to pkg_resources that goes through this egg's required dependencies and does a require() on them. Something like pkg_resources.requireAll("MyProjectName"). What do you think? Kevin

3 3

Returned mail: see transcript for details
by gallwey＠sun.com 26 Jul '05

26 Jul '05

The original message was received at Tue, 26 Jul 2005 08:11:25 +0200 from sun.com [69.151.169.189] ----- The following addresses had permanent fatal errors ----- <distutils-sig(a)python.org>

1 0

Re: [Distutils] [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
by Phillip J. Eby 25 Jul '05

25 Jul '05

[cc:'d to distutils-sig because this is mostly about cool uses for the new EntryPoint facility of setuptools/pkg_resources] At 12:01 AM 7/26/2005 +0100, James Gardner wrote: >Hi Phillip, > >>There's one other advantage: this format will hopefully become as >>successful as WSGI itself in adoption by servers and applications. >>Hopefully within a year or so, *the* normal way to deploy a Python web >>app will be using a .wsgi file. >> >>Beyond that, we can hopefully begin to see "Python" rather than >>"framework X" as being what people write their web apps with. > >Well that would be absolutely wonderful but also looking fairly likely >which is great news. I've got to say a massive thank you for the eggs >format and easy install as well.. Python was really crying out for it and >it will be phenomenally useful. I've split all my code up as a result >because there is no need to worry about people having to install lots of >packages if it is all done automatically. > >One thought: I'd ideally like to be able to backup a WSGI deployment to >allow it to be easily redeployed on another server with a different >configuration or just restored in the event of data loss. This would >probably just involve making a zip file of all data files (including an >SQL dump) and then redistributing it with the .wsgi file. Have you had any >thoughts on how that could be achieved or is that something you wouldn't >want the .wsgi file to be used for? Whatever software installed the >dependencies of the .wsgi file would need to be aware of the data file and >what to do with it, perhaps simply by calling an install handler >somewhere? Umm, all getting a bit complicated but I just wondered if you >had had any thoughts of that nature? Well, you could define another set of entry point groups, like "wsgi.middleware.backup_handlers", which would contain entry points with the same names as in middleware, that would get called with the same configuration arguments as the application factories, but would then do some kind of backing up. Similarly, you could have an entry point group for restoration functions. These would have to defined by the same egg as the one with the factory, of course, although perhaps we could make the entry point names be the entry point targets instead of using the original entry point names. That additional level of indirection would let one egg define backup and restore services for another's factories. Perhaps the backup functions would return the name of a directory tree to back up, and the restore functions would receive some kind of zipfile or archive. Obviously that's a totally unbaked idea that would need a fair amount of thought, but there's nothing stopping anybody from fleshing it out as a tool and publishing a spec for the entry points it uses. >Oh sorry, another quick question: Is there any work underway auto-document >eggs using some of the docutils code if an appropriate specifier is made >in the egg_info file saying the egg is in restructured text or similar? >Would that be something you would be willing to include as a feature of >easy_install or is it a bit too messy? I'd love to be able to distribute a >.wsgi file and have all the documentation for the downloaded modules auto >created. If only some of the modules supported it it would still be quite >handy. I'm having a little trouble envisioning what you mean exactly. All that's really coming across is the idea that "there's some way to generate documentation from eggs". I'd certainly like to be able to see tools like epydoc or pydoc support generating documentation for an egg. However, there's a fair amount of balkanization in how you specify inputs for Python documentation tools, not unlike the previous balkanization of web servers and web apps/frameworks. Maybe somebody will come up with a similar lingua franca for documentation tools. With respect to adding more cool features to setup(), I plan to add a couple of entry point groups to setuptools that would support what you have in mind, though. There's already a distutils.commands group that allows you to register setup commands, but I also plan to add egg_info.writers and distutils.setup_args. The setup_args entry points would have the name of a parameter you'd like setup() to have, and would be a function that would get called on the Distribution object during its finalize_options() (so you can validate the argument). The egg_info.writers group will define entry points for writing metadata files as part of the egg_info command. Last, but not least, I'll add a 'setup_requires' argument to setup() that will specify eggs that need to be present for the setup script to run. With these three things in place, tools like the build_utils or py2exe and py2app won't have to monkeypatch the distutils in order to install themselves; they can instead just define entry points for setup() args and the new commands they add. And for your documentation concept, this could include document-specification arguments and an egg_info.writers entry point to put it in the EGG-INFO. Packages using the arguments would have to use 'setup_requires', however, to list the eggs needed to process those arguments. My idea for stuff like this was mainly to support frameworks; for example if an application needs plugin metadata other than entry points, it can define an egg that extends setuptools with the necessary setup arguments and metadata writers. Then, when you're building a plugin for the tool, you just setup_requires=["AppSetup"], where "AppSetup" is the egg with the setuptools extensions for "App". (Most apps will want their setuptools extensions in a separate egg, because the app itself may need the same extensions in order to be built, which would lead to a hairy chicken-and-egg problem. setuptools itself was a little tricky to bootstrap since it finds its own commands via entry points now!) >Thanks for the answers anyway, the whole thing looks tremendously exciting! That's because it is. :)

1 0

Re: [Distutils] setuptools weird install issue
by Phillip J. Eby 25 Jul '05

25 Jul '05

At 10:34 PM 7/24/2005 -0500, Ian Bicking wrote: >So, I just got an updated version of setuptools (running behind due to SF >anon CVS stupidness). Now it's like setuptools isn't using itself... Make sure that you get rid of your old setuptools.egg-info directory, because I just added one to CVS, and it *must* get checked out or setuptools won't run. The issue is that setuptools now uses "entry points" to discover its own commands. That means it has to have an entry_points.txt file in the .egg-info directory in order to run. But, if you have a local, non-CVS-controlled copy of setuptools.egg-info (or you update without using -d) CVS won't check out the entry_points.txt file. Try: rm -rf setuptools.egg-info cvs upd -d This should get you the entry_points.txt file, and it should work again.

1 0

[Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
by Phillip J. Eby 24 Jul '05

24 Jul '05

[cc:ed to distutils-sig because much of the below is about a new egg feature; follow-ups about the web stuff should stay on web-sig] At 04:04 AM 7/24/2005 -0500, Ian Bicking wrote: >So maybe here's a deployment spec we can start with. It looks like: > > [feature1] > someapplication.somemodule.some_function > > [feature2] > someapplication.somemodule.some_function2 > >You can't get dumber than that! There should also be a "no-feature" >section; maybe one without a section identifier, or some special section >identifier. > >It goes in the .egg-info directory. This way elsewhere you can say: > > application = SomeApplication[feature1] I like this a lot, although for a different purpose than the format Chris and I were talking about. I see this fitting into that format as maybe: [feature1 from SomeApplication] # configuration here >And it's quite unambiguous. Note that there is *no* "configuration" in >the egg-info file, because you can't put any configuration related to a >deployment in an .egg-info directory, because it's not specific to any >deployment. Obviously we still need a way to get configuration in >there, but lets say that's a different matter. Easily fixed via what I've been thinking of as the "deployment descriptor"; I would call your proposal here the "import map". Basically, an import map describes a mapping from some sort of feature name to qualified names in the code. I have an extension that I would make, though. Instead of using sections for features, I would use name/value pairs inside of sections named for the kind of import map. E.g.: [wsgi.app_factories] feature1 = somemodule:somefunction feature2 = another.module:SomeClass ... [mime.parsers] application/atom+xml = something:atom_parser ... In other words, feature maps could be a generic mechanism offered by setuptools, with a 'Distribution.load_entry_point(kind,name)' API to retrieve the desired object. That way, we don't end up reinventing this idea for dozens of frameworks or pluggable applications that just need a way to find a few simple entry points into the code. In addition to specifying the entry point, each entry in the import map could optionally list the "extras" that are required if that entry point is used. It could also issue a 'require()' for the corresponding feature if it has any additional requirements listed in the extras_require dictionary. So, I'm thinking that this would be implemented with an entry_points.txt file in .egg-info, but supplied in setup.py like this: setup( ... entry_points = { "wsgi.app_factories": dict( feature1 = "somemodule:somefunction", feature2 = "another.module:SomeClass [extra1,extra2]", ), "mime.parsers": { "application/atom+xml": "something:atom_parser [feedparser]" } }, extras_require = dict( feedparser = [...], extra1 = [...], extra2 = [...], ) ) Anyway, this would make the most common use case for eggs-as-plugins very easy: an application or framework would simply define entry points, and plugin projects would declare the ones they offer in their setup script. I think this is a fantastic idea and I'm about to leap into implementing it. :) >This puts complex middleware construction into the function that is >referenced. This function might be, in turn, an import from a >framework. Or it might be some complex setup specific to the >application. Whatever. > >The API would look like: > > wsgiapp = wsgiref.get_egg_application('SomeApplication[feature1]') > >Which ultimately resolves to: > > wsgiapp = some_function() > >get_egg_application could also take a pkg_resources.Distribution object. Yeah, I'm thinking that this could be implemented as something like: import pkg_resources def get_wsgi_app(project_name, app_name, *args, **kw): dist = pkg_resources.require(project_name)[0] return dist.load_entry_point('wsgi.app_factories', app_name)(*args,**kw) with all the heavy lifting happening in the pkg_resources.Distribution class, along with maybe a new EntryPoint class (to handle parsing entry point specifiers and to do the loading of them. >Open issues? Yep, there's a bunch. This requires the rest of the >configuration to be done quite lazily. Not sure I follow you; the deployment descriptor could contain all the configuration; see the Web-SIG post I made just previous to this one. > But I can fit this into source >control; it is about *all* I can fit into source control (I can't have >any filenames, I can't have any installation-specific pipelines, I can't >have any other apps), but it is also enough that the deployment-specific >parts can avoid many complexities of pipelining and factories and all >that -- presumably the factory functions handle that. +1. > I don't think >this is useful without the other pieces (both in front of this >configuration file and behind it) but maybe we can think about what >those other pieces could look like. I'm particularly open to >suggestions that some_function() take some arguments, but I don't know >what arguments. At this point, I think this "entry points" concept weighs in favor of having the deployment descriptor configuration values be Python expressions, meaning that a WSGI application factory would accept keyword arguments that can be whatever you like in order to configure it. However, after more thought, I think that the "next application" argument should be a keyword argument too, like 'wsgi_next' or some such. This would allow a factory to have required arguments in its signature, e.g.: def some_factory(required_arg_x, required_arg_y, optional_arg="foo", ....): ... The problem with my original idea to have the "next app" be a positional argument is that it would prevent non-middleware applications from having any required arguments. Anyway, I think we're now very close to being able to define a useful deployment descriptor format for establishing pipelines and setting options, that leaves open the possibility to do some very sophisticated things. Hm. Interesting thought... we could have a function to read a deployment descriptor (from a string, stream, or filename) and then return the WSGI application object. You could then wrap this in a simple WSGI app that does filesystem-based URL routing to serve up *.wsgi files from a directory. This would let you bootstrap a deployment capability into existing WSGI servers, without them having to add their own support for it! Web servers and frameworks that have some kind of file extension mapping mechanism could do this directly, of course. I can envision putting *.wsgi files in my web directories and then configuring Apache to run them using either mod_python or FastCGI or even as a CGI, just by tweaking local .htaccess files. However, once you have Apache tweaked the way you want, .wsgi files can be just dropped in and edited. Of course, there are still some open design issues, like caching of .wsgi files (e.g. should the file be checked for changes on each hit? I guess that could be a setting under "WSGI options", and would only work if the descriptor parser was given an actual filename to load from.)

2 6

--develop => --editable + --build-directory
by Phillip J. Eby 23 Jul '05

23 Jul '05

I tried to implement a --develop option for EasyInstall, but ran into a few snags that forced me to rethink the option a bit. Here's what I've settled on, that can actually be made to work. I'm adding an --editable/-e option, which checks out or extracts a package to a project-named subdirectory of the --build-directory. Unlike the original concept of a "--develop" option, this does *not* run the "develop" command on the checked out or extracted package. It simply fetches and extracts it for you. The problem I had was that it's very difficult to integrate the target package's setup script with the parent easy_install command in order to maintain state properly between the two. In particular, there's no way to tell if a setup script uses setuptools or not, and no way to ensure that the setup script isn't going to just modify stuff on its own, as some packages' setup scripts do. Using the sandbox facility doesn't really help, since the "develop" command *needs* to be able to write to the install directory. So, in the end, the simplest thing was just to make it "--editable" and then you can go to the directory and run "setup.py develop" yourself, which will also take care of that package's dependencies. (Or of course you can run "easy_install ." to install it normally, again with its dependencies.) In order to use --editable, you must also specify a --build-directory. Checked-out or extracted packages are copied into that directory. If you specify --build-directory but not --editable, then if there is already a package with the desired name in the build directory, you'll just get a warning. But if you specify --editable and a package of that name exists, you'll get an error message and an abort. To use --editable, you must also specify only project names and versions on the command line. If you want to download from a URL or filename, you must specify it as a --find-links/-f value instead. (Note that this means you need to tag checkout URLs with a "#egg=" suffix.) Finally, if you want to force EasyInstall to check out a subversion URL instead of a binary distribution, you'll need to ensure that you specify a version that matches the checkout link. I recommend using #egg=projectname-dev to tag links to development checkouts, so that people can do:: easy_install -eb ~/projects projectname==dev to specifically check out the development version in editable form to their ~/projects/projectname directory. The --editable flag causes easy_install to only look for source distributions or checkouts, but a source distribution will take precedence over a checkout if both match the specified version. All in all, there some caveats to using --editable, but that's mainly because there aren't many #egg links out there yet, nor are conventions for using them established as yet. Anyway, this is all now in the CVS HEAD of setuptools; see EasyInstall.txt for more docs and examples.

1 0

Easy Install source download
by Vincenzo Di Massa 23 Jul '05

23 Jul '05

Hi, I'm developing an easy_install based packager. It will create distribution specific packages containing eggs. (deb, rpm...) I would like to generate source packages from which binary egg-packages are built. I spent one day looking at setuptools code, but I did not find how to force easy_install to download a source distribution even if an egg is available. I think package_index has code I can call to download just the tar.gz or zip source... but how? thanks in advance Vincenzo

2 1

Re: [Distutils] Mail Delivery (failure ricardo.vieira@fmglobal.com) (FMGMSWRET-02EX)
by BounceReplies＠fmglobal.com 22 Jul '05

22 Jul '05

Your message has not been delivered to its intended recipient at FM Global. The e-mail's content and/or its attached files may not comply with FM Global's Electronic Communications Policy. If this is a legitimate business e-mail, please forward this e-mail to ReturnedMail(a)fmglobal.com. You will receive an acknowledgement once it has been received. We will then review the message content and determine if it should be released to the intended recipient. You may also contact the FM Global Corporate Help Desk in Johnston, R.I., USA at +1 (1)401 275 3000 x 1299.

1 0

easy_install: application description files
by Ian Bicking 20 Jul '05

20 Jul '05

Are there currently any thoughts about application descriptions? For reasons actually unrelated to easy_install, I'm going to be adding files describing applications at work. These files contain all sorts of company-specific metadata. I'd like to include descriptions of dependencies, for use by various tools (or maybe just by easy_install, should it do everything I want). Right now all the metadata goes in setup(), which is perhaps okay, but requires distutils/setuptools to implement all the features, because the file itself isn't (realistically) parseable without running it. But in this context an ini file would probably be better. Phillip, I know you like the ini files, so you must have something in mind for this. I'm not seeing anything particular in the documentation, though. Or do I put everything in setup() and use some of the files produced by that? I see .egg-info/requires.txt contains the requirement information (BTW, it's missing a trailing newline). Should I just use that directory as my application-metadata storage area? Would I add that directory to my repository? If so, is it recommended I ignore top_level.txt and requires.txt as derivative? Completely unrelated, easy_install installs executable .pyc files. Is that intended? They actually don't seem to be .pyc files either, just copies of the .py files. -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

2 1

Returned mail: see transcript for details
by xemacs-beta＠xemacs.org 20 Jul '05

20 Jul '05

This message was undeliverable due to the following reason(s): Your message could not be delivered because the destination computer was unreachable within the allowed queue period. The amount of time a message is queued before it is returned depends on local configura- tion parameters. Most likely there is a network problem that prevented delivery, but it is also possible that the computer is turned off, or does not have a mail system running right now. Your message was not delivered within 8 days: Host 105.68.58.128 is not responding. The following recipients did not receive this message: <distutils-sig(a)python.org> Please reply to postmaster(a)python.org if you feel this message to be in error.

1 0