Mailman 3 July 2007 - Distutils-SIG

Re: [Distutils] Why so many zc.buildout versions?
by Jim Fulton July 11, 2007

July 11, 2007

You raise a really good point, which is especially relevant in light of pypi performance issues and discussions. I'm copying the distutils and catalog sigs to get some wider discussion. I apologize for the cross posting. I'm beginning to wonder about the strategy that setuptools uses, or maybe about the way we are using the index. It's important to note that there is nothing specific about the buildout package here. It is very important to make multiple versions available to support requirements for specific package versions. It make builds/installs repeatable, whether talking about buildout or other systems built on setuptools. When someone has tested and wants to release an application built from a collection of distributions, they will want to specify those *specific* versions for future builds or installs. This means that we need to retain any versions published indefinitely in a way that can be found by setuptools. Currently, the only way to support multiple versions with the cheeseshop is to unhide past releases. This has a fairly severe effect on performance. As the example below shows, setuptools will fetch the package page and then fetch the pages for each release. That's a lot of requests. What makes it worse is that the individual package pages can be fairly long. I've gotten in the habit of including full documentation on every release page. For example, recent release pages for zc.buildout are around 200K. This is a fairly significant amount of data to transfer. This will certainly make the scanning process take a long time for clients. (Obviously, if we keep doing things the way we are, I'll need to stop doing that.) All of this aggravates any performance problems we might have. Up to now, setuptools has tried hard to use existing systems without change. This means that it reuses systems designed primarily for people, not software. I think that setuptools rightly took the approach it has up to now so that progress could be made without making people change other systems. This was appropriate when setuptools was evolving and people were figuring out ways to use it. I think it is time to take a step back and think a lot harder about how we'd want to structure an index to support setuptools. IMO, a setuptools-aware index would have a single page for each package: - The single page would be published in a case-insensitive way. It would be nice to find a way to avoid this, or maybe we should use a windows-based web server. :) It would also be served very cheaply, for example statically. - The single page would list links for all available distributions, which should include all distributions published. It would also list any other URLs that should be scanned for releases, when releases aren't all uploaded to PyPI. - The single page would contain very little additional information. It would be for use by software, not humans. In addition, the root page with a trailing / would be empty and very cheap. There are a lot of ways we could achieve this pretty cheaply while keeping the existing system pretty much as it is. For example, the current effort to bake static pages could bake these pages instead. We could make the new index available at a different URL for people to play with while we worked the kinks out of the process. Of course, those of us who use the cheesehop and setuptools extensively can also achieve much of this by changing the way we work. Thoughts? Jim On Jul 10, 2007, at 8:44 AM, Philipp von Weitershausen wrote: > When easy_installing zc.buildout I realized that the CheeseShop > still lists a gazillion old versions of zc.buildout. That makes it > take quite some time to install zc.buildout (see below), and I > reckon the same sort of check has to happen each time it looks for > a new version of that egg... > > Is there any reason for having so many old versions around? > > > $ easy_install zc.buildout > Searching for zc.buildout > Reading http://cheeseshop.python.org/pypi/zc.buildout/ > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b19 > Reading http://svn.zope.org/zc.buildout > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b22 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b23 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b20 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b21 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b26 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b27 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b24 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b25 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b28 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b17 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b16 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b18 > Best match: zc.buildout 1.0.0b28 > ... -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

4 9

Merge catalog and distutils sigs
by Jim Fulton July 10, 2007

July 10, 2007

Is there are good reason for the distutils and catalog sigs to be separate? Now, that PyPI is an integral part of the distribution system, I find most topics are really of of interested to both sigs, and I bet that the overlap between the sigs is significant. Would anyone object to combining them? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

1 0

Buildout not updating setuptools
by Graham Stratton July 9, 2007

July 9, 2007

Hi, I'm having a problem with buildout. My system has setuptools 0.6c5 installed. The buildout bootstrap.py script doesn't install the latest version, and nor does running bin/buildout. The result is that the Routes install fails with The required version of setuptools (>=0.6c6) is not available, ... The problem seems to be that bootstrap.py creates a link for setuptools in develop-eggs. Why would that be? Thanks, Graham

4 11

Re: [Distutils] [Web-SIG] entry points, etc
by Ian Bicking July 7, 2007

July 7, 2007

Jim Fulton wrote: > > On Jul 6, 2007, at 11:55 PM, Ian Bicking wrote: > >> Incidentally, something that would be nice with wsgiconfig is if we >> could all agree on how to specify things like entry points and objects. >> Specifically Paste Deploy uses egg:Distribution#ep_name, and >> zc.buildout uses Distribution:ep_name. And Paste Deploy defaults to >> ep_name=main while zc.buildout defaults to ep_name=default. > > Yup. > > Some notes. # is unattractive to me because it looks like a comment. > ConfigParser is a bit odd in it's treatment of #s. It treates them as > comments after empty lines and after section names, but not after comments. If ConfigParser did pay attention to comments it would make everything much more complicated. But that's another issue. > I used ":" because setuptools uses module:name when defining entry > points. That may not have been a good reason. Well, it could be good or bad overlap. Mostly it has to be something illegal in a distribution name, and ideally in any specifier (so you can do Package==0.5:ep_name). > If we agree on some standard, I'll support it. That should happen over > on the distutils-sig list. OK, copied over. >> Paste Deploy uses "entry_point_type = object.name" when you aren't using >> an entry point, but I'd like to switch to just "object.name" with an >> optional "object.name [ep_type]". This helps out those people who have >> some hangup with writing their own setup.py. So having a clear way to >> distinguish between an object reference and an entry point reference >> would be ideal. I still would prefer the entry points, as they make it >> easier to search the system for providing objects and easier to handle >> backward compatibility, but I don't have any reason to *require* entry >> points in my code generally. > > It is hard to assess this out of context. I will note that setuptools > uses dotted_module_name:dotted_object_name to name objects and I'd be > inclined to be consistent with that. If we used a different delimiter > between eggs and entry points and between modules and names, then we'd > be able to tell the refrecnes apart. Again, I think this is more > general than web applications. Another case came up as I was thinking about this, which is referring to an object in a specific Python file (not module). I find this very useful when I'm doing some kind of configuration that involves deployment-specific code. I really don't want to create a package and module and fiddle with sys.path, for a bit of code that is purely deployment specific. So I'd like to also allow for a specifier like filename:object_name. We could use prefixes for all of these. I'm fine supporting : instead of # for entry point names. We could use prefixes, e.g., "egg:Package:ep". Somehow that reads a little funny to me, too many colons. We could also just use "egg Package:ep", which maybe reads better. Then there'd also be "python module:object", and "file filename.py:object". Maybe "python " would be the default, I'm not sure. I encourage people to use entry points, but I don't mind including "egg " -- if anything, I think the name is helpful to people who aren't always clear on the distinction between distributions and packages and modules. The default entry point name is also important. I prefer "main" because I think it better describes the purpose of the entry-point-that-need-not-be-named. "default" feels like it's just describing the mechanism of the entry point loader. -- Ian Bicking : ianb(a)colorstudy.com : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers

2 1

Preferring final releases
by Jim Fulton July 7, 2007

July 7, 2007

Currently, when looking for a new release, easy_install and buildout look for the newest distributions that satisfy a requirement. They will, for example, prefer a newer development release or preliminary release (dev, alpha, beta, or rc) over a final release. I don't think this is a good default policy. In buildout, I'm going to add an option, prefer-final, to control this. If this option is true, then buildout will prefer final distributions over non-final distributions. It will choose the latest final version that meets a requirement if there is one. Maybe buildout should make this the default. (Thoughts?) In any case, since buildout lets people create their own defaults for buildout options, people will be able to choose any default they wish. With this option enabled, you'll be able to get a non-final distribution by explicitly requiring it. I think easy_install and setuptools should provide a similar option and provide a way (presumably through setup.cfg) to make it the default. For backward compatibility reasons, it may not be possible to make this the default policy for setuptools, although, arguably, it is a better default. Jim Note that for buildout, what I said above isn't accurate. :) Buildout prefers develop distributions over non-develop distributions. It will currently choose a develop distribution that meets a requirement, if there is one. If prefer-final is enabled, then buildout will choose the latest develop distribution that satisfies the requirement, if there is one, otherwise, it will pick the latest final distribution that meets a requirement, if there is one, otherwise, it will pick the latest distribution that meets a requirement. -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

3 2

Setuptools feature request: simplified version specification
by Jim Fulton July 6, 2007

July 6, 2007

We (me and a bunch of my Zope friends) find that we'll often want version specifications of the form: project_name >=Vdev, <V+1dev (e.g. "foo >=1dev, <2dev") We think this is so common that we'd like a short-hand way of spelling it. I'll note that I'm not even sure I got the spelling above right. The intent is to request version 1, meaning any release of version 1. I think that's what I spelled above, although I'm not sure. If I got it wrong, maybe someone will correct me. Aside from the verbosity of the spelling above, I think the difficulty in spelling it is a strike against it. Note that a naive spelling: "foo >=1, <2" is wrong because it excludes pre-releases of 1 and includes pre-releases of 2. I propose that a valid version that ends in a number and that isn't preceded by an operator be a valid version specifier and be interpreted as a range. So, assuming that I know how to spell the range, a specification of: project_name V (e.g. "foo 1") would be equivalent to: project_name >=Vdev, <V+1 (e.g. "foo >=1dev, <2dev") This would work with multi-part versions, so "foo 1.2" would be equivalent to "foo >=1.2dev <1.3dev". Note that this could be combined with other version specifiers. For example, to require any version 1 or 2 of foo or versions 3.2 final or later: foo 1, 2, >=3.2 Also note that any version will do, so: foo 1.2a1 would be equivalent to: foo >=1.2a1dev <1.2a2dev And note that versions that don't end in numbers wouldn't be valid version specifiers, so: foo 1a would not be a valid specifier. Thoughts? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

5 7

setuptools: Problem with auto-scripts and standalone scripts
by Frank McIngvale July 5, 2007

July 5, 2007

Hi, I'm trying to use setuptools auto-script feature with standalone scripts and am having problems. Say I have a script "mypkg/scripts/foo.py": --------------------- import sys print "Hello foo! My args are:",sys.argv --------------------- When I try wrapping it like this: 'console_scripts': [ 'foo = mypkg.scripts.foo', ] Then run like this: $ foo aa bb cc It appears to start OK (and it gets the cmdline args) but crashes with a traceback: ------------ Hello foo! My args are: ['c:\\frank\\py25\\Scripts\\foo-script.py', 'aa', 'bb', 'cc'] Traceback (most recent call last): File "c:\frank\py25\Scripts\foo-script.py", line 8, in <module> load_entry_point('mypkg==0.99.50-rc1', 'console_scripts', 'foo')() TypeError: 'module' object is not callable ------------ Am I setting up 'console_scripts' incorrectly, or is this kind of usage not supported? thanks, Frank

2 2

disable register and upload commands for a setup.py
by Jeff Kowalczyk July 4, 2007

July 4, 2007

What would be the most reliable way to disable, with error output, the PyPI 'register' and/or 'upload' commands for a given setup.py? This is for developing certain python egg/packages having setup.py:setup() including full package metadata, but which should never (or not yet) accidentally be registered with PyPI. Thanks.

2 1

Looking for a presentation on Egg
by Andreas Jung July 4, 2007

July 4, 2007

Hi, I have to give a 30 minute presentation on Python egg at our next company developer meeting. Does anyone know of a presentation or slides on this topic to be used as a basis? Thanks, Andreas

3 3

Re: [Distutils] [PEAK] Dependencies before calling setup()
by Phillip J. Eby July 4, 2007

July 4, 2007

At 06:16 PM 7/3/2007 +0400, Timur Izhbulatov wrote: >Hi all, > >I use Cheetah in my project and want to precompile templates in my >project's setup.py. So, I need to have Cheetah installed *before* >setup() is called. Don't do that. setup scripts must *never* perform such actions before setup() is called. What if someone is just running "setup.py --help", after all? NEVER do this. easy_install's sandboxing code will also reject your package as unsafe to run if you attempt something like this. >I'm trying to use setuptools.command.easy_install.main(['Cheetah']) >but I can't find any way to update sys.path after that. Is there any >'official' way to do this? The official way to require that a package is available during setup() execution is viat the 'setup_requires' keyword, similar to the 'install_requires' keyword. If you need the same package in both situations, you must list it in both keywords. To actually do the precompile, you should subclass the install_data or install_lib command, and override its run() method to do the compilation, as well as doing whatever it did before. You use the 'cmdclass' keyword to setup() to pass in a dictionary mapping command names to your subclass(es). Your subclasses *must* honor the options they are given, and not attempt to install anything except where they are told. The options will be available as attributes of the instance created by the distutils. (Oh, and speaking of the distutils, the correct place for discussion/support of setuptools is the distutils-SIG. Please direct future correspondence there. Thanks.)

2 1