Mailman 3 May 2009 - Distutils-SIG

setup tools maximum recursion depth exceeded
by A. Cavallo 18 May '09

18 May '09

There's been a previous post (Tarek, "Fixing the mess in sdist/egg_info") about the fact that setuptools fails due to a circular dependency: distutils.sdist.run() -> setuptools.build_py.data_files -> setuptools.egg_info.run() -> distutils.sdist.add_defaults() -> setuptools.build_py.data_files -> etc Actually I'm working packaging sphinx and it fails for this reason. Has anything moved to fix this problem? Regards, Antonio

1 0

Re: [Distutils] Version string normalization
by P.J. Eby 18 May '09

18 May '09

At 04:00 PM 5/18/2009 -0400, Jean-Paul Calderone wrote: >Hello, > >I've just noticed that when using setuptools, the name of the file created >by the sdist command changes. The version passed to setup is "0.9.33+r17283" >but the file written is "Nevow-0.9.33-r17283.tar.gz". > >This change causes various problems for me. How can I avoid it? You would need to subclass the sdist command, or use a post-processing step to rename it.

1 0

Version string normalization
by Jean-Paul Calderone 18 May '09

18 May '09

Hello, I've just noticed that when using setuptools, the name of the file created by the sdist command changes. The version passed to setup is "0.9.33+r17283" but the file written is "Nevow-0.9.33-r17283.tar.gz". This change causes various problems for me. How can I avoid it? Thanks, Jean-Paul

1 0

Strict/Loose Version and toolchains checks
by David Cournapeau 16 May '09

16 May '09

Hi, Since we are talking about versioning, here is one more complaint which may fall into the bugreport category or not, I am not sure. There are some version checks in some codepaths which do not make much sense IMHO, in particular: - distutils checks the versions of the toolchain. This is broken. If I want to use gcc built from svn to build my extensions, distutils should just do what I ask it to do, and not barf as it currently does. - bdist_msi also fails to run when the project version is not a released version. I can't see the rationale for that behavior, and it is not consistent with others bdist_ anyway. thanks, David

2 2

Adding entry points into Distutils ?
by Tarek Ziadé 15 May '09

15 May '09

Hello I am making a separate email for this topic to make sure no one misses it. There are *many* benefits of adding entry points into Distutils. In fact, adding a plugin system in there, will help make the commands more extendable and therefore will help us in the long term to remove things out of Distutils. So any strong opinion against them ? If not, I'll add a section in PEP 376 - and the first use case I can see is having a pluggable way to extend the list of files contained in .egg-info. The work to be done would be to extract entry points from pkg_resources (the discovery work that consists of looking for entry_points.txt files will be covered by the APIs already described in that PEP) This is not hard. Cheers Tarek -- Tarek Ziadé | http://ziade.org

16 52

Re: [Distutils] PEP 376 - site-directories and site.addsitedir
by P.J. Eby 15 May '09

15 May '09

At 11:38 PM 5/14/2009 +0200, Tarek Ziadé wrote: >That happened because I got a bit lost in the way finders were working >in setuptools, which seemed >a bit over-engineered at that time to me (but I might end up changing >my mind at sime point when I understand it better). The finders mechanism is there to allow support for arbitrary PEP 302 importers -- i.e., sys.path strings that don't go to the filesystem. The main reason why there are two finders currently (find_on_path and find_in_zip) is that find_in_zip supports egg directory bundling. That is, an .egg zipfile may contain arbitrarily nested .egg directories, and an .egg directory may contain arbitrarily nested .egg directories or .egg zipfiles. However, if you're not implementing dependency resolution, you don't need that support. If such a nested item were in use, it'd be on sys.path already. (In other words, the nesting lookups are only useful for alternate version discovery, not simple metadata lookups.)

2 1

including a third party egg without "installing" it
by Randall Smith 14 May '09

14 May '09

I'd like to bundle some third party eggs with my application without installing them. Typically, I'd read __file__ and alter sys.path at runtime to include the third party packages, but I understand that will not work with zipped eggs. I don't know the best way to go about this, but I at least want to know a clean way to do it without requiring the third party eggs to be downloaded and installed separately. Consider this layout. theapp/ docs/ setup.py theapp/ __init__.py app.py ext/ thirdparty1.egg thirdparty2.egg thirdparty3.egg tests/ From my current understanding about how setuptools works, I think I could treat the third party eggs as data files and and runtime (somehow?) insert them into the path and import them. I'm trying to work from this example. http://peak.telecommunity.com/DevCenter/PythonEggs#accessing-package-resour… Is this a workable approach? Is there a better way? --Randall

2 2

Re: [Distutils] PEP 376 - site-directories and site.addsitedir
by P.J. Eby 14 May '09

14 May '09

At 11:10 PM 5/14/2009 +0200, Tarek Ziadé wrote: >2009/5/14 P.J. Eby <pje(a)telecommunity.com>: > > IOW, this approach keeps simple things simple, and leaves complex things > > possible. It also does less I/O than what you're proposing, since in the > > normal case the directories are only ever searched once, and the actual > > metadata reads are both lazy and cached. > >Makes a lot of sense yes. I think I'll just start a prototype for that >code, propose 376 on python-dev and see. Notice, by the way, that this is a good example of pkg_resources' code only looking complex if you haven't gone through all the requirements and design issues yourself. The further you go with this, the more likely you're going to end up with something very much resembling pkg_resources itself. The bad news is that you'll be reinventing a lot of wheels. The good news is, you should also be able to understand and maintain pkg_resources internals by the time you're done. ;-) The relevant classes in pkg_resources, btw, are WorkingSet, Distribution, and EntryPoint. You might want to just copy them and strip out the parts you don't need. (i.e. sys.path manipulation and requirements-resolving stuff)

2 1

Re: [Distutils] PEP 376 - site-directories and site.addsitedir
by P.J. Eby 14 May '09

14 May '09

At 12:03 PM 5/14/2009 +0200, Tarek Ziadé wrote: >Hello > >for PEP 376, I have one last fuzzy point. > >http://svn.python.org/view/peps/trunk/pep-0376.txt?view=markup > >The "get_egg_info" api is currently based on scanning the whole >sys.path. And since sys.path can be modified by people, >so the algorithm is linear and can slow down when there are a lot of paths. > >I have a proposal: let's restrict the search for this API to >site-package directories only. (directories added with >site.addsitedir) > >People will be able to mark add any directory (like the per-user >site-package directory - http://www.python.org/dev/peps/pep-0370) > >This requires to add in site.py a registry to keep track of all >directories added through site.addsitedir > >Any thoughts ? What tradeoffs are you optimizing for? Note that a single scan of every directory on sys.path is exactly what happens when an import doesn't find its target until the *last* directory on sys.path. So this is not really a big deal if you're only doing it *once*. If you want to optimize for repeated searches, the best way to do this is with a structure like pkg_resources' WorkingSet object - it simply reads the directories once and makes an object for each installed package. These objects don't do any further I/O, so really we're just talking about caching a list of .egg-info filenames. Each object in the set can be queried for its metadata -- in which case it reads it exactly once, and caches it. With this setup, the full directory scan is only ever done once -- and it's basically equivalent to adding an extra import at the time you first import the metadata management module. Yes, it does mean a global, unless you want to hand off cache management to the application. But the way pkg_resources does it, with WorkingSet and Distribution objects, allows an app with special needs to do its own path management and search operations. IOW, this approach keeps simple things simple, and leaves complex things possible. It also does less I/O than what you're proposing, since in the normal case the directories are only ever searched once, and the actual metadata reads are both lazy and cached. Note, too, that site-packages dirs are likely to have more packages on them than other directories, which means you're not necessarily saving much I/O to start with, and even that small savings evaporates as soon as you do more than one lookup for plugins.

2 1

Re: [Distutils] RFC : Version comparison
by P.J. Eby 14 May '09

14 May '09

At 10:19 AM 5/14/2009 +0200, Tarek Ziadé wrote: >from a setuptools user point of view, the benefit I can see is that >they will have better >version numbers Better how? That was my question. I personally find the common version patterns in use (e.g. '-' and '-r' for post-releases) less clunky and easier to read than 'post'. I also don't care for '.' in front of dev and post. So, I don't see the changes as all "better" from a readability POV. That's not to say it's a bad thing, if it gives other benefits. But it has not been stated what the benefits are supposed to be. >that will comply with a documented standard, that is This isn't actually a change; setuptools also has a documented standard. >usable and understandable by os packagers for example. I agree that a canonical form is useful. However, if that canonical form can be generated from their existing version numbers, then why should they bother changing? That's the open question, IOW. >Now I suppose setuptools will have to propose both for some time, I don't see why. AFAICT, RationalVersion is a strict subset of the syntax supported by setuptools, and most setuptools versions in use should be convertible. >But what I am scared of is : who will work on setuptools side ? can >you bless someone to do the >work when we agree on a common roadmap ? I don't see that there is any work *to* do in setuptools core, since RationalVersion is a strict subset, and we have setup-argument validation providers already. I.e., anyone can make a plugin that validates or converts a setup(version="...") string, put it on sys.path, and force canonical versions.

2 1