Mailman 3 May 2009 - Distutils-SIG

setup tools maximum recursion depth exceeded
by A. Cavallo May 18, 2009

May 18, 2009

There's been a previous post (Tarek, "Fixing the mess in sdist/egg_info") about the fact that setuptools fails due to a circular dependency: distutils.sdist.run() -> setuptools.build_py.data_files -> setuptools.egg_info.run() -> distutils.sdist.add_defaults() -> setuptools.build_py.data_files -> etc Actually I'm working packaging sphinx and it fails for this reason. Has anything moved to fix this problem? Regards, Antonio

1 0

Re: [Distutils] Version string normalization
by P.J. Eby May 18, 2009

May 18, 2009

At 04:00 PM 5/18/2009 -0400, Jean-Paul Calderone wrote: >Hello, > >I've just noticed that when using setuptools, the name of the file created >by the sdist command changes. The version passed to setup is "0.9.33+r17283" >but the file written is "Nevow-0.9.33-r17283.tar.gz". > >This change causes various problems for me. How can I avoid it? You would need to subclass the sdist command, or use a post-processing step to rename it.

1 0

Version string normalization
by Jean-Paul Calderone May 18, 2009

May 18, 2009

Hello, I've just noticed that when using setuptools, the name of the file created by the sdist command changes. The version passed to setup is "0.9.33+r17283" but the file written is "Nevow-0.9.33-r17283.tar.gz". This change causes various problems for me. How can I avoid it? Thanks, Jean-Paul

1 0

Strict/Loose Version and toolchains checks
by David Cournapeau May 16, 2009

May 16, 2009

Hi, Since we are talking about versioning, here is one more complaint which may fall into the bugreport category or not, I am not sure. There are some version checks in some codepaths which do not make much sense IMHO, in particular: - distutils checks the versions of the toolchain. This is broken. If I want to use gcc built from svn to build my extensions, distutils should just do what I ask it to do, and not barf as it currently does. - bdist_msi also fails to run when the project version is not a released version. I can't see the rationale for that behavior, and it is not consistent with others bdist_ anyway. thanks, David

2 2

Adding entry points into Distutils ?
by Tarek Ziadé May 15, 2009

May 15, 2009

Hello I am making a separate email for this topic to make sure no one misses it. There are *many* benefits of adding entry points into Distutils. In fact, adding a plugin system in there, will help make the commands more extendable and therefore will help us in the long term to remove things out of Distutils. So any strong opinion against them ? If not, I'll add a section in PEP 376 - and the first use case I can see is having a pluggable way to extend the list of files contained in .egg-info. The work to be done would be to extract entry points from pkg_resources (the discovery work that consists of looking for entry_points.txt files will be covered by the APIs already described in that PEP) This is not hard. Cheers Tarek -- Tarek Ziadé | http://ziade.org

16 52

Re: [Distutils] PEP 376 - site-directories and site.addsitedir
by P.J. Eby May 15, 2009

May 15, 2009

At 11:38 PM 5/14/2009 +0200, Tarek Ziadé wrote: >That happened because I got a bit lost in the way finders were working >in setuptools, which seemed >a bit over-engineered at that time to me (but I might end up changing >my mind at sime point when I understand it better). The finders mechanism is there to allow support for arbitrary PEP 302 importers -- i.e., sys.path strings that don't go to the filesystem. The main reason why there are two finders currently (find_on_path and find_in_zip) is that find_in_zip supports egg directory bundling. That is, an .egg zipfile may contain arbitrarily nested .egg directories, and an .egg directory may contain arbitrarily nested .egg directories or .egg zipfiles. However, if you're not implementing dependency resolution, you don't need that support. If such a nested item were in use, it'd be on sys.path already. (In other words, the nesting lookups are only useful for alternate version discovery, not simple metadata lookups.)

2 1

including a third party egg without "installing" it
by Randall Smith May 14, 2009

May 14, 2009

I'd like to bundle some third party eggs with my application without installing them. Typically, I'd read __file__ and alter sys.path at runtime to include the third party packages, but I understand that will not work with zipped eggs. I don't know the best way to go about this, but I at least want to know a clean way to do it without requiring the third party eggs to be downloaded and installed separately. Consider this layout. theapp/ docs/ setup.py theapp/ __init__.py app.py ext/ thirdparty1.egg thirdparty2.egg thirdparty3.egg tests/ From my current understanding about how setuptools works, I think I could treat the third party eggs as data files and and runtime (somehow?) insert them into the path and import them. I'm trying to work from this example. http://peak.telecommunity.com/DevCenter/PythonEggs#accessing-package-resour… Is this a workable approach? Is there a better way? --Randall

2 2

Re: [Distutils] PEP 376 - site-directories and site.addsitedir
by P.J. Eby May 14, 2009

May 14, 2009

At 11:10 PM 5/14/2009 +0200, Tarek Ziadé wrote: >2009/5/14 P.J. Eby <pje(a)telecommunity.com>: > > IOW, this approach keeps simple things simple, and leaves complex things > > possible. It also does less I/O than what you're proposing, since in the > > normal case the directories are only ever searched once, and the actual > > metadata reads are both lazy and cached. > >Makes a lot of sense yes. I think I'll just start a prototype for that >code, propose 376 on python-dev and see. Notice, by the way, that this is a good example of pkg_resources' code only looking complex if you haven't gone through all the requirements and design issues yourself. The further you go with this, the more likely you're going to end up with something very much resembling pkg_resources itself. The bad news is that you'll be reinventing a lot of wheels. The good news is, you should also be able to understand and maintain pkg_resources internals by the time you're done. ;-) The relevant classes in pkg_resources, btw, are WorkingSet, Distribution, and EntryPoint. You might want to just copy them and strip out the parts you don't need. (i.e. sys.path manipulation and requirements-resolving stuff)

2 1

Re: [Distutils] PEP 376 - site-directories and site.addsitedir
by P.J. Eby May 14, 2009

May 14, 2009

At 12:03 PM 5/14/2009 +0200, Tarek Ziadé wrote: >Hello > >for PEP 376, I have one last fuzzy point. > >http://svn.python.org/view/peps/trunk/pep-0376.txt?view=markup > >The "get_egg_info" api is currently based on scanning the whole >sys.path. And since sys.path can be modified by people, >so the algorithm is linear and can slow down when there are a lot of paths. > >I have a proposal: let's restrict the search for this API to >site-package directories only. (directories added with >site.addsitedir) > >People will be able to mark add any directory (like the per-user >site-package directory - http://www.python.org/dev/peps/pep-0370) > >This requires to add in site.py a registry to keep track of all >directories added through site.addsitedir > >Any thoughts ? What tradeoffs are you optimizing for? Note that a single scan of every directory on sys.path is exactly what happens when an import doesn't find its target until the *last* directory on sys.path. So this is not really a big deal if you're only doing it *once*. If you want to optimize for repeated searches, the best way to do this is with a structure like pkg_resources' WorkingSet object - it simply reads the directories once and makes an object for each installed package. These objects don't do any further I/O, so really we're just talking about caching a list of .egg-info filenames. Each object in the set can be queried for its metadata -- in which case it reads it exactly once, and caches it. With this setup, the full directory scan is only ever done once -- and it's basically equivalent to adding an extra import at the time you first import the metadata management module. Yes, it does mean a global, unless you want to hand off cache management to the application. But the way pkg_resources does it, with WorkingSet and Distribution objects, allows an app with special needs to do its own path management and search operations. IOW, this approach keeps simple things simple, and leaves complex things possible. It also does less I/O than what you're proposing, since in the normal case the directories are only ever searched once, and the actual metadata reads are both lazy and cached. Note, too, that site-packages dirs are likely to have more packages on them than other directories, which means you're not necessarily saving much I/O to start with, and even that small savings evaporates as soon as you do more than one lookup for plugins.

2 1

Re: [Distutils] RFC : Version comparison
by P.J. Eby May 14, 2009

May 14, 2009

At 10:19 AM 5/14/2009 +0200, Tarek Ziadé wrote: >from a setuptools user point of view, the benefit I can see is that >they will have better >version numbers Better how? That was my question. I personally find the common version patterns in use (e.g. '-' and '-r' for post-releases) less clunky and easier to read than 'post'. I also don't care for '.' in front of dev and post. So, I don't see the changes as all "better" from a readability POV. That's not to say it's a bad thing, if it gives other benefits. But it has not been stated what the benefits are supposed to be. >that will comply with a documented standard, that is This isn't actually a change; setuptools also has a documented standard. >usable and understandable by os packagers for example. I agree that a canonical form is useful. However, if that canonical form can be generated from their existing version numbers, then why should they bother changing? That's the open question, IOW. >Now I suppose setuptools will have to propose both for some time, I don't see why. AFAICT, RationalVersion is a strict subset of the syntax supported by setuptools, and most setuptools versions in use should be convertible. >But what I am scared of is : who will work on setuptools side ? can >you bless someone to do the >work when we agree on a common roadmap ? I don't see that there is any work *to* do in setuptools core, since RationalVersion is a strict subset, and we have setup-argument validation providers already. I.e., anyone can make a plugin that validates or converts a setup(version="...") string, put it on sys.path, and force canonical versions.

2 1

Re: [Distutils] RFC : Version comparison
by P.J. Eby May 14, 2009

May 14, 2009

At 10:59 AM 5/14/2009 +0200, Tarek Ziadé wrote: >2009/5/14 P.J. Eby <pje(a)telecommunity.com>: > > At 02:34 PM 5/13/2009 +0200, Tarek Ziadé wrote: > >> > >> Any feedback on this "dev version of post release" use case > > > > It looks like the 'suggest' function doesn't handle svn revisions properly > > for conversion from setuptools versions. Setuptools '-r###' is a > > post-release tag, as is any alpha string alphabetically after > 'final'. That > > is, '0.4a1.r10' is actually '0.4a1.post10'. (Notice, for example, that > > setuptools' trunk version is '0.7a1dev-r66608'.) > > > > See > > > http://peak.telecommunity.com/DevCenter/setuptools#specifying-your-project-… > > for the full syntax; I'm a bit concerned that there may be other incorrect > > interpretations taking place in this function. (Notice, for > example, that a > > '-' in a version number indicates a post-release, but your code is treating > > '-' as identical to '.'.) > >Ok, I have changed the suggest function accordingly, > > > def test_suggest_rational_version(self): > > self.assertEquals(suggest('1.0'), '1.0') > self.assertEquals(suggest('1.0-alpha1'), '1.0a1') > self.assertEquals(suggest('1.0rc2'), '1.0c2') > self.assertEquals(suggest('walla walla washington'), None) > > # from setuptools > self.assertEquals(suggest('0.4a1.r10'), '0.4a1.post10') > self.assertEquals(suggest('0.7a1dev-r66608'), '0.7a1.dev66608') > self.assertEquals(suggest('0.6a9.dev-r41475'), '0.6a9.dev41475') > self.assertEquals(suggest('2.4preview1'), '2.4c1') > self.assertEquals(suggest('2.4rc1'), '2.4c1') > self.assertEquals(suggest('2.4pre1') , '2.4c1') All looking pretty good, except: > self.assertEquals(suggest('2.1-rc2'), None) # no suggestion This should be equivalent to '2.1c2': >>> from pkg_resources import parse_version as pv >>> pv('2.1-rc2')==pv('2.1c2') True

1 0

PEP 376 - site-directories and site.addsitedir
by Tarek Ziadé May 14, 2009

May 14, 2009

Hello for PEP 376, I have one last fuzzy point. http://svn.python.org/view/peps/trunk/pep-0376.txt?view=markup The "get_egg_info" api is currently based on scanning the whole sys.path. And since sys.path can be modified by people, so the algorithm is linear and can slow down when there are a lot of paths. I have a proposal: let's restrict the search for this API to site-package directories only. (directories added with site.addsitedir) People will be able to mark add any directory (like the per-user site-package directory - http://www.python.org/dev/peps/pep-0370) This requires to add in site.py a registry to keep track of all directories added through site.addsitedir Any thoughts ? (In the meantime, I'll propose the inclusion of a sitedir list in site, in Python-ideas) ++ Tarek -- Tarek Ziadé | http://ziade.org

2 1

[issue69] No 2.6 install setup
by mkesper May 14, 2009

May 14, 2009

New submission from mkesper <mkesper(a)schokokeks.org>: If you want to install 0.6c9 on windows, you face a hen-and-egg-Problem: An .eeg exists, but no installer. ---------- messages: 287 nosy: mkesper priority: feature status: unread title: No 2.6 install setup _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue69> _______________________________________________

1 0

Re: [Distutils] RFC : Version comparison
by P.J. Eby May 14, 2009

May 14, 2009

At 02:34 PM 5/13/2009 +0200, Tarek Ziadé wrote: >Any feedback on this "dev version of post release" use case It looks like the 'suggest' function doesn't handle svn revisions properly for conversion from setuptools versions. Setuptools '-r###' is a post-release tag, as is any alpha string alphabetically after 'final'. That is, '0.4a1.r10' is actually '0.4a1.post10'. (Notice, for example, that setuptools' trunk version is '0.7a1dev-r66608'.) See http://peak.telecommunity.com/DevCenter/setuptools#specifying-your-project-… for the full syntax; I'm a bit concerned that there may be other incorrect interpretations taking place in this function. (Notice, for example, that a '-' in a version number indicates a post-release, but your code is treating '-' as identical to '.'.) >or I move >forward and integrate it ? Into the stdlib? I'm a bit surprised that something with such global consequences is having its bleeding edge development done in the stdlib, rather than as an external package. I sort of assumed at first the idea was to provide a normalized or canonical form for setuptools versions, but of course it's a bit more restrictive than that, and at the moment the only mapping between the two seems broken -- and in a way that suggests the designers neither read the setuptools version documentation nor examined what people were actually doing with their versioning schemes. To be clear, I'm not opposed to blessing a normalized version scheme -- there are solid benefits to doing such a thing. I'm opposed to putting one in the stdlib without a clear rationale for the choices, and an upgrade path that either lets setuptools users ignore the new scheme, or gives them some benefit for switching. Meanwhile, I haven't seen any indication that this new format is easier for non-Python tools to parse. Is the plan for other tools to use the stdlib parsing code, because I don't see how existing tools like RPM et al could use the proposed scheme.

2 2

Eggs, zc.buildout, and Distributed Version Control, oh my!
by Ross Patterson May 14, 2009

May 14, 2009

...and of course the elephant in the living room, svn:externals. :) I've been using DVC (git mostly) more and more these days and enjoying it very much. It works particularly well when I have a self-contained software package. I tend to start a project in SVN with the standard trunk/branches/tags layout and then use git in front of the SVN repo. I make the egg root itself a buildout for a test environment with my favorite but not required development tools. (Of course I test with "python setup.py test" before release when the egg doesn't require any additional deployment details provided by the buildout. I don't want to get into that discussion.) It all works much better than trying to work directly with SVN far beyond the benefits of just being able to work offline. What I want to ask about and still haven't worked out an answer to is how to best manage framework or deployment projects where I am developing a number of eggs in parallel under a buildout that needs to be versioned as well. In this case, I'm finding moving away from svn:externals more painful than I expected. Having an overview of all version control status for all code being developed is very valuable. Now some of this comes down to eggs being overused and I've certainly been guilty of that myself. For example, I used to make separate eggs for all my client specific code (policy, theme, etc.). This is clearly unnecessary and I've switched to using one egg that contians multiple sub-packages for factoring. Even with that, however, I still find the lack of a good "what changes have I made to all sub projects" overview to be rather painful. Maybe this is outside the scope of the individual DVC tool and should be handled by an independent tool that aggregates status across projects. How do others handle this kind of situation? Ross

2 2

Re: [Distutils] RFC : Version comparison
by P.J. Eby May 13, 2009

May 13, 2009

At 08:58 PM 5/6/2009 +0200, Tarek Ziadé wrote: >Sorry to post again that mail, but it seems that this is the last remaining >issue for the versionning proposal. > >Phillip, could you check my proposal for your development versions of >postreleases >use case ? Is the code up-to-date with the below? >thx > >On Thu, Apr 23, 2009 at 11:19 AM, Tarek Ziadé <ziade.tarek(a)gmail.com> wrote: > > 2009/4/22 P.J. Eby <pje(a)telecommunity.com>: > >> I don't see how it can manage, e.g. a development version of a > postrelease, > >> with an SVN rev or date stamp on it. Such versions might not be found on > >> PyPI or on RPMs, but would be needed in development. > > > > So, instead of having 'dev' and 'post', we would require a third case > > for "pos+dev" version > > > > so (dev|post)N+ could become, ((dev|post)N+)|(postN+devN+) > > > > example: > > > > 1.0.dev459 < 1.0 < 1.0.post456dev463 < 1.0.post456 < 1.0.post489 > > > > > >> > >> (Btw, the wiki page pseudo-regex doesn't match what the code actually > >> parses, either.) > >> > > > > i'll check it thx > > > >> > > > > > > > > -- > > Tarek Ziadé | http://ziade.org > > > > > >-- >Tarek Ziadé | http://ziade.org

2 3

Distutils roadmap : 8th may
by Tarek Ziadé May 13, 2009

May 13, 2009

Hello, I would like to make some decisions and move forward on three topics for Distutils : 1/ version comparison : http://wiki.python.org/moin/Distutils/VersionComparison 2/ standardization of egg-info : PEP 376 + http://wiki.python.org/moin/Distutils/StandardizeEggInfo 3/ changes in PKG-INFO : PEP 345 + http://svn.python.org/projects/peps/branches/jim-update-345/pep-0345.txt + http://wiki.python.org/moin/Distutils/Metadata I propose to settle on these three topics and set next friday (8th may) as a deadline. Cheers Tarek -- Tarek Ziadé | http://ziade.org

2 1

Buildout: adding a download API?
by Thomas Lotze May 12, 2009

May 12, 2009

Buildout as well as a number of buildout recipes need to download files from the net. They need to take care of doing the actual download, using the download cache and honouring the offline option. A couple of years ago, we also wrote the gocept.download recipe that does all that as well as MD5 checks and some other stuff. In order to simplify this situatin, we propose adding a download API to buildout itself. A function like this: def download(url, use_cache=True, md5sum=None) ... return filename might be sufficient to remove the download logic from recipes and make optional checksum testing available without needing to depend on a separate download recipe. As a consequence, zc.recipe.cmmi would be able to do MD5 checks, which would basically make gocept.cmmi redundant. Using gocept.download's capabilities is the one feature of it which still currently keeps us from dropping it in favour of zc.recipe.cmmi. If there's no objection against a download API being added to buildout, we'd like to implement one soon. Otherwise, we'd propose implementing MD5 testing in zc.recipe.cmmi since we consider it a good thing to reduce the zoo of cmmi-related recipes. We do think that a reusable API would be the better choice, though. Viele Grüße, Thomas -- Thomas Lotze · tl(a)gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development

2 1

Setuptools release (svn 1.6)
by Tarek Ziadé May 12, 2009

May 12, 2009

Phillip, What about blessing someone to review and commit the patch for svn 1.6 support (and maybe other patches) and do a setuptools release ? Regards Tarek -- Tarek Ziadé | http://ziade.org

2 1

Possible bug in setuptools - test command?
by Iwan Vosloo May 11, 2009

May 11, 2009

Hi there, We've been having some trouble with a script which invokes setuptools.setup(script_args=['test']) more than once in a single process. During each invocation, python imports different, new instances of the same module. We've narrowed it down to setuptools/command/test.py, in the method with_project_on_sys_path: It saves sys.modules like this: old_modules = sys.modules.copy() Then does its thing and restores sys.modules like this: sys.modules.clear() sys.modules.update(old_modules) But, as the code below illustrates, this results in multiple imports of the same modules: import sys def i(): old_modules = sys.modules.copy() from decimal import Decimal sys.modules.clear() sys.modules.update(old_modules) return Decimal assert i() is i(), 'Two different Decimal instances returned' # Fails Is this a bug? (We are on version 0.6c8 of setuptools, python 2.5) There is an easy solution: Save sys.modules like before: old_modules = sys.modules.copy() But restore it like this: sys.modules = old_modules That seems to fix our problem (not sure of other effects though). Regards - Iwan Vosloo

1 1

Re: [Distutils] Adding entry points into Distutils ?
by P.J. Eby May 9, 2009

May 9, 2009

At 10:21 AM 5/8/2009 +1200, Noah Gift wrote: >1. Different versions of Python conflict with previous versions of >console scripts. Take paste for example. I don't understand what you mean. >2. The entry point mechanism IIRC recursively scans the site-packages >directory and loads up the system path with eggs. This is too >expensive of an operation for the current environment I work in. The scan is not recursive; only files that are actually *on* sys.path are scanned: i.e., either an .egg that is directly on sys.path, or an .egg-info in a directory that is directly on sys.path. Now, it's possible that some application you are using does such a scan explicitly; I'm just noting that merely querying or loading entry points doesn't cause any recursive scans, and it most definitely does not add anything new to sys.path, unless the entry point to be loaded has declared an additional dependency that's *not* on sys.path yet. >3. There doesn't seem to be a clean way to inject user specific >environment details to the console script. >I often need the ability to alter the sys.path in a user specific way >for the entry point without needing to mess up the global sys.path >permanently. I don't understand what you mean here, either.

4 3

Organize e diminua riscos - NS
by Mariana Costa May 8, 2009

May 8, 2009

1 0

Re: [Distutils] Adding entry points into Distutils ?
by P.J. Eby May 7, 2009

May 7, 2009

At 01:14 PM 5/7/2009 -0700, Wheat wrote: >But then having both 'scripts' and 'entry_points/console_scripts' is >less than perfect since it introduces (mostly) unneccessary TIMTOWTDI. I interpret 'scripts' as meaning "non-Python scripts", so there is still only one obvious way to do each thing. ;-)

1 0

Distutils global status
by Tarek Ziadé May 7, 2009

May 7, 2009

Hi, Here's a quick summary of the ongoing work in Distutils, on the issues we seem to reach a consensus. The goal would be to finish the discussions on those and to push them in some kind of "Distutils official roadmap" then look at implementation details (for instance, the pkgutil APIs will be linear unless we use some kind of index) - task : - link - remaining work --- - Introduction the new version comparison algorithm in Distutils - http://bitbucket.org/tarek/distutilsversion - Phillip needs to check the tarek-postdev branch for his use case, then Trent and al. - We need to add the "bits" constructor - Standardize the .egg-info directory, provide APIs, uninstall command - http://svn.python.org/view/peps/trunk/pep-0376.txt?revision=72420&view=mark… - we need to peer-review it here again - Change PKG-INFO content (PEP 345 changes) - http://svn.python.org/view/peps/branches/jim-update-345/pep-0345.txt?revisi… - Tres needs to push the latest changes - we need to peer-review it here again - Finish the massive cleanup / test coverage - not much to say - work in progress by Tarek Cheers Tarek -- Tarek Ziadé | http://ziade.org

1 0

Re: [Distutils] RFC : Version comparison
by P.J. Eby May 6, 2009

May 6, 2009

At 11:12 AM 4/22/2009 +0200, Tarek Ziadé wrote: >Hi, > >We worked during Pycon on version comparisons. Distutils has one but >it is a bit strict, setuptools has another one, but it's a bit >heuristic. > >We would like to propose the inclusion for Python 2.7 of a new version >comparison algorithm, based on the discussion >Fedora, Ubuntu and Python people had. The plan would be to deprecate >the current one (which is not really used anyway) >and provide, promote this one. > >Trent Mick took the lead on this work at the end of Pycon, and worked >on a prototype. > >It's explained here, and there's an implementation (I've put it at >the top of the page for conveniency): > >http://wiki.python.org/moin/Distutils/VersionComparison > >Please comment, I don't see how it can manage, e.g. a development version of a postrelease, with an SVN rev or date stamp on it. Such versions might not be found on PyPI or on RPMs, but would be needed in development. (Btw, the wiki page pseudo-regex doesn't match what the code actually parses, either.)

3 4