Mailman 3 May 2009 - Distutils-SIG

Re: [Distutils] Adding entry points into Distutils ?
by P.J. Eby 09 May '09

09 May '09

At 10:21 AM 5/8/2009 +1200, Noah Gift wrote: >1. Different versions of Python conflict with previous versions of >console scripts. Take paste for example. I don't understand what you mean. >2. The entry point mechanism IIRC recursively scans the site-packages >directory and loads up the system path with eggs. This is too >expensive of an operation for the current environment I work in. The scan is not recursive; only files that are actually *on* sys.path are scanned: i.e., either an .egg that is directly on sys.path, or an .egg-info in a directory that is directly on sys.path. Now, it's possible that some application you are using does such a scan explicitly; I'm just noting that merely querying or loading entry points doesn't cause any recursive scans, and it most definitely does not add anything new to sys.path, unless the entry point to be loaded has declared an additional dependency that's *not* on sys.path yet. >3. There doesn't seem to be a clean way to inject user specific >environment details to the console script. >I often need the ability to alter the sys.path in a user specific way >for the entry point without needing to mess up the global sys.path >permanently. I don't understand what you mean here, either.

4 3

Organize e diminua riscos - NS
by Mariana Costa 08 May '09

08 May '09

1 0

Re: [Distutils] Adding entry points into Distutils ?
by P.J. Eby 07 May '09

07 May '09

At 01:14 PM 5/7/2009 -0700, Wheat wrote: >But then having both 'scripts' and 'entry_points/console_scripts' is >less than perfect since it introduces (mostly) unneccessary TIMTOWTDI. I interpret 'scripts' as meaning "non-Python scripts", so there is still only one obvious way to do each thing. ;-)

1 0

Distutils global status
by Tarek Ziadé 07 May '09

07 May '09

Hi, Here's a quick summary of the ongoing work in Distutils, on the issues we seem to reach a consensus. The goal would be to finish the discussions on those and to push them in some kind of "Distutils official roadmap" then look at implementation details (for instance, the pkgutil APIs will be linear unless we use some kind of index) - task : - link - remaining work --- - Introduction the new version comparison algorithm in Distutils - http://bitbucket.org/tarek/distutilsversion - Phillip needs to check the tarek-postdev branch for his use case, then Trent and al. - We need to add the "bits" constructor - Standardize the .egg-info directory, provide APIs, uninstall command - http://svn.python.org/view/peps/trunk/pep-0376.txt?revision=72420&view=mark… - we need to peer-review it here again - Change PKG-INFO content (PEP 345 changes) - http://svn.python.org/view/peps/branches/jim-update-345/pep-0345.txt?revisi… - Tres needs to push the latest changes - we need to peer-review it here again - Finish the massive cleanup / test coverage - not much to say - work in progress by Tarek Cheers Tarek -- Tarek Ziadé | http://ziade.org

1 0

Re: [Distutils] RFC : Version comparison
by P.J. Eby 06 May '09

06 May '09

At 11:12 AM 4/22/2009 +0200, Tarek Ziadé wrote: >Hi, > >We worked during Pycon on version comparisons. Distutils has one but >it is a bit strict, setuptools has another one, but it's a bit >heuristic. > >We would like to propose the inclusion for Python 2.7 of a new version >comparison algorithm, based on the discussion >Fedora, Ubuntu and Python people had. The plan would be to deprecate >the current one (which is not really used anyway) >and provide, promote this one. > >Trent Mick took the lead on this work at the end of Pycon, and worked >on a prototype. > >It's explained here, and there's an implementation (I've put it at >the top of the page for conveniency): > >http://wiki.python.org/moin/Distutils/VersionComparison > >Please comment, I don't see how it can manage, e.g. a development version of a postrelease, with an SVN rev or date stamp on it. Such versions might not be found on PyPI or on RPMs, but would be needed in development. (Btw, the wiki page pseudo-regex doesn't match what the code actually parses, either.)

3 4

logging into distutils
by A. Cavallo 06 May '09

06 May '09

Hi, I'm playing lately with distutils (on the 2.7a branch). This "sudden" interest has been triggered by the python bugs: http://bugs.python.org/issue5941 http://bugs.python.org/issue5940 What I've found (IMHO) is the lack of proper logging while stepping through all the loops inside the code. In fact distutils.log (the log module used internally) It is loosely written resembling the logging python module but missing the full elegance in the api: is there any reason it could not be made very similar to it? I think this is one of the reason is not more deeply spread into the distutils package (a useful tool in diagnostic, IMHO). Because I'm out of work at the moment (It might not last forever tough) I can do it if there is any interest and send a patch set to be reviewed. Beside this to the people interested I can provide a binary compiled python version with the patches hosted on the suse build server system: http://download.opensuse.org/repositories/home:/cavallo71:/python-opt/ This is totally system transparent so it can be installed and de-installed without impact on the host system. Let me know your toughts please, Antonio cavallo

2 1

Re: [Distutils] Setuptools, namespace packages, --single-version-externally-managed
by P.J. Eby 06 May '09

06 May '09

At 04:35 PM 5/5/2009 -0500, Ian Bicking wrote: >Any ideas on this?Â Phillip? > >On Fri, May 1, 2009 at 5:07 PM, Ian Bicking ><<mailto:ianb@colorstudy.com>ianb(a)colorstudy.com> wrote: >So, a bit of a problem came up with pip and namespace >packages.Â Here's my understanding of what's going on: > >When you install a namespace package with pip, it uses install >--single-version-externally-managed, and generally the namespace >directory is empty and there's a *.nspkg.pth file that has this: > >import sys,new,os; p = >os.path.join(sys._getframe(1).f_locals['sitedir'], >*('<NAMESPACE>',)); ie = >os.path.exists(os.path.join(p,'__init__.py')); m = not ie and >sys.modules.setdefault('zope',new.module('zope')); mp = (m or []) >and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) > >So the lack of an __init__.py file doesn't really matter, because >it's created right there, and has its __path__ added to.Â But >there's a problem when there's another namespace package elsewhere >on the path, that wasn't installed with pip (or Setuptools) and uses >pkgutils.extend_path(__path__, __name__).Â This doesn't get >imported because of that .pth file, and the .pth file doesn't itself >use extend_path, so the path isn't searched.Â This is currently >happening with Zope packages installed with plain distutils, then >another package installed with the zope namespace elsewhere with >pip.Â (When using easy_install, I think >pkg_resource.declare_namespace comes into play somewhere, and this >seems to handle this case, but I'm not sure why the installation is >different with pip.) > >So... what should pip be doing differently to make this work? Heck if I know. IIUC, this should work as long as the namespace declaration gets invoked at some point. But I don't know of any way to ensure that, except for pip to write its own __init__.py (invoking declare_namespace()) and gets rid of the nspkg.pth file.

1 0

Setuptools, namespace packages, --single-version-externally-managed
by Ian Bicking 05 May '09

05 May '09

So, a bit of a problem came up with pip and namespace packages. Here's my understanding of what's going on: When you install a namespace package with pip, it uses install --single-version-externally-managed, and generally the namespace directory is empty and there's a *.nspkg.pth file that has this: import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('<NAMESPACE>',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('zope',new.module('zope')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) So the lack of an __init__.py file doesn't really matter, because it's created right there, and has its __path__ added to. But there's a problem when there's another namespace package elsewhere on the path, that wasn't installed with pip (or Setuptools) and uses pkgutils.extend_path(__path__, __name__). This doesn't get imported because of that .pth file, and the .pth file doesn't itself use extend_path, so the path isn't searched. This is currently happening with Zope packages installed with plain distutils, then another package installed with the zope namespace elsewhere with pip. (When using easy_install, I think pkg_resource.declare_namespace comes into play somewhere, and this seems to handle this case, but I'm not sure why the installation is different with pip.) So... what should pip be doing differently to make this work? -- Ian Bicking | http://blog.ianbicking.org

1 1

Re: [Distutils] Adding entry points into Distutils ?
by P.J. Eby 05 May '09

05 May '09

At 06:57 PM 5/4/2009 -0500, Ian Bicking wrote: >* I'm uncomfortable with the way entry points are scanned.Â I >haven't looked close enough to back it up with numbers, but I think >there's a noticeable performance degradation when the number of >installed packages becomes large.Â (Given the algorithm this would >be expected.) It's linear in the number of entry_points.txt files, yes, but in most apps it should occur at most once, since pkg_resources has a single WorkingSet object holding Distribution objects which cache their entry point data upon first access. There are all sorts of ways you could make different tradeoffs, but this particular set of tradeoffs was optimized for a single-application environment, rather than a massive global shared site-packages where there are plugins for every application on the system. It was also optimized for the zipimport case, because you can tell whether a project has entry points from its cached zip directory, that's needed at startup anyhow.

3 2

Re: [Distutils] Anyone stuck with easy_install / has .pth file issues -- this is for you!
by P.J. Eby 05 May '09

05 May '09

At 01:11 PM 5/4/2009 -0700, Garrett Cooper wrote: >You're right -- it doesn't protect against the following (><): > >/full/path/to/package.egg >./package.egg By duplicates, I meant 'package-1.0.egg' and 'package-1.1.egg', not alternate paths to the same file. (As for the '.' replacement, you might want to look at what I said more closely.) >BTW, wasn't -m deprecated? No. > I mean, -m's `delivered' functionality >doesn't even work You'll need to be more specific there: I don't know what you're saying "doesn't work", let alone *how*, what behavior you expect, or what you're seeing instead. >because python stores all loaded package / module >entries in .pth files as keys in a dictionary _anyhow_ I also don't know what you mean here. .pth file contents are added to sys.path, not a dictionary. Unless you're talking about sys.path_importer_cache or something, in which case I still don't see the connection to anything else at hand. >I have no idea how -xN factor into this though The original docs said to use -m before uninstalling, but they really should say -mxN to exclude scripts and avoid installing (or uninstalling!) dependencies. The docs are broken (or at any rate, suboptimal) and should be fixed. >- The duplicates erasure was just gravy, and purely an exercise on my >part getting used to set's in python. The real point was the missing >package in .pth functionality, caused by easy_install / setuptools >lacking proper MP-safe logic. I'm going to guess that by "MP" you mean multiprocessing, in which case I would say, yes, easy_install assumes that you are not running multiple simultaneous installations to the same directory.

2 1