Mailman 3 May 2005 - Distutils-SIG

EasyInstall: verbosity
by Ian Bicking May 30, 2005

May 30, 2005

As different kinds of indirection and detection are added to easy_install, it should probably have a verbosity option. I think this would probably best work by using the logging module, so that Installer takes a logger object or name, and reports all messages to that logger, and in main() it directs the logger to stdout and sets the log level based on the verbosity indicated. Tools that work with easy_install outside of the command-line could probably also make good use of the logging output. I could add this, but the number of parallel patches is probably going to leave everyone (including me) confused. (I'm working on installation of packages without setup.py files right now) -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

2 3

EasyInstall: SF installs
by Ian Bicking May 30, 2005

May 30, 2005

Attached is a patch so it can install from Sourceforge Download pages, choosing a random mirror. Sadly I find the file I want to install from SF doesn't include version information in its setup file, so it gets the version 0.0.0. Bah. It's an abandoned package anyway (zpt.sf.net), but the up-to-date Zope version hasn't *quite* been separately distributed (though I guess it's possible to build the package from the Zope sources). -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org --- orig/setuptools-0.3a2/easy_install.py 2005-05-29 18:01:42.000000000 -0500 +++ easy_install.py 2005-05-29 20:40:34.734190272 -0500 @@ -464,14 +464,20 @@ def _download_html(self, url, headers, filename): - # Check if it is a subversion index page: + sf_url = url.startswith('http://prdownloads.') file = open(filename) for line in file: if line.strip(): + # Check if it is a subversion index page: if re.search(r'<title>Revision \d+:', line): file.close() os.unlink(filename) return self._download_svn(url, filename) + elif sf_url and re.search(r'^<HTML><HEAD>', line, re.I): + continue + elif sf_url and re.search(r'\s*<TITLE>Select a Mirror for File: ', line): + # Sourceforge mirror page: + return self._download_sourceforge(url, file.read()) else: break # not an index page file.close() @@ -482,8 +488,33 @@ os.system("svn checkout -q %s %s" % (url, filename)) return filename + def _download_sourceforge(self, source_url, sf_page): + """ + Return a (randomly-selected) (scheme URL) for downloading the + package, given the SourceForge mirror-selection page. + """ + import random + import urlparse + import urllib + mirror_regex = re.compile(r'HREF=(/.*?\?use_mirror=[^>]*)') + urls = [m.group(1) for m in mirror_regex.finditer(sf_page)] + if not urls: + raise RuntimeError( + "URL appears to be a Sourceforge mirror page, but no URLs found") + url = urlparse.urljoin(source_url, random.choice(urls)) + f = urllib.urlopen(url) + mirror_page = f.read() + f.close() + match = re.search(r'<META HTTP-EQUIV="refresh" content=".*?URL=(.*?)"', mirror_page) + if not match: + raise RuntimeError( + 'No META HTTP-EQUIV="refresh" found in Sourceforge page at %s' % url) + download_url = match.group(1) + scheme = URL_SCHEME(download_url) + print download_url + return self._download_url(scheme.group(1), download_url) - +

2 2

EasyInstall: svn support
by Ian Bicking May 29, 2005

May 29, 2005

I just tried out EasyInstall, and it seems to work pretty well. Well, I've only tried downloading packages, not really using them, but I'll try that next... I added support to download from subversion repositories. Basically it detects the svn index page (for http repositories, which can't be detected based on the URL), or the svn: URL type, and downloads from there. I'd like for it to fix up the version based on the svn revision, but I haven't looked closely enough at that part yet, or if it's even possible since setup.py typically has a version hardcoded in it. I'm not even sure what the version number should look like, so that it sorts properly with released versions (or even if it should sort with released versions at all; should versions also indicate branches, like stable vs. development?) Anyway, I've attached a diff against 0.3a1 and the compete modified easy_install.py file. -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org --- easy_install.py 2005-05-29 14:30:03.858797032 -0500 +++ orig/setuptools-0.3a1/easy_install.py 2005-05-28 21:36:42.000000000 -0500 @@ -154,7 +154,6 @@ """ import sys, os.path, pkg_resources, re, zipimport -import shutil from pkg_resources import * @@ -220,7 +219,7 @@ if isinstance(spec,Requirement): pass else: - scheme = URL_SCHEME(spec).group(1) + scheme = URL_SCHEME(spec) if scheme: # It's a url, download it to self.tmpdir return self._download_url(scheme, spec) @@ -251,20 +250,19 @@ return self.install_egg(dist_filename,True) # Anything else, try to extract and build - if not os.path.isdir(dist_filename): - import zipfile, tarfile - if zipfile.is_zipfile(dist_filename): - self._extract_zip(dist_filename, self.tmpdir) + import zipfile, tarfile + if zipfile.is_zipfile(dist_filename): + self._extract_zip(dist_filename, self.tmpdir) + else: + import tarfile + try: + tar = tarfile.open(dist_filename) + except tarfile.TarError: + raise RuntimeError( + "Not a valid tar or zip archive: %s" % dist_filename + ) else: - import tarfile - try: - tar = tarfile.open(dist_filename) - except tarfile.TarError: - raise RuntimeError( - "Not a valid tar or zip archive: %s" % dist_filename - ) - else: - self._extract_tar(tar) + self._extract_tar(tar) # Find the setup.py file from glob import glob @@ -374,7 +372,7 @@ destination = os.path.join(self.instdir, os.path.basename(egg_path)) ensure_directory(destination) - if not os.path.exists(destination) or not self.samefile(egg_path, destination): + if not self.samefile(egg_path, destination): if os.path.isdir(destination): shutil.rmtree(destination) elif os.path.isfile(destination): @@ -416,16 +414,15 @@ name = filter(None,urlparse(url)[2].split('/'))[-1] while '..' in name: name = name.replace('..','.').replace('\\','_') - filename = os.path.join(self.tmpdir,name) # Download the file - if scheme == 'svn': - return self._download_svn(scheme, url, filename) from urllib import FancyURLopener, URLopener class opener(FancyURLopener): http_error_default = URLopener.http_error_default try: - filename,headers = opener().retrieve(url, filename) + filename,headers = opener().retrieve( + url,os.path.join(self.tmpdir,name) + ) except IOError,v: if v.args and v.args[0]=='http error': raise RuntimeError( @@ -433,37 +430,10 @@ ) else: raise - # Check if it is a subversion index page: - f = open(filename) - first_line = '' - while not first_line: - first_line = f.readline() - if not first_line: - break - first_line = first_line.strip() - f.close() - if re.search(r'<title>Revision \d+:', first_line): - os.unlink(filename) - return self._download_svn(scheme, url, filename) # and return its filename return filename - def _download_svn(self, scheme, url, filename): - os.system("svn checkout -q %s %s" % (url, filename)) - stdin, stdout = os.popen4("svn info %s" % filename) - result = stdout.read() - match = re.search(r'Last Changed Rev: (\d+)', result) - if not match: - raise RuntimeError( - "svn info error: %s" % result.strip()) - # Actually, this probably doesn't do anything; but somehow this - # information should get put into the version string... - revision = match.group(1) - new_filename = filename + '-svn%s' % revision - shutil.move(filename, new_filename) - return new_filename - - +

2 5

EasyInstall/Egg platforms
by Phillip J. Eby May 29, 2005

May 29, 2005

Currently, Python Eggs and EasyInstall use distutils.get_platform() to tag platform-specific eggs. However, I've heard that this string isn't all that useful on some platforms. (For example, I gather that it's pretty useless info on Mac OS X.) Also, some things might in principle be compatible, even though they have a different platform string. For example, a 'linux-i586' build should in principle be usable on a 'linux-i686' machine, right? Does anybody have more familiarity with the subtleties of binary compatibility, such that we could maybe devise a better platform string approach *before* the existing string is so widely used that it'll be difficult to change?

1 0

EasyInstall: easy-install.pth, active version
by Ian Bicking May 29, 2005

May 29, 2005

Reading through "Changing the Active Version (site-packages installs only)", I'm guessing that the implementation changes easy-install.pth to point to a specific version. But no easy-install.pth is created when you use the --install-dir option. And maybe that's just an oversight... it seems like it easy-install.pth makes just as much sense in a local directory as site-packages...? -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

2 1

EasyInstall - automated download/install/manage Python packages
by Phillip J. Eby May 29, 2005

May 29, 2005

Just a heads-up: http://peak.telecommunity.com/DevCenter/EasyInstall I would like to encourage package authors to try using EasyInstall to download and build the packages they distribute, and to let me know if they encounter any issues. I'd like to make EasyInstall able to build as many Python packages "out of the box" as possible. I've also been updating the Python Eggs documentation: http://peak.telecommunity.com/DevCenter/PythonEggs But there is still a fair amount to write about the runtime API, and as you can see from the "Implementation Status" there are plenty of tasks still "available" with respect to getting the zip-based runtime as robust as it should be.

1 0

non-"standard" compilers...
by Nicholas Bastin May 28, 2005

May 28, 2005

I am in serious need of some help (mental?) with building a distutils-distributed python extension (PyXML). I am on Solaris 8, with the SunPro Forte 10 compiler. I tried the following command line, thinking that it was reasonable: python setup.py build --compiler=unix but the definition of "standard UNIX-style compiler" appears to be 'gcc'. Although I don't quite understand that...the dictionary in unixcompiler.py is initialized to the compiler being 'cc', which would be the Right Thing(tm) on quite a lot of unix platforms, including Solaris. However, by the time the compile command is actually issued on the command line, it's trying to use gcc. I don't follow quite how distutils decides what compiler to use, so I was wondering if someone could point me in the right direction to build with SunPro instead of gcc. For extra credit, and help with my vast amount of mental anguish, someone could try to tell how to build with the Intel C Compiler on windows... -- Nick

6 14

Distutils projects page
by Richard Jones May 26, 2005

May 26, 2005

In an effort to try to understand what projects are out there trying to improve distutils / packaging / make the world a better place, I've put up a wiki page: http://wiki.python.org/moin/DistutilsProjects It contains all the projects I'm aware of. I'm not on the distutils SIG list, so I might have missed some. I did scan the archive for this year, and didn't see anything. Please visit the page and edit it for correctness - or include your project if it's not in there! Richard

2 1

Questions about Python Eggs
by Ian Bicking May 22, 2005

May 22, 2005

So, I'm looking at Python Eggs, and I have some questions... Why does it create a Package.egg-info/ directory? It seems odd; is this meant to replace all the metadata arguments to setup.py? That would be fine, I'm happy to get rid of those arguments, but if it's just a copy of that data it seems odd to install it alongside the distribution files. I think I'm generally going to prefer non-zip installations of Eggs, this way I can apply it to projects that weren't written to use pkg_resource, and people can see the source more easily. By any chance does anyone have code on hand to unpack an egg appropriately? I'm sure it's not hard, but it's not a one-liner I'm trying to cut down on the typing... What's the state of depends? What is depends anyway? I notice the comment on pkg_resources.require aren't very confident ;) It actually doesn't look functional to me, though I haven't tried running it. Is it just meant to raise an ImportError when a requirement isn't met, or can it search some directories for appropriate Egg files? This is something I'm very interested in, so if I have the intention correct I'd like to help move this function along. Anyway ideas about how to apply setuptools to other people's packages? Is it safe to do nasty monkeypatching wherein I replace distutils.setup with setuptools.setup? Or, more directly, anyone have code around to build eggs programmatically from other people's packages? -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

2 7

distutils bug.
by Titus Brown May 21, 2005

May 21, 2005

Hi all, it seems that distutils doesn't recognize '.zip' files as being part of packages, so if I have the following structure, twill/__init__.py twill/pyparsing.zip twill/mechanize.zip twill/shell.py twill/commands.py then packages = ['twill',] doesn't install the .zip files. Quixote has a simple solution that uses the cmdclass option to respecify things for (in their case) *.ptl files, and it works fine for .zip files. I'm pretty sure .zip files should be considered as valid as .py files, by default. Any thoughts? cheers, --titus

2 2