I just tried out EasyInstall, and it seems to work pretty well. Well, I've only tried downloading packages, not really using them, but I'll try that next... I added support to download from subversion repositories. Basically it detects the svn index page (for http repositories, which can't be detected based on the URL), or the svn: URL type, and downloads from there. I'd like for it to fix up the version based on the svn revision, but I haven't looked closely enough at that part yet, or if it's even possible since setup.py typically has a version hardcoded in it. I'm not even sure what the version number should look like, so that it sorts properly with released versions (or even if it should sort with released versions at all; should versions also indicate branches, like stable vs. development?) Anyway, I've attached a diff against 0.3a1 and the compete modified easy_install.py file. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org --- easy_install.py 2005-05-29 14:30:03.858797032 -0500 +++ orig/setuptools-0.3a1/easy_install.py 2005-05-28 21:36:42.000000000 -0500 @@ -154,7 +154,6 @@ """ import sys, os.path, pkg_resources, re, zipimport -import shutil from pkg_resources import * @@ -220,7 +219,7 @@ if isinstance(spec,Requirement): pass else: - scheme = URL_SCHEME(spec).group(1) + scheme = URL_SCHEME(spec) if scheme: # It's a url, download it to self.tmpdir return self._download_url(scheme, spec) @@ -251,20 +250,19 @@ return self.install_egg(dist_filename,True) # Anything else, try to extract and build - if not os.path.isdir(dist_filename): - import zipfile, tarfile - if zipfile.is_zipfile(dist_filename): - self._extract_zip(dist_filename, self.tmpdir) + import zipfile, tarfile + if zipfile.is_zipfile(dist_filename): + self._extract_zip(dist_filename, self.tmpdir) + else: + import tarfile + try: + tar = tarfile.open(dist_filename) + except tarfile.TarError: + raise RuntimeError( + "Not a valid tar or zip archive: %s" % dist_filename + ) else: - import tarfile - try: - tar = tarfile.open(dist_filename) - except tarfile.TarError: - raise RuntimeError( - "Not a valid tar or zip archive: %s" % dist_filename - ) - else: - self._extract_tar(tar) + self._extract_tar(tar) # Find the setup.py file from glob import glob @@ -374,7 +372,7 @@ destination = os.path.join(self.instdir, os.path.basename(egg_path)) ensure_directory(destination) - if not os.path.exists(destination) or not self.samefile(egg_path, destination): + if not self.samefile(egg_path, destination): if os.path.isdir(destination): shutil.rmtree(destination) elif os.path.isfile(destination): @@ -416,16 +414,15 @@ name = filter(None,urlparse(url)[2].split('/'))[-1] while '..' in name: name = name.replace('..','.').replace('\\','_') - filename = os.path.join(self.tmpdir,name) # Download the file - if scheme == 'svn': - return self._download_svn(scheme, url, filename) from urllib import FancyURLopener, URLopener class opener(FancyURLopener): http_error_default = URLopener.http_error_default try: - filename,headers = opener().retrieve(url, filename) + filename,headers = opener().retrieve( + url,os.path.join(self.tmpdir,name) + ) except IOError,v: if v.args and v.args[0]=='http error': raise RuntimeError( @@ -433,37 +430,10 @@ ) else: raise - # Check if it is a subversion index page: - f = open(filename) - first_line = '' - while not first_line: - first_line = f.readline() - if not first_line: - break - first_line = first_line.strip() - f.close() - if re.search(r'<title>Revision \d+:', first_line): - os.unlink(filename) - return self._download_svn(scheme, url, filename) # and return its filename return filename - def _download_svn(self, scheme, url, filename): - os.system("svn checkout -q %s %s" % (url, filename)) - stdin, stdout = os.popen4("svn info %s" % filename) - result = stdout.read() - match = re.search(r'Last Changed Rev: (\d+)', result) - if not match: - raise RuntimeError( - "svn info error: %s" % result.strip()) - # Actually, this probably doesn't do anything; but somehow this - # information should get put into the version string... - revision = match.group(1) - new_filename = filename + '-svn%s' % revision - shutil.move(filename, new_filename) - return new_filename - - +
At 02:46 PM 5/29/2005 -0500, Ian Bicking wrote:
I just tried out EasyInstall, and it seems to work pretty well. Well, I've only tried downloading packages, not really using them, but I'll try that next...
I added support to download from subversion repositories.
Thanks! I was going to suggest you might want to look at contributing that. :)
Basically it detects the svn index page (for http repositories, which can't be detected based on the URL), or the svn: URL type, and downloads from there. I'd like for it to fix up the version based on the svn revision, but I haven't looked closely enough at that part yet, or if it's even possible since setup.py typically has a version hardcoded in it. I'm not even sure what the version number should look like, so that it sorts properly with released versions (or even if it should sort with released versions at all; should versions also indicate branches, like stable vs. development?)
Yeah, I think sticking with the setup.py version is best; EasyInstall and pkg_resources use the generated PKG-INFO to find out stuff like that.
Anyway, I've attached a diff against 0.3a1 and the compete modified easy_install.py file.
I was a bit puzzled by the diff at first; looks like you created a reversed diff.
--- easy_install.py 2005-05-29 14:30:03.858797032 -0500 +++ orig/setuptools-0.3a1/easy_install.py 2005-05-28 21:36:42.000000000 -0500 @@ -154,7 +154,6 @@ """
import sys, os.path, pkg_resources, re, zipimport -import shutil from pkg_resources import *
@@ -220,7 +219,7 @@ if isinstance(spec,Requirement): pass else: - scheme = URL_SCHEME(spec).group(1)
This won't work with non-URL specs. Note that spec is allowed to be a local filename, or a version requirement (e.g. "FooPackage==1.2"). Anyway, I moved the group() to the _download_url() call, where it should accomplish the intent here. As for this next bit below, you're just skipping the archive checks if the source is a directory, right? There aren't any changes to the archive handling parts?
# Anything else, try to extract and build - if not os.path.isdir(dist_filename): - import zipfile, tarfile - if zipfile.is_zipfile(dist_filename): - self._extract_zip(dist_filename, self.tmpdir) + import zipfile, tarfile + if zipfile.is_zipfile(dist_filename): + self._extract_zip(dist_filename, self.tmpdir) + else: + import tarfile + try: + tar = tarfile.open(dist_filename) + except tarfile.TarError: + raise RuntimeError( + "Not a valid tar or zip archive: %s" % dist_filename + ) else: - import tarfile - try: - tar = tarfile.open(dist_filename) - except tarfile.TarError: - raise RuntimeError( - "Not a valid tar or zip archive: %s" % dist_filename - ) - else: - self._extract_tar(tar) + self._extract_tar(tar)
# Find the setup.py file from glob import glob @@ -374,7 +372,7 @@ destination = os.path.join(self.instdir, os.path.basename(egg_path)) ensure_directory(destination)
- if not os.path.exists(destination) or not self.samefile(egg_path, destination): + if not self.samefile(egg_path, destination):
I don't understand what this change is for. Since install_egg is only called with an existing .egg file or directory, this means that if the destination doesn't exist, it can't be the same file. So, the addition here seems redundant.
- # Check if it is a subversion index page:
Before doing this, I think it would be a good idea to check the headers for a text/html Content-Type, so as not to be readline() a big chunk of binary. :)
- def _download_svn(self, scheme, url, filename): - os.system("svn checkout -q %s %s" % (url, filename))
Why checkout instead of export? Oh, I see, it's so you can use svn info later. Never mind.
- # Actually, this probably doesn't do anything; but somehow this - # information should get put into the version string...
It indeed doesn't do anything; you'd be better off munging the setup.py to change the setup version. Maybe what I can do is have the build stuff look for a .svn dir in the build area, and if one is present, grab the revision and add it to the end of the version string supplied to setup(). Or simpler still, I could add a '--tag-svn-revision' option to bdist_egg, that would make *it* do that. I've been thinking it would be nice to have a '--tag-build=NUMBER' or '--tag-date' option for bdist_egg anyway, to support daily builds and such. So adding an option to "get the tag value from 'svn info'" would just be a minor variation on the theme. Anyway, do you have some handy svn: and http: subversion URLs that I could play with for testing?
Phillip J. Eby wrote:
Basically it detects the svn index page (for http repositories, which can't be detected based on the URL), or the svn: URL type, and downloads from there. I'd like for it to fix up the version based on the svn revision, but I haven't looked closely enough at that part yet, or if it's even possible since setup.py typically has a version hardcoded in it. I'm not even sure what the version number should look like, so that it sorts properly with released versions (or even if it should sort with released versions at all; should versions also indicate branches, like stable vs. development?)
Yeah, I think sticking with the setup.py version is best; EasyInstall and pkg_resources use the generated PKG-INFO to find out stuff like that.
Well, at least for my own packages I'll try to make the version better, by dynamically generating it in setup.py; I tried to do this a little with Paste, but there's some improvement to be done there as well. What form should it take? E.g., if the last revision was 0.1, what should the trunk version be? 0.2alpha-svn2822? Does the released version need to be 0.2final then? Or do are version numbers sorted based on a better algorithm than string comparison?
Anyway, I've attached a diff against 0.3a1 and the compete modified easy_install.py file.
I was a bit puzzled by the diff at first; looks like you created a reversed diff.
Oops. I have a hard time with diff for some reason. Like some kind of chronological dyslexia.
--- easy_install.py 2005-05-29 14:30:03.858797032 -0500 +++ orig/setuptools-0.3a1/easy_install.py 2005-05-28 21:36:42.000000000 -0500 @@ -154,7 +154,6 @@ """
import sys, os.path, pkg_resources, re, zipimport -import shutil from pkg_resources import *
@@ -220,7 +219,7 @@ if isinstance(spec,Requirement): pass else: - scheme = URL_SCHEME(spec).group(1)
This won't work with non-URL specs. Note that spec is allowed to be a local filename, or a version requirement (e.g. "FooPackage==1.2"). Anyway, I moved the group() to the _download_url() call, where it should accomplish the intent here.
OK. Or it could be: scheme = URL_SCHEME(spec) scheme = scheme and scheme.group(1) It seems like a match object is a little odd to pass around.
As for this next bit below, you're just skipping the archive checks if the source is a directory, right? There aren't any changes to the archive handling parts?
Yes, that's all -- since svn co creates a directory, it doesn't need to be unpacked.
- if not os.path.exists(destination) or not self.samefile(egg_path, destination): + if not self.samefile(egg_path, destination):
I don't understand what this change is for. Since install_egg is only called with an existing .egg file or directory, this means that if the destination doesn't exist, it can't be the same file. So, the addition here seems redundant.
I was getting an error when calling self.samefile(filename, filename_that_doesnt_exist). Maybe self.samefile needs to check os.path.exists(destination).
- # Check if it is a subversion index page:
Before doing this, I think it would be a good idea to check the headers for a text/html Content-Type, so as not to be readline() a big chunk of binary. :)
Ah, yes, indeed.
- def _download_svn(self, scheme, url, filename): - os.system("svn checkout -q %s %s" % (url, filename))
Why checkout instead of export? Oh, I see, it's so you can use svn info later. Never mind.
Yeah, not a big reason. Though it would be nice if, based on some option, it could be an "svn up" instead of a checkout, if there was some cache of checkouts. And annoyingly, you can't use svn info on a remote repository; the only way I've figured to get the last changed revision of a remote repository is with "svn ls -v parent_dir".
- # Actually, this probably doesn't do anything; but somehow this - # information should get put into the version string...
It indeed doesn't do anything; you'd be better off munging the setup.py to change the setup version.
Maybe what I can do is have the build stuff look for a .svn dir in the build area, and if one is present, grab the revision and add it to the end of the version string supplied to setup(). Or simpler still, I could add a '--tag-svn-revision' option to bdist_egg, that would make *it* do that. I've been thinking it would be nice to have a '--tag-build=NUMBER' or '--tag-date' option for bdist_egg anyway, to support daily builds and such. So adding an option to "get the tag value from 'svn info'" would just be a minor variation on the theme.
Yes, that would work well.
Anyway, do you have some handy svn: and http: subversion URLs that I could play with for testing?
svn://colorstudy.com/trunk/SQLObject and http://svn.colorstudy.com/trunk/SQLObject should both be workable, and point to the same location. It should really look for a scheme of 'svn+ssh' as well, but those are a pain to make readable ;) And technically file: should work for svn too, which I suppose you could detect because it points to a directory with a db/fs-type file. Eh, those can wait. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org
At 04:26 PM 5/29/2005 -0500, Ian Bicking wrote:
Phillip J. Eby wrote:
Basically it detects the svn index page (for http repositories, which can't be detected based on the URL), or the svn: URL type, and downloads from there. I'd like for it to fix up the version based on the svn revision, but I haven't looked closely enough at that part yet, or if it's even possible since setup.py typically has a version hardcoded in it. I'm not even sure what the version number should look like, so that it sorts properly with released versions (or even if it should sort with released versions at all; should versions also indicate branches, like stable vs. development?)
Yeah, I think sticking with the setup.py version is best; EasyInstall and pkg_resources use the generated PKG-INFO to find out stuff like that.
Well, at least for my own packages I'll try to make the version better, by dynamically generating it in setup.py;
Note that it needs to be robust if the build machine doesn't have svn.
I tried to do this a little with Paste, but there's some improvement to be done there as well. What form should it take?
E.g., if the last revision was 0.1, what should the trunk version be? 0.2alpha-svn2822? Does the released version need to be 0.2final then? Or do are version numbers sorted based on a better algorithm than string comparison?
See pkg_resources.parse_version() for the version translator. But since the algorithm is a bit obscure, here are the basic ideas: * The string is converted to a tuple of strings * Runs of numbers are split from non-numeric runs * The string "*final" is added to the end of the tuple * Dots are not kept in the string; only '-' and non-numerics * Certain alpha strings are tweaked; pre, rc, and so on. * Numeric runs are zero-padded for numeric comparision * Elements that equal zero are dropped from each tuple-level numeric run In other words, the revision system knows that '2.0' and '2.0.0' are equivalent version numbers. It knows that alpha, beta, and candidate releases come *before* final releases, but that 'nb', 'pl' releases are *after*, as are '-' releases. So, it knows that '2.0-1' is an "older" release than '2.0.1'. Thus, the best and simplest way to add build/date/revision tags is just to pop a '-' and the extra data. At this point, I've added '--tag-date', '--tag-build=TAG' and '--tag-svn-revision' options to bdist_egg that can be used individually or in combination. If you did something like: setup.py bdist_egg --tag-build=svn --tag-date --tag-svn-revision then today you'd get an egg with a revision like '0.2a-svn-2822-20050529'. In practice, I suspect most people will use either tag-date or tag-svn-revision, because using both is kind of redundant unless the package is also integrating external components.
I was a bit puzzled by the diff at first; looks like you created a reversed diff.
Oops. I have a hard time with diff for some reason. Like some kind of chronological dyslexia.
difflexia, perhaps? :) Actually, difflexia sounds like a problem *reading* diffs. Ah well.
It seems like a match object is a little odd to pass around.
That's because it was an error. It should've read 'self._download_url(scheme.group(1),...' instead.
I was getting an error when calling self.samefile(filename, filename_that_doesnt_exist). Maybe self.samefile needs to check os.path.exists(destination).
Aha. That's a platform-specific issue that I'll need to fix. Windows doesn't have 'os.samefile', so I didn't see the problem. I'll fix self.samefile() to check whether both paths exist before using os.samefile.
Before doing this, I think it would be a good idea to check the headers for a text/html Content-Type, so as not to be readline() a big chunk of binary. :)
Ah, yes, indeed.
I've done this in my version now.
- # Actually, this probably doesn't do anything; but somehow this - # information should get put into the version string...
It indeed doesn't do anything; you'd be better off munging the setup.py to change the setup version. Maybe what I can do is have the build stuff look for a .svn dir in the build area, and if one is present, grab the revision and add it to the end of the version string supplied to setup(). Or simpler still, I could add a '--tag-svn-revision' option to bdist_egg, that would make *it* do that. I've been thinking it would be nice to have a '--tag-build=NUMBER' or '--tag-date' option for bdist_egg anyway, to support daily builds and such. So adding an option to "get the tag value from 'svn info'" would just be a minor variation on the theme.
Yes, that would work well.
I've decided I don't want this to happen automatically for EasyInstall egg builds, though. If I later let you pass build options through to the setup.py, you'll be able to do it that way. I just don't think it should invoke --tag-svn-revision by default. For one thing, I don't have a way to know if subversion is installed, and you could've downloaded a source .zip with .svn dirs accidentally included. (I've seen this from time to time with CVS dirs.)
It should really look for a scheme of 'svn+ssh' as well, but those are a pain to make readable ;)
I'm actually checking for 'scheme=="svn" or scheme.startswith("svn+")', so they ought to work too.
And technically file: should work for svn too, which I suppose you could detect because it points to a directory with a db/fs-type file. Eh, those can wait.
I'd need more details before I could implement that. In the meantime, I'm going to go ahead and build/upload an EasyInstall/setuptools-0.3a2 release with the new subversion support, revision tagging, and bug fixes. Thanks for your help!
Phillip J. Eby wrote:
Well, at least for my own packages I'll try to make the version better, by dynamically generating it in setup.py;
Note that it needs to be robust if the build machine doesn't have svn.
Yeah... I'm not sure what that needs to do. For release branches I'll comment or otherwise disable that code, since the whole point of a release is to avoid svn-anything. If it's not a release, then I'm not sure how they'd acquire it without svn. If I set up something to get a snapshot, then I'll have to be sure to add that svn metadata somewhere to the snapshot, and that will make the svn access unnecessary.
It indeed doesn't do anything; you'd be better off munging the setup.py to change the setup version. Maybe what I can do is have the build stuff look for a .svn dir in the build area, and if one is present, grab the revision and add it to the end of the version string supplied to setup(). Or simpler still, I could add a '--tag-svn-revision' option to bdist_egg, that would make *it* do that. I've been thinking it would be nice to have a '--tag-build=NUMBER' or '--tag-date' option for bdist_egg anyway, to support daily builds and such. So adding an option to "get the tag value from 'svn info'" would just be a minor variation on the theme.
Yes, that would work well.
I've decided I don't want this to happen automatically for EasyInstall egg builds, though. If I later let you pass build options through to the setup.py, you'll be able to do it that way. I just don't think it should invoke --tag-svn-revision by default. For one thing, I don't have a way to know if subversion is installed, and you could've downloaded a source .zip with .svn dirs accidentally included. (I've seen this from time to time with CVS dirs.)
Well, one way is to have _download_svn() put that metadata into the downloaded files -- some special file -- to later be read by setuptools. That would fit into a downloadable snapshot as well. And obviously _download_svn() can't work without svn installed. It should also be reasonably easy to translate to other version control systems, insofar as they have useful revision numbers. Though I don't think many of the distributed systems do :( Unless you are okay with *really* long versions that essentially embed a UUID -- but I don't encounter any of those repositories, so it's not a problem for me yet.
And technically file: should work for svn too, which I suppose you could detect because it points to a directory with a db/fs-type file. Eh, those can wait.
I'd need more details before I could implement that.
It's pretty low priority. I can look into it at some point, but it's only useful for the uncommon case where people are doing development on their own machine from a private repository, and want to test the process.
In the meantime, I'm going to go ahead and build/upload an EasyInstall/setuptools-0.3a2 release with the new subversion support, revision tagging, and bug fixes. Thanks for your help!
Cool, I'll clear out the old one and start again with the new. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org
At 07:40 PM 5/29/2005 -0500, Ian Bicking wrote:
Well, one way is to have _download_svn() put that metadata into the downloaded files -- some special file -- to later be read by setuptools. That would fit into a downloadable snapshot as well. And obviously _download_svn() can't work without svn installed. It should also be reasonably easy to translate to other version control systems, insofar as they have useful revision numbers. Though I don't think many of the distributed systems do :( Unless you are okay with *really* long versions that essentially embed a UUID -- but I don't encounter any of those repositories, so it's not a problem for me yet.
I think this is all YAGNI; I'd like to encourage people to distribute ready-to-use eggs for their users. If you're involved with a project that has active development and you need to build-to-rev, I think it suffices to allow arguments to be passed through to the setup script (like --tag-date and --tag-svn-revision). Really, though, I don't see why you couldn't just set up your svn repository to request building a new egg (and/or source distribution, appropriately tagged) whenever a commit takes place under the project tree.
participants (2)
-
Ian Bicking -
Phillip J. Eby