Re: [Distutils] New code snapshot
[cc'd to the distutils-sig: Corran and I have exchanged private email about Distutils on the Mac in the past; in fact, all the Mac-supporting code currently in the Distutils is thanks to Corran. I think it's worth violating netiquette a bit to inform the rest of the sig that Corran is working on Distutils-on-Mac support, to see if there are any other Mac people lurking out there, wishing Distutils worked on Mac OS. Please speak up!] On 31 July 2000, Corran Webster said:
I've finally gotten back to a position where I can spend a little time looking at distutils for the macintosh again. I took this snapshot an dtried a simple install as a first step to getting things going again.
Excellent!
There were a few bugs, one of which is serious, and I don't know what the solution is.
Firstly an easy bug: on line 80 of sysconfig.py, "platform_specific" needs to be changed to "plat_specific".
Yep, too easy. Fixed -- thanks.
The major bug has to do with macintosh paths. The notation to indicate that you move up one level in the directory heirarchy on Mac OS is a double colon, ie.
dir1:dir2::dir3:file
is the same as "dir1:dir3:file" in the same way that "dir1/dir2/../dir3/file" is the same as "dir1/dir3/file" on posix systems. This is important, because sometimes macintosh paths are specified with a trailing ":" if it is a directory, and sometimes they are not. For example, on my system,
Interesting. "Trailing slash means directory" is a (weak) convention on Unix as well, but I've avoided it in the Distutils because it breaks on Windows. (In particular: if os.path.isdir("\\foo\\bar") is true, os.path.isdir("\\foo\\bar\\") is not. ISTR that this may be fixed in the current CVS Python, but don't remember for sure.)
import sys sys.prefix 'Macintosh HD:Applications:Python 1.5.2c1:' sys.path ['Macintosh HD:Applications:Python 1.5.2c1:', 'Macintosh HD:Applications:Python 1.5.2c1:', 'Macintosh HD:Applications:Python 1.5.2c1:Lib', 'Macintosh HD:Applications:Python 1.5.2c1:Lib:site-packages', 'Macintosh HD:Applications:Python 1.5.2c1:Lib:lib-tk', ... etc ]
Well, one thing I learn from this is that you really ought to upgrade to 1.5.2final. ;-)
Notice how sometimes the paths have trailing ":" and sometimes they don't.
I guess the convention is a weak one in Mac OS too.
While os.path.join does the right thing no matter how the paths end, but os.path.normpath does not do anything, one way or the other. This leads to two problems:
1. with the install schemes of the install.py module, you get something like:
"$base:Libs" expanding to "Macintosh HD:Applications:Python 1.5.2c1::Libs"
which is the same as "Macintosh HD:Applications:Libs", ie. very much the wrong place.
Ahh, I see. You can be lazy on Unix, because "foo//bar" is the same as "foo/bar" (except in Emacs!). If I understand correctly, on Mac OS it's as though "foo//bar" was the same as "foo/../bar", which makes life a bit harder. I try hard not to rely on the collapsibility of "//" though -- just seems a bit dodgy (and kernel-dependent). Note that on Unix:
os.path.normpath("foo//bar") 'foo/bar' os.path.normpath("foo//bar/") 'foo/bar'
which argues in favour of fixing normpath to strip trailing colons on Mac OS.
2. I hacked a fix for 1 (see below), but this led to a second problem: just because you do os.path.normpath, you still can't assume that two different paths aren't really the same. The result of this was having install.py tell me that distutils wasn't installed somewhere in sys.path, when in fact it was. This may have other consequences where distutils compares paths to each other.
Ick! The whole point of normpath, as I see it, is to guarantee that string equality means filesystem equality. If it doesn't do that on Mac OS, then it's broken. Offhand, I can't think of other places where Distutils compares paths. Maybe the code that excludes setup.py from the modules to install -- that's in build_py.py; see the 'find_package_modules()' method.
(a) The simple hack I used to get around problem 1 was simply to expand the variable substitution system to accept "${base}Lib". The problem with this is that we can't guarantee that base will end with ":", so we could still get mangled pathnames.
Yup.
(b) A more robust solution is probebly to write a normpath for distutils which guarantees that pathnames either do or do not end in a ":" on the mac. Not sure the best place to put it, or how much code this is going to affect, but it could be a real pain to get right.
(c) Lobby for normpath to be "fixed" on the mac - almost certainly doable, but leaves the problem of 1.5.* versions not working properly.
I think we can do both (b) and (c). Eg. somewhere we'd say: if python < 1.6 and Mac OS: # maybe 2.0 def normpath (...): ... os.path.normpath = normpath and hopefully the "Distutils-specific" normpath would be right there in macpath.py in 2.0 (maybe 1.6). Can you whip up a patch to macpath.py and submit it to the SourceForge patch manager? (Or just mail it to me if you don't want to deal with SF.) Make sure you add a test too!
(d) Some change to the install scheme mechanism so that it uses os.path.join(). For example, use a list or tuple of strings, rather than a single string:
'mac': { 'purelib': ('$base', 'Lib'), 'platlib': ('$base', 'Mac', 'PlugIns'), 'headers': ('$base', 'Include', '$dist_name'), 'scripts': ('$base', 'Scripts'), 'data' : ('$base',), }
I'm not keen on that -- getting that stuff working was fairly tricky (although not nearly as tricky as designing it!), and I really don't want to revisit it. Greg -- Greg Ward - Unix bigot gward@python.net http://starship.python.net/~gward/ I'd like some JUNK FOOD ... and then I want to be ALONE --
At 10:03 PM -0400 1/8/00, Greg Ward wrote:
[cc'd to the distutils-sig: Corran and I have exchanged private email about Distutils on the Mac in the past; in fact, all the Mac-supporting code currently in the Distutils is thanks to Corran. I think it's worth violating netiquette a bit to inform the rest of the sig that Corran is working on Distutils-on-Mac support, to see if there are any other Mac people lurking out there, wishing Distutils worked on Mac OS. Please speak up!]
I agree that this should probably be discussed here... indeed any additional insight would be useful at this point. Unfortunately, I just did some mucking around with os and os.path on the mac, and things are going to be very ugly I fear...
The major bug has to do with macintosh paths. The notation to indicate that you move up one level in the directory heirarchy on Mac OS is a double colon, ie.
dir1:dir2::dir3:file
is the same as "dir1:dir3:file" in the same way that "dir1/dir2/../dir3/file" is the same as "dir1/dir3/file" on posix systems. This is important, because sometimes macintosh paths are specified with a trailing ":" if it is a directory, and sometimes they are not. For example, on my system,
Interesting. "Trailing slash means directory" is a (weak) convention on Unix as well, but I've avoided it in the Distutils because it breaks on Windows. (In particular: if os.path.isdir("\\foo\\bar") is true, os.path.isdir("\\foo\\bar\\") is not. ISTR that this may be fixed in the current CVS Python, but don't remember for sure.)
Unfortunately, it seems like sometimes the trailing ":" is required. To explain why, I need to explain a little bit more about mac pathnames. On the mac, a path starting with a colon is relative, while one which doesn't start with a colon is absolute (ie. the reverse of the posix convention), _unless_ it's just a single name (ie. no colons whatsoever) in which case it's relative. The end result of this is that "Macintosh HD" refers to a file in the local directory called "Macintosh HD" (as does ":Macintosh HD"), while "Macintosh HD:" refers to the root directory of the hard disk "Macintosh HD". However, "Macintosh HD:Applications:" and "Macintosh HD:Applications" are the same thing as each other. Similarly ":Macintosh HD:Applications:" and ":Macintosh HD:Applications" are the same as each other. This seems to be a system-level convention (ie. we aren't going to be able to change it). Observe also:
os.path.join("Macintosh HD", "Applications") ':Macintosh HD:Applications' os.path.join("Macintosh HD:", "Applications") 'Macintosh HD:Applications'
There's a similar problem with ":", "::", etc. Removing a trailing ":" in these cases directly affects the meaning.
import sys sys.prefix 'Macintosh HD:Applications:Python 1.5.2c1:' sys.path ['Macintosh HD:Applications:Python 1.5.2c1:', 'Macintosh HD:Applications:Python 1.5.2c1:', 'Macintosh HD:Applications:Python 1.5.2c1:Lib', 'Macintosh HD:Applications:Python 1.5.2c1:Lib:site-packages', 'Macintosh HD:Applications:Python 1.5.2c1:Lib:lib-tk', ... etc ]
Well, one thing I learn from this is that you really ought to upgrade to 1.5.2final. ;-)
1.5.2c1 _is_ 1.5.2final on the mac ;)
Notice how sometimes the paths have trailing ":" and sometimes they don't.
I guess the convention is a weak one in Mac OS too.
Yes and no :(
Note that on Unix:
os.path.normpath("foo//bar") 'foo/bar' os.path.normpath("foo//bar/") 'foo/bar'
which argues in favour of fixing normpath to strip trailing colons on Mac OS.
Having looked at this, I can see why Mac normpath can't just strip trailing colons. However, the cases where we have to leave a trailing colon are fairly easily identified (ie. paths which have just one ":" and it's at the end should have the trailing colon left intact, as should paths which are entirely colons...) Another solution would be to leave trailing ":" on known directories, but this would fail for working with directory paths prior to their actual creation.
2. I hacked a fix for 1 (see below), but this led to a second problem: just because you do os.path.normpath, you still can't assume that two different paths aren't really the same. The result of this was having install.py tell me that distutils wasn't installed somewhere in sys.path, when in fact it was. This may have other consequences where distutils compares paths to each other.
Ick! The whole point of normpath, as I see it, is to guarantee that string equality means filesystem equality. If it doesn't do that on Mac OS, then it's broken.
I have to agree with this.
Offhand, I can't think of other places where Distutils compares paths. Maybe the code that excludes setup.py from the modules to install -- that's in build_py.py; see the 'find_package_modules()' method.
OK. Hopefully a suitable replacement for normpath will do the trick - just grep for "import os" and patch it in...
(b) A more robust solution is probebly to write a normpath for distutils which guarantees that pathnames either do or do not end in a ":" on the mac. Not sure the best place to put it, or how much code this is going to affect, but it could be a real pain to get right.
(c) Lobby for normpath to be "fixed" on the mac - almost certainly doable, but leaves the problem of 1.5.* versions not working properly.
I think we can do both (b) and (c). Eg. somewhere we'd say:
if python < 1.6 and Mac OS: # maybe 2.0 def normpath (...): ... os.path.normpath = normpath
and hopefully the "Distutils-specific" normpath would be right there in macpath.py in 2.0 (maybe 1.6).
Can you whip up a patch to macpath.py and submit it to the SourceForge patch manager? (Or just mail it to me if you don't want to deal with SF.) Make sure you add a test too!
I think I have a suitable replacement function (see the attachment normpath.py) and test suite. I'll patch it into macpath.py and see if I can submit it to sourceforge (I assume that macpath.py is part of the stuff archived there, rather than the macpython CVS archive - if not I'll forward stuff to Jack Jansen). Even with this improvement, we're not out of the woods. This fixes problem 2, but we'll still need to worry about substituting correctly in the cases where we have a trailing ":". At least these shouldn't be too hard to detect - I think it's just install.py that needs to take this into account, since os.path.join() is used everywhere else, right? I'll fiddle with this in the next day or two.
(d) Some change to the install scheme mechanism so that it uses os.path.join(). For example, use a list or tuple of strings, rather than a single string:
'mac': { 'purelib': ('$base', 'Lib'), 'platlib': ('$base', 'Mac', 'PlugIns'), 'headers': ('$base', 'Include', '$dist_name'), 'scripts': ('$base', 'Scripts'), 'data' : ('$base',), }
I'm not keen on that -- getting that stuff working was fairly tricky (although not nearly as tricky as designing it!), and I really don't want to revisit it.
Understandably. I think a solution is in sight. Regards, Corran
On 01 August 2000, Corran Webster said:
Unfortunately, it seems like sometimes the trailing ":" is required. To explain why, I need to explain a little bit more about mac pathnames. On the mac, a path starting with a colon is relative, while one which doesn't start with a colon is absolute (ie. the reverse of the posix convention), _unless_ it's just a single name (ie. no colons whatsoever) in which case it's relative. The end result of this is that "Macintosh HD" refers to a file in the local directory called "Macintosh HD" (as does ":Macintosh HD"), while "Macintosh HD:" refers to the root directory of the hard disk "Macintosh HD".
However, "Macintosh HD:Applications:" and "Macintosh HD:Applications" are the same thing as each other. Similarly ":Macintosh HD:Applications:" and ":Macintosh HD:Applications" are the same as each other.
Ooh, tricky! Now I can see why Mac OS doesn't have a command-line interface: if users can't understand the difference between "/foo" and "foo", how on earth will they understand that "foo" == ":foo" != "foo:", while "foo:bar" == ":foo:bar" (but "foo:bar" == "foo:bar:" and ":foo:bar" == ":foo:bar:"). Aieee! Doesn't quite rival VMS for complexity, but it's getting there.
There's a similar problem with ":", "::", etc. Removing a trailing ":" in these cases directly affects the meaning. [...] Having looked at this, I can see why Mac normpath can't just strip trailing colons. However, the cases where we have to leave a trailing colon are fairly easily identified (ie. paths which have just one ":" and it's at the end should have the trailing colon left intact, as should paths which are entirely colons...)
Agreed. Sounds fairly straightforward to implement Mac OS' rules. I've glanced at the patch you mailed me privately and will comment in more detail when I respond to that.
OK. Hopefully a suitable replacement for normpath will do the trick - just grep for "import os" and patch it in...
Changed my mind on that: any Distutils code that uses normpath should use the Distutils version, which will be identical to os.path.normpath except for OS/Python combinations where normpath is broken (specifically, Python 1.5 on Mac OS). Details when I respond to your patch.
Even with this improvement, we're not out of the woods. This fixes problem 2, but we'll still need to worry about substituting correctly in the cases where we have a trailing ":". At least these shouldn't be too hard to detect - I think it's just install.py that needs to take this into account, since os.path.join() is used everywhere else, right?
Ugh. Head hurts. Have to think about this more once normpath is working. Greg -- Greg Ward - Linux nerd gward@python.net http://starship.python.net/~gward/ ... I think I'd better go back to my DESK and toy with a few common MISAPPREHENSIONS ...
At 9:30 PM -0400 2/8/00, Greg Ward wrote:
On 01 August 2000, Corran Webster said:
Unfortunately, it seems like sometimes the trailing ":" is required. To explain why, I need to explain a little bit more about mac pathnames. On the mac, a path starting with a colon is relative, while one which doesn't start with a colon is absolute (ie. the reverse of the posix convention), _unless_ it's just a single name (ie. no colons whatsoever) in which case it's relative. The end result of this is that "Macintosh HD" refers to a file in the local directory called "Macintosh HD" (as does ":Macintosh HD"), while "Macintosh HD:" refers to the root directory of the hard disk "Macintosh HD".
However, "Macintosh HD:Applications:" and "Macintosh HD:Applications" are the same thing as each other. Similarly ":Macintosh HD:Applications:" and ":Macintosh HD:Applications" are the same as each other.
Ooh, tricky! Now I can see why Mac OS doesn't have a command-line interface: if users can't understand the difference between "/foo" and "foo", how on earth will they understand that "foo" == ":foo" != "foo:", while "foo:bar" == ":foo:bar" (but "foo:bar" == "foo:bar:" and ":foo:bar" == ":foo:bar:"). Aieee! Doesn't quite rival VMS for complexity, but it's getting there.
Actually it's ":foo" == "foo" != "foo:" but ":foo:bar" != "foo:bar" == "foo:bar:"
There's a similar problem with ":", "::", etc. Removing a trailing ":" in these cases directly affects the meaning. [...] Having looked at this, I can see why Mac normpath can't just strip trailing colons. However, the cases where we have to leave a trailing colon are fairly easily identified (ie. paths which have just one ":" and it's at the end should have the trailing colon left intact, as should paths which are entirely colons...)
Agreed. Sounds fairly straightforward to implement Mac OS' rules. I've glanced at the patch you mailed me privately and will comment in more detail when I respond to that.
For the record, I posted to the MacPython list about this, and it looks like people are happy with the new version of os.path.normpath for the mac, so hopefully that'll be in the CVS tree in a day or two (I'll send it to Jack Jansen to add it to the tree). Regards, Corran
OK, let's see if I understand.
":foo" == "foo" != "foo:" rel rel abs
but
":foo:bar" != "foo:bar" == "foo:bar:" rel abs abs
Right? And the oddball is "foo", which is special-cased to be relative. I think I get it now...
For the record, I posted to the MacPython list about this, and it looks like people are happy with the new version of os.path.normpath for the mac, so hopefully that'll be in the CVS tree in a day or two (I'll send it to Jack Jansen to add it to the tree).
Cool. I wonder if this should be committed on the 1.6 branch as well, since it's "just a bug fix". Greg -- Greg Ward - just another /P(erl|ython)/ hacker gward@python.net http://starship.python.net/~gward/ Know thyself. If you need help, call the CIA.
At 9:33 PM -0400 3/8/00, Greg Ward wrote:
OK, let's see if I understand.
":foo" == "foo" != "foo:" rel rel abs
but
":foo:bar" != "foo:bar" == "foo:bar:" rel abs abs
Right? And the oddball is "foo", which is special-cased to be relative. I think I get it now...
Spot-on.
For the record, I posted to the MacPython list about this, and it looks like people are happy with the new version of os.path.normpath for the mac, so hopefully that'll be in the CVS tree in a day or two (I'll send it to Jack Jansen to add it to the tree).
Cool. I wonder if this should be committed on the 1.6 branch as well, since it's "just a bug fix".
I think that's the plan. I have no idea what the situation is for MacPython 1.6 vs. 2.0 at the moment, particularly since Jack Jansen is at CWI. I imagine that there will be a 1.6 release, since a lot of stuff has been added in 1.6 alpha (threading, particularly). Regards, Corran
participants (2)
-
Corran Webster
-
Greg Ward