[Distutils] New code snapshot

[Corran Webster]

--============_-1246924501==_============
Content-Type: text/plain; charset="us-ascii"

At 10:03 PM -0400 1/8/00, Greg Ward wrote:
> [cc'd to the distutils-sig: Corran and I have exchanged private email
> about Distutils on the Mac in the past; in fact, all the Mac-supporting
> code currently in the Distutils is thanks to Corran.  I think it's worth
> violating netiquette a bit to inform the rest of the sig that Corran is
> working on Distutils-on-Mac support, to see if there are any other Mac
> people lurking out there, wishing Distutils worked on Mac OS.  Please
> speak up!]

I agree that this should probably be discussed here... indeed any
additional insight would be useful at this point.

Unfortunately, I just did some mucking around with os and os.path on the
mac, and things are going to be very ugly I fear...

> > The major bug has to do with macintosh paths.  The notation to indicate
> > that you move up one level in the directory heirarchy on Mac OS is a double
> > colon, ie.
> >
> > dir1:dir2::dir3:file
> >
> > is the same as "dir1:dir3:file" in the same way that
> > "dir1/dir2/../dir3/file" is the same as "dir1/dir3/file" on posix systems.
> > This is important, because sometimes macintosh paths are specified with a
> > trailing ":" if it is a directory, and sometimes they are not.  For
> > example, on my system,
>
> Interesting.  "Trailing slash means directory" is a (weak) convention on
> Unix as well, but I've avoided it in the Distutils because it breaks on
> Windows.  (In particular: if os.path.isdir("\\foo\\bar") is true,
> os.path.isdir("\\foo\\bar\\") is not.  ISTR that this may be fixed in
> the current CVS Python, but don't remember for sure.)

Unfortunately, it seems like sometimes the trailing ":" is required.  To
explain why, I need to explain a little bit more about mac pathnames.  On
the mac, a path starting with a colon is relative, while one which doesn't
start with a colon is absolute (ie. the reverse of the posix convention),
_unless_ it's just a single name (ie. no colons whatsoever) in which case
it's relative.  The end result of this is that "Macintosh HD" refers to a
file in the local directory called "Macintosh HD" (as does ":Macintosh
HD"), while "Macintosh HD:" refers to the root directory of the hard disk
"Macintosh HD".

However, "Macintosh HD:Applications:" and "Macintosh HD:Applications" are
the same thing as each other. Similarly ":Macintosh HD:Applications:" and
":Macintosh HD:Applications" are the same as each other.

This seems to be a system-level convention (ie. we aren't going to be able
to change it).

Observe also:

>>> os.path.join("Macintosh HD", "Applications")
':Macintosh HD:Applications'
>>> os.path.join("Macintosh HD:", "Applications")
'Macintosh HD:Applications'

There's a similar problem with ":", "::", etc.  Removing a trailing ":" in
these cases directly affects the meaning.

> > >>> import sys
> > >>> sys.prefix
> > 'Macintosh HD:Applications:Python 1.5.2c1:'
> > >>> sys.path
> > ['Macintosh HD:Applications:Python 1.5.2c1:', 'Macintosh
> > HD:Applications:Python 1.5.2c1:', 'Macintosh HD:Applications:Python
> > 1.5.2c1:Lib', 'Macintosh HD:Applications:Python 1.5.2c1:Lib:site-packages',
> > 'Macintosh HD:Applications:Python 1.5.2c1:Lib:lib-tk', ... etc ]
>
> Well, one thing I learn from this is that you really ought to upgrade to
> 1.5.2final.  ;-)

1.5.2c1 _is_ 1.5.2final on the mac  ;)

> > Notice how sometimes the paths have trailing ":" and sometimes they don't.
>
> I guess the convention is a weak one in Mac OS too.

Yes and no  :(

> Note that on Unix:
>
> >>> os.path.normpath("foo//bar")
> 'foo/bar'
> >>> os.path.normpath("foo//bar/")
> 'foo/bar'
>
> which argues in favour of fixing normpath to strip trailing colons on
> Mac OS.

Having looked at this, I can see why Mac normpath can't just strip trailing
colons.  However, the cases where we have to leave a trailing colon are
fairly easily identified (ie. paths which have just one ":" and it's at the
end should have the trailing colon left intact, as should paths which are
entirely colons...)

Another solution would be to leave trailing ":" on known directories, but
this would fail for working with directory paths prior to their actual
creation.

> > 2. I hacked a fix for 1 (see below), but this led to a second problem: just
> > because you do os.path.normpath, you still can't assume that two different
> > paths aren't really the same.  The result of this was having install.py
> > tell me that distutils wasn't installed somewhere in sys.path, when in fact
> > it was.  This may have other consequences where distutils compares paths to
> > each other.
>
> Ick!  The whole point of normpath, as I see it, is to guarantee that
> string equality means filesystem equality.  If it doesn't do that on Mac
> OS, then it's broken.

I have to agree with this.

> Offhand, I can't think of other places where Distutils compares paths.
> Maybe the code that excludes setup.py from the modules to install --
> that's in build_py.py; see the 'find_package_modules()' method.

OK.  Hopefully a suitable replacement for normpath will do the trick - just
grep for "import os" and patch it in...

> > (b) A more robust solution is probebly to write a normpath for distutils
> > which guarantees that pathnames either do or do not end in a ":" on the
> > mac.  Not sure the best place to put it, or how much code this is going to
> > affect, but it could be a real pain to get right.
> >
> > (c) Lobby for normpath to be "fixed" on the mac - almost certainly doable,
> > but leaves the problem of 1.5.* versions not working properly.
>
> I think we can do both (b) and (c).  Eg. somewhere we'd say:
>
>     if python < 1.6 and Mac OS:              # maybe 2.0
>         def normpath (...):
>             ...
>         os.path.normpath = normpath
>
> and hopefully the "Distutils-specific" normpath would be right there in
> macpath.py in 2.0 (maybe 1.6).
>
> Can you whip up a patch to macpath.py and submit it to the SourceForge
> patch manager?  (Or just mail it to me if you don't want to deal with
> SF.)  Make sure you add a test too!

I think I have a suitable replacement function (see the attachment
normpath.py) and test suite.  I'll patch it into macpath.py and see if I
can submit it to sourceforge (I assume that macpath.py is part of the stuff
archived there, rather than the macpython CVS archive - if not I'll forward
stuff to Jack Jansen).

Even with this improvement, we're not out of the woods.  This fixes problem
2, but we'll still need to worry about substituting correctly in the cases
where we have a trailing ":".  At least these shouldn't be too hard to
detect - I think it's just install.py that needs to take this into account,
since os.path.join() is used everywhere else, right?

I'll fiddle with this in the next day or two.

> > (d) Some change to the install scheme mechanism so that it uses
> > os.path.join().  For example, use a list or tuple of strings, rather than a
> > single string:
> >
> >     'mac': {
> >         'purelib': ('$base', 'Lib'),
> >         'platlib': ('$base', 'Mac', 'PlugIns'),
> >         'headers': ('$base', 'Include', '$dist_name'),
> >         'scripts': ('$base', 'Scripts'),
> >         'data'   : ('$base',),
> >         }
>
> I'm not keen on that -- getting that stuff working was fairly tricky
> (although not nearly as tricky as designing it!), and I really don't
> want to revisit it.

Understandably.

I think a solution is in sight.

Regards,
Corran

--============_-1246924501==_============
Content-Type: text/plain; name="normpath.py"; charset="us-ascii"
Content-Disposition: attachment; filename="normpath.py"

import string

def normpath(s):
	"""Normalize a pathname.  Will return the same result for
	equivalent paths."""

	if ":" not in s:
		return ":"+s

	comps = string.splitfields(s, ":")
	i = 1
	while i < len(comps)-1:
		if comps[i] == "" and comps[i-1] != "":
			if i > 1:
				del comps[i-1:i+1]
				i = i-1
			else:
				# best way to handle this is to raise an
exception
				raise norm_error, 'path starts with volume::'
		else:
			i = i + 1
		print i, comps

	s = string.join(comps, ":")

	# remove trailing ":" except for ":" and "Volume:"
	if s[-1] == ":" and len(comps) > 2 and s != ":"*len(s):
		s = s[:-1]
	return s

def _test_normpath():
	pass

if __name__ == "__main__":
	_test_normpath()
	
--============_-1246924501==_============
Content-Type: text/plain; name="test_normpath.py"; charset="us-ascii"
Content-Disposition: attachment; filename="test_normpath.py"

from test_support import *

import normpath

errors = 0

def testit(name, value, expected):
	if value != expected:
		raise TestFailed, '%s returned %f, expected %f'%\
			(name, value, expected)


testit("normpath", normpath.normpath(":"), ":")
testit("normpath", normpath.normpath("::"), "::")
testit("normpath", normpath.normpath(":::"), ":::")

testit("normpath", normpath.normpath("test"), ":test")
testit("normpath", normpath.normpath(":test"), ":test")
testit("normpath", normpath.normpath("test:"), "test:")
testit("normpath", normpath.normpath(":test:test1:"), ":test:test1")
testit("normpath", normpath.normpath("test:test1:"), "test:test1")

testit("normpath", normpath.normpath(":test::"), ":")
testit("normpath", normpath.normpath(":test:::"), "::")
testit("normpath", normpath.normpath(":test:::test1:"), "::test1")
testit("normpath", normpath.normpath(":test:test1::"), ":test")
testit("normpath", normpath.normpath("test:test1::"), "test:")
testit("normpath", normpath.normpath(":test:test1:::"), ":")

if errors:
	print str(errors) + " errors."
else:
	print "No errors.  Thank your lucky stars."


--============_-1246924501==_============--