[Pythonmac-SIG] Some thoughts on Python and OS X

[Kenneth McDonald]

Bob,

Thanks for the responses. Sorry I was wrong on some points: I'd received
the impression that Apple's Python was somehow "special", and for that
I apologise. In my own defense, I definitely received that impression
on this group due to messages from folks at Apple; they weren't
being misleading, they simply said, "Don't muck around with the
Apple installation, because replacing it with your own Python
distro will do bad things", or words to that effect. Without further 
details,
it's not difficult to conclude that this must mean Apple distro != 
Python distro.

And while I haven't actually used Linux in a long time (and never
particularly enthusiastically), you are partly right about expecting
certain things to be done in a UNIXish manner. There never was
a real "standard" for the UNIX file structure, but Apple's structure
is, uh, somewhat less standard than standard :-).

What would really help--for me at least, and perhaps for others--
is a good document explaining how and why Apple has implemented
certain structures, how they map to "old" structures. I've looked
through a number of the OS X books available, and though I can
find bits and pieces here and there, I've never seen a good,
coherent overview. For example, I'd love to see a good explanation
as to why Frameworks exist, why to put things into a framework
structure rather than placing libraries, binaries, etc. into
/.../lib, /.../bin, how to deal with Makefiles that don't know about
frameworks (most of them), and so on.

I also apologise for getting /usr/local wrong. As I say, I've ended up
creating a number of 'old-style' UNIX directories. I guess it must've
been another one.

Definitely, the fact that /Library/Python/2.3 and 
/Library/Frameworks/Python/...{somewhere}
are supposed to be the same place is news to me. They aren't the same
on my system, I have no idea whether due to me or to some installer
script, and now I'm scared to try to fix this :-)

As regards to 'distutils', well first, I'll absolutely claim stupidity 
on my
part about distutils always installing into a 'versioned' subdirectory,
that's useful knowledge, and I'm certainly turning bright red.
However, simply saying that "distutils understands OS X, so anything
written properly using distutils should work on OS X" is a little too
facile. I've spent too many hours trying to tell Python (and other 
things)
where to find (non-Python) libs, files, etc. because the OS X default
paths/file structure didn't seem adequate for 'legacy' UNIX software.

And again, this was not intended to be criticism, at least not
in the derogatory sense. I like OS X, I like programming on it, and
I'm extremely glad to be back using Python after several years away
from it. However, I have spent some time wrestling with problems
largely caused by the "Apple-ness" of OS X at the UNIX level, and
I know others have had similar problems, and I thought it would be
a good idea to talk about some of these. I've certainly learned some
useful things from your response.

Thanks,
Ken


On Jun 14, 2004, at 12:54 AM, Bob Ippolito wrote:

>
> On Jun 14, 2004, at 1:28 AM, Kenneth McDonald wrote:
>
>> The following comes from some experiences I've had recently whilst
>> wrangling with wxPython/SPE, plus some others encountered in the
>> past while doing custom installs of Tk and tkinter. I don't want to 
>> start
>> any flame wars, and I certainly don't want to hurt any feelings--the 
>> Python
>> people at Apple are doing a great job, and I really appreciate their
>> effort. However, the "Apple Philosophy" as applied to Python (and
>> indeed as applied to much of the UNIX subsurface in OS X) is making
>> things painfully, and in my opinion unnecessarily, difficult to use.
>>
>> First, the obvious: Having a "required, custom" build of Python on
>> OS X is a Bad Idea. Last I heard, this was the official OS X way of
>> doing things, i.e. the Python you got was what you should use, don't
>> try replacing it, and no, it's not a stock Python. I believe this has 
>> been
>> recognized to be a problem and is being worked on--sorry if I'm
>> ignorant of recent fixes, I "threw out" the stock OS X Python some
>> time ago, as I needed my own custom compilations.
>
> It's not custom.  It's highly suggested not to replace it because 
> there are various bugs here and there on the Python side that make it 
> a bad idea to do so.
>
>> In general (and I'm sure the Apple techs understand this, but perhaps
>> you could pass this on to those in management who don't?), DON'T
>> EVER take a community standard tool such as Python and "improve"
>> it in such a manner as to make it incompatible with what people have
>> come to expect. If you must produce an Apple-specific version which
>> requires special consideration when being used with other packages,
>> then observe the following:
>>
>> 	1) The _standard_ version should be the standard; even on OS X.
>> 	People who want to use Python without caring about the OS its
>> 	on will then be able to use it properly. Those who want to make use
>> 	of the os-enhanced version will know what to do.
>>
>> 	2) Make sure that an 'enhanced' version can coexist peacefully with
>> 	the standard version.
>
> It *is* standard.  It's not "improved".  The only difference between 
> what Apple distributes and what was in 2.3.0 is the location that it 
> is installed in (/System rather than /) and the additional 
> CoreGraphics extension, which has nothing to do with how it was 
> compiled.
>
>> One of the things that is expected is that I can download, compile,
>> and install a more recent version of Python without problem (assuming
>> the newer version itself doesn't have problems.) Another is that
>> add-ons for Python which have been written to install nicely in a
>> generic UNIX environment should continue to do so under OS X.
>
> It will work just fine, but it must supersede the system installation 
> due to bugs that were in 2.3.0 that we can not fix.  Once you install 
> your own Python, you can no longer expect to be able to compile 
> software for the stock 2.3.0.
>
> Any Python extensions that have been written properly already use 
> distutils, and thus already work fine on OS X.  If anything is 
> designed to work in a "generic UNIX environment" and does NOT work on 
> OS X is simply not done correctly.  Things should work on Windows too, 
> and that means using distutils.
>
>> Next, paths. I can understand why Apple (actually, probably NeXT)
>> decided to come up with improvements on the UNIX file structure.
>> But why, oh why, couldn't they have maintained some sort of backward
>> compatibility using links? I've effectively recreated my own UNIX file
>> structure using links, to make up for this, and would be surprised if 
>> others
>> haven't as well. In particular, why the heck isn't there the 
>> equivalent of
>> /usr/local? Having this basic de facto standard sort of file 
>> structure in
>> place would save us UNIX software users countless hours in headaches.
>> Yes, it's easy to argue that the problems are the fault of bad 
>> installation
>> scripts and makefiles. Well, yes, in theory that's probably true; but 
>> so
>> what? The point of software is to be a useful tool, not to drive the 
>> customer
>> crazy by insisting that somebody else, in assuming a de facto 
>> standard of
>> the time would continue into the future, made a mistake.
>
> Uh, there is a /usr/local.
>
>> I find this a particular problem when it comes to Frameworks and 
>> Libraries.
>> Here's a typical example of installing a couple of Python modules 
>> recently:
>>
>> 1) Compile module A--works, great. Install it. Where the heck are the
>> files? Do a search on the disk. Ah, there they are in
>> /Library/Frameworks/Python/.../.../... and so on 
>> .../2.3/site-packages.
>> Create a link from a more reasonable location so they'll be on the
>> python path.
>
> /Library/Python/2.3 is the more reasonable link, it already exists.
>
>> 2) Compile and install module B. Where is it? Disk search. Ah, it's in
>> /Library/Python/2.3. Create link. Why did it end up here, vs. Module
>> A in Frameworks? Beats me.
>
> They're the SAME PLACE unless you or some broken installation script 
> screwed that up.
>
>> And why, for that matter, are all of these modules always ending up in
>> a '2.3' directory? I happen to know that some of them aren't dependent
>> on the Python binary, and it looks to me as if, when 2.3 changes to 
>> 2.4,
>> I'm gonna have some unnecessary work, again. Trust me, if a module
>> is incompatible with 2.4, I'll know it soon enough; you don't have to
>> force me to move everything manually to the 2.4 path.
>
> This happens on EVERY Python platform.  If you want to install 
> "version independent" Python files somewhere else, feel free to do so. 
>  There are many options that you may pass to distutils to control this 
> behavior.  Then you simply need to make a pth file that points to your 
> "version independent" site directory.
>
>> None of this is going to make me move off of OS X, of course; it still
>> is, by far, the best OS for most users IMHO, myself included. But this
>> situation will cost Apple some money; I generally keep fairly 
>> up-to-date
>> with hardware, and was planning on getting a new Powerbook when
>> they hit 2 GHz. However, I've had to do so much manual tweaking to my
>> system that the effort needed to recreate that on a new drive is a 
>> little too
>> much to want to repeat in the forseeable future, so Apple's probably 
>> going
>> to have to hit 3GHz for me to decide it's worth the pain. (Yes, I know
>> I don't have to do a "clean" install of everything when I go to a new 
>> system,
>> but that's one of the things I do to help me clean all the cruft out.)
>>
>> I'm looking forward to what comes out in the future, I have a lot of
>> confidence in the Python people at Apple, and I have no problems at
>> all in continuing to pay for new versions of OS X which bring new
>> features and improvements. I just hope that the above is of some help
>> in deciding how some of those features should be implemented in the
>> future.
>
> There aren't really "Python people at Apple", at least not that I'm 
> aware of.  Any problems with the Python distribution for OS X are at 
> least partially our fault.  Start filing bugs and feature requests, 
> and making patches if you REALLY want a particular issue fixed in a 
> way that suits you.
>
> I think a lot of your troubles are just because you expect things to 
> work as they do in Linux, and that's just not the case.  There are 
> equivalent or better ways of doing things on OS X, that you either 
> don't accept or haven't figured out yet.
>
> -bob