[Pythonmac-SIG] Extensions for static versus framework MachoPython

[bill fancher]

On Sunday, July 7, 2002, at 03:12  PM, Jack Jansen wrote:

>> As for _environ, there's a way around that. From Mach-O Runtime 
>> Architecture, p.24:
>
> Tried this, but didn't get it to work without major surgery to the Python 
> core (removing all direct references to the _environ global).

AFAIK, the only references are in posixmodule.c. I just conditionally 
defined "_environ" as "(*_NSGetEnviron())" in the same place Cygwin 
implements its support for _environ. Seems to me to work well: it's fairly 
simple surgery and "the leading platform" has to do something special here 
too, so we've got lots of (perhaps questionable) company. (Pardon also my 
chauvinism.)

>  Because when I simply added a global _environ to Python.framework I 
> couldn't link the interpreter anymore because of the multiply-defined 
> _environ (one from Python.framework, one from crt0).

By #defining _environ as above, we don't end up with "_environ" as a 
symbol in the Python library so there's no link problem. We link to the 
function implemented in the standard runtime library and the function 
returns a pointer to _environ, which is dereffed by the macro. This works 
for static libraries too, so there's no need to treat that case specially.

>
>> Assuming the PYTHON_API_VERSION	varies with what Apple calls major 
>> versions (as it does in Python 2.2 vs. 2.3), then it should be safe. I'd 
>> rather have the module not get linked against the library, but use 
>> -flat_namespace -undefined suppress. That keeps dyld from looking for a 
>> library that might not be needed and might not even be there. That patch 
>> I sent you off list that allows one to build Python as a standalone 
>> dynamic shared library uses that approach, and doesn't have the problem 
>> discussed here.
>
> [I hadn't replied to your mods yet, as I didn't get around to it, but I 
> might as well do so now]
>
> I know, but it has a major problem in that -flat_namespace will take any 
> symbol with the correct name and simply assume it is the symbol needed. 
> This is an accident waiting to happen, and I'm heavily opposed to it. I 
> know that for unix-Python this is the norm, but it leads to very obscure 
> bugs.

But we'd have lots of good company...
>
> And note that these bugs aren't theoretical, either. I've been bitten by 
> it twice over the years, once because both SGI's original GL library and 
> curses had a clear() function, and once because the SGI compression 
> library contained a modified version of the IJG JPEG library with 
> unmodified external names so anything that was linked against the normal 
> jpeg library would be dead.

If something fails everywhere else, it should, by default, fail on a Mac. 
If we have special Mac machinery that would allow it to succeed, we should 
engage that machinery consciously and be aware that software depending on 
such Mac specific features isn't portable. IMO, the default installation 
should be bug compatible across platforms.

We can always tweak various things for the binary distribution, which 
should cater to the majority, and we might find, e.g., that most people 
want two-level namespaces. Such adjustments can be made at build time by 
setting environment variables.

>
> The first case (some Python script importing both the gl and curses 
> modules) happened in exactly one Python script in the world: 
> test.regrtest. The second case was really nasty, though, because image 
> processing apps might easily import both PIL and cl without being aware 
> of it. I'm very glad that MacPython has never had these problems (because 
> external symbol and library are matched up), and so when Apple made this 
> functionality available for OSX with the two-level namespaces I 
> immediately jumped on the bandwagon.

There's no compelling reason I can see to get off the bandwagon. The price 
of admission is that the libs you link against are going to get loaded 
along with the module. The main library doesn't care how modules are 
linked.

One reason the "two Pythons" problem even comes up is that there are now 
two fundamentally different ways to install Python, resulting in two sets 
of everything. I think this is a bad situation. We can avoid the 
duplication with linked frameworks.

>
> But again: if enough people here feel that the benefits of 
> -flat_namespace outweigh these problems, please speak up! I'm always will 
> to (grudgingly:-) bend to the majority...

I think cross platform modules should, by default, be linked to behave as 
they do on other platforms even if that means behaving badly in some ways.
  Mac docs should note the default, outline its practical consequences, 
mention the workarounds possible on Mac OS X, and remind users that 
software depending on these platform specific tricks is not portable.

For Mac only stuff like the binary distribution, just link the best way. 
We have complete control over the Mac only build process and how things 
get compiled for the binary distribution.

>
> I have a completely unrelated problem with your patch (maybe you should 
> post it to sourceforge as Tony suggested?)


As soon as I fix the bugs he found and change things to get install_name 
from the -enable-shared value so it's independent of prefix...

>  and that is that you build the shared library in /usr/local and then 
> symlink to it from the Python.framework. In my opinion this defeats the 
> purpose of using the framework in the first place, and that is having 
> everything contained in a single location, so that install/uninstall 
> becomes a trivial copy or remove.

Linked frameworks don't have that bit of useful behavior, but AFAICT, it's 
the only one they lack. That doesn't stop Apple from using them. One 
reason, among many, why you can't do drag and drop system installations, 
as on Mac OS 9, is that there are Frameworks in System with links to files 
in /usr.

It could even be argued that *versioning* is the principal advantage of 
frameworks. Apple's description of Frameworks begins:

"A framework is a type of bundle that packages a dynamic shared library 
with the resources that the library requires, including header files and 
reference documentation. This consolidation of code and resources brings 
with it a number of benefits. For example, it makes it easier for the 
library to locate its resources, and it makes installation and 
uninstallation easier on the user."

<http://developer.apple.com/techpubs/macosx/Essentials/SystemOverview/Frameworks/
 >

This doesn't say exactly how the "packaging" is to be done (through 
inclusion or linkage), though it's apparent from the examples that 
inclusion is taken as the norm.

Note that "ease of installation" is mentioned after "location of resources"
. And that's all the mention it gets. The passage then goes on to describe 
the benefits of shared libraries and versioning (a product of how 
resources are "consolidated" and which resources get "located") at length.

Linked frameworks can fully support versioning, and frameworks of any sort 
are easy to add to Project Builder projects and allow convenient access to 
headers and documentation. So to take drag and drop as the only framework 
advantage is to ignore other benefits.

(Speaking of docs, the Macintosh binary distribution should include the 
html docs in the fromework and a help index. Very nice for module builders 
or adventurous scripters using PB. Dropping a few links here and there 
would put them in Help Viewer and make them searchable.)

Once we're not considering installation or developer questions, nobody 
cares much about frameworks. From inside, Python looks the pretty much the 
same with a real framework, a linked framework, or no framework at all.

>  From historical data (usage of MacPython and Python on Windows) I assume 
> that most OSX Python users will not build Python from source, and will 
> probably not have installed the developer tools (they may not even have 
> them if their OSX came bundled with their machine). I want MachoPython to 
> make life easy for these people primarily, as the people who build from 
> source are probably also more knowledgeable.

In keeping with the trend toward a "standard" Macho binary distribution, 
which I applaud, I suggest the first step should be to move all the mac 
specific stuff (or as much as we can) out of the core build mechanism. In 
particular, there should be no hint of frameworks there. On a Mac, running 
"./configure; make; make install" should produce what we'd expect on any 
other *NIX: a library in /usr/local/lib etc. Call this the "generic" 
installation.

The Mach-O binary distribution should, IMO, install over the generic 
installation. (Or it could do the generic installation as a first step. 
Whatever.) We might want a minimal "Darwin only" option and a "Mac OS X" 
option to install pieces that link to libraries not found on Darwin.

I would urge that supporting two totally different installation schemes 
such as "standard" and "real framework" is asking for trouble.

A linked framework would make the Mac binary distribution an Installer 
package, but I'd note that Apple recommends drag and drop only for "simple"
  installations. If uninstallation is a requirement, we can provide an 
"Uninstall" option with the package.

It's also worth mentioning that the "simple" drag and drop installation of 
Apple's Tcl/Tk stuff  (using "real" frameworks) involves dragging various 
pieces to various locations, as would a similar Python distribution. Not 
very user friendly.

>
> The question is really one of "should MachoPython behave like a Macintosh 
> package or like a unix package", and my answer would be "like a Macintosh 
> package".

Given a "generic" build process to produce a shared library, we could 
easily tweak things in the Mac specific stuff to create an installation 
inside a framework, if that's the way we end up going. Making the 
appropriate links to create a "real" framework is then trivial. Whatever 
we do about frameworks, building a shared library with an appropriate 
install_name is a necessary step. We put that much in the core 
distribution and hash out the remainder among ourselves, without bothering 
the neighbors.

To pursue the latter goal a bit, let me again advocate following Apple's 
lead in adopting a more lenient conception of "behaving like a Macintosh 
package".

One anecdote in support: My first attempt to compile against the Mac 
Tcl/Tk frameworks from Apple failed because I hadn't defined "TCLFRAMEWORK"
.  Again, these are "real" frameworks.

A linked framework needn't care whether you say #include 
<SpiffyStuff/SpiffyStuff.h> or #include <SpiffyStuff.h>, and there's no 
need modify library code to check whether SPIFFYSTUFF_FRAMEWORK is defined 
and perform conditional compiliation as was done with Tcl/Tk. So the 
advantages of drag and drop install/uninstall are offset by the need for 
changes to library code and to code that compiles and links against that 
library. A developer can say "I'll take the hit by changing my library 
code to deal with real frameworks", but that decision affects other 
developers too; the library developer only takes PART of the hit and 
IMPOSES the rest on third-party coders, who are, in turn, free to pass it 
on to those who compile their product.

Apple recommends that "legacy code" be compiled in the usual way, and that 
no modifications be made to library code to accommodate frameworks. It 
then ignores its own (to me, sensible,) advice with Mac Tcl/Tk, providing,
  at the same time, an example of the pitfalls of doing so. That doesn't 
give us much guidance.

>  If you want the more traditional unix behavior you can ignore the whole 
> framework business and do "configure; make" and have a perfectly normal 
> unix Python on your OSX box.

Can't we get that with the Macho binary distribution, so developer tools 
aren't required to get a "normal unix Python"?

I'll close with this observation, hinted at above: currently there's no 
way to get a normal unix *shared* library installation *at all*. I see 
that as the fundamental shortcoming of the current Mac platform support 
(hence, the patch I sent for comment).

--
bill

> --
> - Jack Jansen        <Jack.Jansen@oratrix.com>        
> http://www.cwi.nl/~jack -
> - If I can't dance I don't want to be part of your revolution -- Emma 
> Goldman -
>