I recall having the discussion but I don't quite recall the resolution: Is Next support now officially dropped from the distribution ? I have a revised dynamic loading module that strips out all of the dead branches ( as well as better error reporting ): I was going to call it dynload_darwin.c and add support to configure, but grepping thru configure I only saw darwin as triggering dynload_next.c -- it *looks* like the Next has bee dropped. Should we rename the file anyway ? ( to make it easier for folks to know where to look. ) There has also been some discussion on the pythonmac-sig list about dynamic loading. There are some other problems that this module doesn't fix yet. If someone wants to subit a better one, that's fine by me, but we REALLY need to get the better error reporting in there so we can at least find the problem. The other thing that's been discussed is adding configure support to build with the dlopen compatability libs if that is available. ( doing config with --without-dyld doesn't seem to change anything. ) -- Steve
Steven Majewski
I recall having the discussion but I don't quite recall the resolution: Is Next support now officially dropped from the distribution ?
AFAIR, yes.
Should we rename the file anyway ? ( to make it easier for folks to know where to look. )
Yes.
The other thing that's been discussed is adding configure support to build with the dlopen compatability libs if that is available.
Can you please explain what that would provide to module users or end users? Would there be additional modules available that otherwise wouldn't be available? If not, I don't think that this option should be provided. Regards, Martin
On 30 Jan 2002, Martin v. Loewis wrote:
The other thing that's been discussed is adding configure support to build with the dlopen compatability libs if that is available.
Can you please explain what that would provide to module users or end users? Would there be additional modules available that otherwise wouldn't be available? If not, I don't think that this option should be provided.
dlcompat libs are used by Apple to build Apache and some other programs. The libs are not included in Mac OSX, although the sources are available in the Darwin CVS, and an improved version is distributed on Fink and maybe other places. Since additional libs are required, I would not make that the default. ( unless, since there's already a check for libdl in config, we make it dependent on that. ) The problem is that the current dynload_next is broken, and we've had some problems replicating tests and solutions because, among other problems, of the very poor error reporting in dynload_next, everyone is starting from a differently hacked version of the 2.2 distribution. (The other variable is which modules and packages people are loading.) Reportedly, using the dlcompat libs fixes some problems for some people. Obviously, the best solution would be an even better dynload_darwin that fixes all of the problems. But it the interim, I'ld like to at least get everone debugging from the same baseline. If there's a string objection to adding optional libdl support, I can live without that. Adding it would just make it easier for folks to test that configuration and build. Getting a less broken dynload module is probably more important. -- Steve.
I did another version of dynload_darwin that took a >10 line function from the dlcompat/dlopen.c code which uses an undocumented (at least in the man pages -- there's probably comments in the Darwin source code) non-public way around the public/private namespace problem we were having with the previous version. I'm waiting for some folks on pythonmac-sig to test it and report back. I'm guessing that this solves the problem without requiring libdl. However it gets into the possible problem of including another license. Could someone who undestands these issues a bit more than I, look at this: Apple Public Source License: http://www.publicsource.apple.com/apsl/ -- Steve BTW: Here's the magic code I added from dlcompat/dlopen.c: ( On the one hand, it's fairly short and trivial. On the other, I wouldn't have had a clue about this without reading the code! ) /* * NSMakePrivateModulePublic() is not part of the public dyld API so we define * it here. The internal dyld function pointer for * __dyld_NSMakePrivateModulePublic is returned so thats all that maters to get * the functionality need to implement the dlopen() interfaces. */ static enum bool NSMakePrivateModulePublic( NSModule module) { static enum bool (*p)(NSModule module) = NULL; if(p == NULL) _dyld_func_lookup("__dyld_NSMakePrivateModulePublic", (unsigned long *)&p); if(p == NULL){ #ifdef DEBUG printf("_dyld_func_lookup of __dyld_NSMakePrivateModulePublic " "failed\n"); #endif return(FALSE); } return(p(module)); }
On Wednesday, January 30, 2002, at 09:42 PM, Steven Majewski wrote:
dlcompat libs are used by Apple to build Apache and some other programs. The libs are not included in Mac OSX, although the sources are available in the Darwin CVS, and an improved version is distributed on Fink and maybe other places. Since additional libs are required, I would not make that the default. ( unless, since there's already a check for libdl in config, we make it dependent on that. )
The problem is that the current dynload_next is broken, and we've had some problems replicating tests and solutions because, among other problems, of the very poor error reporting in dynload_next, everyone is starting from a differently hacked version of the 2.2 distribution. (The other variable is which modules and packages people are loading.)
Reportedly, using the dlcompat libs fixes some problems for some people.
I'm not too thrilled with dlcompat. First and foremost, it fixes
some problems for some people but may introduce problems for
others (if I understand correctly). And then there's the issue
of it not being part of the base MacOSX distribution.
I now have a dynload_next.c (that I'll check in tomorrow) that
can behave in two ways based on a #define.
With the define off it loads every extension module in a
separate namespace, i.e. two independent modules can never break
each other by supplying external symbols the other module
expected to load from a completely different place.
With the define on it loads all extension modules into the
application namespace. Some people want this (despite the
problems sketched above) because they have modules that refer to
external symbols defined in modules that have been loaded
earlier (and I assume there's magic that ensures their modules
are loaded in the right order).
While I think this is an accident waiting to happen [*] the
latter behaviour is more-or-less the standard unix behaviour, so
it should probably be supportable in some way. I prefer the new
(OSX 10.1) preferred Apple way of linking plugins (which is also
the common way to do so on all other non-unix platforms) where
the plugin has to be linked against the application and dynamic
libraries it is going to be plugged into, so none of this
dynamic behaviour goes on.
[*] I know of two cases where this already happened: both the
curses library and the SGI gl library defined a function
clear(), so you were hosed when you used both in the same Python
script. And the SGI compression library contains a private
version of libjpeg with no symbol renaming, so if you used the
cl module and a module which linked against the normal libjpeg
you were also hosed.
--
- Jack Jansen
On Thu, 31 Jan 2002, Jack Jansen wrote:
I'm not too thrilled with dlcompat. First and foremost, it fixes some problems for some people but may introduce problems for others (if I understand correctly). And then there's the issue of it not being part of the base MacOSX distribution.
I now have a dynload_next.c (that I'll check in tomorrow) that can behave in two ways based on a #define.
With the define off it loads every extension module in a separate namespace, i.e. two independent modules can never break each other by supplying external symbols the other module expected to load from a completely different place.
With the define on it loads all extension modules into the application namespace. [...]
Did you see the version I posted a day or two ago: If I fixed up the #ifdef macros, you could compile that three ways (at least): Global public symbols, Private Symbols, or the dlcompat trick. ( But it uses that magic hook into the non-public API from dlcompat. ) My main requirement is the better error reporting. -- Steve
Jack Jansen
With the define on it loads all extension modules into the application namespace. Some people want this (despite the problems sketched above) because they have modules that refer to external symbols defined in modules that have been loaded earlier (and I assume there's magic that ensures their modules are loaded in the right order).
On Unix, this is a runtime option via sys.setdlopenflags (RTLD_GLOBAL turns on import into application namespace). Do you think you could emulate this API?
While I think this is an accident waiting to happen [*] the latter behaviour is more-or-less the standard unix behaviour, so it should probably be supportable in some way.
It is not at all standard unix behaviour. Since Python 1.5.2, Python
loads extensions with RTLD_LOCAL on
I prefer the new (OSX 10.1) preferred Apple way of linking plugins (which is also the common way to do so on all other non-unix platforms) where the plugin has to be linked against the application and dynamic libraries it is going to be plugged into, so none of this dynamic behaviour goes on.
I'm not sure linking with a libpython.so is desirable, I'm quite fond of the approach to let the executable export symbols to the extensions. If that is possible on OS X, I'd encourage you to follow such a strategy (in unix gcc/ld, this is enabled through -Wl,--export-dynamic).
[*] I know of two cases where this already happened: both the curses library and the SGI gl library defined a function clear(), so you were hosed when you used both in the same Python script.
On Unix, the originally trigger might have been the problem with initsocket, which was also exported in an Oracle library, thus breaking Oracle (the Python symbol is now init_socket, but that does not change the principle). Regards, Martin
On Friday, February 1, 2002, at 12:09 , Martin v. Loewis wrote:
Jack Jansen
writes: With the define on it loads all extension modules into the application namespace. Some people want this (despite the problems sketched above) because they have modules that refer to external symbols defined in modules that have been loaded earlier (and I assume there's magic that ensures their modules are loaded in the right order).
On Unix, this is a runtime option via sys.setdlopenflags (RTLD_GLOBAL turns on import into application namespace). Do you think you could emulate this API?
Shouldn't be a problem. I had never heard of sys.setdlopenflags(), otherwise I would have done so already.
I prefer the new (OSX 10.1) preferred Apple way of linking plugins (which is also the common way to do so on all other non-unix platforms) where the plugin has to be linked against the application and dynamic libraries it is going to be plugged into, so none of this dynamic behaviour goes on.
I'm not sure linking with a libpython.so is desirable, I'm quite fond of the approach to let the executable export symbols to the extensions. If that is possible on OS X, I'd encourage you to follow such a strategy (in unix gcc/ld, this is enabled through -Wl,--export-dynamic).
Indeed, you link against the embedder (be it .so, framework or
application) in a special way that say "this is going to be the host
application".
--
- Jack Jansen
On Friday, February 1, 2002, at 03:52 , Jack Jansen wrote:
On Unix, this is a runtime option via sys.setdlopenflags (RTLD_GLOBAL turns on import into application namespace). Do you think you could emulate this API?
Shouldn't be a problem. I had never heard of sys.setdlopenflags(), otherwise I would have done so already.
Hmm. I had a look at the setdlopenflags() and accompanying
infrastructure, and it seems you can
set many flags to dlopen() through this call, is that right?
If it is, is it a good idea to call the OSX-specific routine
setdlopenflags() too, even though
it will only support the "use global namespace" flag? Or is that the
only flag you can reasonably
pass to dlopen() anyway?
--
- Jack Jansen
Jack Jansen
Hmm. I had a look at the setdlopenflags() and accompanying infrastructure, and it seems you can set many flags to dlopen() through this call, is that right?
Correct. It is admittedly very Unixish at the moment.
If it is, is it a good idea to call the OSX-specific routine setdlopenflags() too, even though it will only support the "use global namespace" flag? Or is that the only flag you can reasonably pass to dlopen() anyway?
Effectively, yes. There is also a symbol RTLD_LOCAL, which is 0 on most systems, and it may be reasonable to add RTLD_LAZY (defer resolution of function symbols until they are called the first time). Anyway, my main point is that this should be a run-time option. If the APIs can merge, that might be a good thing (even if it means to deprecate setdlopenflags); if that is not feasible, I'd atleast recommend that you put the control over extension loading also into sys. Regards, Martin
participants (4)
-
Jack Jansen
-
Jack Jansen
-
martin@v.loewis.de
-
Steven Majewski