
I recall having the discussion but I don't quite recall the resolution: Is Next support now officially dropped from the distribution ? I have a revised dynamic loading module that strips out all of the dead branches ( as well as better error reporting ): I was going to call it dynload_darwin.c and add support to configure, but grepping thru configure I only saw darwin as triggering dynload_next.c -- it *looks* like the Next has bee dropped. Should we rename the file anyway ? ( to make it easier for folks to know where to look. ) There has also been some discussion on the pythonmac-sig list about dynamic loading. There are some other problems that this module doesn't fix yet. If someone wants to subit a better one, that's fine by me, but we REALLY need to get the better error reporting in there so we can at least find the problem. The other thing that's been discussed is adding configure support to build with the dlopen compatability libs if that is available. ( doing config with --without-dyld doesn't seem to change anything. ) -- Steve

Steven Majewski <sdm7g@Virginia.EDU> writes:
AFAIR, yes.
Should we rename the file anyway ? ( to make it easier for folks to know where to look. )
Yes.
Can you please explain what that would provide to module users or end users? Would there be additional modules available that otherwise wouldn't be available? If not, I don't think that this option should be provided. Regards, Martin

On 30 Jan 2002, Martin v. Loewis wrote:
dlcompat libs are used by Apple to build Apache and some other programs. The libs are not included in Mac OSX, although the sources are available in the Darwin CVS, and an improved version is distributed on Fink and maybe other places. Since additional libs are required, I would not make that the default. ( unless, since there's already a check for libdl in config, we make it dependent on that. ) The problem is that the current dynload_next is broken, and we've had some problems replicating tests and solutions because, among other problems, of the very poor error reporting in dynload_next, everyone is starting from a differently hacked version of the 2.2 distribution. (The other variable is which modules and packages people are loading.) Reportedly, using the dlcompat libs fixes some problems for some people. Obviously, the best solution would be an even better dynload_darwin that fixes all of the problems. But it the interim, I'ld like to at least get everone debugging from the same baseline. If there's a string objection to adding optional libdl support, I can live without that. Adding it would just make it easier for folks to test that configuration and build. Getting a less broken dynload module is probably more important. -- Steve.

I did another version of dynload_darwin that took a >10 line function from the dlcompat/dlopen.c code which uses an undocumented (at least in the man pages -- there's probably comments in the Darwin source code) non-public way around the public/private namespace problem we were having with the previous version. I'm waiting for some folks on pythonmac-sig to test it and report back. I'm guessing that this solves the problem without requiring libdl. However it gets into the possible problem of including another license. Could someone who undestands these issues a bit more than I, look at this: Apple Public Source License: http://www.publicsource.apple.com/apsl/ -- Steve BTW: Here's the magic code I added from dlcompat/dlopen.c: ( On the one hand, it's fairly short and trivial. On the other, I wouldn't have had a clue about this without reading the code! ) /* * NSMakePrivateModulePublic() is not part of the public dyld API so we define * it here. The internal dyld function pointer for * __dyld_NSMakePrivateModulePublic is returned so thats all that maters to get * the functionality need to implement the dlopen() interfaces. */ static enum bool NSMakePrivateModulePublic( NSModule module) { static enum bool (*p)(NSModule module) = NULL; if(p == NULL) _dyld_func_lookup("__dyld_NSMakePrivateModulePublic", (unsigned long *)&p); if(p == NULL){ #ifdef DEBUG printf("_dyld_func_lookup of __dyld_NSMakePrivateModulePublic " "failed\n"); #endif return(FALSE); } return(p(module)); }

On Wednesday, January 30, 2002, at 09:42 PM, Steven Majewski wrote:
I'm not too thrilled with dlcompat. First and foremost, it fixes some problems for some people but may introduce problems for others (if I understand correctly). And then there's the issue of it not being part of the base MacOSX distribution. I now have a dynload_next.c (that I'll check in tomorrow) that can behave in two ways based on a #define. With the define off it loads every extension module in a separate namespace, i.e. two independent modules can never break each other by supplying external symbols the other module expected to load from a completely different place. With the define on it loads all extension modules into the application namespace. Some people want this (despite the problems sketched above) because they have modules that refer to external symbols defined in modules that have been loaded earlier (and I assume there's magic that ensures their modules are loaded in the right order). While I think this is an accident waiting to happen [*] the latter behaviour is more-or-less the standard unix behaviour, so it should probably be supportable in some way. I prefer the new (OSX 10.1) preferred Apple way of linking plugins (which is also the common way to do so on all other non-unix platforms) where the plugin has to be linked against the application and dynamic libraries it is going to be plugged into, so none of this dynamic behaviour goes on. [*] I know of two cases where this already happened: both the curses library and the SGI gl library defined a function clear(), so you were hosed when you used both in the same Python script. And the SGI compression library contains a private version of libjpeg with no symbol renaming, so if you used the cl module and a module which linked against the normal libjpeg you were also hosed. -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

On Thu, 31 Jan 2002, Jack Jansen wrote:
Did you see the version I posted a day or two ago: If I fixed up the #ifdef macros, you could compile that three ways (at least): Global public symbols, Private Symbols, or the dlcompat trick. ( But it uses that magic hook into the non-public API from dlcompat. ) My main requirement is the better error reporting. -- Steve

Jack Jansen <Jack.Jansen@oratrix.nl> writes:
On Unix, this is a runtime option via sys.setdlopenflags (RTLD_GLOBAL turns on import into application namespace). Do you think you could emulate this API?
It is not at all standard unix behaviour. Since Python 1.5.2, Python loads extensions with RTLD_LOCAL on <dlfcn.h> systems, so that each module has its own namespace. People often requested that this is changed, but we successfully managed to turn down all these requests. Eventually, somebody came up with sys.setdlopenflags; this was good enough for me.
I'm not sure linking with a libpython.so is desirable, I'm quite fond of the approach to let the executable export symbols to the extensions. If that is possible on OS X, I'd encourage you to follow such a strategy (in unix gcc/ld, this is enabled through -Wl,--export-dynamic).
On Unix, the originally trigger might have been the problem with initsocket, which was also exported in an Oracle library, thus breaking Oracle (the Python symbol is now init_socket, but that does not change the principle). Regards, Martin

On Friday, February 1, 2002, at 12:09 , Martin v. Loewis wrote:
Shouldn't be a problem. I had never heard of sys.setdlopenflags(), otherwise I would have done so already.
Indeed, you link against the embedder (be it .so, framework or application) in a special way that say "this is going to be the host application". -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

On Friday, February 1, 2002, at 03:52 , Jack Jansen wrote:
Hmm. I had a look at the setdlopenflags() and accompanying infrastructure, and it seems you can set many flags to dlopen() through this call, is that right? If it is, is it a good idea to call the OSX-specific routine setdlopenflags() too, even though it will only support the "use global namespace" flag? Or is that the only flag you can reasonably pass to dlopen() anyway? -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

Jack Jansen <jack@oratrix.com> writes:
Correct. It is admittedly very Unixish at the moment.
Effectively, yes. There is also a symbol RTLD_LOCAL, which is 0 on most systems, and it may be reasonable to add RTLD_LAZY (defer resolution of function symbols until they are called the first time). Anyway, my main point is that this should be a run-time option. If the APIs can merge, that might be a good thing (even if it means to deprecate setdlopenflags); if that is not feasible, I'd atleast recommend that you put the control over extension loading also into sys. Regards, Martin

Steven Majewski <sdm7g@Virginia.EDU> writes:
AFAIR, yes.
Should we rename the file anyway ? ( to make it easier for folks to know where to look. )
Yes.
Can you please explain what that would provide to module users or end users? Would there be additional modules available that otherwise wouldn't be available? If not, I don't think that this option should be provided. Regards, Martin

On 30 Jan 2002, Martin v. Loewis wrote:
dlcompat libs are used by Apple to build Apache and some other programs. The libs are not included in Mac OSX, although the sources are available in the Darwin CVS, and an improved version is distributed on Fink and maybe other places. Since additional libs are required, I would not make that the default. ( unless, since there's already a check for libdl in config, we make it dependent on that. ) The problem is that the current dynload_next is broken, and we've had some problems replicating tests and solutions because, among other problems, of the very poor error reporting in dynload_next, everyone is starting from a differently hacked version of the 2.2 distribution. (The other variable is which modules and packages people are loading.) Reportedly, using the dlcompat libs fixes some problems for some people. Obviously, the best solution would be an even better dynload_darwin that fixes all of the problems. But it the interim, I'ld like to at least get everone debugging from the same baseline. If there's a string objection to adding optional libdl support, I can live without that. Adding it would just make it easier for folks to test that configuration and build. Getting a less broken dynload module is probably more important. -- Steve.

I did another version of dynload_darwin that took a >10 line function from the dlcompat/dlopen.c code which uses an undocumented (at least in the man pages -- there's probably comments in the Darwin source code) non-public way around the public/private namespace problem we were having with the previous version. I'm waiting for some folks on pythonmac-sig to test it and report back. I'm guessing that this solves the problem without requiring libdl. However it gets into the possible problem of including another license. Could someone who undestands these issues a bit more than I, look at this: Apple Public Source License: http://www.publicsource.apple.com/apsl/ -- Steve BTW: Here's the magic code I added from dlcompat/dlopen.c: ( On the one hand, it's fairly short and trivial. On the other, I wouldn't have had a clue about this without reading the code! ) /* * NSMakePrivateModulePublic() is not part of the public dyld API so we define * it here. The internal dyld function pointer for * __dyld_NSMakePrivateModulePublic is returned so thats all that maters to get * the functionality need to implement the dlopen() interfaces. */ static enum bool NSMakePrivateModulePublic( NSModule module) { static enum bool (*p)(NSModule module) = NULL; if(p == NULL) _dyld_func_lookup("__dyld_NSMakePrivateModulePublic", (unsigned long *)&p); if(p == NULL){ #ifdef DEBUG printf("_dyld_func_lookup of __dyld_NSMakePrivateModulePublic " "failed\n"); #endif return(FALSE); } return(p(module)); }

On Wednesday, January 30, 2002, at 09:42 PM, Steven Majewski wrote:
I'm not too thrilled with dlcompat. First and foremost, it fixes some problems for some people but may introduce problems for others (if I understand correctly). And then there's the issue of it not being part of the base MacOSX distribution. I now have a dynload_next.c (that I'll check in tomorrow) that can behave in two ways based on a #define. With the define off it loads every extension module in a separate namespace, i.e. two independent modules can never break each other by supplying external symbols the other module expected to load from a completely different place. With the define on it loads all extension modules into the application namespace. Some people want this (despite the problems sketched above) because they have modules that refer to external symbols defined in modules that have been loaded earlier (and I assume there's magic that ensures their modules are loaded in the right order). While I think this is an accident waiting to happen [*] the latter behaviour is more-or-less the standard unix behaviour, so it should probably be supportable in some way. I prefer the new (OSX 10.1) preferred Apple way of linking plugins (which is also the common way to do so on all other non-unix platforms) where the plugin has to be linked against the application and dynamic libraries it is going to be plugged into, so none of this dynamic behaviour goes on. [*] I know of two cases where this already happened: both the curses library and the SGI gl library defined a function clear(), so you were hosed when you used both in the same Python script. And the SGI compression library contains a private version of libjpeg with no symbol renaming, so if you used the cl module and a module which linked against the normal libjpeg you were also hosed. -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

On Thu, 31 Jan 2002, Jack Jansen wrote:
Did you see the version I posted a day or two ago: If I fixed up the #ifdef macros, you could compile that three ways (at least): Global public symbols, Private Symbols, or the dlcompat trick. ( But it uses that magic hook into the non-public API from dlcompat. ) My main requirement is the better error reporting. -- Steve

Jack Jansen <Jack.Jansen@oratrix.nl> writes:
On Unix, this is a runtime option via sys.setdlopenflags (RTLD_GLOBAL turns on import into application namespace). Do you think you could emulate this API?
It is not at all standard unix behaviour. Since Python 1.5.2, Python loads extensions with RTLD_LOCAL on <dlfcn.h> systems, so that each module has its own namespace. People often requested that this is changed, but we successfully managed to turn down all these requests. Eventually, somebody came up with sys.setdlopenflags; this was good enough for me.
I'm not sure linking with a libpython.so is desirable, I'm quite fond of the approach to let the executable export symbols to the extensions. If that is possible on OS X, I'd encourage you to follow such a strategy (in unix gcc/ld, this is enabled through -Wl,--export-dynamic).
On Unix, the originally trigger might have been the problem with initsocket, which was also exported in an Oracle library, thus breaking Oracle (the Python symbol is now init_socket, but that does not change the principle). Regards, Martin

On Friday, February 1, 2002, at 12:09 , Martin v. Loewis wrote:
Shouldn't be a problem. I had never heard of sys.setdlopenflags(), otherwise I would have done so already.
Indeed, you link against the embedder (be it .so, framework or application) in a special way that say "this is going to be the host application". -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

On Friday, February 1, 2002, at 03:52 , Jack Jansen wrote:
Hmm. I had a look at the setdlopenflags() and accompanying infrastructure, and it seems you can set many flags to dlopen() through this call, is that right? If it is, is it a good idea to call the OSX-specific routine setdlopenflags() too, even though it will only support the "use global namespace" flag? Or is that the only flag you can reasonably pass to dlopen() anyway? -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

Jack Jansen <jack@oratrix.com> writes:
Correct. It is admittedly very Unixish at the moment.
Effectively, yes. There is also a symbol RTLD_LOCAL, which is 0 on most systems, and it may be reasonable to add RTLD_LAZY (defer resolution of function symbols until they are called the first time). Anyway, my main point is that this should be a run-time option. If the APIs can merge, that might be a good thing (even if it means to deprecate setdlopenflags); if that is not feasible, I'd atleast recommend that you put the control over extension loading also into sys. Regards, Martin
participants (4)
-
Jack Jansen
-
Jack Jansen
-
martin@v.loewis.de
-
Steven Majewski