Upgrading Python Breaks Extensions; Fix proposal

[John Roth]

"Ignacio Vazquez-Abrams" <ignacio at openservices.net> wrote in message
news:mailman.999031543.25940.python-list at python.org...
> On 28 Aug 2001, Bryan Olson wrote:
>
> > I've done a Google search and found that others have had the same
> > problem I'm having.  Any upgrade of Python, even from 2.0 to 2.1, breaks
> > extension modules.  I'm working on MS-Windows, and the extension cannot
> > find python20.dll, which is not surprising because it was replace with
> > python21.dll.  I suppose I could put back the old DLL, but I don't know
> > what that would do, since the rest of the system is using the newer DLL.
> >
> > I don't the workings of Windows DLL's are to blame; Unix shared
> > libraries would have the same problem.
>
> Actually they don't. In the Unix version the .so's link back to the
> executable; presumably they don't do that in Windows. Does anyone know
why?

Yes, as a matter of fact. The design goals for Unix and Windows were totally
different. Under Unix, each program is its own address space, and .so
modules are private resources once they're loaded. Under Windows, they
are shared resources - the same physical memory copy can be used
simultaniously by multiple users, and appears in multiple address spaces.
Consequently, it's impossible for them to link back. Also, much of the
thinking that went into the mechanisms came from CORBA, where they
wanted to run an application on one system, and link to a module on
another system.

> > If the interface to Python extensions actually changed, then certainly
> > there's no avoiding updating the extensions.  The current system is much
> > worse; any upgrade to Python breaks extensions, in many cases
> > needlessly.
> >
> > I don't think that giving different versions of dynamic libraries the
> > same name, such as "python.dll", is a good idea.  It would just cause
> > other problems.
>
> Not nearly as many problems as 5 different versions of msvcrt40.dll ;)
>
> > So here's my proposal for a solution: don't make the extension load the
> > library.  When the interpreter loads the extension, it should give it a
> > pointer through which it can find all the library objects. The pointer
> > could just be the system-dependent library pointer/handle, but more
> > elegant would be a table of functions, defined the same on all
> > platforms.

I agree. This would solve the problem rather neatly.

> What should happen is that the extensions under Windows should have the
same
> behaviour as extensions on Unix; they should reach back to the executable
for
> run-time bindings. IIRC, the DllMain() function is passed the handle of
the
> executable when the DLL is loaded; at that point the bindings should be
done.

As I mentioned above, it can't be done, at least without implementing a
completely
independent loader. I'd much rather see development resources go to
something
more useful.

> > I think the change could be made without forcing code changes to
> > extensions (though of course they'd need one last re-compile).  We just
> > have to change the statically-linked stubs library.  We need one
> > function that always gets linked in, and that the interpreter will call
> > with the function-table pointer.  The utility functions that the library
> > offers would be stubs that get the function pointers from the table, and
> > call through them.
> >
> > Of course we should think this through and try to find a way to
> > negotiate versions, so that extensions last as long as possible.
> >
> >
> > Is this idea PEP-worthy?

Go for it. Anything that saves me from having to distribute a new version
of extension DLLs when it's technically unnecessary is a great idea.

John Roth

> > --Bryan
>
> --
> Ignacio Vazquez-Abrams  <ignacio at openservices.net>
>
>
>