Re: Add multiple frozen modules
I think I finally understand the work Greg, Gordon and the distutils sig folks have done with building the Python library into the Python binary. But I am still having problems. I still think we may need to be able to have multiple frozen module arrays. When Python starts up (Py_Initialize()) it: Performs an import of exceptions.py. Calls initsigs(). The implication is that someday something_else.py may be imported here. Calls initmain(). Performs an import of site.py. Greg's (wonderful) imputil.py must be "turned on" in site.py. So the conclusion is that the default Python import logic can not be replaced until after it has been used to find exceptions.py and site.py. This seems to be unfortunate. It would be nice to replace it for all modules. It would be nice if all this were identical on Unix and Windows, and that no C compiler were required. And that Python versions could be changed by replacing python15.dll/.so. And that having a frozen "__main__" still worked... Here are a few ideas: 1) Use freeze to include exceptions.py, site.py and imputil.py in the binary. Frozen modules are always (???) found before identical modules on PYTHONPATH/Registry. There is no chicken-and-egg problem because site.py has already been hacked to turn on imputil.py and to provide for custom imports. This works now. But there can be only one frozen module array, thus my suggestion to allow multiple arrays. OTOH, maybe this is OK since imports are now customized. But we are taking away the freeze feature including frozen "__main__". 2) Declare that imputil.py is part of the distribution, and hack Py_Initialize() to turn it on before anything is imported. I am not sure how to solve the chicken-and-egg problem. Perhaps sys.executable could be used as the initial search path in Py_Initialize(), and the regular path be used iff sys.executable fails. At least on Windows 95 and later, sys.executable is the highly reliable path to the Python interpreter binary. I am not sure how reliable it is on Unix. The idea is that site.py (etc.) is always in the same directory as the binary. For this, we need to guarantee that *.py in the directory of sys.executable is always found first. This is not currently the case. 3) Somehow have the user create special built-in modules which actually contain the *.pyc code, and hack Py_Initialize() to load and initialize them first. 4) Use pickle to create the magic file "python.py0" which contains site.py etc. in a standard format, I guess a dictionary. Hack Py_Initialize() to load its modules if it is located in the directory of sys.executable. So the start-up Python code lives in "python.py0" with its binary. Maybe we could automagically load python.py1, 2, ... too thus providing a hook for the user to add frozen code without a compiler. sys.executable must be bullet-proof. Does the executable mean python.exe or python15.dll?? Both?? Another problem is that tools/Freeze/*.py seems to require one of the frozen modules to be named "__main__" and thus be executed on start up. This means we can't put all this into python15.dll. It seems that (on Windows) exceptions.py and imputils.py should go into python15.dll, and not into python.exe. Site.py might go into either python15.dll or python.exe. FWIW, I am currently using three frozen module arrays. Site.py, exceptions.py and the Python library goes in python15.dll. My WPY modules go in python.exe, and the application main modules go in another DLL. This enables replacing python15.dll to update Python. But I am not that happy with this scheme. Jim Ahlstrom
Sorry to jump into this discussion this late... Jim kindly made me take a closer look at the thread. As far as I have skimmed the discussion, you are mostly talking about freeze'ing on Windows platforms. Well, then I'm the wrong guy to say anything, because my knowledge of that platform doesn't go too deep. All I can say is that Florent Heyworth has been able to create a cgipython binary (see the mxCGIPython project on my Python Pages) for NT which includes the Python lib as frozen modules in the EXE. Here are some instructions which I copied from a mail by Florent (hope this is ok, Florent): """
BTW: What did you have to do to compile the thingie on WinNT ? I read that freeze.py works on WinNT too, but have never tried it.
Well , it works but I needed to make a few changes as the whole build process on windows is geared to producing dlls - what i did was this (I had to do this by hand - maybe I'll try to find the time to automate this): - - removed the dllmain routine in the python15 classes - - undefined the USE_DL_IMPORT macro in config.h - - ran freeze.py from the command line as defined im your makefile - - added the *.c files created into the python15 project - - replaced the standard frozen.c with the newly created frozen.c in CGIPython - the "main" method here calls Py_FrozenMain - - added frozen_dllmain.c from the "PC" directory because that's where PyWinFreeze_ExeInit and PyWinFreeze_ExeTerm are defined - - changed the static struct _frozen _PyImport_FrozenModules[] definition to extern struct _frozen _PyImport_FrozenModules[] - - changed the preprocessor definitions to: NDEBUG,WIN32,_CONSOLE,_MBCS - - changed the link options to /subsystem:console And that was it. """ More information on installing such a binary is on the mxCGIPython web-page. Hope this helps a bit... Florent is the one to ask for details ;-) -- Something else which might also be related: The problem with this binary is that it fails to load shared libs (DLLs on WinNT). This fails for both Unix and Windows platforms. I haven't dug any deeper into this, but its either a linker problem (Python's symbols are not found) or some problem with the interaction of the import mechanism and frozen modules (I had some reports about the loader not finding the initXXX() functions of the shared extension modules). Does anybody have a clue on this one ? Maybe some tips on how to set the linker options ? Cheers, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 169 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
"M.-A. Lemburg" wrote:
As far as I have skimmed the discussion, you are mostly talking about freeze'ing on Windows platforms.
Actually, we are trying to get something which works identically on Windows and Unix if possible.
The problem with this binary is that it fails to load shared libs (DLLs on WinNT). This fails for both Unix and Windows
I would guess the problem on Windows is that the shared libs have been compiled to need Python, and so the only way they can get Python is in a DLL. If the main.exe and a shared lib both need Python (the usual case) then Python MUST be in a shared lib too. Making an exe will not work. On Windows, the solution is to put the frozen Python libs into python15.dll. Then everything will work. Jim Ahlstrom
James C. Ahlstrom wrote:
"M.-A. Lemburg" wrote:
As far as I have skimmed the discussion, you are mostly talking about freeze'ing on Windows platforms.
Actually, we are trying to get something which works identically on Windows and Unix if possible.
Ehh... I thought this was only a problem on Windows where the default is to build a Python DLL instead of an EXE. On Unix, freezing multiple modules into one binary is not much of a problem (except the shared lib thing below): this is how all the cgipython binaries for the mxCGIPython project where created.
The problem with this binary is that it fails to load shared libs (DLLs on WinNT). This fails for both Unix and Windows
I would guess the problem on Windows is that the shared libs have been compiled to need Python, and so the only way they can get Python is in a DLL. If the main.exe and a shared lib both need Python (the usual case) then Python MUST be in a shared lib too. Making an exe will not work. On Windows, the solution is to put the frozen Python libs into python15.dll. Then everything will work.
Hmm, but that wouldn't be all that elegant :-) since the code is included twice. Even worse, I think this will mess up the memory allocation, if I remember old mails on c.l.p regarding malloc and DLLs on Windows correctly. Cheers, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 169 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
"M.-A. Lemburg" wrote:
Ehh... I thought this was only a problem on Windows where the default is to build a Python DLL instead of an EXE. On Unix, freezing multiple modules into one binary is not much of a problem (except the shared lib thing below): this is how all the cgipython binaries for the mxCGIPython project where created.
I think you are right that this topic is mostly of interest to Windows people. On Unix you usually use freeze to include everything in the binary as you say. On Windows the interpreter is in python15.dll, a shared library. The advantage is that you can add other DLL's without recompiling anything. So "built-in" modules become dynamic. There is no reason Unix could not do this too if shared modules are available. For those of us distributing applications, the executable programs could be 1) main1.exe, main2.exe, etc. with a frozen "__main__" module included; 2) or a single executable "python.exe" but with a specified program on the command line "python.exe C:/path/myprog.py" as specified in a Windows icon (shortcut). This exposes the myprog source. What I am trying to do is put the Python library into python15.dll but still allow other *.pyc files to be in main1.exe. Thus the subject line, add multiple frozen modules. Putting the library into python15.dll guarantees version match, and means you can upgrade Python versions just by replacing python15.dll. You upgrade your application by replacing main1.exe. Simple.
The problem with this binary is that it fails to load shared libs (DLLs on WinNT). This fails for both Unix and Windows
I would guess the problem on Windows is that the shared libs have been compiled to need Python, and so the only way they can get Python is in a DLL. If the main.exe and a shared lib both need Python (the usual case) then Python MUST be in a shared lib too. Making an exe will not work. On Windows, the solution is to put the frozen Python libs into python15.dll. Then everything will work.
Hmm, but that wouldn't be all that elegant :-) since the code is included twice. Even worse, I think this will mess up the memory allocation, if I remember old mails on c.l.p regarding malloc and DLLs on Windows correctly.
I do not understand. Nothing is included twice. The only Python interpreter is in python15.dll, and all clients such as main1.exe, cool_stuff.pyd, etc. use it. Jim Ahlstrom
James C. Ahlstrom wrote:
"M.-A. Lemburg" wrote:
Ehh... I thought this was only a problem on Windows where the default is to build a Python DLL instead of an EXE. On Unix, freezing multiple modules into one binary is not much of a problem (except the shared lib thing below): this is how all the cgipython binaries for the mxCGIPython project where created.
I think you are right that this topic is mostly of interest to Windows people. On Unix you usually use freeze to include everything in the binary as you say.
On Windows the interpreter is in python15.dll, a shared library. The advantage is that you can add other DLL's without recompiling anything. So "built-in" modules become dynamic. There is no reason Unix could not do this too if shared modules are available.
For those of us distributing applications, the executable programs could be 1) main1.exe, main2.exe, etc. with a frozen "__main__" module included; 2) or a single executable "python.exe" but with a specified program on the command line "python.exe C:/path/myprog.py" as specified in a Windows icon (shortcut). This exposes the myprog source.
What I am trying to do is put the Python library into python15.dll but still allow other *.pyc files to be in main1.exe. Thus the subject line, add multiple frozen modules. Putting the library into python15.dll guarantees version match, and means you can upgrade Python versions just by replacing python15.dll. You upgrade your application by replacing main1.exe. Simple.
Just a note here: the new DLL should *not* be named python15.dll, since this is likely to cause incompatible setups -- why not call it pylib15.dll or something similar. This DLL would then only include the frozen modules and live happily alongside of the standard python15.dll. Only the EXE would know about the frozen modules in pylib15.dll (with the extra magic to include them as frozen modules in sys). The same could be done to your Python files. As a result you'd have these files: main.exe -- special main for frozen app which links to the DLLs myapp.dll -- frozen application modules python15.dll -- the standard python15.dll pylib15.dll -- the standard lib in form of frozen modules This setup should work on all platforms providing shared libs since the hard part of linking in the different DLLs is handled by the linker (*). main.exe would only need to know the names of all the frozen modules. (*) Using the lib in this form is also likely to reduce startup times once the DLL is loaded.
The problem with this binary is that it fails to load shared libs (DLLs on WinNT). This fails for both Unix and Windows
I would guess the problem on Windows is that the shared libs have been compiled to need Python, and so the only way they can get Python is in a DLL. If the main.exe and a shared lib both need Python (the usual case) then Python MUST be in a shared lib too. Making an exe will not work. On Windows, the solution is to put the frozen Python libs into python15.dll. Then everything will work.
Hmm, but that wouldn't be all that elegant :-) since the code is included twice. Even worse, I think this will mess up the memory allocation, if I remember old mails on c.l.p regarding malloc and DLLs on Windows correctly.
I do not understand. Nothing is included twice. The only Python interpreter is in python15.dll, and all clients such as main1.exe, cool_stuff.pyd, etc. use it.
Sorry, I misread you reply and was still thinking of how the cgipython EXE works (it includes the lib as frozen modules). Adding the python15.dll to cgipython would probably lead to the malloc problems. If the lib were included in python15.dll that would be different, I suppose. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 169 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
"M.-A. Lemburg" wrote:
Just a note here: the new DLL should *not* be named python15.dll, since this is likely to cause incompatible setups -- why not call it pylib15.dll or something similar. This DLL would then only include the frozen modules and live happily alongside of the standard python15.dll. Only the EXE would know about the frozen modules in pylib15.dll (with the extra magic to include them as frozen modules in sys).
I don't have a problem with including the frozen libs as a seperate DLL. But I don't think the main.exe needs to be involved at all on Windows. Remember on Windows all of Python including import.c is in python15.dll. We may be having Unix/Windows mind set problems. I am saying that the frozen libs (however they are packaged) are controlled by the module containing Python and import.c. On Windows that is python15.dll, on Unix it is often the main. By the way, if you can test a Windows CGI setup, I would be happy to send you a hacked python15.dll which might solve your module load problems. Jim Ahlstrom
James C. Ahlstrom wrote:
"M.-A. Lemburg" wrote:
Just a note here: the new DLL should *not* be named python15.dll, since this is likely to cause incompatible setups -- why not call it pylib15.dll or something similar. This DLL would then only include the frozen modules and live happily alongside of the standard python15.dll. Only the EXE would know about the frozen modules in pylib15.dll (with the extra magic to include them as frozen modules in sys).
I don't have a problem with including the frozen libs as a seperate DLL. But I don't think the main.exe needs to be involved at all on Windows. Remember on Windows all of Python including import.c is in python15.dll.
AFAIK (and that only refers to Unix), freeze.py builds a custom main program which includes the frozen mods as static data. I'm not sure whether it also includes a custom import.c -- I think the hooks are there for the modified main.c to use... have to look into this a little closer sometime.
We may be having Unix/Windows mind set problems. I am saying that the frozen libs (however they are packaged) are controlled by the module containing Python and import.c. On Windows that is python15.dll, on Unix it is often the main.
By the way, if you can test a Windows CGI setup, I would be happy to send you a hacked python15.dll which might solve your module load problems.
Thanks for the offer, but I'm running Unix for CGI things. There are some people out on c.l.p which would like to use cgipython + DLLs though, so you might want to publish your modified DLL somewhere for them to download. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 168 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
participants (2)
-
James C. Ahlstrom -
M.-A. Lemburg