Import extensions from zipfiles (windows only)
I have a first working version of an importer which can import extension modules from zipfiles, avoiding to unpack them to the file system. License is still LGPL, unfortunately. Subscribers to the py2exe-users list already know that it uses this code which simulates the windows LoadLibrary call: http://www.joachim-bauch.de/tutorials/load_dll_memory.html It works in simple cases, the only ones that I have tested so far. Shall I publish it for experimentation? Thomas
At 09:15 PM 12/15/2004 +0100, Thomas Heller wrote:
I have a first working version of an importer which can import extension modules from zipfiles, avoiding to unpack them to the file system. License is still LGPL, unfortunately.
Subscribers to the py2exe-users list already know that it uses this code which simulates the windows LoadLibrary call:
http://www.joachim-bauch.de/tutorials/load_dll_memory.html
It works in simple cases, the only ones that I have tested so far.
Shall I publish it for experimentation?
Doesn't this technique require an extension module, which would mean in turn that we can't bootstrap it from the zipfile? That is, it would have to be included in Python, which the LGPL would rule out anyway. Still, it sounds most interesting. I hope in another week or two to have some time to hammer out a prototype for a self-extract API and a setuptools extension to build a basic archive format. I'm thinking that rather than allowing metadata to be a holdup, I'd like to get a base implementation we can experiment with. In that regard, your technique sounds useful too, but I'm kind of wary about the licensing issue.
Thomas
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Phillip J. Eby wrote:
At 09:15 PM 12/15/2004 +0100, Thomas Heller wrote:
I have a first working version of an importer which can import extension modules from zipfiles, avoiding to unpack them to the file system. License is still LGPL, unfortunately.
Subscribers to the py2exe-users list already know that it uses this code which simulates the windows LoadLibrary call:
http://www.joachim-bauch.de/tutorials/load_dll_memory.html
It works in simple cases, the only ones that I have tested so far.
Shall I publish it for experimentation?
Doesn't this technique require an extension module, which would mean in turn that we can't bootstrap it from the zipfile? That is, it would have to be included in Python, which the LGPL would rule out anyway.
Still, it sounds most interesting. I hope in another week or two to have some time to hammer out a prototype for a self-extract API and a setuptools extension to build a basic archive format. I'm thinking that rather than allowing metadata to be a holdup, I'd like to get a base implementation we can experiment with. In that regard, your technique sounds useful too, but I'm kind of wary about the licensing issue.
I wonder why you put so much effort into avoiding the unzip of the file ? What's so bad about it ? In the end, the user will want "plugins" to be easily installable, e.g. have the application install them for him. For that to work, the most important part is a download manager. The rest (unzip into the plugin directory) can easily be done using standard distutils tools. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 15 2004)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
At 10:14 AM 12/16/2004 +0100, M.-A. Lemburg wrote:
I wonder why you put so much effort into avoiding the unzip of the file ? What's so bad about it ?
I mostly don't want it to be a manual step. I think Thomas would also like to have the option of py2exe generating single-file "installation-free" executables, even if an application includes extensions. (That's not a plugins issue, of course.)
In the end, the user will want "plugins" to be easily installable, e.g. have the application install them for him. For that to work, the most important part is a download manager. The rest (unzip into the plugin directory) can easily be done using standard distutils tools.
Well, as long as they're either unzipped into different directories, or there is only one version of each plugin needed. But, it'd also be nice to be able to just put zipfiles on the Python path and share them between applications for some plugins. One big advantage to this is that platform-specific packagers (RPM et al) could be made to simply dump the zipfiles in an all-purpose location like /lib/pythonXX/site-packages, and not have to worry about version conflicts. A script or application could call a utility routine to add the needed items to sys.path at runtime. Something like 'require("Twisted>=1.1.0")' to search the existing sys.path for an appropriately-named directory or zipfile, and push it on the front of sys.path. Of course, with the proper metadata in the zipfiles (or their unpacked versions), the routine could also search for the necessary dependencies, and check for version conflicts.
Phillip J. Eby wrote:
At 10:14 AM 12/16/2004 +0100, M.-A. Lemburg wrote:
I wonder why you put so much effort into avoiding the unzip of the file ? What's so bad about it ?
I mostly don't want it to be a manual step.
Isn't that an application feature rather than a distutils one ? I mean distutils can help the application by packing everything up nicely in a ZIP file, but the application has to take care of getting the ZIP file and installing it - like most over-the-web installers do nowadays.
I think Thomas would also like to have the option of py2exe generating single-file "installation-free" executables, even if an application includes extensions. (That's not a plugins issue, of course.)
That's a valid case. However, I'd consider that a cosmetic thing: the days of one .exe does it all, drop-into c:\windows are long over :-/
In the end, the user will want "plugins" to be easily installable, e.g. have the application install them for him. For that to work, the most important part is a download manager. The rest (unzip into the plugin directory) can easily be done using standard distutils tools.
Well, as long as they're either unzipped into different directories, or there is only one version of each plugin needed. But, it'd also be nice to be able to just put zipfiles on the Python path and share them between applications for some plugins.
Sharing plugins is a different concept than having managing multiple plugins for a single application. It's a matter of who owns the plugin installation location. If all applications come from the same vendor, then the usual approach is to have a vendor specific base dir and then a shared/ directory for shared resources. If the applications come from different vendors, you'd have to define a standard location for the applications to look for plugins. The standard then owns the location. Most likely the installation will have to be done by an administrator, unless you want to run into permission problems.
One big advantage to this is that platform-specific packagers (RPM et al) could be made to simply dump the zipfiles in an all-purpose location like /lib/pythonXX/site-packages, and not have to worry about version conflicts. A script or application could call a utility routine to add the needed items to sys.path at runtime. Something like 'require("Twisted>=1.1.0")' to search the existing sys.path for an appropriately-named directory or zipfile, and push it on the front of sys.path.
Right, but that's all possible today: simply create directories under site-packages/ that are not Python package directories and contain the version number, e.g. site-packages/ egenix-mx-base-2.0.90/ mx/ __init__.py egenix-mx-base-2.1.0/ mx/ __init__.py The rest of the logic can then be done in Python and placed into a helper module: import syspathtools syspathtools.use('egenix-mx-base >= 2.1') from mx import DateTime
Of course, with the proper metadata in the zipfiles (or their unpacked versions), the routine could also search for the necessary dependencies, and check for version conflicts.
You'd probably want to place the meta data into the version directories as PKG-INFO file. However, this is only necessary if you plan to do the dependency checking *before* actually using the plugin. In the normal situation, you'd just do dynamic checking and then report the problem as run-time error. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 15 2004)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
(If someone feels the cross-posting is no longer appropriate, feel free
to snip te recipients list)
"M.-A. Lemburg"
Phillip J. Eby wrote:
I think Thomas would also like to have the option of py2exe generating single-file "installation-free" executables, even if an application includes extensions.
Yes.
(That's not a plugins issue, of course.)
Sure.
That's a valid case. However, I'd consider that a cosmetic thing: the days of one .exe does it all, drop-into c:\windows are long over :-/
Single file executables are not only a cosmetic thing: Chances are that an application breaks because one of the files are damaged, missing, overwritten or so. And them are damned simple to handle - not always does one want to create an installer. But the real reason (for me, anyway) are maybe more than one dll python com servers in one process, which may even use python itself - but this has been discussed before, and Philip even has his own ideas an this topic. Back to plugins: wouldn't it be nice if I simply copy the file PyOpenGL.2.5.99.plugin into sys/site-packages and can then use it? (Crazy idea, probably, but with the zip extension importer it may even be possible to import the bdist_wininst created installer exe files directly) Thomas
"Phillip J. Eby"
At 09:15 PM 12/15/2004 +0100, Thomas Heller wrote:
I have a first working version of an importer which can import extension modules from zipfiles, avoiding to unpack them to the file system. License is still LGPL, unfortunately.
Subscribers to the py2exe-users list already know that it uses this code which simulates the windows LoadLibrary call:
http://www.joachim-bauch.de/tutorials/load_dll_memory.html
It works in simple cases, the only ones that I have tested so far.
Shall I publish it for experimentation?
Doesn't this technique require an extension module, which would mean in turn that we can't bootstrap it from the zipfile? That is, it would have to be included in Python, which the LGPL would rule out anyway.
Still, it sounds most interesting.
In case you haven't seen the announcement I made here's the link: http://starship.python.net/crew/theller/moin.cgi/Hacks_2fZipExtImporter
I hope in another week or two to have some time to hammer out a prototype for a self-extract API and a setuptools extension to build a basic archive format. I'm thinking that rather than allowing metadata to be a holdup, I'd like to get a base implementation we can experiment with. In that regard, your technique sounds useful too, but I'm kind of wary about the licensing issue.
participants (3)
-
M.-A. Lemburg -
Phillip J. Eby -
Thomas Heller