BUG: Stub loaders not created for non distutils .pyd files when using setuptools.
All I have come across what I believe is a bug in setuputils bdist_egg command. According to this page http://peak.telecommunity.com/ DevCenter/EggFormats#zip-file-issues it is not possible for python's zip loader to load shared libraries directly from the zip files themselves therefore a special stub loader py file is generated for each C extension included in a package that works around this issue. The problem I have found becomes apparent when pyd extension libraries are compiled using a process that lives outside of distutils itself (ie not using the Extension class). The code in bdist_egg.py only generates stub loaders for extensions that are built by distutils - any pyd files included in another manner are not processed. This is actually inconsistent with a method elsewhere in the distribution called exe_to_egg or somesuch where all resources are scanned and any .pyd files get a stub generated. I've created a simple patch (attached) to fix the issue. It checks all the resources found in the all_outputs list and generates stubs for any pyd files found - this covers off cases where you have pyd files included that are not generated by distutils itself. I understand that the preferred method would be to use distutils to do all extension builds but in my case this is not possible or desirable. I'm dynamically generating a set of extensions as the result of a toolchain that includes SWIG and needs to be refused across multiple languages so we use Ribosome to build the actual extension files in a portable manner instead of distutils. I then copy them into the final package as datafiles. I hope that either my patch (or a modified version) can be incorporated or someone can suggest another workaround that would fix the issue allowing externally generated pyd files to have wrappers auto-generated by bdist_egg. Many thanks in advance JKP
At 12:26 PM 4/20/2007 +0100, Jamie Kirkpatrick wrote:
I hope that either my patch (or a modified version) can be incorporated or someone can suggest another workaround that would fix the issue allowing externally generated pyd files to have wrappers auto-generated by bdist_egg.
Have you tried something like this? from setuptools.command.build_ext import build_ext class my_build_ext(build_ext): def run(self): pass setup( cmdclass = {'build_ext':my_build_ext}, ext_modules = [Extension(...), ....] } (That is, simply using a custom build_ext subclass that doesn't do anything... or perhaps invokes your custom build process.)
I haven't tried this - looks like it would be worth a go for sure. If so then I guess it can be considered a workaround for the "issue". Question though - how will the setuptools machinery know what the .pyd file is called? Jamie On 20 Apr 2007, at 15:26, Phillip J. Eby wrote:
At 12:26 PM 4/20/2007 +0100, Jamie Kirkpatrick wrote:
I hope that either my patch (or a modified version) can be incorporated or someone can suggest another workaround that would fix the issue allowing externally generated pyd files to have wrappers auto-generated by bdist_egg.
Have you tried something like this?
from setuptools.command.build_ext import build_ext class my_build_ext(build_ext): def run(self): pass
setup( cmdclass = {'build_ext':my_build_ext}, ext_modules = [Extension(...), ....] }
(That is, simply using a custom build_ext subclass that doesn't do anything... or perhaps invokes your custom build process.)
At 04:15 PM 4/20/2007 +0100, Jamie Kirkpatrick wrote:
I haven't tried this - looks like it would be worth a go for sure. If so then I guess it can be considered a workaround for the "issue". Question though - how will the setuptools machinery know what the .pyd file is called?
By you giving the correct information to Extension() as if you were going to have the distutils do the building.
participants (2)
-
Jamie Kirkpatrick
-
Phillip J. Eby