Re: [Cython] Store list of imported modules in .so file
On Tue, Nov 24, 2015 at 12:50 PM, Hartmut Goebel <h.goebel@crazy-compilers.com> wrote:
Hi,
I'm a core-developer for PyInstaller. We would like to enhance cython support in PyInstaller and resolve dependencies of cython-compiled modules automatically. This would ease freezing cython modules with PyInstaller a lot and thus help users including cythonized-modules into their shipped application.
For us, the most elegant way to get the dependencies would be to have the list of imported modules in the .so-file. E.g. some simple string marked by some cookie which is easily grepped out of the binary.
Is there any chance to get suche a list into the .so/.dll-file?
Import statements are "just code." They can appear inside functions and more generally can occur at runtime. In the absolute worst case, you could have something like (Python 3.x): importlib.import_module(input('What should I import today?')) This is not specific to Cython, but I imagine it's a lot harder to pull out via static analysis when you don't have Python-like source (see also: pkgutil.get_data(), any use of __file__, etc.). OTOH, if you're dealing with importlib et al., you're already in a bad place for static analysis since people can do things like: import importlib as foo; bar = foo.import_module; baz = bar('qux') (which is roughly equivalent to import qux as baz, but good luck spotting it automatically) Anyway, I think it might help to clarify what *exactly* you mean by "dependency." If you mean "any module that might possibly get imported by module X," you're going to have a hard time with this. Setuptools takes a fully manual approach to this problem, requiring developers to explicitly list all dependencies. That may not work for your needs, but it does seem to be more flexible since it lets you specify PEP 440 version constraints.
On Fri, Nov 27, 2015 at 9:13 AM, Kevin Norris <nykevin.norris@gmail.com> wrote:
On Tue, Nov 24, 2015 at 12:50 PM, Hartmut Goebel <h.goebel@crazy-compilers.com> wrote:
Hi,
I'm a core-developer for PyInstaller. We would like to enhance cython support in PyInstaller and resolve dependencies of cython-compiled modules automatically. This would ease freezing cython modules with PyInstaller a lot and thus help users including cythonized-modules into their shipped application.
For us, the most elegant way to get the dependencies would be to have the list of imported modules in the .so-file. E.g. some simple string marked by some cookie which is easily grepped out of the binary.
Is there any chance to get suche a list into the .so/.dll-file?
Import statements are "just code." They can appear inside functions and more generally can occur at runtime. In the absolute worst case, you could have something like (Python 3.x):
importlib.import_module(input('What should I import today?'))
This is not specific to Cython, but I imagine it's a lot harder to pull out via static analysis when you don't have Python-like source (see also: pkgutil.get_data(), any use of __file__, etc.). OTOH, if you're dealing with importlib et al., you're already in a bad place for static analysis since people can do things like:
import importlib as foo; bar = foo.import_module; baz = bar('qux')
(which is roughly equivalent to import qux as baz, but good luck spotting it automatically)
Anyway, I think it might help to clarify what *exactly* you mean by "dependency." If you mean "any module that might possibly get imported by module X," you're going to have a hard time with this. Setuptools takes a fully manual approach to this problem, requiring developers to explicitly list all dependencies. That may not work for your needs, but it does seem to be more flexible since it lets you specify PEP 440 version constraints.
Yes, you can do all sorts of crazy stuff. On the other hand, setuptools explicitness might be overkill for the 98% of projects that whose dependencies could be easily inferred, and at least give you a starting point. - Robert
Am 27.11.2015 um 18:13 schrieb Kevin Norris:
Anyway, I think it might help to clarify what *exactly* you mean by "dependency." If you mean "any module that might possibly get imported by module X," you're going to have a hard time with this.
We know about the problems you wrote -- this is PyInstallers job and we know where he limits are. So we would be happy to get a list od fully qualified module names for each "import" cython finds (and as I assume generated code for). PyInstaller too can only analyse the import-statements, which is enough for 90% of the cases. (We do a bit more, e.g. finding libraries used via ctypes.) Setuptools is not an option, since this are meant to list al* requirements of some package, while PyInstaller only includes the *used* modules - which might be quite a difference :-) -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |
participants (3)
-
Hartmut Goebel -
Kevin Norris -
Robert Bradshaw