prevent extension module code from being executed during startup?
Hi, when starting pypy-c, I get some extension module code executed before that module is imported. The code in question is in pypy/module/cppyy/pythonify.py: gbl = make_cppnamespace(None, "::", None, False) # global C++ namespace gbl.__doc__ = "Global C++ namespace." sys.modules['cppyy.gbl'] = gbl This code is not rpython code. The module is first mentioned in the __init__.py of cppyy: appleveldefs = { 'gbl' : 'pythonify.gbl', 'load_reflection_info' : 'pythonify.load_reflection_info', 'add_pythonization' : 'pythonify.add_pythonization', } How can I prevent that code from being executed at startup, i.e. have it deferred all the way until "import cppyy"? The use case is as follows: I have a prototype where I load the Reflex library (and associated C-API) at run-time, using _cffi_backend. That way, it is no longer necessary to link pypy-c with libReflex.so. I can handle a missing .so silently, but would prefer to raise an ImportError. However, for that to work, I need to make sure nothing of the module gets touched until it's imported, so that the error is reported at the right time. Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On Fri, Mar 29, 2013 at 9:14 PM, <wlavrijsen@lbl.gov> wrote:
Indeed, this runs ahead of time --- maybe already at translation time, actually. You can move this logic into a function _initialize_gbl(), and remove the appleveldefs entry for 'gbl'. Then call this new function at import time, by adding a method like this in the class Module in __init__.py: def startup(self, space): space.call_method(space.wrap(self), "_initialize_gbl") A bientôt, Armin.
Hi Armin,
Indeed, this runs ahead of time --- maybe already at translation time, actually.
yes, but that's fine: the code has no side effects. If it does not run at translation time, the value of "gbl" seems to be frozen at its initial one, so that's not an option.
That actually doesn't work: the translator complains about the setup of applevel objects being circular (don't have the exact error message handy). I've tried a range of variations, but no luck to get the desired outcome. But anyway after I decided to leave gbl as-is, and simply put the check in startup(), I found that it is run at program load, not on import, so that doesn't help in the first place. My current thinking is to weed the initialization down to the absolute minimum (that's a good thing regardless :) ), then fail on first use if the library is not available. Not ideal, but I guess in practice it makes little or no difference. Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On Sat, Mar 30, 2013 at 7:08 AM, <wlavrijsen@lbl.gov> wrote:
At this point, you need to provide more information (as I can't run the tests myself). Fwiw the patch I have in mind would look like this: http://bpaste.net/show/87755/ and I have no clue about which part of the code contains an error message about being circular. A bientôt, Armin.
Hi Armin,
Fwiw the patch I have in mind would look like this: http://bpaste.net/show/87755/
this line: - 'gbl' : 'pythonify.gbl', can not be removed, as w/o it 'gbl' will not show up at the module level.
and I have no clue about which part of the code contains an error message about being circular.
Looks like this (and I get it when running the translation with the patch as above, but with 'gbl' still in the applevel defs). Sorry, the message said recursive, not circular: [translation:ERROR] File "/home/wlav/pypy-dev/pypy/pypy/interpreter/mixedmodule.py", line 81, in getdictvalue [translation:ERROR] return self._load_lazily(space, name) [translation:ERROR] File "/home/wlav/pypy-dev/pypy/pypy/interpreter/mixedmodule.py", line 91, in _load_lazily [translation:ERROR] w_value = loader(space) [translation:ERROR] File "/home/wlav/pypy-dev/pypy/pypy/interpreter/mixedmodule.py", line 245, in afileloader [translation:ERROR] return app.wget(space, attrname) [translation:ERROR] File "/home/wlav/pypy-dev/pypy/pypy/interpreter/gateway.py", line 999, in wget [translation:ERROR] w_globals = self.getwdict(space) [translation:ERROR] File "/home/wlav/pypy-dev/pypy/pypy/interpreter/gateway.py", line 992, in getwdict [translation:ERROR] return space.fromcache(ApplevelCache).getorbuild(self) [translation:ERROR] File "/home/wlav/pypy-dev/pypy/rpython/rlib/cache.py", line 48, in getorbuild [translation:ERROR] self, key) [translation:ERROR] Exception: ("<Cache 'ApplevelCache' (18 items)> recursive building of <ApplevelClass filename='/home/wlav/pypy-dev/pypy/pypy/module/cppyy/pythonify.py'>", < [translation:ERROR] v42 = getattr(self_2, ('builtin_modules')) [translation:ERROR] In <FunctionGraph of (pypy.interpreter.baseobjspace:407)getbuiltinmodule at 0x2fe303a0>: [translation:ERROR] Happened at file /home/wlav/pypy-dev/pypy/pypy/interpreter/baseobjspace.py line 422 [translation:ERROR] [translation:ERROR] ==> w_mod = self.builtin_modules[name] [translation:ERROR] [translation:ERROR] Known variable annotations: [translation:ERROR] self_2 = SomePBC(can_be_None=False, const=StdObjSpace, subset_of=None)>) [translation:ERROR] Processing block: [translation:ERROR] block@120 is a <class 'rpython.flowspace.flowcontext.SpamBlock'> [translation:ERROR] in (pypy.interpreter.baseobjspace:407)getbuiltinmodule [translation:ERROR] containing the following operations: [translation:ERROR] v42 = getattr(self_2, ('builtin_modules')) [translation:ERROR] v43 = getitem_key(v42, name_0) [translation:ERROR] --end-- But again, the real problem is that startup() still runs immediately when I type './pypy-c' and not when 'import cppyy' later on the prompt, so even if the above problem gets solved, I still can't raise an ImportError in startup() when the library is missing. That is, if I leave the setup as-is and simply call verify_backend() in the startup() function in cppyy/__init__.py, this is how it looks if the library is missing: $ ./pypy-c Traceback (most recent call last): File "app_main.py", line 52, in run_toplevel ImportError: missing reflection module rflxlib.so! Python 2.7.3 (2377fb34943d+3ef424d4281f+, Mar 30 2013, 03:53:47) [PyPy 2.0.0-beta1 with GCC 4.6.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``things worked nicely till today just by chance''
(And copying rflxlib.so to an accessible location nicely makes the error message go away, so the checking does work properly, unless that message I got on the prompt is being prophetic. :) ) Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On Sat, Mar 30, 2013 at 5:40 PM, <wlavrijsen@lbl.gov> wrote:
- 'gbl' : 'pythonify.gbl', can not be removed, as w/o it 'gbl' will not show up at the module level.
Ah I see, pythonify.py does "sys.modules['cppyy.gbl'] = gbl". I somehow misread that. It should also do in the same place "cppyy.gbl = gbl" to force the value to show up when this initialization code runs.
But again, the real problem is that startup() still runs immediately when I type './pypy-c' and not when 'import cppyy' later on the prompt
Uh, that's strange. The docstrings in interpreter/module.py say specifically the opposite. But the truth looks a bit more complicated indeed, e.g. it depends if getbuiltinmodule('cppyy') was already called during translation or not... A bientôt, Armin.
Hi Armin,
yes, that is called, and it is being called by having applevel defs, b/c pythonify.py does an "import cppyy" at the module level. So, the solution then, is to not do that. :) I now have a _init_pythonify applevel def that does all setup that touches cppyy. That is then called in startup() as you told me to do. All other references to cppyy have an import just before them (i.e. inside the function) and the module-level one has been removed. Looks a bit strange, but does work as I want it: now nothing runs on ./pypy-c startup, and gbl & friends are build on import. Thanks! Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On Fri, Mar 29, 2013 at 9:14 PM, <wlavrijsen@lbl.gov> wrote:
Indeed, this runs ahead of time --- maybe already at translation time, actually. You can move this logic into a function _initialize_gbl(), and remove the appleveldefs entry for 'gbl'. Then call this new function at import time, by adding a method like this in the class Module in __init__.py: def startup(self, space): space.call_method(space.wrap(self), "_initialize_gbl") A bientôt, Armin.
Hi Armin,
Indeed, this runs ahead of time --- maybe already at translation time, actually.
yes, but that's fine: the code has no side effects. If it does not run at translation time, the value of "gbl" seems to be frozen at its initial one, so that's not an option.
That actually doesn't work: the translator complains about the setup of applevel objects being circular (don't have the exact error message handy). I've tried a range of variations, but no luck to get the desired outcome. But anyway after I decided to leave gbl as-is, and simply put the check in startup(), I found that it is run at program load, not on import, so that doesn't help in the first place. My current thinking is to weed the initialization down to the absolute minimum (that's a good thing regardless :) ), then fail on first use if the library is not available. Not ideal, but I guess in practice it makes little or no difference. Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On Sat, Mar 30, 2013 at 7:08 AM, <wlavrijsen@lbl.gov> wrote:
At this point, you need to provide more information (as I can't run the tests myself). Fwiw the patch I have in mind would look like this: http://bpaste.net/show/87755/ and I have no clue about which part of the code contains an error message about being circular. A bientôt, Armin.
Hi Armin,
Fwiw the patch I have in mind would look like this: http://bpaste.net/show/87755/
this line: - 'gbl' : 'pythonify.gbl', can not be removed, as w/o it 'gbl' will not show up at the module level.
and I have no clue about which part of the code contains an error message about being circular.
Looks like this (and I get it when running the translation with the patch as above, but with 'gbl' still in the applevel defs). Sorry, the message said recursive, not circular: [translation:ERROR] File "/home/wlav/pypy-dev/pypy/pypy/interpreter/mixedmodule.py", line 81, in getdictvalue [translation:ERROR] return self._load_lazily(space, name) [translation:ERROR] File "/home/wlav/pypy-dev/pypy/pypy/interpreter/mixedmodule.py", line 91, in _load_lazily [translation:ERROR] w_value = loader(space) [translation:ERROR] File "/home/wlav/pypy-dev/pypy/pypy/interpreter/mixedmodule.py", line 245, in afileloader [translation:ERROR] return app.wget(space, attrname) [translation:ERROR] File "/home/wlav/pypy-dev/pypy/pypy/interpreter/gateway.py", line 999, in wget [translation:ERROR] w_globals = self.getwdict(space) [translation:ERROR] File "/home/wlav/pypy-dev/pypy/pypy/interpreter/gateway.py", line 992, in getwdict [translation:ERROR] return space.fromcache(ApplevelCache).getorbuild(self) [translation:ERROR] File "/home/wlav/pypy-dev/pypy/rpython/rlib/cache.py", line 48, in getorbuild [translation:ERROR] self, key) [translation:ERROR] Exception: ("<Cache 'ApplevelCache' (18 items)> recursive building of <ApplevelClass filename='/home/wlav/pypy-dev/pypy/pypy/module/cppyy/pythonify.py'>", < [translation:ERROR] v42 = getattr(self_2, ('builtin_modules')) [translation:ERROR] In <FunctionGraph of (pypy.interpreter.baseobjspace:407)getbuiltinmodule at 0x2fe303a0>: [translation:ERROR] Happened at file /home/wlav/pypy-dev/pypy/pypy/interpreter/baseobjspace.py line 422 [translation:ERROR] [translation:ERROR] ==> w_mod = self.builtin_modules[name] [translation:ERROR] [translation:ERROR] Known variable annotations: [translation:ERROR] self_2 = SomePBC(can_be_None=False, const=StdObjSpace, subset_of=None)>) [translation:ERROR] Processing block: [translation:ERROR] block@120 is a <class 'rpython.flowspace.flowcontext.SpamBlock'> [translation:ERROR] in (pypy.interpreter.baseobjspace:407)getbuiltinmodule [translation:ERROR] containing the following operations: [translation:ERROR] v42 = getattr(self_2, ('builtin_modules')) [translation:ERROR] v43 = getitem_key(v42, name_0) [translation:ERROR] --end-- But again, the real problem is that startup() still runs immediately when I type './pypy-c' and not when 'import cppyy' later on the prompt, so even if the above problem gets solved, I still can't raise an ImportError in startup() when the library is missing. That is, if I leave the setup as-is and simply call verify_backend() in the startup() function in cppyy/__init__.py, this is how it looks if the library is missing: $ ./pypy-c Traceback (most recent call last): File "app_main.py", line 52, in run_toplevel ImportError: missing reflection module rflxlib.so! Python 2.7.3 (2377fb34943d+3ef424d4281f+, Mar 30 2013, 03:53:47) [PyPy 2.0.0-beta1 with GCC 4.6.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``things worked nicely till today just by chance''
(And copying rflxlib.so to an accessible location nicely makes the error message go away, so the checking does work properly, unless that message I got on the prompt is being prophetic. :) ) Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On Sat, Mar 30, 2013 at 5:40 PM, <wlavrijsen@lbl.gov> wrote:
- 'gbl' : 'pythonify.gbl', can not be removed, as w/o it 'gbl' will not show up at the module level.
Ah I see, pythonify.py does "sys.modules['cppyy.gbl'] = gbl". I somehow misread that. It should also do in the same place "cppyy.gbl = gbl" to force the value to show up when this initialization code runs.
But again, the real problem is that startup() still runs immediately when I type './pypy-c' and not when 'import cppyy' later on the prompt
Uh, that's strange. The docstrings in interpreter/module.py say specifically the opposite. But the truth looks a bit more complicated indeed, e.g. it depends if getbuiltinmodule('cppyy') was already called during translation or not... A bientôt, Armin.
Hi Armin,
yes, that is called, and it is being called by having applevel defs, b/c pythonify.py does an "import cppyy" at the module level. So, the solution then, is to not do that. :) I now have a _init_pythonify applevel def that does all setup that touches cppyy. That is then called in startup() as you told me to do. All other references to cppyy have an import just before them (i.e. inside the function) and the module-level one has been removed. Looks a bit strange, but does work as I want it: now nothing runs on ./pypy-c startup, and gbl & friends are build on import. Thanks! Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
participants (2)
-
Armin Rigo -
wlavrijsen＠lbl.gov