Function-like Modules (better control/introspection)
hello, could we make modules work more like this... https://gist.github.com/xtfxme/9556806 tl;dr: __code__ object available (potentially a __self__ object), reload by calling, makes working with modules more natural/safe! ENCLOSED: modules-next.py -- C Anthony #----------------------------------------------------------------------------------------( modules-next.py ) # encoding: utf-8 # # modules-next # # C Anthony Risinger import sys m_name = 'modules_next' m_source = r"""\ if __name__ == '{0}': from . import __call__ as reexec from . import __code__ as code from . import __dict__ as ns print('>>> {{0}}: {{1}}\n{{2}}\n'.format(__name__, code, ns.keys())) __name__ = 'modules_next_reloaded' reexec() print('>>> {{0}}: {{1}}\n{{2}}\n'.format(__name__, code, ns.keys())) else: print('!!! __name__ == {{0}}\n'.format(__name__)) def throw(): raise Exception(__name__) """.format(m_name) m_code = compile(m_source, m_name, 'exec') m_dict = { '__file__': __file__ + '.next', '__name__': m_name, '__path__': list(), '__doc__': None, } #...bind namespace/functions module = eval('lambda: None', m_dict) #...automatically expose as attributes module.__dict__ = module.__globals__ #...must be set twice, __code__ sees __dict__, but we see descriptor module.__name__ = m_name #...redefine the function body! module.__code__ = m_code #...yay! we haz legitimate module! sys.modules[m_name] = module print('--* importing...') import modules_next print('>>> {0}: {1}\n{2}\n'.format( __name__, modules_next.__code__, modules_next.__dict__.keys(), )) print('--* importing (for real this time)...') #...import/reload by calling our func-y-module module() print('>>> {0}: {1}\n{2}\n'.format( __name__, modules_next.__code__, modules_next.__dict__.keys(), )) print('--* throwing...') modules_next.throw()
ok, let me try this again, sans distractions: could we consider adding __code__ and __call__ to module objects? __code__ would contain the code executed in the module __dict__, and __call__ would reload the module. thoughts? -- C Anthony On Fri, Mar 14, 2014 at 4:03 PM, C Anthony Risinger <anthony@xtfx.me> wrote:
hello,
could we make modules work more like this...
https://gist.github.com/xtfxme/9556806
tl;dr: __code__ object available (potentially a __self__ object), reload by calling, makes working with modules more natural/safe!
ENCLOSED: modules-next.py
--
C Anthony
#----------------------------------------------------------------------------------------( modules-next.py ) # encoding: utf-8 # # modules-next # # C Anthony Risinger
import sys
m_name = 'modules_next' m_source = r"""\ if __name__ == '{0}': from . import __call__ as reexec from . import __code__ as code from . import __dict__ as ns print('>>> {{0}}: {{1}}\n{{2}}\n'.format(__name__, code, ns.keys())) __name__ = 'modules_next_reloaded' reexec() print('>>> {{0}}: {{1}}\n{{2}}\n'.format(__name__, code, ns.keys())) else: print('!!! __name__ == {{0}}\n'.format(__name__))
def throw(): raise Exception(__name__) """.format(m_name) m_code = compile(m_source, m_name, 'exec') m_dict = { '__file__': __file__ + '.next', '__name__': m_name, '__path__': list(), '__doc__': None, } #...bind namespace/functions module = eval('lambda: None', m_dict) #...automatically expose as attributes module.__dict__ = module.__globals__ #...must be set twice, __code__ sees __dict__, but we see descriptor module.__name__ = m_name #...redefine the function body! module.__code__ = m_code #...yay! we haz legitimate module! sys.modules[m_name] = module
print('--* importing...') import modules_next print('>>> {0}: {1}\n{2}\n'.format( __name__, modules_next.__code__, modules_next.__dict__.keys(), ))
print('--* importing (for real this time)...') #...import/reload by calling our func-y-module module() print('>>> {0}: {1}\n{2}\n'.format( __name__, modules_next.__code__, modules_next.__dict__.keys(), ))
print('--* throwing...') modules_next.throw()
On 18 Mar 2014 01:59, "C Anthony Risinger" <anthony@xtfx.me> wrote:
ok, let me try this again, sans distractions:
could we consider adding __code__ and __call__ to module objects?
__code__ would contain the code executed in the module __dict__, and __call__ would reload the module.
thoughts?
The question to ask is: what would such a change make possible that importlib doesn't already handle? This specific proposal wouldn't work right for extension modules, so people will still need to go through the importlib APIs anyway. If you haven't seen it yet, you may also want to give PEP 451 a read - we already overhauled several aspects of the import system for 3.4. Cheers, Nick.
--
C Anthony
On Fri, Mar 14, 2014 at 4:03 PM, C Anthony Risinger <anthony@xtfx.me>
wrote:
hello,
could we make modules work more like this...
https://gist.github.com/xtfxme/9556806
tl;dr: __code__ object available (potentially a __self__ object), reload
by calling, makes working with modules more natural/safe!
ENCLOSED: modules-next.py
--
C Anthony
#----------------------------------------------------------------------------------------( modules-next.py )
# encoding: utf-8 # # modules-next # # C Anthony Risinger
import sys
m_name = 'modules_next' m_source = r"""\ if __name__ == '{0}': from . import __call__ as reexec from . import __code__ as code from . import __dict__ as ns print('>>> {{0}}: {{1}}\n{{2}}\n'.format(__name__, code, ns.keys())) __name__ = 'modules_next_reloaded' reexec() print('>>> {{0}}: {{1}}\n{{2}}\n'.format(__name__, code, ns.keys())) else: print('!!! __name__ == {{0}}\n'.format(__name__))
def throw(): raise Exception(__name__) """.format(m_name) m_code = compile(m_source, m_name, 'exec') m_dict = { '__file__': __file__ + '.next', '__name__': m_name, '__path__': list(), '__doc__': None, } #...bind namespace/functions module = eval('lambda: None', m_dict) #...automatically expose as attributes module.__dict__ = module.__globals__ #...must be set twice, __code__ sees __dict__, but we see descriptor module.__name__ = m_name #...redefine the function body! module.__code__ = m_code #...yay! we haz legitimate module! sys.modules[m_name] = module
print('--* importing...') import modules_next print('>>> {0}: {1}\n{2}\n'.format( __name__, modules_next.__code__, modules_next.__dict__.keys(), ))
print('--* importing (for real this time)...') #...import/reload by calling our func-y-module module() print('>>> {0}: {1}\n{2}\n'.format( __name__, modules_next.__code__, modules_next.__dict__.keys(), ))
print('--* throwing...') modules_next.throw()
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Tue, 18 Mar 2014 02:36:31 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
On 18 Mar 2014 01:59, "C Anthony Risinger" <anthony@xtfx.me> wrote:
ok, let me try this again, sans distractions:
could we consider adding __code__ and __call__ to module objects?
__code__ would contain the code executed in the module __dict__, and __call__ would reload the module.
thoughts?
The question to ask is: what would such a change make possible that importlib doesn't already handle?
I suppose module.__code__ would be a lot less wasteful than module.__loader__.get_code(module.__name__) (which seems to reload the bytecode from disk every time). (while looking at this, I was surprised that dis.dis(some_module) 1) doesn't show a disassembly for the module toplevel 2) shows disassembly for objects which some_module imports from other modules) Regards Antoine.
I don't know whether it's worth the savings, but the __code__ for a module's toplevel is thrown away because it is not needed after it has run. Reading it back from disk is wasteful if you do it every time for every module, but how often is it used in a situation where speed matters? Saving __code__ might be useful so you can tell with certainty which code was run -- the file may have been modified since it was imported. On Mon, Mar 17, 2014 at 10:11 AM, Antoine Pitrou <solipsis@pitrou.net>wrote:
On Tue, 18 Mar 2014 02:36:31 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
On 18 Mar 2014 01:59, "C Anthony Risinger" <anthony@xtfx.me> wrote:
ok, let me try this again, sans distractions:
could we consider adding __code__ and __call__ to module objects?
__code__ would contain the code executed in the module __dict__, and __call__ would reload the module.
thoughts?
The question to ask is: what would such a change make possible that importlib doesn't already handle?
I suppose module.__code__ would be a lot less wasteful than module.__loader__.get_code(module.__name__) (which seems to reload the bytecode from disk every time).
(while looking at this, I was surprised that dis.dis(some_module) 1) doesn't show a disassembly for the module toplevel 2) shows disassembly for objects which some_module imports from other modules)
Regards
Antoine.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
On 18 Mar 2014 03:16, "Antoine Pitrou" <solipsis@pitrou.net> wrote:
On Tue, 18 Mar 2014 02:36:31 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
On 18 Mar 2014 01:59, "C Anthony Risinger" <anthony@xtfx.me> wrote:
ok, let me try this again, sans distractions:
could we consider adding __code__ and __call__ to module objects?
__code__ would contain the code executed in the module __dict__, and __call__ would reload the module.
thoughts?
The question to ask is: what would such a change make possible that importlib doesn't already handle?
I suppose module.__code__ would be a lot less wasteful than module.__loader__.get_code(module.__name__) (which seems to reload the bytecode from disk every time).
Yep, it's a space/speed trade-off - keeping this info around could waste a lot of memory, and most code will never need it. It's a similar reason to why we throw source code away after compiling it, but then also route inspect.getsource() through the linecache module. Cheers, Nick.
(while looking at this, I was surprised that dis.dis(some_module) 1) doesn't show a disassembly for the module toplevel 2) shows disassembly for objects which some_module imports from other modules)
I don't think we have any tests to see how dis.dis behaves with module objects - it's likely just hitting the path that is designed to disassemble all methods on a class. (We throw away the bytecode after execution for classes as well) Cheers, Nick.
Regards
Antoine.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Tue, 18 Mar 2014 03:38:47 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
(while looking at this, I was surprised that dis.dis(some_module) 1) doesn't show a disassembly for the module toplevel 2) shows disassembly for objects which some_module imports from other modules)
I don't think we have any tests to see how dis.dis behaves with module objects - it's likely just hitting the path that is designed to disassemble all methods on a class. (We throw away the bytecode after execution for classes as well)
A comment in dis.dis's source code does talk about modules, so apparently someone thought about them, but not very hard :-) (edit: that someone is you - b2a9a59c2d57 ;-P) Regards Antoine.
On 18 Mar 2014 03:45, "Antoine Pitrou" <solipsis@pitrou.net> wrote:
On Tue, 18 Mar 2014 03:38:47 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
(while looking at this, I was surprised that dis.dis(some_module) 1) doesn't show a disassembly for the module toplevel 2) shows disassembly for objects which some_module imports from other modules)
I don't think we have any tests to see how dis.dis behaves with module objects - it's likely just hitting the path that is designed to
disassemble
all methods on a class. (We throw away the bytecode after execution for classes as well)
A comment in dis.dis's source code does talk about modules, so apparently someone thought about them, but not very hard :-)
(edit: that someone is you - b2a9a59c2d57 ;-P)
Huh, clearly I put a lot of thought into that one... One day I will learn to go check the source rather than speculating from memory, but apparently yesterday is not that day :) Cheers, Nick.
Regards
Antoine.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
participants (4)
-
Antoine Pitrou -
C Anthony Risinger -
Guido van Rossum -
Nick Coghlan