Technique for safely reloading dynamically generated module
Peter Otten
__peter__ at web.de
Thu Jul 21 11:40:16 EDT 2016
Malcolm Greene wrote:
> We're designing a server application that parses a custom DSL (domain
> specific language) source file, generates a Python module with the
> associated logic, and runs the associated code. Since this is a server
> application, we need to reload the module after each regeneration. Is
> this process is simple as the following pseudo code or are there other
> issues we need to be aware of? Are there better techniques for this
> workflow (eval, compile, etc)?
>
> We're working in Python 3.5.1.
>
> import importlib
>
> # custom_code is the module our code will generate - a version of this
> # file will always be present
> # if custom_code.py is missing, a blank version of this file is created
> # before this step
> import custom_code
>
> while True:
> # (re)generates custom_code.py visible in sys.path
> generate_custom_code( source_file )
>
> # reload the module whose source we just generated
> importlib.reload( custom_code )
>
> # run the main code in generated module
> custom_code.go()
If the go() function in that loop is the only place where the generated
module is used you can avoid writing code to the file system altogether.
Here's a really simple alternative suggestion based on exec():
SOURCEFILE = "whatever.dsl"
def make_func(sourcefile):
ns = {}
exec(generated_code(sourcefile), ns)
return ns["go"]
def generated_code(sourcefile):
# placeholder for your actual source generation
return """
def go():
print("Generated from", {!r})
""".format(sourcefile)
while True:
# XXX should we wait for updates of sourcefile here?
go = make_func(SOURCEFILE)
go()
More information about the Python-list
mailing list