Isolated (?transactional) exec (?subinterpreter) calls
Hi, Having a lot of ideas is a curse, because I can barely follow up on them, but I try - I really read replies, just don't have enough energy to answer immediately (as it usually requires some time for research). Here is another one that ripes too long to become rotten: Make exec(code[, globals[, locals]]) calls consistent, optionally isolated from parent environment and transactional. Consistent: - it should not matter if the code is executed with command line interpreter or from exec(), code should not be modified to successfully run in exec if it successfully runs in intepreter session http://bugs.python.org/issue13557 http://bugs.python.org/issue14049 http://bugs.python.org/issue1167300 http://bugs.python.org/issue991196 Optionally isolated from parent environment: - a feature to execute user script in a snapshot of current environment and have a choice whenever to merge its modifications back to environment or not real user story - read system configuration settings, where optional detection rules are written in Python (Blender/SCons build scripts) - autodetection probes can affect environment while detection takes place and it can lead to more problems later (think of virtualenv on Python process level with defined data exchange protocol through globals/locals variables) Transactional: - well, if it is isolated - it is already transactional - an ability to discard results if an error or an exception inside exec() occurs - getting back to the state right before exec. -- anatoly t.
2012/6/8 anatoly techtonik <techtonik@gmail.com>
It would be a really interesting feature, but seems very difficult to implement. Do you have the slightest idea how this would work? What about global state, environment variables, threads, and all kinds of side-effects? Or are you thinking about a solution based on the multiprocessing module? -- Amaury Forgeot d'Arc
On Fri, Jun 8, 2012 at 4:00 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
For my original user story both approaches will suffice. I've never used multiprocessing (mostly because 2.6 only compatibility) and it looks like it is capable to do what I want with some tweaks. But first approach with fine-grained environment control (object space, state, memory) will be more beneficial for Python as it can bring a nice research methodology for interpreter improvements (and hopefully some pictures). Two things are required: 1. Execution rollback 2. Scope control Execution rollback (or transaction) can be either "save the state and restore" or "keep track of changes and discard". On a lowest possible level it is something like using memory copy-on-write while Python bytecode modifies it and discarding the copied stuff in the end. Like in OSI model for networking, this low level memory and code abstraction is the 1st layer. But you're absolutely right about global state, environment variables, threads and other stuff - when we jump to a higher layer - rolling back execution pointer to a saved checkpoint and discarding memory will not be enough. We need to ensure that reverted operation did not affect state of the system outside the execution scope. "Scope control" means that every pathway when execution can alter global state outside needs to be carefully recorded and classified. It will then be possible to detect "escaped" transactions automatically and detect if the operation is safe to revert or not.
On 6/8/2012 4:16 AM, anatoly techtonik wrote:
These 4 duplicate issues are all about misuse of exec. The code in each *does* run if exec is passed just one namespace instead of two. When people pass two separate namespaces, the code executes as if embedded in a class definition. Since 'docs@python' never applied my suggested doc patch on 13557, I will take a stab at it.
You can do at least some of that now. -- Terry Jan Reedy
2012/6/8 anatoly techtonik <techtonik@gmail.com>
It would be a really interesting feature, but seems very difficult to implement. Do you have the slightest idea how this would work? What about global state, environment variables, threads, and all kinds of side-effects? Or are you thinking about a solution based on the multiprocessing module? -- Amaury Forgeot d'Arc
On Fri, Jun 8, 2012 at 4:00 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
For my original user story both approaches will suffice. I've never used multiprocessing (mostly because 2.6 only compatibility) and it looks like it is capable to do what I want with some tweaks. But first approach with fine-grained environment control (object space, state, memory) will be more beneficial for Python as it can bring a nice research methodology for interpreter improvements (and hopefully some pictures). Two things are required: 1. Execution rollback 2. Scope control Execution rollback (or transaction) can be either "save the state and restore" or "keep track of changes and discard". On a lowest possible level it is something like using memory copy-on-write while Python bytecode modifies it and discarding the copied stuff in the end. Like in OSI model for networking, this low level memory and code abstraction is the 1st layer. But you're absolutely right about global state, environment variables, threads and other stuff - when we jump to a higher layer - rolling back execution pointer to a saved checkpoint and discarding memory will not be enough. We need to ensure that reverted operation did not affect state of the system outside the execution scope. "Scope control" means that every pathway when execution can alter global state outside needs to be carefully recorded and classified. It will then be possible to detect "escaped" transactions automatically and detect if the operation is safe to revert or not.
On 6/8/2012 4:16 AM, anatoly techtonik wrote:
These 4 duplicate issues are all about misuse of exec. The code in each *does* run if exec is passed just one namespace instead of two. When people pass two separate namespaces, the code executes as if embedded in a class definition. Since 'docs@python' never applied my suggested doc patch on 13557, I will take a stab at it.
You can do at least some of that now. -- Terry Jan Reedy
participants (4)
-
Amaury Forgeot d'Arc -
anatoly techtonik -
Simon Sapin -
Terry Reedy