Armin Rigo posted an URL to a C extension module he has recently developed, and used. That module is able to copy running instances of a generator. Currently, it does so by just "knowing", and type by type dealing with, several types whose instances could be values referred to by the generator's frame. I was suggesting extending this so that, when values of other types are met, instead of automatically failing (or, as I believe I recall Armin's extension does today, copying the reference rather than the value), the copy process would...:

I haven't seen Armin's code, but I don't believe that the type alone gives enough information about whether they should be copied. Consider a generator that uses a dict as a cache or memo for values it computes. Multiple instances of the generator share the dict, but for efficiency the generator references it in a local variable. This dict should not be copied when copying the generator's stack frame. But consider another generator that uses a dict to hold some of its state. This dict should be copied.

...

...

...

Cool! Why don't you try copy.copy on types you don't automatically recognize and know how to deal with, BTW? That might make this

and Armin said that __copy__ seems weak to him but __deepcopy__ might not be:

...

...

I will try. Note that only __deepcopy__ makes sense, as far as I can tell, because there is too much state that really needs to be copied and not shared in a generator (in particular, the sequence iterators from 'for' loops).

So, now he went on to ask about __deepcopy__ and you answered:

...

...

I'm not sure about how deep-copying should be defined for built-in types. Should a built-in __deepcopy__ method try to import and call copy.deepcopy() on the sub-items? This doesn't seem to be right.

Almost -- you have to pass the memo argument that your __deepcopy__ received as the second argument to the recursive deepcopy() calls, to avoid looping on cycles.

Now, if Armin's code can only provide __deepcopy__ and not __copy__, then it's probably of little use wrt the __copy__ functionality I talk about in PEP 323 (which I still must revise to take into account your feedback and Raymond's -- plan to get to that as soon as I've cleared my mbox) -- the memory and time expenditure is likely to be too high for that.

Right.

It's still going to be a cool hack, well worth "publishing" as such,

As a third-party module? Sure.

and probably able to be "user-installed" as the way deepcopy deals with generators even though generators themselves may not sprout a __deepcopy__ method themselves (fortunately, copy.copy does a lot of "ad-hoc protocol adaptation" -- it's occasionally a bit rambling or hard to follow, but often allows plugging in "third party copiers" for types which their authors hadn't imagined would be copied or deep copied, so that other innocent client code which just calls copy.copy(x) will work... essentially how "real" adaptation would work, except that registering a third-party protocol adapter would be easier:-).

--Guido van Rossum (home page: http://www.python.org/~guido/)

This is a quite deep problem, actually. I admit I have never used copy.py because in all cases I needed more control about what should be copied or not. This generator-copier module that we are talking about is no exception: its existence is not only due to the fact that it can copy generators, but also that I needed precise control over what I copied and what I shared. Putting this information in __getstate__ or __copy__ methods of instances or in copy_reg only goes so far, because sometimes you want to do different things with the same instances in the same program -- e.g. you may want at some point only a copy of a small number of objects (e.g. to be able to rollback a small transaction), and at some other point a more complete copy of the state of the same program.

Nevertheless, I can surely make a C module that registers in copy_reg a deep copier for generators.

I'm not sure that there would be a general use for this... --Guido van Rossum (home page: http://www.python.org/~guido/)

Armin Rigo posted an URL to a C extension module he has recently developed, and used. That module is able to copy running instances of a generator. Currently, it does so by just "knowing", and type by type dealing with, several types whose instances could be values referred to by the generator's frame. I was suggesting extending this so that, when values of other types are met, instead of automatically failing (or, as I believe I recall Armin's extension does today, copying the reference rather than the value), the copy process would...:

I haven't seen Armin's code, but I don't believe that the type alone gives enough information about whether they should be copied. Consider a generator that uses a dict as a cache or memo for values it computes. Multiple instances of the generator share the dict, but for efficiency the generator references it in a local variable. This dict should not be copied when copying the generator's stack frame. But consider another generator that uses a dict to hold some of its state. This dict should be copied.

...

...

...

Cool! Why don't you try copy.copy on types you don't automatically recognize and know how to deal with, BTW? That might make this

and Armin said that __copy__ seems weak to him but __deepcopy__ might not be:

...

...

I will try. Note that only __deepcopy__ makes sense, as far as I can tell, because there is too much state that really needs to be copied and not shared in a generator (in particular, the sequence iterators from 'for' loops).

So, now he went on to ask about __deepcopy__ and you answered:

...

...

I'm not sure about how deep-copying should be defined for built-in types. Should a built-in __deepcopy__ method try to import and call copy.deepcopy() on the sub-items? This doesn't seem to be right.

Almost -- you have to pass the memo argument that your __deepcopy__ received as the second argument to the recursive deepcopy() calls, to avoid looping on cycles.

Now, if Armin's code can only provide __deepcopy__ and not __copy__, then it's probably of little use wrt the __copy__ functionality I talk about in PEP 323 (which I still must revise to take into account your feedback and Raymond's -- plan to get to that as soon as I've cleared my mbox) -- the memory and time expenditure is likely to be too high for that.

Right.

It's still going to be a cool hack, well worth "publishing" as such,

As a third-party module? Sure.

and probably able to be "user-installed" as the way deepcopy deals with generators even though generators themselves may not sprout a __deepcopy__ method themselves (fortunately, copy.copy does a lot of "ad-hoc protocol adaptation" -- it's occasionally a bit rambling or hard to follow, but often allows plugging in "third party copiers" for types which their authors hadn't imagined would be copied or deep copied, so that other innocent client code which just calls copy.copy(x) will work... essentially how "real" adaptation would work, except that registering a third-party protocol adapter would be easier:-).

--Guido van Rossum (home page: http://www.python.org/~guido/)

This is a quite deep problem, actually. I admit I have never used copy.py because in all cases I needed more control about what should be copied or not. This generator-copier module that we are talking about is no exception: its existence is not only due to the fact that it can copy generators, but also that I needed precise control over what I copied and what I shared. Putting this information in __getstate__ or __copy__ methods of instances or in copy_reg only goes so far, because sometimes you want to do different things with the same instances in the same program -- e.g. you may want at some point only a copy of a small number of objects (e.g. to be able to rollback a small transaction), and at some other point a more complete copy of the state of the same program.

Nevertheless, I can surely make a C module that registers in copy_reg a deep copier for generators.

I'm not sure that there would be a general use for this... --Guido van Rossum (home page: http://www.python.org/~guido/)