wrapped std::vector slice affecting items in a python list
Hi, I've come across some unusual behaviour that I was hoping some one might be able to explain to me. In python I have a list and a wrapped std::vector<MyCustomType> (using the vector_indexing_suite) where the list contains the items in the vector. If I slice insert items into the vector then the items in the python list change. The attached example illustrates this. Is this behaviour expected ? I'm quite confused as to what's going on so any help clarifying this would be greatly appreciated. thanks, Babak http://boost.2283326.n4.nabble.com/file/n3738972/test.py test.py http://boost.2283326.n4.nabble.com/file/n3738972/source.cpp source.cpp -- View this message in context: http://boost.2283326.n4.nabble.com/wrapped-std-vector-slice-affecting-items-... Sent from the Python - c++-sig mailing list archive at Nabble.com.
On 08/12/2011 04:28 AM, babak wrote:
Hi,
I've come across some unusual behaviour that I was hoping some one might be able to explain to me.
In python I have a list and a wrapped std::vector<MyCustomType> (using the vector_indexing_suite) where the list contains the items in the vector. If I slice insert items into the vector then the items in the python list change. The attached example illustrates this.
Is this behaviour expected ?
I'm quite confused as to what's going on so any help clarifying this would be greatly appreciated. thanks, Babak
http://boost.2283326.n4.nabble.com/file/n3738972/test.py test.py http://boost.2283326.n4.nabble.com/file/n3738972/source.cpp source.cpp
Some of the stuff vector_indexing_suite is sufficiently complex that at least I'd say this behavior is not unexpected. After all, when you slice a Python list, and modify the elements in the sliced bit, you modify the original list as well (unless your elements are immutable things like strings or numbers, and in that case you aren't really modifying them, you're replacing them). I think the thing you get back from slicing a std::vector is probably a proxy object designed to have exactly that behavior (note: I haven't checked; that's just how I'd have implemented it). In any case, I'd recommend against wrapping a std::vector of a mutable C++ type directly in Python. It's almost always unsafe, because any operation on the vector that invalidates iterators will destroy the elements of the vector, even if there are Python objects that point to those elements elsewhere with nonzero reference counts. It's much safer to use a vector of shared_ptr, or always convert to Python lists and avoid vector_indexing_suite. This isn't a Boost.Python-specific problem, it's an inherent incompatibility between C++'s value-based containers and Python's shared-reference ones. Jim Bosch
Hi Jim Thanks for your response. I'm abit surprised that this is the case, I kind of thought that indexing_suite was there to bridge this incompatibility. Is it normal then for large projects reliant on value based containers to copy data when going from python to c++ ? thanks babak -- View this message in context: http://boost.2283326.n4.nabble.com/wrapped-std-vector-slice-affecting-items-... Sent from the Python - c++-sig mailing list archive at Nabble.com.
On 08/16/2011 07:32 AM, babak wrote:
Hi Jim
Thanks for your response.
I'm abit surprised that this is the case, I kind of thought that indexing_suite was there to bridge this incompatibility. Is it normal then for large projects reliant on value based containers to copy data when going from python to c++ ?
I don't really know what normal is; my particular area is scientific computing, so I actually use numpy arrays in almost all the cases where I'm dealing with huge containers. I do tend to copy data when dealing with small containers of more complex objects, and when it's really an issue, I make my C++ objects cheap to copy (via copy on write or something). Things are also a lot safer if you only wrap vectors as const; it's exposing the mutators to Python that gets really difficult. Finally, if your C++ code works on iterators rather than containers, you can avoid copying by using stl_input_iterator to iterate directly over the Python containers. One large project I'm a part of uses vectors of shared_ptr extensively on the C++ side to combat this problem (even though they use SWIG, not Boost.Python). If you have control over the C++, that's often the best solution, regardless of how you're exposing them to Python. I would that any large project that uses SWIG with value-based containers is probably mostly ignoring the safety issues; unlike Boost.Python, SWIG simply doesn't have the facilities to protect you from null pointers and dangling references, so its STL wrappers don't do anything to prevent such things. The onus is on the Python user to be safe. On the Boost.Python side, indexing_suite essentially does the best it can to make things safe and full-featured, and while it can't totally foolproof your code, it does make it a lot more difficult to segfault accidentally. If you find yourself needing it a lot, you may want to looking into indexing suite v2, which I believe is best obtained through the Py++ package. Jim
Okay thanks for the info. Things are also a lot safer if you only wrap vectors as const; it's exposing
the mutators to Python that gets really difficult.
Is the const-ness of a wrapped object just a side effect of not exposing methods which modify it or is it due to something else more explicit ? thanks!
On 08/16/2011 11:29 AM, Babak Khataee wrote:
Okay thanks for the info.
Things are also a lot safer if you only wrap vectors as const; it's exposing the mutators to Python that gets really difficult.
Is the const-ness of a wrapped object just a side effect of not exposing methods which modify it or is it due to something else more explicit ?
The former. It's really only the methods that can invalidate iterators (i.e. those that add or remove elements) that you need to worry about. Non-const access to individual elements is totally safe on its own. Jim
okay cool. I think that's a reasonable compromise. It should also (hopefully) make it possible to move away from the indexing suite as well. thanks for your help! On 16 August 2011 21:41, Jim Bosch <talljimbo@gmail.com> wrote:
On 08/16/2011 11:29 AM, Babak Khataee wrote:
Okay thanks for the info.
Things are also a lot safer if you only wrap vectors as const; it's exposing the mutators to Python that gets really difficult.
Is the const-ness of a wrapped object just a side effect of not exposing methods which modify it or is it due to something else more explicit ?
The former. It's really only the methods that can invalidate iterators (i.e. those that add or remove elements) that you need to worry about. Non-const access to individual elements is totally safe on its own.
Jim
Hi, Thought you might be interested to know that I think the cause of the problem in my example is the iter method that the indexing suite adds. Unlike getitem it doesn't seem to return proxy objects, so when I slice insert into the cpp container some of the references made by iterating through it change, instead of becoming detached. babak -- View this message in context: http://boost.2283326.n4.nabble.com/wrapped-std-vector-slice-affecting-items-... Sent from the Python - c++-sig mailing list archive at Nabble.com.
participants (3)
-
babak -
Babak Khataee
-
Jim Bosch