[C++-sig] Re: New slice implementation

[Raoul Gough]

Jonathan Brandmeyer <jbrandmeyer at earthlink.net> writes:

> On Thu, 2004-01-08 at 07:22, Raoul Gough wrote:
>> I guess the main difference is whether this is returned via a
>> separate container, or via iterators into the existing
>> container. Note the potential problems from the Python side,
>> though, if the existing container disappears while those iterators
>> still exist.
>
> That bit about container lifetime is a very good point, but what I
> have in mind is performing modifying operations.  That is, I want to
> be able to write a function that uses a Python slice object to
> address which elements of the container that I want to operate on,
> such as this:
>
> double
> partial_sum( std::vector<double>* Foo, slice index)
> {
>     slice::range<std::vector<double> > bounds;
>     try {
>         bounds = index.get_indicies( Foo->begin(), Foo->end());
>     } catch (std::invalid_argument)
>         return 0.0;
>     double ret = 0.0;
>     while (bounds.start != bounds.stop) {
>         ret += *bounds.start;
>         std::advance( bounds.start, bounds.step);
>     }
>     ret += bounds.start;
>     return ret;
> }

Yes, I guess it makes a good deal of sense to use iterators in this
case.  However, how would you make use of them from Python code?

>> BTW, wouldn't it be a good idea to have a slice constructor that takes
>> a PySliceObject * as parameter?
>
> I try to avoid raw PyObject*'s whenever possible, but I think that the
> answer is "no".  The reason is that you have no idea how to properly
> manage it.  That is partially what the detail::new_reference<>,
> detail::borrowed_reference<>, and detail::new_non_null_reference<> are
> for, right?.  Feel free to correct me if I'm wrong.

I didn't necessarily mean a raw PySliceObject. All I'm getting at is,
if you want to implement __getitem__ then you will end up with a
PySliceObject created by the Python interpreter. AFAICS, you don't
have a way of generating one of your slice objects from this. Am I
missing something here?

>>> part = v[1::4]

calls v.__getitem__ with a PySliceObject (1, None, 4)

Regarding the borrowed_reference and so on, I did things that way at
first myself (I guess you just copied the existing code like I
did?). Apparently the preferred (and documented) way of doing this
kind of thing is via boost::python::handle instead.

>> > ---crash_test.py---
>> > # Uses the existing vector_indexing_suite_ext.cpp test modul
>> > from vector_indexing_suite_ext import *
>> > foo = FloatVec()
>> > # Weird initialization, why not supported by a constructor?
>> 
>> That's a good question, but it isn't necessarily that easy to
>> answer. At least, not if you want to use the container's
>> iterator-based constructor template. e.g. std::vector has a
>> constructor
>
> I don't think you reasonably can use those iterator-based constructors
> unless you have a way of creating a [begin,end) pair of generic
> iterators descended from boost::python::object.  Something that, when
> dereferenced, automatically calls extract<value_type>.  The 'begin'
> iterator would also have to trap for IndexErroror and StopIteration and
> compare equal to the 'end' iterator afterwords.  I smell another code
> contribution coming in a day or so for something just like this.

That would be great! I never quite convinced myself that it could be
done reliably - in particular, you have to convert one from-Python
parameter to two iterators before calling the constructor. You have to
know somehow when to do this, and when to convert the object to a
single C++ parameter (e.g. constructing vector(5)).

>
>> template <class InputIterator> 
>> vector(InputIterator f, InputIterator l, const Allocator& a = Allocator())
>> 
>> which would be the best one to use. I still haven't figured out the
>> details of providing this.
>
> Well, I don't know much about the metaprogramming guts of either suite,
> but I wrote this simple template to create a preallocated vector that I
> initialize with extract<>(), and exported it using "injected
> constructors" support:
>
> template<typename Container>
> boost::shared_ptr<Container>
> create_from_pysequence( object seq)
> {
>   boost::shared_ptr<Container> ret( 
>     new Container(extract<int>(seq.attr("__len__")())));
>   object seq_i = seq.attr("__iter__")();
>   for ( typename Container::iterator i = ret->begin(); i != ret->end();
> ++i) {
>         *i = extract< typename Container::value_type>( 
>             seq_i.attr("next")());
>     }
>     return ret;
> }

What does the constructor injection look like?

>
>
>> An easier way:
>> 
>> def print_vec(foo):
>>      print [x for x in foo]
>
> Oooh.  Nice.
>
>> >
>> > # Should raise IndexError, or print backwards; actually prints the 
>> > # original
>> > print_vector(foo[::-1])
>> 
>> That would be because the step value is ignored, right? 
>
> Yes.
>
>> In any case,
>> it's very useful to try this kind of test with a real Python list to
>> see what "should" happen:
>> 
>> >>> v = [1,2,3,4,5]
>> >>> print v[::-1]
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in ?
>> TypeError: sequence index must be integer
>> 
>> Would have to look that one up in the Python reference to see if it's
>> documented! The indexing_v2 suite prints [5,4,3,2] which also can't
>> really be right.
>
> Try it with Python 2.3.  In Python 2.2, slicing support was extended for
> builtin sequences to support __getitem__(slice(start,stop,step)),
> whereas in Python 2.2 you only have the __getslice__(start,stop) form.

Ah, OK. Just tried it in 2.3 and got [5,4,3,2,1]. Must be an
off-by-one error for the None case in my __getitem__.

>
> I think that the right way to handle negative step sizes (when you can
> choose the algorithm) is to use reverse iterators.  The reason is that
> when provided a negative step, the stop value defaults to "one before
> the beginning" and the start value defaults to the last element.  So
> long as you are using the [begin,end) ranges for iterators, the only way
> to make that work safely is with a reverse iterator and algorithm that
> carefully accounts for the effects of non-singular step size.

I don't try to support slices unless the container has random access
(and then there's no need for a reverse iterator).

>
>> >
>> > # I think this should raise IndexError; crashes.
>> > foo[-1:0] = [7, 8, 9]
>> 
>> With a real python list, it inserts 7, 8, 9 before the last element:
>> 
>> >>> v = [1,2,3,4]
>> >>> v[-1:0] = [7,8,9]
>> >>> print v
>> [1, 2, 3, 7, 8, 9, 4]
>
> Yes, that is what happens: performing an insertion before the provided
> start value.  However, I think that it should be an undefined operation
> since the expression is utter nonsense.  I've looked at the source code
> for PyListObject, and I think that this behavior is the result of bounds
> limiting rather than real error checking.

I don't understand this. Assigning something into an empty slice in a
container always performs an insertion. More generally, assigning a
longer list to a plain slice with fewer elements performs insertion.
e.g.

>>> l = [1,2,3,4]
>>> l[3:0] = [7,8,9]
>>> print l
[1, 2, 3, 7, 8, 9, 4]

so why shouldn't this be exactly the same (in this case) as
>>> l[-1:0] = [7,8,9]

>
> Furthermore if you try it with Numeric (the original source of rich
> slices), you will find that it is a no-op, which is what I would rather
> see in Boost now that I think a little more about it.

I haven't used Numeric before - is it documented somewhere?

>
> See Python bug# 873305 at http://sourceforge.net/tracker/?group_id=5470

I don't see any place to enter the bug number - how do I get to see
bug 873305?

-- 
Raoul Gough.
export LESS='-X'