[Python-Dev] new buffer in python2.7

[Kristján Valur Jónsson]

So, I did some profiling.  It turns out that the difference is not due to the creation of the MemoryView object as such, but the much more complicated slice handling for generic objects.
Here is the interesting part of the inclusive sample measurements for the MemoryView:
Function Name	Inclusive Samples	Exclusive Samples	Inclusive Samples %	Exclusive Samples %
apply_slice	8.997	468	62,23	3,24
_PyObject_GetItem	6.257	400	43,28	2,77
memory_subscript	5.857	1.051	40,51	7,27
_PyMemoryView_FromBuffer	2.642	455	18,27	3,15
memory_dealloc	1.572	297	10,87	2,05
_PyObject_Malloc	1.374	1.374	9,50	9,50
__PyObject_GC_New	1.256	236	8,69	1,63
_PySlice_New	1.211	333	8,38	2,30
slice_dealloc	1.061	769	7,34	5,32
__PyObject_GC_Malloc	1.022	293	7,07	2,03
bytearray_getbuffer	987	354	6,83	2,45
dup_buffer	932	932	6,45	6,45

Notice how  a Slice object is generated.  Then a PyObject_GetItem() is done.  The salient code path is from apply_slice().  A slice object must be constructed and destroyed.

In contrast, when done on a string directly (or a bytes object in py3k) you go directly to PySequence_GetSlice.
Function Name	Inclusive Samples	Exclusive Samples	Inclusive Samples %	Exclusive Samples %
apply_slice	3.888	502	48,44	6,25
_PySequence_GetSlice	3.039	350	37,86	4,36
string_slice	2.689	281	33,50	3,50
_PyString_FromStringAndSize	2.409	575	30,01	7,16
[MSVCR90.dll]	1.413	1.407	17,61	17,53
string_dealloc	467	150	5,82	1,87
_PyObject_Malloc	379	379	4,72	4,72
_PyObject_Free	378	378	4,71	4,71
__PyEval_SliceIndex	347	347	4,32	4,32

(The measurements for the MemoryView above already contain some optimizations I've done on naïve functions).
This area is ripe for improvements.

K
-----Original Message-----
From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Kristján Valur Jónsson
Sent: Wednesday, October 27, 2010 20:00
To: Antoine Pitrou; python-dev at python.org
Subject: Re: [Python-Dev] new buffer in python2.7

Ah, well in 2.7 you don't have the luxury of a bytes object.  A str() would be similar in 2.7 Anyway, isn't the "bytes" object immutable? In that case, it is not a useful target for a sock.recv_into() call.
Calling getbuffer on a bytearray or a bytes object should be really cheap, so I still don't accept this as a matter of fact situation.

Btw, going to 3.2 isn't a real option for us any time soon.  A lot of companies probably find themselves in a similar situation.  This is why I spend so much effort applying some love to 2.7.  Most of the stuff I locally do to 2.7 I then contribute to python as 3.2 patches.  Someday they'll get backported, no doubt :)

K

-----Original Message-----
From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Antoine Pitrou
Sent: Wednesday, October 27, 2010 19:16
To: python-dev at python.org
Subject: Re: [Python-Dev] new buffer in python2.7


> >Here are micro-benchmarks under 3.2:
> 
> > $ ./python -m timeit -s "x = b'x'*10000" "x[:100]"
> > 10000000 loops, best of 3: 0.134 usec per loop $ ./python -m timeit 
> > -s "x = memoryview(b'x'*10000)" "x[:100]"
> > 10000000 loops, best of 3: 0.151 usec per loop
> 
> That's weird.  The greedy slice needs two memory allocations.  One for 
> the ByteArray object itself, one for its cargo.  In total, more that
> 100 bytes.  In contrast, creating the MemoryView object requires only 
> one allocation of a few dozen bytes.

It's not a bytearray object, it's a bytes object. It requires only a single allocation since the data is allocated inline.
The memoryview object must also call getbuffer() again on the original object.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com