[Python-ideas] Length hinting and preallocation for container types

[Stefan Behnel]

Daniel Holth, 08.03.2013 16:17:
> On Fri, Mar 8, 2013 at 10:02 AM, Stefan Behnel wrote:
>> Daniel Holth, 08.03.2013 15:55:
>>> On Fri, Mar 8, 2013 at 9:48 AM, Stefan Behnel wrote:
>>>> Eli Bendersky, 08.03.2013 15:40:
>>>>> On Fri, Mar 8, 2013 at 6:04 AM, Daniel Holth wrote:
>>>>>> On Fri, Mar 8, 2013 at 8:35 AM, Antoine Pitrou wrote:
>>>>>>> Le Fri, 8 Mar 2013 08:18:07 -0500,
>>>>>>> Daniel Holth a écrit :
>>>>>>>> I am a fan of the proposal. Imagine you are programming for a
>>>>>>>> memory-constrained system. By telling the list how big it needs to be
>>>>>>>> you can save precious RAM.
>>>>>>>
>>>>>>> Is it an actual use case or are you just imagining it? :)
>>>>>>> I'm asking because, unless you are only allocating that list and all
>>>>>>> the objects contained it in it already exist, limiting the list's size
>>>>>>> won't do much for the process' memory occupation.
>>>>>>
>>>>>> It might help if it was a list of integers between -1 and 99 and
>>>>>> 1-character strings.
>>>>>
>>>>> That's not what you should use lists for if memory consumption matters. Use
>>>>> http://docs.python.org/dev/library/array.html, especially if your integers
>>>>> are in such a limited range.
>>>>
>>>> Yep, and regarding the second part, a string is a very efficient way to
>>>> store many 1-character strings.
>>>
>>> I do know C
>>
>> So do I. This thread is about Python, though. At least, that's what I think
>> it is.
> 
> IIUC the JIT is smart enough to give me a very efficient list of
> unboxed integers without having to change the type, increasing the
> Pythonicity of my program.

It may or may not. It's a runtime optimiser, there's no guarantee that it
will always perform "as expected". For example, it may decide to optimise
your list for integer values up to 255, and when you add a value 256 for
some reason, it may have to reallocate and copy the whole list. And when
you remove the last 256 value from the list, there is no guarantee that it
will shrink your list back to the optimal size, it may just keep wasting
memory. Oh, and it may actually waste memory right from the start, by not
optimising your list for values up to 255 but for values up to 2**31,
although all you actually wanted to store was values between 1 and 99, right?

It's always a good idea to put some thoughts into the choice of the right
data structure for your use case.

So, that being said, should we discuss extending this proposal to add a new
API for Python lists that allows defining the maximum value of integer
values that you want to store in them? That would allow for some serious
optimisations.

Stefan