[Python-ideas] Length hinting and preallocation for container types

[Daniel Holth]

On Fri, Mar 8, 2013 at 10:45 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Daniel Holth, 08.03.2013 16:17:
>> On Fri, Mar 8, 2013 at 10:02 AM, Stefan Behnel wrote:
>>> Daniel Holth, 08.03.2013 15:55:
>>>> On Fri, Mar 8, 2013 at 9:48 AM, Stefan Behnel wrote:
>>>>> Eli Bendersky, 08.03.2013 15:40:
>>>>>> On Fri, Mar 8, 2013 at 6:04 AM, Daniel Holth wrote:
>>>>>>> On Fri, Mar 8, 2013 at 8:35 AM, Antoine Pitrou wrote:
>>>>>>>> Le Fri, 8 Mar 2013 08:18:07 -0500,
>>>>>>>> Daniel Holth a écrit :
>>>>>>>>> I am a fan of the proposal. Imagine you are programming for a
>>>>>>>>> memory-constrained system. By telling the list how big it needs to be
>>>>>>>>> you can save precious RAM.
>>>>>>>>
>>>>>>>> Is it an actual use case or are you just imagining it? :)
>>>>>>>> I'm asking because, unless you are only allocating that list and all
>>>>>>>> the objects contained it in it already exist, limiting the list's size
>>>>>>>> won't do much for the process' memory occupation.
>>>>>>>
>>>>>>> It might help if it was a list of integers between -1 and 99 and
>>>>>>> 1-character strings.
>>>>>>
>>>>>> That's not what you should use lists for if memory consumption matters. Use
>>>>>> http://docs.python.org/dev/library/array.html, especially if your integers
>>>>>> are in such a limited range.
>>>>>
>>>>> Yep, and regarding the second part, a string is a very efficient way to
>>>>> store many 1-character strings.
>>>>
>>>> I do know C
>>>
>>> So do I. This thread is about Python, though. At least, that's what I think
>>> it is.
>>
>> IIUC the JIT is smart enough to give me a very efficient list of
>> unboxed integers without having to change the type, increasing the
>> Pythonicity of my program.
>
> It may or may not. It's a runtime optimiser, there's no guarantee that it
> will always perform "as expected". For example, it may decide to optimise
> your list for integer values up to 255, and when you add a value 256 for
> some reason, it may have to reallocate and copy the whole list. And when
> you remove the last 256 value from the list, there is no guarantee that it
> will shrink your list back to the optimal size, it may just keep wasting
> memory. Oh, and it may actually waste memory right from the start, by not
> optimising your list for values up to 255 but for values up to 2**31,
> although all you actually wanted to store was values between 1 and 99, right?
>
> It's always a good idea to put some thoughts into the choice of the right
> data structure for your use case.
>
> So, that being said, should we discuss extending this proposal to add a new
> API for Python lists that allows defining the maximum value of integer
> values that you want to store in them? That would allow for some serious
> optimisations.

Definitely. list(hint=prime_numbers_only, length=42)