[Python-ideas] Length hinting and preallocation for container types

Eli Bendersky eliben at gmail.com
Fri Mar 8 14:27:59 CET 2013


On Fri, Mar 8, 2013 at 5:18 AM, Daniel Holth <dholth at gmail.com> wrote:

> On Fri, Mar 8, 2013 at 5:07 AM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> > Le Fri, 08 Mar 2013 08:16:59 +0900,
> > "Stephen J. Turnbull" <stephen at xemacs.org>
> > a écrit :
> >>
> >>  > > But why do you think allocation is slow in the general case?
> >>  > > Sure, it involves system calls which indeed do slow things down.
> >>  >
> >>  > It depends what one calls a system call. A memory allocation
> >>  > shouldn't always incur a call to the kernel. Library calls can be
> >>  > quite fast.
> >>
> >> If the process itself is growing, eventually it does.
> >
> > I think most allocators would request big chunks of memory from the
> > kernel, and then carve out the small blocks requested by the user from
> > that. Therefore, my intuition is that a long-running process, if not
> > leaky, should end up not stressing mmap / sbrk system calls too much.
> >
> >> And I suspect that Alex (and Christian!) would consider saving a few
> >> CPU cycles in an inner loop very important.
> >
> > They probably would, but that is not a design point for Python.
> >
> > Regards
> >
> > Antoine.
>
> I am a fan of the proposal. Imagine you are programming for a
> memory-constrained system. By telling the list how big it needs to be
> you can save precious RAM. It's a pretty standard feature to be able
> to hint and trim the size of data structures, just like you can
> usually choose the buffer size for stream operations.
> __________________________________
>

If it's voting time, I'm -1. Having programmed a lot of memory-constrained
systems (not in Python, though) - this is not how things usually work
there. In a memory-constrained system, you don't "grow and shrink" your
data structures. That's because growing often needs to reallocate the whole
chunk and do a copy, and shrinking only helps memory fragmentation. In such
systems, you usually know in advance or at least limit the size of data
structures and pre-allocate, which is perfectly possible in Python today.
Shrinking is rarely, if ever, useful. If it is, you can implement concrete
data structures for your concrete needs. And Python has a lot of ways to
save memory for large arrays of things (array, numpy, encoding in bytes,
etc) if one really wants to. I just don't believe the proposal will help in
a lot of realistic code, and it certainly goes against the way of Python.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130308/fb4c210a/attachment.html>


More information about the Python-ideas mailing list