[Python-ideas] Length hinting and preallocation for container types
eliben at gmail.com
Fri Mar 8 14:27:59 CET 2013
On Fri, Mar 8, 2013 at 5:18 AM, Daniel Holth <dholth at gmail.com> wrote:
> On Fri, Mar 8, 2013 at 5:07 AM, Antoine Pitrou <solipsis at pitrou.net>
> > Le Fri, 08 Mar 2013 08:16:59 +0900,
> > "Stephen J. Turnbull" <stephen at xemacs.org>
> > a écrit :
> >> > > But why do you think allocation is slow in the general case?
> >> > > Sure, it involves system calls which indeed do slow things down.
> >> >
> >> > It depends what one calls a system call. A memory allocation
> >> > shouldn't always incur a call to the kernel. Library calls can be
> >> > quite fast.
> >> If the process itself is growing, eventually it does.
> > I think most allocators would request big chunks of memory from the
> > kernel, and then carve out the small blocks requested by the user from
> > that. Therefore, my intuition is that a long-running process, if not
> > leaky, should end up not stressing mmap / sbrk system calls too much.
> >> And I suspect that Alex (and Christian!) would consider saving a few
> >> CPU cycles in an inner loop very important.
> > They probably would, but that is not a design point for Python.
> > Regards
> > Antoine.
> I am a fan of the proposal. Imagine you are programming for a
> memory-constrained system. By telling the list how big it needs to be
> you can save precious RAM. It's a pretty standard feature to be able
> to hint and trim the size of data structures, just like you can
> usually choose the buffer size for stream operations.
If it's voting time, I'm -1. Having programmed a lot of memory-constrained
systems (not in Python, though) - this is not how things usually work
there. In a memory-constrained system, you don't "grow and shrink" your
data structures. That's because growing often needs to reallocate the whole
chunk and do a copy, and shrinking only helps memory fragmentation. In such
systems, you usually know in advance or at least limit the size of data
structures and pre-allocate, which is perfectly possible in Python today.
Shrinking is rarely, if ever, useful. If it is, you can implement concrete
data structures for your concrete needs. And Python has a lot of ways to
save memory for large arrays of things (array, numpy, encoding in bytes,
etc) if one really wants to. I just don't believe the proposal will help in
a lot of realistic code, and it certainly goes against the way of Python.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-ideas