[Tutor] Acceleration of long lists

[Alan Gauld]

On 29/07/2021 19:02, Jan Kenin wrote:

> Often I append instances of objects in list-objects. Then I realize that
> the progress of my program slows down, when the list becomes larger and
> larger, e.g. larger than 3000.

3000 is nothing to a modern computer.
Remember the list is not holding your objects, it is simply holding a
reference to each object. When you append an object you are only adding
a new reference - a few bytes.

Also, I believe that in the implementation details Python
allocates memory for lists in chunks so you only actually
create new memory when an existing chunk runs out, not
for every append operation. (That's certainly how I would
build it!)

>  How can I accerlerate this? Perhaps it is
> a question of memory. Can I initialize the list in order to do the
> memory allocation only once?

No, that's a memory optimisation that's in the hands of the
implementers and indeed may well work differently
in different versions of Python (CPython, Jython, IronPython etc)

The solution with any kind of performance problem is to measure
to find out what is slowing things down. Have you tried using
the profiler to see which functions are taking the time?

Also, what kind of slow-down do you see? fractions of a second?
seconds?, minutes? And is it only certain operations? If an operation
is slow you probably won't notice it for a single object, but
once you get into thousands repeating it then the delay
becomes obvious. Again profiling is your friend. But without
much more details (and ideally code) we can't be more specific.

Also, if your code is doing embedded loops on these lists then
that can slow things dramatically. And in Python it's easy
to introduce loops without realizing - for example using
'in' tests causes a loop. If you repeat the same 'in' test
multiple times you will loop over the entire list each time.
Also searching for items might well be faster if you use
a dictionary rather than a loop. But without sight of your
code we can only make generic suggestions.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos