[ python-Bugs-1025127 ] shutils.rmtree() uses excessive amounts of memory

[SourceForge.net]

Bugs item #1025127, was opened at 2004-09-09 15:52
Message generated for change (Comment added) made by jlgijsbers
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1025127&group_id=5470

Category: Python Library
Group: Python 2.3
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: James Henstridge (jhenstridge)
Assigned to: Johannes Gijsbers (jlgijsbers)
Summary: shutils.rmtree() uses excessive amounts of memory

Initial Comment:
The shutils.rmtree() implementation uses an excessive
amount of memory when deleting large directory heirarchies.

Before actually deleting any files, it builds up a list
of (function, filename) tuples for all the files that
it is going to remove.  If there are a lot of files,
this results in a lot of memory for a large heirarchy
(I had a Python process using 800MB in one case).

I'm not sure why it is doing things this way.  It isn't
using the list to avoid recursion, so the depth of
directories it can remove is still limited by Python's
recursion limit.

Replacing _build_cmdtuple() with a generator might be a
good way to reduce the memory usage while leaving the
rest of the code unchanged.

I checked in CVS, and this issue is still present on HEAD.

----------------------------------------------------------------------

>Comment By: Johannes Gijsbers (jlgijsbers)
Date: 2004-10-07 23:13

Message:
Logged In: YES 
user_id=469548

I just looked at my own patch again, added the _raise_err
function back in and checked it in as rev 1.33 of shutil.py. 

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-09-12 00:56

Message:
Logged In: YES 
user_id=31435

I don't really have time for a thorough review.  I'll note that 
stuff like

func = something1
arg = something2
func(arg)

looks, to my eye, like a convoluted way to say

something1(something2)

I suppose that's to keep the onerror= gimmick working, 
though.

----------------------------------------------------------------------

Comment By: Johannes Gijsbers (jlgijsbers)
Date: 2004-09-11 23:36

Message:
Logged In: YES 
user_id=469548

Please attach changes as a patch next time. I've attached
shutils-rmtree.py as a patch this time.

I gave it a quick review and added a test to test_shutil.py
to ensure some not-very-obvious behavior (don't delete a
path passed to rmtree if it's a file) would be preserved.
The new version seems fine to me. Tim, could you take a look
at it as well?

----------------------------------------------------------------------

Comment By: James Henstridge (jhenstridge)
Date: 2004-09-10 03:33

Message:
Logged In: YES 
user_id=146903

Attached is a Python file including a fixed up
shutils.rmtree() using os.walk().  It seems to work for me,
and should have the same error behaviour.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-09-09 16:17

Message:
Logged In: YES 
user_id=31435

Rewrite it using os.walk() (not os.path.walk()) with 
topdown=False.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1025127&group_id=5470