[ python-Bugs-1025127 ] shutils.rmtree() uses excessive amounts of memory

SourceForge.net noreply at sourceforge.net
Thu Sep 9 16:17:09 CEST 2004


Bugs item #1025127, was opened at 2004-09-09 09:52
Message generated for change (Comment added) made by tim_one
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1025127&group_id=5470

Category: Python Library
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: James Henstridge (jhenstridge)
Assigned to: Nobody/Anonymous (nobody)
Summary: shutils.rmtree() uses excessive amounts of memory

Initial Comment:
The shutils.rmtree() implementation uses an excessive
amount of memory when deleting large directory heirarchies.

Before actually deleting any files, it builds up a list
of (function, filename) tuples for all the files that
it is going to remove.  If there are a lot of files,
this results in a lot of memory for a large heirarchy
(I had a Python process using 800MB in one case).

I'm not sure why it is doing things this way.  It isn't
using the list to avoid recursion, so the depth of
directories it can remove is still limited by Python's
recursion limit.

Replacing _build_cmdtuple() with a generator might be a
good way to reduce the memory usage while leaving the
rest of the code unchanged.

I checked in CVS, and this issue is still present on HEAD.

----------------------------------------------------------------------

>Comment By: Tim Peters (tim_one)
Date: 2004-09-09 10:17

Message:
Logged In: YES 
user_id=31435

Rewrite it using os.walk() (not os.path.walk()) with 
topdown=False.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1025127&group_id=5470


More information about the Python-bugs-list mailing list