walking a directory with very many files

Lie Ryan lie.1296 at gmail.com
Thu Jun 18 14:48:27 EDT 2009


Mike Kazantsev wrote:
> On Wed, 17 Jun 2009 03:42:02 GMT
> Lie Ryan <lie.1296 at gmail.com> wrote:
> 
>> Mike Kazantsev wrote:
>>> In fact, on modern filesystems it doesn't matter whether you
>>> accessing /path/f9e95ea4926a4 with million files in /path
>>> or /path/f/9/e/95ea with only hundred of them in each path. Former
>>> case (all-in-one-path) would even outperform the latter with ext3
>>> or reiserfs by a small margin.
>>> Sadly, that's not the case with filesystems like FreeBSD ufs2 (at
>>> least in sixth branch), so it's better to play safe and create
>>> subdirs if the app might be run on different machines than keeping
>>> everything in one path.
>> It might not matter for the filesystem, but the file explorer (and ls)
>> would still suffer. Subfolder structure would be much better, and much
>> easier to navigate manually when you need to.
> 
> It's an insane idea to navigate any structure with hash-based names
> and hundreds of thousands files *manually*: "What do we have here?
> Hashies?" ;)
> 

Like... when you're trying to debug a code that generates an error with
a specific file...

Yeah, it might be possible to just mv the file from outside, but not
being able to enter a directory just because you've got too many files
in it is kind of silly.



More information about the Python-list mailing list