walking a directory with very many files

[Lawrence D'Oliveiro]

In message <20090619134015.349ba199 at malediction>, Mike Kazantsev wrote:

> On Fri, 19 Jun 2009 17:53:40 +1200
> Lawrence D'Oliveiro <ldo at geek-central.gen.new_zealand> wrote:
> 
>> In message <20090618081423.2e0356b9 at coercion>, Mike Kazantsev wrote:
>> 
>> > On Thu, 18 Jun 2009 10:33:49 +1200
>> > Lawrence D'Oliveiro <ldo at geek-central.gen.new_zealand> wrote:
>> > 
>> >> In message <20090617214535.108667ca at coercion>, Mike Kazantsev
>> >> wrote:
>> >> 
>> >>> On Wed, 17 Jun 2009 23:04:37 +1200
>> >>> Lawrence D'Oliveiro <ldo at geek-central.gen.new_zealand> wrote:
>> >>> 
>> >>>> In message <20090617142431.2b25faf5 at malediction>, Mike Kazantsev
>> >>>> wrote:
>> >>>> 
>> >>>>> On Wed, 17 Jun 2009 17:53:33 +1200
>> >>>>> Lawrence D'Oliveiro <ldo at geek-central.gen.new_zealand> wrote:
>> >>>>> 
>> >>>>>>> Why not use hex representation of md5/sha1-hashed id as a
>> >>>>>>> path, arranging them like /path/f/9/e/95ea4926a4 ?
>> >>>>>>> 
>> >>>>>>> That way, you won't have to deal with many-files-in-path
>> >>>>>>> problem ...
>> >>>>>> 
>> >>>>>> Why is that a problem?
>> >>>>> 
>> >>>>> So you can os.listdir them?
>> >>>> 
>> >>>> Why should you have a problem os.listdir'ing lots of files?
>> >>> 
>> >>> I shouldn't, and I don't ;)
>> >> 
>> >> Then why did you suggest that there was a problem being able to
>> >> os.listdir them?
>> > 
>> > I didn't, OP did ...
>> 
>> Then why did you reply to my question "Why is that a problem?" with
>> "So that you can os.listdir them?", if you didn't think there was a
>> problem (see above)?
> 
> Why do you think that if I didn't suggest there is a problem, I think
> there is no problem?

It wasn't that you didn't suggest there was a problem, but that you 
suggested a "solution" as though there was a problem.

> Why would you want to listdir them?

It's a common need, to find out what's in a directory.

> I can imagine at least one simple scenario: you had some nasty crash
> and you want to check that every file has corresponding, valid db
> record.

But why would that be relevant to this case?