[Python-Dev] Builtin open() too slow

[Eugene Toder]

Hi

What OS and what file system you are using? Many file systems (e,g.
ext2/3fs) handle large directories very poorly.
A quick way to check if this has anything to do with Python is writing
a small C program that opens these files and time it.

Eugene

On Sat, Mar 12, 2011 at 10:13 AM, Lukas Lueg <lukas.lueg at googlemail.com> wrote:
> Hi,
>
> i've a storage engine that stores a lot of files (e.g. > 10.000) in
> one path. Running the code under cProfile, I found that with a total
> CPU-time of 1,118 seconds, 121 seconds are spent in 27.013 calls to
> open(). The number of calls is not the problem; however I find it
> *very* discomforting that Python spends about 2 minutes out of 18
> minutes of cpu time just to get a file-handle after which it can spend
> some other time to read from them.
>
> May this be a problem with the way Python 2.7 gets filehandles from
> the OS or is it a problem with large directories itself?
>
> Best regards
> Lukas
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/eltoder%40gmail.com
>