[Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5

[Ryan Stuart]

Hi,

On Sun, 8 Mar 2015 at 12:33 Ben Hoyt <benhoyt at gmail.com> wrote:

> Others: if you want to benchmark this, the simplest way is to use my
> os.walk() benchmark.py test program here:
> https://github.com/benhoyt/scandir -- it compares the built-in os.walk()
> implemented with os.listdir() with a version of walk() implemented with
> os.scandir(). I see huge gains on Windows (12-50x) and modest gains on my
> Linux VM (3-5x).
>

I have a MacBook Pro Laptop running OS X 10.10.2. I did the following:

   - hg update -r 8ef4f75a8018
   - patch -p1 < scandir-8.patch
   - ./configure --with-pydebug && make -j7

I then ran ./python.exe ~/Workspace/python/scandir/benchmark.py and I got:

*Creating tree at /Users/rstuart/Workspace/python/scandir/benchtree:
depth=4, num_dirs=5, num_files=50*
*Using slower ctypes version of scandir*
*Comparing against builtin version of os.walk()*
*Priming the system's cache...*
*Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 1/3...*
*Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 2/3...*
*Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 3/3...*
*os.walk took 0.184s, scandir.walk took 0.158s -- 1.2x as fast*

I then did ./python.exe ~/Workspace/python/scandir/benchmark.py -s and got:

*Using slower ctypes version of scandir*
*Comparing against builtin version of os.walk()*
*Priming the system's cache...*
*Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 1/3...*
*Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 2/3...*
*Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 3/3...*
*os.walk size 23400, scandir.walk size 23400 -- equal*
*os.walk took 0.483s, scandir.walk took 0.463s -- 1.0x as fast*

Hope this helps.

Cheers

Note that the actual CPython version of os.walk() doesn't yet use
> os.scandir(). I intend to open a separate issue for that shortly (or Victor
> can). But that part should be fairly straight-forward, as I already have a
> version available in my GitHub project.
>
> -Ben
>
>
> On Sat, Mar 7, 2015 at 9:13 PM, Victor Stinner <victor.stinner at gmail.com>
> wrote:
>
>> Hi,
>>
>> FYI I commited the implementation of os.scandir() written by Ben Hoyt.
>> I hope that it will be part of Python 3.5 alpha 2 (Ben just sent the
>> final patch today).
>>
>> Please test this new feature. You may benchmark here.
>> http://bugs.python.org/issue22524 contains some benchmark tools and
>> benchmark results of older versions of the patch.
>>
>> The implementation was tested on Windows and Linux. I'm now watching
>> for buildbots to see how other platforms like os.scandir().
>>
>> Bad news: OpenIndiana doesn't support d_type: the dirent structure has
>> no d_type field. I already fixed the implementation to support this
>> case. os.scandir() is still useful on OpenIndiana, because the stat
>> result is cached in a DirEntry, so only one syscall is required,
>> instead of multiple, when multiple DirEntry methods are called (ex:
>> entry.is_dir() and not entry.is_symlink()).
>>
>> Victor
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> ryan.stuart.85%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150308/1c1238d9/attachment.html>