[Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

[Steven D'Aprano]

On Fri, Jun 27, 2014 at 03:07:46AM +0300, Paul Sokolovsky wrote:

> With my MicroPython hat on, os.scandir() would make things only worse.
> With current interface, one can either have inefficient implementation
> (like CPython chose) or efficient implementation (like MicroPython
> chose) - all transparently. os.scandir() supposedly opens up efficient
> implementation for everyone, but at the price of bloating API and
> introducing heavy-weight objects to wrap info. 

os.scandir is not part of the Python API, it is not a built-in function. 
It is part of the CPython standard library. That means (in my opinion) 
that there is an expectation that other Pythons should provide it, but 
not an absolute requirement. Especially for the os module, which by 
definition is platform-specific. In my opinion that means you have four 
options:

1. provide os.scandir, with exactly the same semantics as on CPython;

2. provide os.scandir, but change its semantics to be more lightweight 
   (e.g. return an ordinary tuple, as you already suggest);

3. don't provide os.scandir at all; or

4. do something different depending on whether the platform is Linux
   or an embedded system.

I would consider any of those acceptable for a library feature, but not 
for a language feature.


[...]
> But reusing os.stat struct is glaringly not what's proposed. And
> it's clear where that comes from - "[DirEntry.]lstat(): like os.lstat(),
> but requires no system calls on Windows". Nice, but OS "FooBar" can do
> much more than Windows - it has a system call to send a file by email,
> right when scanning a directory containing it. So, why not to have
> DirEntry.send_by_email(recipient) method? I hear the answer - it's
> because CPython strives to support Windows well, while doesn't care
> about "FooBar" OS.

Correct. If there is sufficient demand for FooBar, then CPython may 
support it. Until then, FooBarPython can support it, and offer whatever 
platform-specific features are needed within its standard library.


> And then it again leads to the question I posed several times - where's
> line between "CPython" and "Python"? Is it grounded for CPython to add
> (or remove) to Python stdlib something which is useful for its users,
> but useless or complicating for other Python implementations?

I think so. And other implementations are free to do the same thing.

Of course there is an expectation that the standard library of most 
implementations will be broadly similar, but not that they will be 
identical.

I am surprised that both Jython and IronPython offer an non-functioning 
dis module: you can import it successfully, but if there's a way to 
actually use it, I haven't found it:


steve at orac:~$ jython
Jython 2.5.1+ (Release_2_5_1, Aug 4 2010, 07:18:19)
[OpenJDK Server VM (Sun Microsystems Inc.)] on java1.6.0_27
Type "help", "copyright", "credits" or "license" for more information.
>>> import dis
>>> dis.dis(lambda x: x+1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/share/jython/Lib/dis.py", line 42, in dis
    disassemble(x)
  File "/usr/share/jython/Lib/dis.py", line 64, in disassemble
    linestarts = dict(findlinestarts(co))
  File "/usr/share/jython/Lib/dis.py", line 183, in findlinestarts
    byte_increments = [ord(c) for c in code.co_lnotab[0::2]]
AttributeError: 'tablecode' object has no attribute 'co_lnotab'


IronPython gives a different exception:

steve at orac:~$ ipy
IronPython 2.6 Beta 2 DEBUG (2.6.0.20) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
>>> import dis
>>> dis.dis(lambda x: x+1)
Traceback (most recent call last):
TypeError: don't know how to disassemble code objects


It's quite annoying, I would have rather that they just removed the 
module altogether. Better still would have been to disassemble code 
objects to whatever byte code the Java and .Net platforms use. But 
there's surely no requirement to disassemble to CPython byte code!



-- 
Steven