On 20 June 2013 23:29, Christian Heimes wrote:

Am 20.06.2013 15:21, schrieb Florent:

...

we already have "_pyio.py", we could have "_pystat.py".

Works for me.

I suggest following the guidelines in PEP 399 for cross implementation compatibility of the standard library: http://www.python.org/dev/peps/pep-0399/#details 1. Keep stat.py 2. Make the C version "_stat" 3. Add the following to the end of stat.py: try: from _stat import * except ImportError: pass Run the tests with and without the C module in the test suite (again, as per PEP 399). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith wrote:

On 6/20/2013 9:40 AM, Nick Coghlan wrote:

...

On 20 June 2013 23:29, Christian Heimes wrote:

...

Am 20.06.2013 15:21, schrieb Florent:

...

we already have "_pyio.py", we could have "_pystat.py".

Works for me.

I suggest following the guidelines in PEP 399 for cross implementation compatibility of the standard library: http://www.python.org/dev/peps/pep-0399/#details

1. Keep stat.py 2. Make the C version "_stat" 3. Add the following to the end of stat.py:

try: from _stat import * except ImportError: pass

Run the tests with and without the C module in the test suite (again, as per PEP 399).

Agreed with the above.

But isn't the real problem with this module in Python the fact that the constants might be wrong? I'm not sure what, if anything, we can do about that.

There isn't anything we can do beyond at least trying to provide reasonable defaults when something better isn't available (which is what the stats module has done all this time). It might make testing difficult for the Python code when the C code has the right values, but I don't think it's necessarily worth tossing out the Python code as backup.

cffi makes this kind of constant-grabbing very easy. However this is a tiny module so no problem with having a C version. On Thu, Jun 20, 2013 at 11:04 AM, Thomas Wouters wrote:

On Thu, Jun 20, 2013 at 8:01 AM, Brett Cannon wrote:

...

On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith wrote:

...

On 6/20/2013 9:40 AM, Nick Coghlan wrote:

...

On 20 June 2013 23:29, Christian Heimes wrote:

...

Am 20.06.2013 15:21, schrieb Florent:

...

we already have "_pyio.py", we could have "_pystat.py".

Works for me.

I suggest following the guidelines in PEP 399 for cross implementation compatibility of the standard library: http://www.python.org/dev/peps/pep-0399/#details

1. Keep stat.py 2. Make the C version "_stat" 3. Add the following to the end of stat.py:

try: from _stat import * except ImportError: pass

Run the tests with and without the C module in the test suite (again, as per PEP 399).

Agreed with the above.

But isn't the real problem with this module in Python the fact that the constants might be wrong? I'm not sure what, if anything, we can do about that.

There isn't anything we can do beyond at least trying to provide reasonable defaults when something better isn't available (which is what the stats module has done all this time). It might make testing difficult for the Python code when the C code has the right values, but I don't think it's necessarily worth tossing out the Python code as backup.

If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway.

-- Thomas Wouters

Hi! I'm an email virus! Think twice before sending your email to help me spread!

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

2013/6/20 Charles-François Natali :

2013/6/20 Thomas Wouters :

...

If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway.

That's exactly my rationale for pushing for removal.

+1 to nixing it. -- Regards, Benjamin

On 21/06/13 01:35, Benjamin Peterson wrote:

2013/6/20 Charles-François Natali :

...

2013/6/20 Thomas Wouters :

...

If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway.

That's not correct. Other implementations can do exactly what CPython 3.3 does, namely just use stat.py as given. Not all implementations necessarily care about multiple platforms where stat constants are likely to change.

...

That's exactly my rationale for pushing for removal.

+1 to nixing it.

-1 Reading the Python source code is a very good way for beginner programmers to learn about things like this. Being able to read stat.py in Python makes a good, complementary source of information for those new to stat. Getting rid of stat.py means there's no source code to read at all, unless the user has built Python from source and kept the C source code. And even if they have, it's easy to forget how intimidating C can be to neophytes. I'm with Nick on this one: PEP 399 already answers the question of what to do with stat.py. We keep it as a fallback, which guarantees that any Python implementation that uses the standard library is no worse off than what we have now. When available, we override the default constants with platform specific ones generated from whatever language is used by the implementation. The accuracy of those constants then becomes a matter of quality of implementation. A low quality implementation may take the lazy way out and just use the default constants, while a high quality implementation will not. At the moment, the stat.py module contains ten small functions. Getting rid of the stat.py means that they have to be re-implemented in C/Java/RPython/etc. for every implementation. Even if the re-implementations are individually trivial, it's still a cost for no real gain. Keeping the pure Python implementation also lowers the bar for adding new functions in the future, and for documentation changes. -- Steven

Gustavo Carneiro writes:

On Fri, Jun 21, 2013 at 8:20 PM, Steven D'Aprano wrote:

...

-1 Reading the Python source code is a very good way for beginner programmers to learn about things like this.

On the other hand, it is counter-productive to learn about code that is conceptually _wrong_.

It's hardly _conceptually_ wrong when it's simply a pure-Python version that reflects *exactly* the implementation used by many OSes written in C. It's purely a quality of implementation issue in that it fails to observe the DRY principle and is non-portable. But if you look at on Mac OS X or Linux, you'll see that the visible definitions are protected by a thicket of #ifdefs, and the actual definitions may be drawn from other files #include'd there, and not using the definitions visible in at all. Ie, on those systems the OS implementation possesses exactly the same structure that CPython would have with a stat.py + _stat module implementation. If the presence of the _stat module is documented and the rationale briefly explained in stat.py, I suppose you have the best of both worlds for the novice programmer (at the expense of a test for the presence of _stat at import time).

On 20Jun2013 08:35, Benjamin Peterson wrote: | 2013/6/20 Charles-François Natali : | > 2013/6/20 Thomas Wouters : | >> If the .py file is going to be wrong or incomplete, why would we want to | >> keep it -- or use it as fallback -- at all? If we're dead set on having a | >> .py file instead of requiring it to be part of the interpreter (whichever | >> that is, however it was built), it should be generated as part of the build | >> process. Personally, I don't see the value in it; other implementations will | >> need to do *something* special to use it anyway. | > | > That's exactly my rationale for pushing for removal. | | +1 to nixing it. -1 to nixing it. I think there should be a pure python reference implementation. If it exposes only the portable constants/macros or, better, has a machine generated _section_ for the local platform macros, all to the good; it would not lie. - A huge amount of code only needs to care about the portable stuff (is this a dir, is this a regular file, is it neither). Missing local constants won't break such code. - A reference implementation exposes algorithms and purpose in a common language (Python, of course; everyone caring about such an impl can already read Python:-). - A reference implementation provides a base for other implementations to use outright, or to build on. - A reference implementation provides something to test against for comparison of the common stuff. - The implementation cost is low; the ref implementation already exists! Getting rid of it seems mad. Pruning/fixing/adapting the nonportable bits might be good. Christian Heimes' test patch would make that much easier to identify. So, -1 on removal of stat.py. Cheers, -- Cameron Simpson I will not do it as a hack I will not do it for my friends I will not do it on a Mac I will not write for Uncle Sam I will not do it on weekends I won't do ADA, Sam-I-Am - Gregory Bond

On Fri, Jun 21, 2013 at 11:06 AM, Nick Coghlan wrote:

On 21 June 2013 01:04, Thomas Wouters wrote:

...

If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway.

Because practicality beats purity. This "wrong" Python code has been good enough for all Python version up until 3.4, it makes sense to keep it as a fallback instead of throwing it away.

How would you know if the Python you're running on has an incorrect bitflag? If the "wrong" code is simply incomplete (it has the standard flags but none of the contended ones), that would at least be safe - you'll never get a false flag, but you might be unable to recognize the platform-specific ones. And then the platform-specific modules would always be adding, never overwriting, bitflags. ChrisA

On 21 June 2013 17:25, Victor Stinner wrote:

2013/6/21 Nick Coghlan :

...

Because practicality beats purity. This "wrong" Python code has been good enough for all Python version up until 3.4, it makes sense to keep it as a fallback instead of throwing it away.

How do you plan to handle the following case in Python?

"Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines 0xe000 as S_IFWHT (whiteout), but Solaris defines it as S_IFPORT (event port)."

We may keep the Python module if it is kept unchanged, but the Python and C modules should have the same public API (the C module should not have more features).

I think it's OK to expose additional platform specific features in the C version, and have them fail cleanly with the pure Python version (rather than silently giving the wrong answer). What's not OK is for the standard library to regress for other implementations just because we added a C implementation for the benefit of CPython. That's exactly the kind of thing PEP 399 says we *won't* do. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Am 21.06.2013 13:45, schrieb Antoine Pitrou:

For me, PEP 399 should not be considered a requirement but a guideline. stat.py is algorithmically trivial and I don't think it saves much work for authors of third-party Python implementations.

The module's content is rather boring. It's just a bunch of constants, some wrapper functions for macros and one trivial function that turns st_mode into "-rwxr-xr-x" string. In my opinion stat module falls into the same line as the errno module. The latter is also implemented in C in order to pull a bunch of constants from header files. I have yet another argument in favor of a C implementation. POSIX standards won't gain any new S_IF* numeric constants for new files types. Some file types such as mq, semaphore, shmem and typed memory from POSIX real time extensions aren't testable with S_IF*, too. (But I don't know any platform that implements them as files, though). http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html#tag... Christian

Le Sat, 22 Jun 2013 01:29:40 +1000, Nick Coghlan a écrit :

On 21 June 2013 21:45, Antoine Pitrou wrote:

...

Le Fri, 21 Jun 2013 21:39:10 +1000, Nick Coghlan a écrit :

...

What's not OK is for the standard library to regress for other implementations just because we added a C implementation for the benefit of CPython. That's exactly the kind of thing PEP 399 says we *won't* do.

For me, PEP 399 should not be considered a requirement but a guideline. stat.py is algorithmically trivial and I don't think it saves much work for authors of third-party Python implementations.

So why not just replace the *broken* parts of stat.py and keep the rest of it? Why make pointless work for the other implementations?

I guess the answer is: because it's more work for us :-)

Basically, I want to hear from the Jython, PyPy and IronPython devs that they're OK with us deleting Lib/stat.py. Hearing other CPython devs say they're fine with it doesn't mean anything, since we're not the ones that will have to do additional work as a result.

Yes, I agree with that. Regards Antoine.

Am 21.06.2013 17:47, schrieb Benjamin Peterson:

<hat kind="pypy"> Kill it. I would argue having incorrect constants makes the implementation incompatible with CPython anyway. This not that much work (as long as there are tests that the constants exist at least) to emulate. </hat>

My patch adds extensive tests for all features of the stat module. The test check the existence of attributes and the return value of all functions. A missing or ill-defined attribute is going to raise an error. But see for yourself: http://hg.python.org/cpython/rev/f8ff61f44aca Christian

On Fri, Jun 21, 2013 at 9:29 AM, Nick Coghlan wrote:

So why not just replace the *broken* parts of stat.py and keep the rest of it?

In some ways this sounds like yet another use case for what amounts to properties on modules. The sketchy contants that are causing concern could raise NotImplementedError in stat.py, but the C version would implement them. Then PEP 399 remains satisfied and appropriately so. -eric

On 6/21/2013 7:39 AM, Nick Coghlan wrote:

On 21 June 2013 17:25, Victor Stinner wrote:

...

2013/6/21 Nick Coghlan :

...

Because practicality beats purity. This "wrong" Python code has been good enough for all Python version up until 3.4, it makes sense to keep it as a fallback instead of throwing it away.

How do you plan to handle the following case in Python?

"Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines 0xe000 as S_IFWHT (whiteout), but Solaris defines it as S_IFPORT (event port)."

We may keep the Python module if it is kept unchanged, but the Python and C modules should have the same public API (the C module should not have more features).

I think it's OK to expose additional platform specific features in the C version, and have them fail cleanly with the pure Python version (rather than silently giving the wrong answer). What's not OK is for the standard library to regress for other implementations just because we added a C implementation for the benefit of CPython. That's exactly the kind of thing PEP 399 says we *won't* do.

I was just writing up something similar. But as always, Nick said it better than me. -- Eric.

2013/6/20 Eric V. Smith javascript:;>:

But isn't the real problem with this module in Python the fact that the constants might be wrong? I'm not sure what, if anything, we can do about that.

Python is providing a stat module implemented in Python since 10 years, or maybe 20 years, and I don't remember that someone complained that constants are wrong. At the same time, Python distributes IN.py which contains things like that: LONG_MAX = 9223372036854775807 LONG_MAX = 2147483647 LONG_MIN = (-LONG_MAX - 1) In my opinion, the situation of plat-*/*.py modules is worse than stat.py :-) Victor

2013/6/20 Eric V. Smith :

This is serious, not argumentative: If there's really no concern that the values be correct, then why implement it in C?

I read again the issue. The problem is to add new flags. Current flags (type: block device/symlink/..., file mode) are well defined and portable, whereas new flags are usually specific to an OS and no standardize yet. Examples: - Solaris: "door" - Solaris: "event port" - OSX/Darwin/FreeBSD: "whiteout" The other problem is that values of the new flags are not portable. Martin v. Löwis wrote: "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines 0xe000 as S_IFWHT (whiteout), but Solaris defines it as S_IFPORT (event port)." A C module reusing existing S_ISxxx() macros is the obvious way to solve this issue. Victor

2013/6/20 Serhiy Storchaka

Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be implement it on C?

Aha. Should an internal C module fetch the value of the constants, and a public stat.py nicely wrap them in enums? -- Amaury Forgeot d'Arc

But aren't most of these masks? enum,IntEnum doesn't handle those very gracefully (the | operator returns a plain int). On Thu, Jun 20, 2013 at 1:18 PM, Amaury Forgeot d'Arc wrote:

2013/6/20 Serhiy Storchaka

...

Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be implement it on C?

Aha. Should an internal C module fetch the value of the constants, and a public stat.py nicely wrap them in enums?

-- Amaury Forgeot d'Arc

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (python.org/~guido)

2013/6/20 Serhiy Storchaka :

Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be implement it on C?

Numerical values are less important than S_ISxxx() macros. Example: #define S_ISDOOR(mode) (((mode)&0xF000) == 0xd000) 0xd000 is (stat.S_IFSOCK + stat.S_IFIFO). And how do you represent the file mode with enums? I don't think that enum should be used in the stat module. I would prefer a stat object with methods than having to calls low-level functions. Something like: os.stat("document.txt").st_mode.is_reg() versus stat.S_ISREG(os.stat("document.txt").st_mode) The idea was discussed in http://bugs.python.org/issue11406 to solve a real API design issue. How should os.scandir() return the "is a directory" information with a portable API. I'm not saying that stat.S_ISREG should go away. The two approaches are complementary. Victor