stat module in C -- what to do with stat.py?
Hello, I have re-implemented the entire stat module in C. [1] It's a necessary step to fix portability issues. For most constants POSIX standards dictate only the name of the constant and its meaning but not its value. Only the values of permission bits (0644 == rw-r--r--) have fixed values. For example S_IFDIR is usually 0o40000 but it might be 0o20000 on some platforms. Common file types seem to have the same value on all important platforms. It's the only reason we were able to get away with stat.py. But uncommon file types like door, event port and whiteout don't have sensible default values. The C implementation is able to pick up the platform's values easily. What shall I do about stat.py? statmodule.c implements all features of stat.py so there is no point in using it in CPython. It's one Python file less to load on every startup. However it might still be useful to alternative implementations of Python such as Jython or PyPy. 1) keep the file stat.py but let it be shadowed by the builtin stat module. Antoine loathes my hack... 2) rename stat.py to _stat.py 3) remove stat.py Opinions? Christian [1] http://bugs.python.org/issue11016
we already have "_pyio.py", we could have "_pystat.py". my 2c -- Florent Xicluna 2013/6/20 Christian Heimes <christian@python.org>:
Hello,
I have re-implemented the entire stat module in C. [1] It's a necessary step to fix portability issues. For most constants POSIX standards dictate only the name of the constant and its meaning but not its value. Only the values of permission bits (0644 == rw-r--r--) have fixed values.
For example S_IFDIR is usually 0o40000 but it might be 0o20000 on some platforms. Common file types seem to have the same value on all important platforms. It's the only reason we were able to get away with stat.py. But uncommon file types like door, event port and whiteout don't have sensible default values. The C implementation is able to pick up the platform's values easily.
What shall I do about stat.py? statmodule.c implements all features of stat.py so there is no point in using it in CPython. It's one Python file less to load on every startup. However it might still be useful to alternative implementations of Python such as Jython or PyPy.
1) keep the file stat.py but let it be shadowed by the builtin stat module. Antoine loathes my hack... 2) rename stat.py to _stat.py 3) remove stat.py
Opinions?
Christian
[1] http://bugs.python.org/issue11016 _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/florent.xicluna%40gmail.co...
On 20 June 2013 23:29, Christian Heimes <christian@python.org> wrote:
Am 20.06.2013 15:21, schrieb Florent:
we already have "_pyio.py", we could have "_pystat.py".
Works for me.
I suggest following the guidelines in PEP 399 for cross implementation compatibility of the standard library: http://www.python.org/dev/peps/pep-0399/#details 1. Keep stat.py 2. Make the C version "_stat" 3. Add the following to the end of stat.py: try: from _stat import * except ImportError: pass Run the tests with and without the C module in the test suite (again, as per PEP 399). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 6/20/2013 9:40 AM, Nick Coghlan wrote:
On 20 June 2013 23:29, Christian Heimes <christian@python.org> wrote:
Am 20.06.2013 15:21, schrieb Florent:
we already have "_pyio.py", we could have "_pystat.py".
Works for me.
I suggest following the guidelines in PEP 399 for cross implementation compatibility of the standard library: http://www.python.org/dev/peps/pep-0399/#details
1. Keep stat.py 2. Make the C version "_stat" 3. Add the following to the end of stat.py:
try: from _stat import * except ImportError: pass
Run the tests with and without the C module in the test suite (again, as per PEP 399).
Agreed with the above. But isn't the real problem with this module in Python the fact that the constants might be wrong? I'm not sure what, if anything, we can do about that. -- Eric.
On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith <eric@trueblade.com> wrote:
On 6/20/2013 9:40 AM, Nick Coghlan wrote:
On 20 June 2013 23:29, Christian Heimes <christian@python.org> wrote:
Am 20.06.2013 15:21, schrieb Florent:
we already have "_pyio.py", we could have "_pystat.py".
Works for me.
I suggest following the guidelines in PEP 399 for cross implementation compatibility of the standard library: http://www.python.org/dev/peps/pep-0399/#details
1. Keep stat.py 2. Make the C version "_stat" 3. Add the following to the end of stat.py:
try: from _stat import * except ImportError: pass
Run the tests with and without the C module in the test suite (again, as per PEP 399).
Agreed with the above.
But isn't the real problem with this module in Python the fact that the constants might be wrong? I'm not sure what, if anything, we can do about that.
There isn't anything we can do beyond at least trying to provide reasonable defaults when something better isn't available (which is what the stats module has done all this time). It might make testing difficult for the Python code when the C code has the right values, but I don't think it's necessarily worth tossing out the Python code as backup.
On Thu, Jun 20, 2013 at 8:01 AM, Brett Cannon <brett@python.org> wrote:
On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith <eric@trueblade.com>wrote:
On 6/20/2013 9:40 AM, Nick Coghlan wrote:
On 20 June 2013 23:29, Christian Heimes <christian@python.org> wrote:
Am 20.06.2013 15:21, schrieb Florent:
we already have "_pyio.py", we could have "_pystat.py".
Works for me.
I suggest following the guidelines in PEP 399 for cross implementation compatibility of the standard library: http://www.python.org/dev/peps/pep-0399/#details
1. Keep stat.py 2. Make the C version "_stat" 3. Add the following to the end of stat.py:
try: from _stat import * except ImportError: pass
Run the tests with and without the C module in the test suite (again, as per PEP 399).
Agreed with the above.
But isn't the real problem with this module in Python the fact that the constants might be wrong? I'm not sure what, if anything, we can do about that.
There isn't anything we can do beyond at least trying to provide reasonable defaults when something better isn't available (which is what the stats module has done all this time). It might make testing difficult for the Python code when the C code has the right values, but I don't think it's necessarily worth tossing out the Python code as backup.
If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway. -- Thomas Wouters <thomas@python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread!
cffi makes this kind of constant-grabbing very easy. However this is a tiny module so no problem with having a C version. On Thu, Jun 20, 2013 at 11:04 AM, Thomas Wouters <thomas@python.org> wrote:
On Thu, Jun 20, 2013 at 8:01 AM, Brett Cannon <brett@python.org> wrote:
On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith <eric@trueblade.com> wrote:
On 6/20/2013 9:40 AM, Nick Coghlan wrote:
On 20 June 2013 23:29, Christian Heimes <christian@python.org> wrote:
Am 20.06.2013 15:21, schrieb Florent:
we already have "_pyio.py", we could have "_pystat.py".
Works for me.
I suggest following the guidelines in PEP 399 for cross implementation compatibility of the standard library: http://www.python.org/dev/peps/pep-0399/#details
1. Keep stat.py 2. Make the C version "_stat" 3. Add the following to the end of stat.py:
try: from _stat import * except ImportError: pass
Run the tests with and without the C module in the test suite (again, as per PEP 399).
Agreed with the above.
But isn't the real problem with this module in Python the fact that the constants might be wrong? I'm not sure what, if anything, we can do about that.
There isn't anything we can do beyond at least trying to provide reasonable defaults when something better isn't available (which is what the stats module has done all this time). It might make testing difficult for the Python code when the C code has the right values, but I don't think it's necessarily worth tossing out the Python code as backup.
If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway.
-- Thomas Wouters <thomas@python.org>
Hi! I'm an email virus! Think twice before sending your email to help me spread!
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
2013/6/20 Thomas Wouters <thomas@python.org>:
If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway.
That's exactly my rationale for pushing for removal. cf
2013/6/20 Charles-François Natali <cf.natali@gmail.com>:
2013/6/20 Thomas Wouters <thomas@python.org>:
If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway.
That's exactly my rationale for pushing for removal.
+1 to nixing it. -- Regards, Benjamin
Am 20.06.2013 17:35, schrieb Benjamin Peterson:
2013/6/20 Charles-François Natali <cf.natali@gmail.com>:
2013/6/20 Thomas Wouters <thomas@python.org>:
If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway.
That's exactly my rationale for pushing for removal.
+1 to nixing it.
I'm +0 for removal. The stat module contains merely constants, wrappers for macros and one simple functions. Somebody just say the word. Christian
On 21/06/13 01:35, Benjamin Peterson wrote:
2013/6/20 Charles-François Natali <cf.natali@gmail.com>:
2013/6/20 Thomas Wouters <thomas@python.org>:
If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway.
That's not correct. Other implementations can do exactly what CPython 3.3 does, namely just use stat.py as given. Not all implementations necessarily care about multiple platforms where stat constants are likely to change.
That's exactly my rationale for pushing for removal.
+1 to nixing it.
-1 Reading the Python source code is a very good way for beginner programmers to learn about things like this. Being able to read stat.py in Python makes a good, complementary source of information for those new to stat. Getting rid of stat.py means there's no source code to read at all, unless the user has built Python from source and kept the C source code. And even if they have, it's easy to forget how intimidating C can be to neophytes. I'm with Nick on this one: PEP 399 already answers the question of what to do with stat.py. We keep it as a fallback, which guarantees that any Python implementation that uses the standard library is no worse off than what we have now. When available, we override the default constants with platform specific ones generated from whatever language is used by the implementation. The accuracy of those constants then becomes a matter of quality of implementation. A low quality implementation may take the lazy way out and just use the default constants, while a high quality implementation will not. At the moment, the stat.py module contains ten small functions. Getting rid of the stat.py means that they have to be re-implemented in C/Java/RPython/etc. for every implementation. Even if the re-implementations are individually trivial, it's still a cost for no real gain. Keeping the pure Python implementation also lowers the bar for adding new functions in the future, and for documentation changes. -- Steven
On Fri, Jun 21, 2013 at 8:20 PM, Steven D'Aprano <steve@pearwood.info>wrote:
On 21/06/13 01:35, Benjamin Peterson wrote:
2013/6/20 Charles-François Natali <cf.natali@gmail.com>:
2013/6/20 Thomas Wouters <thomas@python.org>:
If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway.
That's not correct. Other implementations can do exactly what CPython 3.3 does, namely just use stat.py as given. Not all implementations necessarily care about multiple platforms where stat constants are likely to change.
That's exactly my rationale for pushing for removal.
+1 to nixing it.
-1
Reading the Python source code is a very good way for beginner programmers to learn about things like this.
On the other hand, it is counter-productive to learn about code that is conceptually _wrong_.
Gustavo Carneiro writes:
On Fri, Jun 21, 2013 at 8:20 PM, Steven D'Aprano <steve@pearwood.info> wrote:
-1 Reading the Python source code is a very good way for beginner programmers to learn about things like this.
On the other hand, it is counter-productive to learn about code that is conceptually _wrong_.
It's hardly _conceptually_ wrong when it's simply a pure-Python version that reflects *exactly* the implementation used by many OSes written in C. It's purely a quality of implementation issue in that it fails to observe the DRY principle and is non-portable. But if you look at <sys/stat.h> on Mac OS X or Linux, you'll see that the visible definitions are protected by a thicket of #ifdefs, and the actual definitions may be drawn from other files #include'd there, and not using the definitions visible in <sys/stat.h> at all. Ie, on those systems the OS implementation possesses exactly the same structure that CPython would have with a stat.py + _stat module implementation. If the presence of the _stat module is documented and the rationale briefly explained in stat.py, I suppose you have the best of both worlds for the novice programmer (at the expense of a test for the presence of _stat at import time).
Hi, On Fri, Jun 21, 2013 at 9:20 PM, Steven D'Aprano <steve@pearwood.info> wrote:
process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway.
That's not correct. Other implementations can do exactly what CPython 3.3 does, namely just use stat.py as given. Not all implementations necessarily care about multiple platforms where stat constants are likely to change.
Thanks Steven. That's PyPy's position: for now we really care only about <insert some list of common platform there> where stat.py has been obviously sufficient, as shown by CPython's long experience with stat.py. A bientôt, Armin.
On 20Jun2013 08:35, Benjamin Peterson <benjamin@python.org> wrote: | 2013/6/20 Charles-François Natali <cf.natali@gmail.com>: | > 2013/6/20 Thomas Wouters <thomas@python.org>: | >> If the .py file is going to be wrong or incomplete, why would we want to | >> keep it -- or use it as fallback -- at all? If we're dead set on having a | >> .py file instead of requiring it to be part of the interpreter (whichever | >> that is, however it was built), it should be generated as part of the build | >> process. Personally, I don't see the value in it; other implementations will | >> need to do *something* special to use it anyway. | > | > That's exactly my rationale for pushing for removal. | | +1 to nixing it. -1 to nixing it. I think there should be a pure python reference implementation. If it exposes only the portable constants/macros or, better, has a machine generated _section_ for the local platform macros, all to the good; it would not lie. - A huge amount of code only needs to care about the portable stuff (is this a dir, is this a regular file, is it neither). Missing local constants won't break such code. - A reference implementation exposes algorithms and purpose in a common language (Python, of course; everyone caring about such an impl can already read Python:-). - A reference implementation provides a base for other implementations to use outright, or to build on. - A reference implementation provides something to test against for comparison of the common stuff. - The implementation cost is low; the ref implementation already exists! Getting rid of it seems mad. Pruning/fixing/adapting the nonportable bits might be good. Christian Heimes' test patch would make that much easier to identify. So, -1 on removal of stat.py. Cheers, -- Cameron Simpson <cs@zip.com.au> I will not do it as a hack I will not do it for my friends I will not do it on a Mac I will not write for Uncle Sam I will not do it on weekends I won't do ADA, Sam-I-Am - Gregory Bond <gnb@bby.com.au>
On 21 June 2013 01:04, Thomas Wouters <thomas@python.org> wrote:
If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway.
Because practicality beats purity. This "wrong" Python code has been good enough for all Python version up until 3.4, it makes sense to keep it as a fallback instead of throwing it away. As Daniel notes, it also means PyPy can just have a small cffi module that adds (or overwrites) the platform specific constants, instead of having to reimplement the whole module. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Jun 21, 2013 at 11:06 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 21 June 2013 01:04, Thomas Wouters <thomas@python.org> wrote:
If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway.
Because practicality beats purity. This "wrong" Python code has been good enough for all Python version up until 3.4, it makes sense to keep it as a fallback instead of throwing it away.
How would you know if the Python you're running on has an incorrect bitflag? If the "wrong" code is simply incomplete (it has the standard flags but none of the contended ones), that would at least be safe - you'll never get a false flag, but you might be unable to recognize the platform-specific ones. And then the platform-specific modules would always be adding, never overwriting, bitflags. ChrisA
2013/6/21 Nick Coghlan <ncoghlan@gmail.com>:
Because practicality beats purity. This "wrong" Python code has been good enough for all Python version up until 3.4, it makes sense to keep it as a fallback instead of throwing it away.
How do you plan to handle the following case in Python? "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines 0xe000 as S_IFWHT (whiteout), but Solaris defines it as S_IFPORT (event port)." We may keep the Python module if it is kept unchanged, but the Python and C modules should have the same public API (the C module should not have more features). Or should play with "if sys.platform == ..."? Victor
On 21 June 2013 17:25, Victor Stinner <victor.stinner@gmail.com> wrote:
2013/6/21 Nick Coghlan <ncoghlan@gmail.com>:
Because practicality beats purity. This "wrong" Python code has been good enough for all Python version up until 3.4, it makes sense to keep it as a fallback instead of throwing it away.
How do you plan to handle the following case in Python?
"Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines 0xe000 as S_IFWHT (whiteout), but Solaris defines it as S_IFPORT (event port)."
We may keep the Python module if it is kept unchanged, but the Python and C modules should have the same public API (the C module should not have more features).
I think it's OK to expose additional platform specific features in the C version, and have them fail cleanly with the pure Python version (rather than silently giving the wrong answer). What's not OK is for the standard library to regress for other implementations just because we added a C implementation for the benefit of CPython. That's exactly the kind of thing PEP 399 says we *won't* do. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le Fri, 21 Jun 2013 21:39:10 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
On 21 June 2013 17:25, Victor Stinner <victor.stinner@gmail.com> wrote:
2013/6/21 Nick Coghlan <ncoghlan@gmail.com>:
Because practicality beats purity. This "wrong" Python code has been good enough for all Python version up until 3.4, it makes sense to keep it as a fallback instead of throwing it away.
How do you plan to handle the following case in Python?
"Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines 0xe000 as S_IFWHT (whiteout), but Solaris defines it as S_IFPORT (event port)."
We may keep the Python module if it is kept unchanged, but the Python and C modules should have the same public API (the C module should not have more features).
I think it's OK to expose additional platform specific features in the C version, and have them fail cleanly with the pure Python version (rather than silently giving the wrong answer).
PEP 399 says we don't do it: "Acting as a drop-in replacement also dictates that no public API be provided in accelerated code that does not exist in the pure Python code. Without this requirement people could accidentally come to rely on a detail in the accelerated code which is not made available to other VMs that use the pure Python implementation."
What's not OK is for the standard library to regress for other implementations just because we added a C implementation for the benefit of CPython. That's exactly the kind of thing PEP 399 says we *won't* do.
For me, PEP 399 should not be considered a requirement but a guideline. stat.py is algorithmically trivial and I don't think it saves much work for authors of third-party Python implementations. Regards Antoine.
Am 21.06.2013 13:45, schrieb Antoine Pitrou:
For me, PEP 399 should not be considered a requirement but a guideline. stat.py is algorithmically trivial and I don't think it saves much work for authors of third-party Python implementations.
The module's content is rather boring. It's just a bunch of constants, some wrapper functions for macros and one trivial function that turns st_mode into "-rwxr-xr-x" string. In my opinion stat module falls into the same line as the errno module. The latter is also implemented in C in order to pull a bunch of constants from header files. I have yet another argument in favor of a C implementation. POSIX standards won't gain any new S_IF* numeric constants for new files types. Some file types such as mq, semaphore, shmem and typed memory from POSIX real time extensions aren't testable with S_IF*, too. (But I don't know any platform that implements them as files, though). http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html#tag... Christian
On 21 June 2013 21:45, Antoine Pitrou <solipsis@pitrou.net> wrote:
Le Fri, 21 Jun 2013 21:39:10 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
What's not OK is for the standard library to regress for other implementations just because we added a C implementation for the benefit of CPython. That's exactly the kind of thing PEP 399 says we *won't* do.
For me, PEP 399 should not be considered a requirement but a guideline. stat.py is algorithmically trivial and I don't think it saves much work for authors of third-party Python implementations.
So why not just replace the *broken* parts of stat.py and keep the rest of it? Why make pointless work for the other implementations? Basically, I want to hear from the Jython, PyPy and IronPython devs that they're OK with us deleting Lib/stat.py. Hearing other CPython devs say they're fine with it doesn't mean anything, since we're not the ones that will have to do additional work as a result. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le Sat, 22 Jun 2013 01:29:40 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
On 21 June 2013 21:45, Antoine Pitrou <solipsis@pitrou.net> wrote:
Le Fri, 21 Jun 2013 21:39:10 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
What's not OK is for the standard library to regress for other implementations just because we added a C implementation for the benefit of CPython. That's exactly the kind of thing PEP 399 says we *won't* do.
For me, PEP 399 should not be considered a requirement but a guideline. stat.py is algorithmically trivial and I don't think it saves much work for authors of third-party Python implementations.
So why not just replace the *broken* parts of stat.py and keep the rest of it? Why make pointless work for the other implementations?
I guess the answer is: because it's more work for us :-)
Basically, I want to hear from the Jython, PyPy and IronPython devs that they're OK with us deleting Lib/stat.py. Hearing other CPython devs say they're fine with it doesn't mean anything, since we're not the ones that will have to do additional work as a result.
Yes, I agree with that. Regards Antoine.
2013/6/21 Nick Coghlan <ncoghlan@gmail.com>:
Basically, I want to hear from the Jython, PyPy and IronPython devs that they're OK with us deleting Lib/stat.py. Hearing other CPython devs say they're fine with it doesn't mean anything, since we're not the ones that will have to do additional work as a result.
<hat kind="pypy"> Kill it. I would argue having incorrect constants makes the implementation incompatible with CPython anyway. This not that much work (as long as there are tests that the constants exist at least) to emulate. </hat> -- Regards, Benjamin
Am 21.06.2013 17:47, schrieb Benjamin Peterson:
<hat kind="pypy"> Kill it. I would argue having incorrect constants makes the implementation incompatible with CPython anyway. This not that much work (as long as there are tests that the constants exist at least) to emulate. </hat>
My patch adds extensive tests for all features of the stat module. The test check the existence of attributes and the return value of all functions. A missing or ill-defined attribute is going to raise an error. But see for yourself: http://hg.python.org/cpython/rev/f8ff61f44aca Christian
On 22/06/13 01:29, Nick Coghlan wrote:
Basically, I want to hear from the Jython, PyPy and IronPython devs that they're OK with us deleting Lib/stat.py. Hearing other CPython devs say they're fine with it doesn't mean anything, since we're not the ones that will have to do additional work as a result.
It's not just established Python implementations. A pure-python standard library makes a good foundation for any future implementations. -- Steven
On Fri, Jun 21, 2013 at 9:29 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
So why not just replace the *broken* parts of stat.py and keep the rest of it?
In some ways this sounds like yet another use case for what amounts to properties on modules. The sketchy contants that are causing concern could raise NotImplementedError in stat.py, but the C version would implement them. Then PEP 399 remains satisfied and appropriately so. -eric
On 6/21/2013 7:45 AM, Antoine Pitrou wrote:
Le Fri, 21 Jun 2013 21:39:10 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
I think it's OK to expose additional platform specific features in the C version, and have them fail cleanly with the pure Python version (rather than silently giving the wrong answer).
PEP 399 says we don't do it:
"Acting as a drop-in replacement also dictates that no public API be provided in accelerated code that does not exist in the pure Python code. Without this requirement people could accidentally come to rely on a detail in the accelerated code which is not made available to other VMs that use the pure Python implementation."
Any C accelerator extensions should by documented as CPython extensions not necessarily available elsewhere. Or the doc should have, in this case, a separate _stat that says "In addition to the above, CPython's _stat, imported by stat, also provides ... -- Terry Jan Reedy
On 6/21/2013 7:39 AM, Nick Coghlan wrote:
On 21 June 2013 17:25, Victor Stinner <victor.stinner@gmail.com> wrote:
2013/6/21 Nick Coghlan <ncoghlan@gmail.com>:
Because practicality beats purity. This "wrong" Python code has been good enough for all Python version up until 3.4, it makes sense to keep it as a fallback instead of throwing it away.
How do you plan to handle the following case in Python?
"Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines 0xe000 as S_IFWHT (whiteout), but Solaris defines it as S_IFPORT (event port)."
We may keep the Python module if it is kept unchanged, but the Python and C modules should have the same public API (the C module should not have more features).
I think it's OK to expose additional platform specific features in the C version, and have them fail cleanly with the pure Python version (rather than silently giving the wrong answer). What's not OK is for the standard library to regress for other implementations just because we added a C implementation for the benefit of CPython. That's exactly the kind of thing PEP 399 says we *won't* do.
I was just writing up something similar. But as always, Nick said it better than me. -- Eric.
On 21Jun2013 21:39, Nick Coghlan <ncoghlan@gmail.com> wrote: | On 21 June 2013 17:25, Victor Stinner <victor.stinner@gmail.com> wrote: | > How do you plan to handle the following case in Python? | > | > "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines | > 0xe000 as S_IFWHT (whiteout), but Solaris defines it as | > S_IFPORT (event port)." | > | > We may keep the Python module if it is kept unchanged, but the Python | > and C modules should have the same public API (the C module should not | > have more features). | | I think it's OK to expose additional platform specific features in the | C version, and have them fail cleanly with the pure Python version | (rather than silently giving the wrong answer). What's not OK is for | the standard library to regress for other implementations just because | we added a C implementation for the benefit of CPython. That's exactly | the kind of thing PEP 399 says we *won't* do. +1 I'm all for the C module exposing any and all of the S_* macros for the local platform, and good with the python module (if used because the C module isn't present, or conceivably is compiled out because it is known broken on this platform) exposing only the portable stuff. At least you can detect "I don't know what to do" rather than ploughing on mistakenly. -- Cameron Simpson <cs@zip.com.au> Ignorance is preferable to error; and he is less remote from the truth who believes nothing, than he who believes what is wrong. - Thomas Jefferson
2013/6/20 Eric V. Smith <eric@trueblade.com <javascript:;>>:
But isn't the real problem with this module in Python the fact that the constants might be wrong? I'm not sure what, if anything, we can do about that.
Python is providing a stat module implemented in Python since 10 years, or maybe 20 years, and I don't remember that someone complained that constants are wrong. At the same time, Python distributes IN.py which contains things like that: LONG_MAX = 9223372036854775807 LONG_MAX = 2147483647 LONG_MIN = (-LONG_MAX - 1) In my opinion, the situation of plat-*/*.py modules is worse than stat.py :-) Victor
On 6/20/2013 4:53 PM, Victor Stinner wrote:
2013/6/20 Eric V. Smith <eric@trueblade.com <javascript:;>>:
But isn't the real problem with this module in Python the fact that the constants might be wrong? I'm not sure what, if anything, we can do about that.
Python is providing a stat module implemented in Python since 10 years, or maybe 20 years, and I don't remember that someone complained that constants are wrong.
This is serious, not argumentative: If there's really no concern that the values be correct, then why implement it in C? Let's just leave the python version with the hard-coded, usually-correct values. My personal opinion is that having the correct values in C is the way to go, and delete stat.py. Eric.
2013/6/20 Eric V. Smith <eric@trueblade.com>:
This is serious, not argumentative: If there's really no concern that the values be correct, then why implement it in C?
I read again the issue. The problem is to add new flags. Current flags (type: block device/symlink/..., file mode) are well defined and portable, whereas new flags are usually specific to an OS and no standardize yet. Examples: - Solaris: "door" - Solaris: "event port" - OSX/Darwin/FreeBSD: "whiteout" The other problem is that values of the new flags are not portable. Martin v. Löwis wrote: "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines 0xe000 as S_IFWHT (whiteout), but Solaris defines it as S_IFPORT (event port)." A C module reusing existing S_ISxxx() macros is the obvious way to solve this issue. Victor
20.06.13 16:05, Christian Heimes написав(ла):
I have re-implemented the entire stat module in C. [1] It's a necessary step to fix portability issues. For most constants POSIX standards dictate only the name of the constant and its meaning but not its value. Only the values of permission bits (0644 == rw-r--r--) have fixed values.
For example S_IFDIR is usually 0o40000 but it might be 0o20000 on some platforms. Common file types seem to have the same value on all important platforms. It's the only reason we were able to get away with stat.py. But uncommon file types like door, event port and whiteout don't have sensible default values. The C implementation is able to pick up the platform's values easily.
What shall I do about stat.py? statmodule.c implements all features of stat.py so there is no point in using it in CPython. It's one Python file less to load on every startup. However it might still be useful to alternative implementations of Python such as Jython or PyPy.
1) keep the file stat.py but let it be shadowed by the builtin stat module. Antoine loathes my hack... 2) rename stat.py to _stat.py 3) remove stat.py
Opinions?
Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be implement it on C?
2013/6/20 Serhiy Storchaka <storchaka@gmail.com>
Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be implement it on C?
Aha. Should an internal C module fetch the value of the constants, and a public stat.py nicely wrap them in enums? -- Amaury Forgeot d'Arc
On 06/20/2013 01:18 PM, Amaury Forgeot d'Arc wrote:
2013/6/20 Serhiy Storchaka wrote:
Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be implement it on C?
Aha. Should an internal C module fetch the value of the constants, and a public stat.py nicely wrap them in enums?
That's pretty much the approach I'm thinking about for sockets. -- ~Ethan~
But aren't most of these masks? enum,IntEnum doesn't handle those very gracefully (the | operator returns a plain int). On Thu, Jun 20, 2013 at 1:18 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
2013/6/20 Serhiy Storchaka <storchaka@gmail.com>
Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be implement it on C?
Aha. Should an internal C module fetch the value of the constants, and a public stat.py nicely wrap them in enums?
-- Amaury Forgeot d'Arc
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On 06/20/2013 01:27 PM, Guido van Rossum wrote:
On Thu, Jun 20, 2013 at 1:18 PM, Amaury Forgeot d'Arc wrote:
2013/6/20 Serhiy Storchaka wrote:
Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be implement it on C?
Aha. Should an internal C module fetch the value of the constants, and a public stat.py nicely wrap them in enums?
But aren't most of these masks? enum.IntEnum doesn't handle those very gracefully (the | operator returns a plain int).
As is, IntEnum would not be a great choice. We could either add a another Enum (BitMaskEnum?) that was also int-based, or flesh out IntEnum with enough smarts to handle |, &, and ^ and return an IntEnum when possible and fall back to a plain int when not possible. -- ~Ethan~
2013/6/20 Serhiy Storchaka <storchaka@gmail.com>:
Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be implement it on C?
Numerical values are less important than S_ISxxx() macros. Example: #define S_ISDOOR(mode) (((mode)&0xF000) == 0xd000) 0xd000 is (stat.S_IFSOCK + stat.S_IFIFO). And how do you represent the file mode with enums? I don't think that enum should be used in the stat module. I would prefer a stat object with methods than having to calls low-level functions. Something like: os.stat("document.txt").st_mode.is_reg() versus stat.S_ISREG(os.stat("document.txt").st_mode) The idea was discussed in http://bugs.python.org/issue11406 to solve a real API design issue. How should os.scandir() return the "is a directory" information with a portable API. I'm not saying that stat.S_ISREG should go away. The two approaches are complementary. Victor
participants (23)
-
Amaury Forgeot d'Arc -
Antoine Pitrou -
Armin Rigo -
Benjamin Peterson -
Brett Cannon -
Cameron Simpson -
Charles-François Natali -
Chris Angelico -
Christian Heimes -
Daniel Holth -
Eric Snow -
Eric V. Smith -
Ethan Furman -
Florent -
Guido van Rossum -
Gustavo Carneiro -
Nick Coghlan -
Serhiy Storchaka -
Stephen J. Turnbull -
Steven D'Aprano -
Terry Reedy -
Thomas Wouters -
Victor Stinner