How to specify optional support of arguments
In Python 3.3 a number of functions in the os module got the support of new options, like dir_fd or follow_symlinks. I well remember this because the initial reason of my contribution to CPython was related to this feature. The support of new options was platform dependent. They were supported only on some Posix platforms, but not on Windows. Sets like supports_dir_fd and supports_follow_symlinks was added in the os module to specify which functions support which options. If say os.stat is contained in os.supports_dir_fd, then os.stat() supports keyword argument dir_fd. Now I want to add the support of dir_fd to glob.glob() and some other functions. It will be platform-dependent, it requires os.open() and os.stat() supporting dir_fd and os.scandir supporting the file descriptor argument. How can I specify that glob.glob() supports dir_fd? 1. Add glob.glob in os.supports_dir_fd? It looks strange that importing the glob module modifies the value of the os module. 2. Introduce the glob.supports_dir_fd set and add glob.glob to it? We will need to add sets supports_dir_fd in all modules which contain functions which can support dir_fd. But what if the function is defined in one module, by normally is imported from other module? Should we merge sets supports_dir_fd from different modules? Also, the common problem with two former options is that when we wrap the function, we need to add it to the corresponding set. 3. Or set the attribute glob.glob.supports_dir_fd = True or glob.glob.__supports_dir_fd__ = True (and add similar attributes to all os functions which support)? What are your thoughts?
(see an answer to the OP's specific question below) I don't quite get the point for <module>.supports_stuff. It seems well-intentioned but misguided. 1. The documentation clearly says that it's supported depending on OS flavor -- so if I want to know if I can supply it, I need to rather check `os.name`. Those members are thus redundant. If the distinction is finer than os.name then I'd need some other, case-specific check depending on what the distinction is; alternatively, I could check the function's signature in its metadata if the support is reflected in it. 2. Support of a parameter is a characteristic of a function rather than a module. So information like this doesn't belong at module level. Moreover, object references can be moved around at will and they are all equal, there's no such thing as "one true reference" (you can get the object's lexical scope but there's no guarantee that it corresponds to the public interface). This is the structural conflict that Sergey ran into when he suddenly discovered that the functions and the flag are two completely unrelated objects that can be in completely different places. So any such flags, if they have the right to exist at all, should be attached to function objects. On 14.06.2020 20:36, Serhiy Storchaka wrote:
This would be the way consistent with the current design. If the public interface is to use the function from another module, the public interface module should have a parameter like this; whether the private module also has one and whether they are different objects is implementation detail.
14.06.20 23:45, Ivan Pozdeev via Python-Dev пише:
Yes, it is finer than os.name. It can depend on the kernel or libc version (and we do not know which versions are required on every platform), and there are a lot of platforms besides the main three. The user should not be expert in all platforms on which his program potentially can be ran. The function's signature is the same on all platforms. Just on some platforms only default value can be passed (None for dir_fd) or only specific types of argument is accepted (path-like, but not int).
On 15.06.2020 8:45, Serhiy Storchaka wrote:
Then a single boolean flag is clearly not enough. Compare: in https://docs.python.org/3/library/resource.html , the set of present RLIMIT_* members shows which rlimits are available in the specific OS. So I guess you want some similar pointers that would show which relevant #define constants (or other C-level entities that govern the availability) were present at the time of compilation. If availability is rather governed by a runtime than compile-time check, then something to perform the same check should be introduced; the distinction from .supports_stuff is it would be named after the check and completely decoupled from any functions that might be affected by the check.
Isn't it more Pythonic to simply call the function and an alternative path to handle the exception, anyway? Half of os needs to be tested for NotImplementedError or OSError if it's going to run anywhere outside the development environment anyway, otherwise you're stuck with only the most basic functions. What _would_ be handy is a list of what each function throws. As for the original question, since it's duplicating or wrapping a module level functionality, it should be module level, rather than function level, and if there's a better design then it would need to be ported back to os as well. Although I could also see good reasoning behind os being implicitly or having to be explicitly imported to test for the less-compatible extensions. On Sun, Jun 14, 2020 at 11:14 PM Ivan Pozdeev via Python-Dev < python-dev@python.org> wrote:
Instead of a bunch of ad-hoc mechanisms for finding out about platform-dependent arguments, maybe there should be a function in the inspect module for testing whether a function has a given argument. Then you could say something like if inspect.hasargument(glob.glob, 'dir_fd'): ... -- Greg
Even without platform differences, limits on the value of parameters really ought to be in the docstring, which is available to inspect. If you're asking for a more specific convention that could actually be used, that still probably still needs to be written.
(see an answer to the OP's specific question below) I don't quite get the point for <module>.supports_stuff. It seems well-intentioned but misguided. 1. The documentation clearly says that it's supported depending on OS flavor -- so if I want to know if I can supply it, I need to rather check `os.name`. Those members are thus redundant. If the distinction is finer than os.name then I'd need some other, case-specific check depending on what the distinction is; alternatively, I could check the function's signature in its metadata if the support is reflected in it. 2. Support of a parameter is a characteristic of a function rather than a module. So information like this doesn't belong at module level. Moreover, object references can be moved around at will and they are all equal, there's no such thing as "one true reference" (you can get the object's lexical scope but there's no guarantee that it corresponds to the public interface). This is the structural conflict that Sergey ran into when he suddenly discovered that the functions and the flag are two completely unrelated objects that can be in completely different places. So any such flags, if they have the right to exist at all, should be attached to function objects. On 14.06.2020 20:36, Serhiy Storchaka wrote:
This would be the way consistent with the current design. If the public interface is to use the function from another module, the public interface module should have a parameter like this; whether the private module also has one and whether they are different objects is implementation detail.
14.06.20 23:45, Ivan Pozdeev via Python-Dev пише:
Yes, it is finer than os.name. It can depend on the kernel or libc version (and we do not know which versions are required on every platform), and there are a lot of platforms besides the main three. The user should not be expert in all platforms on which his program potentially can be ran. The function's signature is the same on all platforms. Just on some platforms only default value can be passed (None for dir_fd) or only specific types of argument is accepted (path-like, but not int).
On 15.06.2020 8:45, Serhiy Storchaka wrote:
Then a single boolean flag is clearly not enough. Compare: in https://docs.python.org/3/library/resource.html , the set of present RLIMIT_* members shows which rlimits are available in the specific OS. So I guess you want some similar pointers that would show which relevant #define constants (or other C-level entities that govern the availability) were present at the time of compilation. If availability is rather governed by a runtime than compile-time check, then something to perform the same check should be introduced; the distinction from .supports_stuff is it would be named after the check and completely decoupled from any functions that might be affected by the check.
Isn't it more Pythonic to simply call the function and an alternative path to handle the exception, anyway? Half of os needs to be tested for NotImplementedError or OSError if it's going to run anywhere outside the development environment anyway, otherwise you're stuck with only the most basic functions. What _would_ be handy is a list of what each function throws. As for the original question, since it's duplicating or wrapping a module level functionality, it should be module level, rather than function level, and if there's a better design then it would need to be ported back to os as well. Although I could also see good reasoning behind os being implicitly or having to be explicitly imported to test for the less-compatible extensions. On Sun, Jun 14, 2020 at 11:14 PM Ivan Pozdeev via Python-Dev < python-dev@python.org> wrote:
Instead of a bunch of ad-hoc mechanisms for finding out about platform-dependent arguments, maybe there should be a function in the inspect module for testing whether a function has a given argument. Then you could say something like if inspect.hasargument(glob.glob, 'dir_fd'): ... -- Greg
Even without platform differences, limits on the value of parameters really ought to be in the docstring, which is available to inspect. If you're asking for a more specific convention that could actually be used, that still probably still needs to be written.
participants (6)
-
Emily Bowman -
Greg Ewing -
Guido van Rossum -
Ivan Pozdeev -
Jim J. Jewett -
Serhiy Storchaka