Support for async read/write
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Current Python lacks support for "aio_*" syscalls to do async IO. I think this could be a nice addition for python 3.3. If you agree, I will create an issue in the tracker. If you think the idea is of no value, please say so for me to move on. Maybe an 3th party module, but I think this functionality sould be available in core python. Thanks!. PS: The function calls are: aio_cancel, aio_error, aio_fsync, aio_read, aio_return, aio_write. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTL3MUplgi5GaxT1NAQKv5QQAnDan88kXJ67fucz2rT/mZze+065lm9E4 +XJ2JfqGMVE1/qMsXwg81l19RHSYReBgBjd7zyXWE9Fk/1Rfq4gjOZEtoG0IpGZG E3CH+Ab5/o/PjJZNKQaPpe0cwGSXFPKkPpgepKS1d8ZXyf6VXMc8UTSWjMI5jIVe 4M+yvw5m4I0= =nsdO -----END PGP SIGNATURE-----
Yes, that would be a nice addition. -peter On 10/19/10 12:50 PM, "Jesus Cea" <jcea@jcea.es> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Current Python lacks support for "aio_*" syscalls to do async IO. I think this could be a nice addition for python 3.3.
If you agree, I will create an issue in the tracker. If you think the idea is of no value, please say so for me to move on. Maybe an 3th party module, but I think this functionality sould be available in core python.
Thanks!.
PS: The function calls are: aio_cancel, aio_error, aio_fsync, aio_read, aio_return, aio_write.
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQCVAwUBTL3MUplgi5GaxT1NAQKv5QQAnDan88kXJ67fucz2rT/mZze+065lm9E4 +XJ2JfqGMVE1/qMsXwg81l19RHSYReBgBjd7zyXWE9Fk/1Rfq4gjOZEtoG0IpGZG E3CH+Ab5/o/PjJZNKQaPpe0cwGSXFPKkPpgepKS1d8ZXyf6VXMc8UTSWjMI5jIVe 4M+yvw5m4I0= =nsdO -----END PGP SIGNATURE----- _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.c...
On 04:50 pm, jcea@jcea.es wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Current Python lacks support for "aio_*" syscalls to do async IO. I think this could be a nice addition for python 3.3.
Adding more platform wrappers is always nice. Keep in mind that the quality of most (all?) aio_* implementations is spotty at best, though. On Linux, they will sometimes block (for example, if you fail to align buffers properly, or open a file without O_DIRECT, or if there are too many other aio operations active on the system at the time, etc). Thus, these APIs are finicky at best, and the Python bindings will be similarly fragile. Jean-Paul
If you agree, I will create an issue in the tracker. If you think the idea is of no value, please say so for me to move on.
Do you have an application in mind?
Maybe an 3th party module, but I think this functionality sould be available in core python.
Starting off as a 3rd party library to try to develop some interest and users (and thus experience) before adding it to the standard library might make sense (as it frequently does).
Thanks!.
PS: The function calls are: aio_cancel, aio_error, aio_fsync, aio_read, aio_return, aio_write.
Jean-Paul
On Oct 19, 2010, at 1:47 PM, exarkun@twistedmatrix.com wrote:
Adding more platform wrappers is always nice. Keep in mind that the quality of most (all?) aio_* implementations is spotty at best, though. On Linux, they will sometimes block (for example, if you fail to align buffers properly, or open a file without O_DIRECT, or if there are too many other aio operations active on the system at the time, etc).
You're thinking of the linux-specific AIO calls. Those have the properties you're describing (which makes them pretty useless for most code too), but they're completely different from the aio_* functions. The POSIX aio_* calls don't do any of that. They aren't syscalls implemented in the kernel, they're implemented in glibc. They "simply" create a threadpool in your process to call the standard synchronous operations, and make it difficult to reliably get completion notification (completion notification takes place via Real-Time signals (SIGEV_SIGNAL), which can be dropped if linux runs out of space in its RT-signal-queue, and when that happens you get no indication that that has occurred. You can also do completion notification via calling a function on a thread (SIGEV_THREAD), but, for that, glibc will always spawns a brand new thread for each notification, which is quite slow.) Basically: you shouldn't ever use those APIs. Especially on linux, but probably everywhere else. So, in conclusion, I disagree that adding wrappers for these would be nice. It wouldn't. It would cause some people to think they would be useful things to call, and they would always be wrong. James
So, in conclusion, I disagree that adding wrappers for these would be nice. It wouldn't. It would cause some people to think they would be useful things to call, and they would always be wrong.
We are all consenting adults. If people want to shoot themselves in their feet, we let them. For example, we have os.open, even though there is no garbage collection for file handles, and we have os._exit, even though it doesn't call finalizers. People should trust that the APIs we expose go *literally* into the library, so we can blame any consequences that this has onto the library (and possibly the kernel, in turn). People readily accept that explanation. Regards, Martin
On Oct 19, 2010, at 6:44 PM, Martin v. Löwis wrote:
So, in conclusion, I disagree that adding wrappers for these would be nice. It wouldn't. It would cause some people to think they would be useful things to call, and they would always be wrong.
We are all consenting adults. If people want to shoot themselves in their feet, we let them. For example, we have os.open, even though there is no garbage collection for file handles, and we have os._exit, even though it doesn't call finalizers.
There's a difference. os._exit is useful. os.open is useful. aio_* are *not* useful. For anything. If there's anything you think you want to use them for, you're wrong. It either won't work properly or it will worse performing than the simpler alternatives. It would absolutely be a waste of time (of both the implementor of the wrapper and the poor users who stumble across them in documentation and try to use them) to bother adding wrappers to these functions for python. James
On Oct 19, 2010, at 8:09 PM, James Y Knight wrote:
There's a difference.
os._exit is useful. os.open is useful. aio_* are *not* useful. For anything. If there's anything you think you want to use them for, you're wrong. It either won't work properly or it will worse performing than the simpler alternatives.
I'd like to echo this sentiment. This is not about providing a 'safe' wrapper to hide some powerful feature of these APIs: the POSIX aio_* functions are really completely useless. To quote the relevant standard <http://www.opengroup.org/onlinepubs/000095399/basedefs/aio.h.html>: APPLICATION USAGE None. RATIONALE None. FUTURE DIRECTIONS None. Not only is the performance usually worse than expected, the behavior of aio_* functions require all kinds of subtle and mysterious coordination with signal handling, which I'm not entirely sure Python would even be able to pull off without some modifications to the signal module. (And, as Jean-Paul mentioned, if your OS kernel runs out of space in a queue somewhere, completion notifications might just never be delivered at all.) I would love for someone to prove me wrong. In particular, I would really love for there to be a solution to asynchronous filesystem I/O better than "start a thread, read until you block". But, as far as I know, there isn't, and wrapping these functions will just confuse and upset anyone who attempts to use them in any way.
On 01:37 am, glyph@twistedmatrix.com wrote:
On Oct 19, 2010, at 8:09 PM, James Y Knight wrote:
There's a difference.
os._exit is useful. os.open is useful. aio_* are *not* useful. For anything. If there's anything you think you want to use them for, you're wrong. It either won't work properly or it will worse performing than the simpler alternatives.
I'd like to echo this sentiment. This is not about providing a 'safe' wrapper to hide some powerful feature of these APIs: the POSIX aio_* functions are really completely useless.
To quote the relevant standard <http://www.opengroup.org/onlinepubs/000095399/basedefs/aio.h.html>:
APPLICATION USAGE
None.
RATIONALE
None.
FUTURE DIRECTIONS
None.
Not only is the performance usually worse than expected, the behavior of aio_* functions require all kinds of subtle and mysterious coordination with signal handling, which I'm not entirely sure Python would even be able to pull off without some modifications to the signal module. (And, as Jean-Paul mentioned, if your OS kernel runs out of space in a queue somewhere, completion notifications might just never be delivered at all.)
Just to be clear, James corrected me there. I thought Jesus was talking about the mostly useless Linux AIO APIs, which have the problems I described. He was actually talking about the POSIX AIO APIs, which have a different set of problems making them a waste of time. Jean-Paul
On Oct 19, 2010, at 9:55 PM, exarkun@twistedmatrix.com wrote:
Not only is the performance usually worse than expected, the behavior of aio_* functions require all kinds of subtle and mysterious coordination with signal handling, which I'm not entirely sure Python would even be able to pull off without some modifications to the signal module. (And, as Jean-Paul mentioned, if your OS kernel runs out of space in a queue somewhere, completion notifications might just never be delivered at all.)
Just to be clear, James corrected me there. I thought Jesus was talking about the mostly useless Linux AIO APIs, which have the problems I described. He was actually talking about the POSIX AIO APIs, which have a different set of problems making them a waste of time.
I know, I'm referring to the behavior of POSIX AIO. Perhaps I'm overstating the case with 'subtle and mysterious', then, but the POISX 'aiocb' structure still includes an 'aio_sigevent' member which is the way to find out about I/O event completion. If you're writing an application that uses AIO, basically all of your logic ends up living in the context of a signal handler, and as <http://www.opengroup.org/onlinepubs/000095399/functions/xsh_chap02_04.html#tag_02_04_01> puts it, "When signal-catching functions are invoked asynchronously with process execution, the behavior of some of the functions defined by this volume of IEEE Std 1003.1-2001 is unspecified if they are called from a signal-catching function." Of course, you could try using signalfd(), but that's not in POSIX. (Or, you could use SIGEV_THREAD, but that would be functionally equivalent to running read() in a thread, except much more difficult.)
On Tue, Oct 19, 2010 at 8:37 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
I'd like to echo this sentiment. This is not about providing a 'safe' wrapper to hide some powerful feature of these APIs: the POSIX aio_* functions are really completely useless. To quote the relevant standard <http://www.opengroup.org/onlinepubs/000095399/basedefs/aio.h.html>:
APPLICATION USAGE None. RATIONALE None. FUTURE DIRECTIONS None.
No comment on the rest of your claim, but this is a silly argument. The standard says the same thing about at least fcntl.h, signal.h, pthread.h, and ucontext.h, which clearly are useful. Jeffrey
On Oct 20, 2010, at 12:31 AM, Jeffrey Yasskin wrote:
No comment on the rest of your claim, but this is a silly argument. The standard says the same thing about at least fcntl.h, signal.h, pthread.h, and ucontext.h, which clearly are useful.
It was meant to be tongue-in-cheek :). Perhaps I should not have assumed that everyone else was as familiar with the POSIX documentation; I figured that most readers would know that most pages say that. But, that was the result of a string of many different searches attempting to find someone explaining why this was a good idea or why anyone would want to use it. I think in this case, it's accurate.
On Wed, Oct 20, 2010 at 2:55 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Oct 20, 2010, at 12:31 AM, Jeffrey Yasskin wrote:
No comment on the rest of your claim, but this is a silly argument. The standard says the same thing about at least fcntl.h, signal.h, pthread.h, and ucontext.h, which clearly are useful.
It was meant to be tongue-in-cheek :). Perhaps I should not have assumed that everyone else was as familiar with the POSIX documentation; I figured that most readers would know that most pages say that.
Oops, sorry. My joke-detector failed.
Am 19.10.2010 19:47, schrieb exarkun@twistedmatrix.com:
On 04:50 pm, jcea@jcea.es wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Current Python lacks support for "aio_*" syscalls to do async IO. I think this could be a nice addition for python 3.3.
Adding more platform wrappers is always nice.
Exactly so. Exposing platform APIs as-is really doesn't need much discussion, except when the platform wrapper is going to deviate significantly from the platform API. There is a long tradition of exposing even obscure unixish APIs in Python, making it the #1 language to write exotic scripts in. We definitely want to continue this tradition. Please refrain from trying to come up with "abstractions", though; this should be outside of Python for the moment. Covering both Unix AIO and Windows completion ports (say) may be quite a challenge (but then, JP may have some view on this). Regards, Martin
You should file a new issue on the bug tracker but unless you have a patch to propose it's unlikely that someone else is gonna implement it. Regards --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ 2010/10/19 Jesus Cea <jcea@jcea.es>:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Current Python lacks support for "aio_*" syscalls to do async IO. I think this could be a nice addition for python 3.3.
If you agree, I will create an issue in the tracker. If you think the idea is of no value, please say so for me to move on. Maybe an 3th party module, but I think this functionality sould be available in core python.
Thanks!.
PS: The function calls are: aio_cancel, aio_error, aio_fsync, aio_read, aio_return, aio_write.
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQCVAwUBTL3MUplgi5GaxT1NAQKv5QQAnDan88kXJ67fucz2rT/mZze+065lm9E4 +XJ2JfqGMVE1/qMsXwg81l19RHSYReBgBjd7zyXWE9Fk/1Rfq4gjOZEtoG0IpGZG E3CH+Ab5/o/PjJZNKQaPpe0cwGSXFPKkPpgepKS1d8ZXyf6VXMc8UTSWjMI5jIVe 4M+yvw5m4I0= =nsdO -----END PGP SIGNATURE----- _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com
Hello Jesus,
Current Python lacks support for "aio_*" syscalls to do async IO. I think this could be a nice addition for python 3.3.
If you agree, I will create an issue in the tracker. If you think the idea is of no value, please say so for me to move on. Maybe an 3th party module, but I think this functionality sould be available in core python.
Well, if you think this is interesting, it would be nice if you could do a bit more research instead of just filing issues for us to solve. After all, you probably already have an idea about why this is useful, how to interface it into the stdlib, etc. Just saying "we should expose these functions" doesn't bring a lot of added value IMO (there are many system functions which could be exposed and currently aren't). Also, the canonical way to do file I/O in Python 3 is the `io` lib, therefore it would be a bit of a shame to have separate, non-integrated `aio_*` functions. Also, since the `io` lib is already supposed to support non-blocking IO, perhaps it would be valuable to stress this support and propose any interesting patches for fixing and/or improving it. Regards Antoine.
Also, the canonical way to do file I/O in Python 3 is the `io` lib, therefore it would be a bit of a shame to have separate, non-integrated `aio_*` functions.
I disagree. We also have posix.open, posix.dup, etc. We always expose POSIX functions in the posix module (except for the socket functions, unfortunately, and a few other exceptions), and I see no reason to break with this tradition. The io module should be thought of as sitting on top of the posix module (and it initially was).
Also, since the `io` lib is already supposed to support non-blocking IO, perhaps it would be valuable to stress this support and propose any interesting patches for fixing and/or improving it.
I think that's entirely independent. Regards, Martin
Le mercredi 20 octobre 2010 à 00:48 +0200, "Martin v. Löwis" a écrit :
Also, the canonical way to do file I/O in Python 3 is the `io` lib, therefore it would be a bit of a shame to have separate, non-integrated `aio_*` functions.
I disagree. We also have posix.open, posix.dup, etc. We always expose POSIX functions in the posix module (except for the socket functions, unfortunately, and a few other exceptions), and I see no reason to break with this tradition. The io module should be thought of as sitting on top of the posix module (and it initially was).
I'm not suggesting to break with any "tradition", but that building duplicate APIs to do file I/O has little value and only makes things more cumbersome.
Also, since the `io` lib is already supposed to support non-blocking IO, perhaps it would be valuable to stress this support and propose any interesting patches for fixing and/or improving it.
I think that's entirely independent.
I don't think it is. If the goal is to give ways to improve file I/O performance for specialized use cases, then it certainly deserves integration with the current I/O stack. If the goal is not that, then I don't really know what it is. It would be nice to know about the use case Jesus has in mind. Regards Antoine.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 20/10/10 00:58, Antoine Pitrou wrote:
It would be nice to know about the use case Jesus has in mind.
I am thinking about the cases where a request come, you do some disk read in behalf of it and you reply. If the read is "slow" (if not cached, you have to deal with a physical harddisk), you stop the main-loop for a while, unless you use threads. If, instead, I can schedule a read and keep processing other requests (possibly queueing new reads), be notified when the read is done and complete the reply, I think this is more simple and performing (and far less memory hungry) that using threads. I just opened a issue, assigned to me. I plan to do the implementation myself, at least for Solaris and possibly (recent) Linux: <http://bugs.python.o rg/issue11117>. Some people say AIO OS implementations are flaky. Well, you must deal with it, like you deal with other flaky OS corners, like exhausting memory or whatever. I would suppose that if Python exposes AIO and in some OS support is underpar, the exposing would incentivate a better support for OS. Like happened to threads and linux years ago. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTUwbeZlgi5GaxT1NAQK4uwP+J3PwGY1dvBi+OyBo9M9UWsma0khgzdUS 6ewHhCrCK+U5HK0e/g9cLbesBSsYvVfNjPe+fb9cQuwMBK0lpF3kOzfsEf82RIxR NlsqOba31CE1tW9uS4wja0TddFDob3IImqgwSB7NptBOZTNVjDvB6k0V7KHqvPWX 9g01GqaQ9uQ= =hx6M -----END PGP SIGNATURE-----
participants (9)
-
"Martin v. Löwis" -
Antoine Pitrou -
exarkun＠twistedmatrix.com -
Giampaolo Rodolà -
Glyph Lefkowitz -
James Y Knight -
Jeffrey Yasskin -
Jesus Cea -
Peter Portante