python sendmsg()/recvmsg() implementation
Hello, my name is Greg. I've just started using python after many years of C programming, and I'm also new to the list. I wanted to clarify this first, so that maybe I will get a little less beating for my stupidity :) I use python3 and Linux on arch x86 (production will be on x86_64, though this shouldn't matter much). The application that I'm presently working on is a network server. It would use separate processes to accept the connections, and to do the work (much like how apache prefork does). One process accept()s on the original socket and the received socket (client socket) will be read for a request. After the request is received and parsed this process (the controller) will choose one from its' children that is most capable of handling the said request. It would then pass the file descriptor through a socketpair to the appropriate children and go handle the next client. All works fine and smooth, but I realized that I need sendmsg()/recvmsg() to pass the FD. Since these are not implemented in the python socket module, and Linux has no other way to do this, I'm stuck. Fell flat on my face, too :) Browsing the net I've found a patch to the python core (http://bugs.python.org/issue1194378), dated 2005. First of all, I would like to ask you guys, whether you know of any way of doing this FD passing magic, or that you know of any 3rd party module / patch / anything that can do this for me. Since I'm fairly familiar with C and (not that much, but I feel the power) python, I would take the challenge of writing it, given that the above code is still somewhat usable. If all else fails I would like to have your help to guide me through this process. Thanks Kalman Gergely
On Tue, 09 Jun 2009 16:46:54 +0200, Kálmán Gergely <kalman.gergely@duodecad.hu> wrote:
Hello, my name is Greg.
I've just started using python after many years of C programming, and I'm also new to the list. I wanted to clarify this first, so that maybe I will get a little less beating for my stupidity :)
Welcome!
[snip]
Browsing the net I've found a patch to the python core (http://bugs.python.org/issue1194378), dated 2005.
First of all, I would like to ask you guys, whether you know of any way of doing this FD passing magic, or that you know of any 3rd party module / patch / anything that can do this for me.
Aside from the patch in the tracker, there are several implementations of these APIs as third-party extension modules.
Since I'm fairly familiar with C and (not that much, but I feel the power) python, I would take the challenge of writing it, given that the above code is still somewhat usable. If all else fails I would like to have your help to guide me through this process.
What would be great is if you could take the patch in the tracker and get it into shape so that it is suitable for inclusion. This would involve three things, I think: 1. Write unit tests for the functionality (since the patch itself provides none) 2. Update the patch so that it again applies cleanly against trunk 3. Add documentation for the new APIs Once this is done, you can get a committer to look at it and either provide more specific feedback or apply it. Thanks, Jean-Paul
Jean-Paul Calderone wrote:
On Tue, 09 Jun 2009 16:46:54 +0200, Kálmán Gergely <kalman.gergely@duodecad.hu> wrote:
Hello, my name is Greg.
I've just started using python after many years of C programming, and I'm also new to the list. I wanted to clarify this first, so that maybe I will get a little less beating for my stupidity :)
Welcome!
[snip]
Browsing the net I've found a patch to the python core (http://bugs.python.org/issue1194378), dated 2005.
First of all, I would like to ask you guys, whether you know of any way of doing this FD passing magic, or that you know of any 3rd party module / patch / anything that can do this for me.
Aside from the patch in the tracker, there are several implementations of these APIs as third-party extension modules.
Since I'm fairly familiar with C and (not that much, but I feel the power) python, I would take the challenge of writing it, given that the above code is still somewhat usable. If all else fails I would like to have your help to guide me through this process.
What would be great is if you could take the patch in the tracker and get it into shape so that it is suitable for inclusion. This would involve three things, I think:
1. Write unit tests for the functionality (since the patch itself provides none) 2. Update the patch so that it again applies cleanly against trunk 3. Add documentation for the new APIs
Once this is done, you can get a committer to look at it and either provide more specific feedback or apply it.
Thanks,
Jean-Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/synapse%40jasmin.hu
Hello again So, after a little cleanup I've managed to integrate the code into socketmodule.c/h. It works fine now, though I needed to add it to Lib/socket.py, otherwise it wouldn't show up in the socket module (I've searched for recvfrom and added it). I've also cleaned up the code a little, fixed some codingstyle issues (which might still exist). Since I am not a python core developer the patch might still be in a pretty outdated state. I'd like someone to look it over and direct me to some documentation (the ones I've found so far were pretty sketchy), for it to be acceptable for inclusion. The sanity of the code is what matters to me the most. I've looked it over though and found it in a sound state, but I guess you guys might have a different opinion about that ;) With writing the test cases, some documentation would be nice. I've attached the patch generated with svn diff for revision 73434, and the test code that I use to pass a file descriptor between processes. It works just fine :). Thanks Kalman Gergely
Kálmán Gergely wrote:
Since I am not a python core developer the patch might still be in a pretty outdated state. I'd like someone to look it over and direct me to some documentation (the ones I've found so far were pretty sketchy), for it to be acceptable for inclusion. The sanity of the code is what matters to me the most. I've looked it over though and found it in a sound state, but I guess you guys might have a different opinion about that ;)
With writing the test cases, some documentation would be nice.
Most unit tests these days are written based on either doctest or unittest. When adding new features to an existing module, it is usually best to follow the testing style already used for the rest of that module (in this case, that should be test/test_socket.py). The relevant question in the dev FAQ gives good pointers: http://www.python.org/dev/faq/#how-to-test-a-patch
I've attached the patch generated with svn diff for revision 73434, and the test code that I use to pass a file descriptor between processes. It works just fine :).
Uploading files to the tracker is generally the best option - patches tend to get lost if they're just sent to the mailing list. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Hello This is the rewritten-from-scratch implementation of the sendmsg()/recvmsg() methods. Any comments / suggestions / flames are very welcome. Currently it supports what I need and I'm only releasing it, because I don't have much time to develop it further in the forseeable future (1-2 months). It is rewritten from scratch, using the python c-api documents. I've tried my best, but I wouldn't bet that it works as it's supposed to. I'd be glad if someone could give me a review on what I've done wrong. The core parts are implemented correctly (I think), the features that are missing: - using scatter/gather - using it with non-stream oriented sockets (doesn't support addresses /msg_name/) These should be very easy to implement though. I will fix the errors that are present right now, and if no one takes up the task I will implement the missing features also. You might have to wait for it a little though. Thanks in advance Cheers, Kalman Gergely
On Thu, Jul 23, 2009, K?lm?n Gergely wrote:
This is the rewritten-from-scratch implementation of the sendmsg()/recvmsg() methods. Any comments / suggestions / flames are very welcome. Currently it supports what I need and I'm only releasing it, because I don't have much time to develop it further in the forseeable future (1-2 months). It is rewritten from scratch, using the python c-api documents. I've tried my best, but I wouldn't bet that it works as it's supposed to. I'd be glad if someone could give me a review on what I've done wrong.
Please post this to bugs.python.org -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "The volume of a pizza of thickness 'a' and radius 'z' is given by pi*z*z*a"
Done, it's at: http://bugs.python.org/issue6560 Kalman Gergely Aahz wrote:
On Thu, Jul 23, 2009, K?lm?n Gergely wrote:
This is the rewritten-from-scratch implementation of the sendmsg()/recvmsg() methods. Any comments / suggestions / flames are very welcome. Currently it supports what I need and I'm only releasing it, because I don't have much time to develop it further in the forseeable future (1-2 months). It is rewritten from scratch, using the python c-api documents. I've tried my best, but I wouldn't bet that it works as it's supposed to. I'd be glad if someone could give me a review on what I've done wrong.
Please post this to bugs.python.org
On Tue, Jun 9, 2009 at 7:46 AM, Kálmán Gergely<kalman.gergely@duodecad.hu> wrote:
Hello, my name is Greg.
I've just started using python after many years of C programming, and I'm also new to the list. I wanted to clarify this first, so that maybe I will get a little less beating for my stupidity :)
I use python3 and Linux on arch x86 (production will be on x86_64, though this shouldn't matter much).
The application that I'm presently working on is a network server. It would use separate processes to accept the connections, and to do the work (much like how apache prefork does). One process accept()s on the original socket and the received socket (client socket) will be read for a request. After the request is received and parsed this process (the controller) will choose one from its' children that is most capable of handling the said request. It would then pass the file descriptor through a socketpair to the appropriate children and go handle the next client. All works fine and smooth, but I realized that I need sendmsg()/recvmsg() to pass the FD. Since these are not implemented in the python socket module, and Linux has no other way to do this, I'm stuck. Fell flat on my face, too :)
Browsing the net I've found a patch to the python core (http://bugs.python.org/issue1194378), dated 2005.
First of all, I would like to ask you guys, whether you know of any way of doing this FD passing magic, or that you know of any 3rd party module / patch / anything that can do this for me.
IIRC, this is already implemented in the multiprocessing package, which comes standard with Python 2.6 and 3.0 . It looks like the test is disabled in test/test_multiprocessing.py , and re-enabling it (on Windows) produces errors that make me think it's more an issue with the tests than with multiprocessing itself. Dig into it, see if you can get the tests to pass :) - Josiah
Since I'm fairly familiar with C and (not that much, but I feel the power) python, I would take the challenge of writing it, given that the above code is still somewhat usable. If all else fails I would like to have your help to guide me through this process.
Thanks
Kalman Gergely _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.com
participants (5)
-
Aahz -
Jean-Paul Calderone -
Josiah Carlson -
Kálmán Gergely -
Nick Coghlan