Sharing file descriptors across processes

dave at pythonapocrypha.com dave at pythonapocrypha.com
Wed Sep 5 12:03:11 EDT 2001


I need to somehow "pass" a file descriptor (socket) from one process to
another process, does anybody know how this can be done? More specifically
I have a program that accepts an incoming socket and then opens another
socket to a server, and then acts as a proxy, passing data between the
two. I would like to find some way to take that pair of sockets and hand
them off to another worker process. Here's some additional info:

- This is a Linux system. Linux-specific solutions are okay but I may need
to eventually find a solution for Solaris.
- I'd prefer a Python-only solution of course, but I don't really have a
problem implementing the hand-off functionality in a C extension module.
- I want to split the work into separate processes to take advantage of
multiple CPUs.
- This is nearly identical to a main thread passing off work to a worker
thread, except that the global interpreter lock nukes a lot of the
multi-CPU advantage. One idea I've considered but not yet pursued is to
start several C threads, each one running an instance of the Python
interpreter, and somehow passing the sockets between the Python
interpreters.
- This is nearly identical to a main process forking off work to a child
worker process, except that forking each time is too expensive. I can fork
off the children initially (if that will help somehow, e.g. if that lets
the children share the file descriptor tables or something), but the
parent and the children need to be long-running, where the parent is
continually passing off work to the kids.

Hmm.. this really isn't all that Python-specific, sorry. I'd be happy just
to find a way to do it in C to begin with.  I'd really appreciate any help
at all, ranging from "oh yeah, just do this" to "try reading up on this".

Thanks!
Dave





More information about the Python-list mailing list