[py-svn] r6832 - py/dist/doc
hpk at codespeak.net
hpk at codespeak.net
Sun Oct 3 12:49:39 CEST 2004
Author: hpk
Date: Sun Oct 3 12:49:38 2004
New Revision: 6832
Added:
py/dist/doc/execnet.txt
Log:
a draft in pseudo-rest for a document describing
the py.execnet library. Armin, you wanted to
have a look at the py.execnet stuff and i think
looking into this document and improving it
is the way to go (apart from looking what is
already there).
Added: py/dist/doc/execnet.txt
==============================================================================
--- (empty file)
+++ py/dist/doc/execnet.txt Sun Oct 3 12:49:38 2004
@@ -0,0 +1,160 @@
+The py.execnet library 0.8.0
+============================
+
+Execnet deals with letting your python programs execute and
+communicate across process and computer barriers. At the
+core it is a very simple and powerful mechanism: executing
+source code at "the other side". The other side is reachable
+through Gateways, currently
+
+- py.execnet.PopenGateway for executing code within a child process
+ on the same computer
+
+- py.execnet.SocketGateway for executing code across a socket connection.
+
+For now, see the py/gateway/gateway_test.py file for how it works
+currently.
+
+High Level Interface for executing code
+---------------------------------------
+
+These gateways offer the following "high level interface"
+
+ [[docfunc(py.__impl__.execnet.gateway.Gateway.remote_exec_oneshot)]]
+
+ [[docfunc(py.__impl__.execnet.gateway.Gateway.remote_exec_async)]]
+
+ [[docfunc(py.__impl__.execnet.gateway.Gateway.remote_exec_sync)]]
+
+ [[docfunc(py.__impl__.execnet.gateway.Gateway.remote_import)]]
+
+(editors note: the mechanism to substitute those "docfunc" calls
+is not there yet, but i want such documentation to be automatically
+connected to the real code, showing its docstrings etc.pp)
+
+The most important property of execnet gateways is that
+the protocol they speak with each other ``are determined by
+the client side``. If you open a gateway on some server
+in order to coordinate with some other programs you
+determine your own protocol, not affecting all the others.
+Multiple clients can run their own gateway versions in
+the same remote process (e.g. connecting through their
+version of a SocketGateway).
+
+You should not need to maintain software on the other sides
+you are running your code at.
+
+...
+
+(Proposed "Channel" VHL interface for exchanging data)
+------------------------------------------------------
+
+While executing custom strings on "the other side" is simple enough
+it is often tricky to deal with. Therefore we want a way
+to send data items to and fro between the distributedly running
+program. The idea is to inject a Channel object for each
+execution of source code. This Channel object allows two
+program parts to send data to each other.
+
+Here is the rough idea for the interface:
+
+ channel.gateway # the gateway through which this channel operates
+
+ channel.send(*items, timeout=None):
+ sends the given items to the other side of the channel,
+ possibly blocking if the sender queue is full.
+ Note that each value V of the items needs to have the
+ following property (all basic types in python have it):
+ eval(repr(V)) == V.
+
+ channel.receive(timeout=None):
+ receives an item that was sent from the other side,
+ possibly blocking if there is none.
+
+ channel.set(**namevaluepairs):
+ set the given namevaluepairs on the channel object.
+ These bindings will appear on both sides of the channel.
+
+ channel.get(name, ignore=_dummy, timeout=None):
+ return the value from a name binding, possibly blocking
+ until it appears. If ignore is provided then its value
+ will be ignored and the function will only return if
+ a different value is received. Obviously, set/get can be
+ used for sending named signals to each other.
+
+ channel.subscribe(itemcallback=None, namevaluecallback=None):
+ register the given callbacks. itemcallback will be called
+ with every received item. namevaluecallback will be called
+ with every received set of namevaluepairs as keyword arguments.
+ You can subscribe multiple callbacks which will be executed
+ in the order of their subscription. Note that the execution
+ of callbacks is serialized and all callbacks will run in the
+ same thread.
+
+ channel.unsubscribe(itemcallback=None, namevaluecallback=None):
+ unregister the given callback from their subscription.
+ If a callback is registered multiple times all instances
+ will be deleted.
+
+
+
+An Example for Channels
+-----------------------
+
+The following proposed example opens a PopenGateway, i.e. a
+child process, starts a socket server within that process and
+then opens a SocketGateway to the freshly started
+socketserver. The "startserver.py" is a small script that
+basically listens and accepts socket connections, receives one
+liners and executes them. This socketserver stuff is supposed
+to never change. Probably it should be a script, callable like
+py.execnet.socketserver from the command line.
+
+ import py
+ socketserverbootstrap = py.code.Source(
+ py.script.getpath('py.execnet.socketserver').read()
+ """
+ import socket
+ portrange = channel.get("portrange")
+ for i in portrange:
+ try:
+ sock = listen(("localhost", i))
+ except socket.error:
+ continue
+ else:
+ channel.set(listenport=i)
+ startserver(sock)
+ print "started server with socket"
+ break
+ else:
+ channel.set(listenport=None)
+ """)
+
+ # open a gateway to a fresh child process
+ proxygw = py.execnet.PopenGateway()
+
+ # execute asynchronously the above socketserverbootstrap on the other
+ channel = proxygw.remote_exec_async(socketserverbootstrap)
+
+ # send parameters for the for-loop
+ channel.set(portrange=(7770, 7800))
+ #
+ # the other side should start the for loop now, we
+ # wait for the result
+ #
+ listenport = channel.get('listenport', timeout=3)
+ if listenport is None:
+ raise IOError, "could not setup remote SocketServer"
+
+ # open another gateway to the freshly installed socket server
+ socketgw = py.execnet.SocketGateway('localhost', listenport)
+ print "initialized socket gateway on port", listenport
+
+ # later you can exit / close down the gateways
+ socketgw.exit()
+ proxygw.exit()
+
+the example is slightly futuristic but apart from the
+channel interface the code for the above functionality
+is there.
+
More information about the pytest-commit
mailing list