[py-dev] outlining what i intend to do with execnet + what additional features could be helpfull

[holger krekel]

On Mon, Aug 24, 2009 at 15:42 +0200, Ronny Pfannschmidt wrote:
> On Mon, 2009-08-24 at 14:34 +0200, holger krekel wrote:
> > Hi Ronny, 
> > 
> > from IRC i also know that you are wondering how/if to best apply 
> > py.execnet for the tasks.  Can you post an example of
> > a frontend command/API you want to implement and some 
> > code of how that translates to one particular API? 
> > I'd then consider and suggest a way/method to apply execnet.
> 
> i think a good initial example for anyvc would be one of the unittests
> 
> def test_build_first_commit(mgr):
>     repo = mgr.make_repo('repo')
>     with repo.transaction(message='initial', author='test') as root:
>         with root.join('test.txt').open('w') as f:
>             f.write("text")
> 
>     with repo.get_default_head() as root:
>         with root.join("test.txt").open() as f:
>             content = f.read()
>             assert content == 'text'
> 
> this test creates a repo, an initial commit, then asserts the content
> mgr is a utility to manage the directories where tests dump their data
> make_repo creates the repository object that should "be" in another
> object when using py.execnet

The execnet spirit is to design your own local protocol.  Let
me give and give some context by means of an example: py/path/gateway 
contains an experimental remote-path implementation, using execnet: 

* remotepath.py: a "RemotePath" class (similar to your frontend "Repo" one)
* channeltest.py: a PathServer backend implementing the fs access code 

Basically the frontend-class is connected through (sub) channels with 
a backend instance.  If you say on the frontend: 

    x = remotepath.join("hello")

this will enter this code on the caller/client/frontend side: 

    def join(self, *args):
        id = ~COUNTER.next()
        self._channel.send(('JOIN', self._id, id) + args)
        return RemotePath(self._channel, id)

which creates a prospective 'id' for the result, sends a command 
to the backend and immediately returns.  This has *zero* latency. 

If you later perform an operation on the returned instance like list()

    def listdir(self, *args):
        self._channel.send(('LIST', self._id) + args)
        return [RemotePath(self._channel, id, basename)
                for (id, basename) in self._channel.receive()]

this will send a LIST command to the backend PathServer which 
itself implements it like this: 

    def command_LIST(self, id, *args):
        path = self.C2P[id]
        answer = [(self.p2c(p), p.basename) for p in path.listdir(*args)]
        self.channel.send(answer)
  
so here we create a local path using the client-provided ID
and instantiate an answer.  C2P and p2c() help to manage the
ID/path relations.   

For implementing our own small custom "command" pattern 
protocol the backend PAthServer implements a simple dispatch
loop based on the frontend/backend connecting channel: 

    def serve(self):
        try:
            while 1:
                msg = self.channel.receive()
                meth = getattr(self, 'command_' + msg[0])
                meth(*msg[1:])
        except EOFError:
            pass

hope this makes sense to you and serves as inspiration.  To summarize:
Design your own protocol, optimized for your special purposes.
You can easily evolve your protocol on a per-need basis -
no need to have global RPC or IDL/RMI interface definitions.
There also is no need for "matching" server/client software etc.
It's a zero-install "local" protocol, also easy to test. 

best,

holger