[py-dev] xdist sending info between master and slave - support for client code

[holger krekel]

On Sat, Apr 17, 2010 at 13:54 +1000, meme dough wrote:
> > Is it possible to modify xdist so that client code on master can send info
> >
> > > to the slave.  Will need access to rsync roots before it goes ahead and
> > > installs the slave though.
> >
> > I think we could add a hook called at the master-side which could return/update
> > an extra dictionary to be transfered to the client (in theory even a
> > pickled-object but i'd like to avoid guaranteeing pickleability for the core
> > test distribution mechanism).  A slave-side hook (sessionstart?) would
> > get this information.
> 
> I created a new hook that gets called just before each test node is installed:
> 
> def pytest_testnodeinstall(nodemanager, gateway, slavedata):

I think we can just reuse testnodeready and carry a "nodemanager"
and a slavedata attribute on each node.  

Sidenote: at some point i'd like to change the distribution model and thus would
like to hide as many xdist-internals as possible for now. The goal for that
change is to do test collection on the clients and co-ordinate only their
execution and reporting from the master. 
 
> Client code on the master can set anything it needs to send to the
> slave on the slavedata object.
>
> The slavedata is passed over the channel in the same way as config
> object and is stored on the slave node such that any client code on
> the slave can access the slavedata.  
> It can do this in any hook that
> the slave client code needs to implement (provided that it can get
> access to the session object which is the slave node object which has
> the slavedata).

I'd like slavedata to be a simple dictionary so that slave code
can access it by "session.slavedata['NAME']".  Btw, I actually consider
it a bit of a hack that 'session == node' on slaves. But apparently useful :) 

> > > Is it possible to modify xdist so that client code on the slave can send
> > > info back to the master.  This looks like must modify xdist somehow since it
> > > doesn't seem to send any report back at finish.
> >
> > We can add a hook or re-use an existing one for allowing a client to send
> > back (e.g. report) data but the question is: which hook sees this information
> > on the master side?
> >
> > Maybe sessionfinish?  So the slave's sessionfinish hook could add whatever
> > extra information and the final sessionfinish sees it.
> 
> When the slave node is created it sets a slavereport to an empty
> slavereport object.
> 
> Client code on the slave can add to the slavereport in any hook that
> it implements.

only in hooks that know about 'session' or nodes respectively, right? 
I wonder if we could settle on a one-to-one config<->node relationship
on the slave side.  Would make the node nicely available. 

> When the slave finishes it already sends a slavefinished event which
> now also sends back the slavereport object.
> 
> The master TXNode stores the slavereport on itself.
> 
> Client code on the master can implement the pytest_testnodedown hook
> and access the slavereport.

sounds ok enough. 

> > Maybe it's better to design and call specific xdist-prefied hooks and not re-use
> > existing ones?  As you have hacked on this topic and for your purposes
> > already - what do you think?
> 
> xdist hooks seem better than the existing ones.  Also many slaves send
> reports back on pytest_testnodedown hook and seems better to deal with
> just the one immediately rather than build up and pass to
> pytest_sessionfinish.

Can you finally briefly describe your use case - i.e. for what are
you using the new master<->slave data exchange? 

> I implemented and if you think it is good enough would you please be
> so kind as to pull the changes in and release pytest and xdist.  You
> can change of course if you have a better idea!
> 
> http://bitbucket.org/memedough/py-trunk/overview
> 
> http://bitbucket.org/memedough/pytest-xdist/overview

Could you see to implement my above suggestions and see
if it still satisfies your use case? 
(reuse testnodeready, slavedata dict, 
config<->node relationship on slaves)

If i see it right that'd confine changes to the xdist code
in which case i'd be willing to release it soon. 

I'd be happy if you could also add tests for the 
new functionality.

cheers & thanks for the patch and explanations, 
holger

> :)
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 

--