[py-dev] OpenStack

[holger krekel]

Hi Brack,

sorry for taking a while.  Am still on travel, currently in the Sonoran
deserts.  My answers up until first half May are thus likely to lag.
(They do have suprisingly good internet connection ... better than in the
three-times more expensive hotel in the San Francisco valley last week).

On Tue, Mar 20, 2012 at 10:27 -0700, Brack, Laurent P. wrote:
> Forking from the [py-dev] pytest-timeout 0.2 e-mail thread.
> 
> > I am interested in OpenStack but you can you detail a bit more of what
> > you want to achieve?
> 
> I have attached a rough diagram of what we are building internally (hopefully it will not be filtered out). 

thanks for sharing.

> About a year ago, we attended a presentation on OpenStack (when it was still driven by Anso Labs before they got acquired by Rackspace). 
> We were (and are) currently using a private cloud (VMWare) but contemplated the idea of scaling up to public clouds. Problem we had was that 
> we had to develop custom code for each cloud vendor whether Amazon EC2, Penguin, VMWare, etc. so OpenStack was really a viable path to not lock 
> ourselves in with a given vendor. 

sounds sensible.

> While a bulk of our tests require embedded devices (in which case virtualization makes little sense) a good chunk can be run 
> on standard workstations (all OS flavors) and in this case it only makes sense to move to a virtual environment. 
> 
> PyTest combined with xdist was the dream come true as we could focus on developing tests meant to run on 
> a single machine and later seamlessly parallelize their execution. There is nothing new here. 
> 
> We were thinking of writing a plugin to xdist (cxdist - c for cloud) that would interface with OpenStack, using the pytest_xdist_setupnodes
> and pytest_configure_node hooks.

ok.

> In those hooks, the plugin would provision the machine (via OpenStack) and then make xdist believe that it is dealing with 
> physical slaves.  

> Finally at the end of the run, we would teardown the slaves, leaving resources for other tests to run. 
> 
> We have not started yet on this as scalability is not an issue but this is our plan. As the diagram shows, the read boxes 
> are plugins we intend to release to the open source community. 

Is your general idea to use the open stack integration to get 
parallelizable test runs with totally controllable environments? 

> Our teams write a lot of "data driven tests" (testing various AV codecs with different configuration parameters using very similar verification procedure)
> and as a result we make heavy use of funcargs. 

Curious, are you using 2.2.3 for this?  There are some plans to further
improve parametrization where i would be interested in your feedback.

> The pytest_testlink plugin is very similar to the pytest Case Conductor plugin from the Mozilla folks (http://blargon7.com/2012/01/case-conductor-pytest-plugin-proposal/) but interacts with TestLink (teamst.org in which we are currently involved in - improving performance of web services etc.) as opposed to Case Conductor. 

Is pytest-testlink a plugin that you plan to work on?

> However in addition to linking a test case ID to a given python test, we have implemented a simple mechanism to generate test cases 
> from what we called Test Description Records. Those are currently stored in excel spreadsheets but will be store in TestLink as 
> test case properties. The object injection is done automatically by the plugin using a custom marker indicating which function argument shall 
> be used for TDR injection as well as correlating TDR with parametrized test functions. 

right.  Driving things from a spreadsheet, maybe even in Google docs, 
sounds good to me, btw :)

> The point is that by using the funcarg feature offered by pytest, we can generate tremendous amount of test cases with little effort
> and this will require scalability and use of virtualization over time. 
> 
> Sorry for the long description, but I wanted to give a complete overview to help comprehend where we are going. 
> Our plan is to eventually release all this to the open source community (when we think the plugins are mature and robust enough). 
> 
> If other people have a need/interest for those plugins (some of them are not illustrated), I would be more than happy to accelerate 
> this process (getting approval from our Open Source Board). Since pytest_cxdist doesn't exist, this could be open source from the 
> get go. 

I wonder if the provisioning of VMs wouldn't better fit at the level of
"tox", see http://tox.testrun.org or even yet a level higher.  tox
creates virtual environments on the Python level.  FYI there are plans to
introduce a plugin system to tox and to make pytest-xdist make use of
tox configuration directly.   

Btw, from May on i (and others) should be available through my company
merlinux for some consulting in case you need some more substantial
support than occassional comments from far away deserts :)

best,
holger