Mailman 3 March 2012 - pytest-dev

[py-dev] pytest-timeout 0.2
by Floris Bruynooghe June 20, 2012

June 20, 2012

Hi all, I've made a second release of the pytest-timeout plugin for py.test which can time out long running tests. This release includes a number of suggestions made on this list, major changes include: * Fixed the activation problem * Set timeout using configuration file * Add a timeout marker to modify timeout of one item * The marker can also choose the method (signal/thread) * Renamed --nosigalrm to --timeout_method to future proof adding of eventlet and gevent timeout methods * Works on python 3, tested on 2.6, 2.7 and 3.2 Not yet done: * Automatic enabling of the plugin, you still need to enable it on the command line or configuration file before you can use the marker. This was probably a bad idea but I felt bad about stealing a marker by default. * eventlet and gevent timeouts As before the release is on pypi: http://pypi.python.org/pypi/pytest-timeout and the development repository and issue tracker on bitbucket: https://bitbucket.org/flub/pytest-timeout/ I'd be pleased to receive any further feedback you may have. Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

3 8

[py-dev] Performance tests with py.test
by Bogdan Opanchuk June 4, 2012

June 4, 2012

Hello, I am trying to wrap performance tests for my module into py.test. Each testcase is, basically, a data preparation stage (can take some time) and the test itself. Is there some way (hooks, plugins) to make py.test print the value returned from the testcase (and make testcases time whatever they want)? What I want is just a table like the one after "py.test -v", but with return values instead of "PASSED". I googled it and haven't found any solutions. At the moment I am doing it as following: 1. Overload "pytest_pyfunc_call()" and save returned result into "pyfuncitem" parameter. 2. Overload "pytest_runtest_makereport()", look for "perf" mark and somehow output "item.result". 3. I haven't decided about the way to output it yet, because I am currently trying to find the place where "PASSED" is printed. But this seems quite clunky and overcomplicated. Is there any better way to do this? If not for the data preparation, the currently existing time measuring system would probably be enough for me (although I'd still miss the ability to print GFLOPS value instead of seconds), but the data preparation is testcase-specific and I do not want to move it to "pytest_generate_tests()" hook. Thank you in advance.

2 4

[py-dev] OpenStack
by Brack, Laurent P. April 1, 2012

April 1, 2012

Forking from the [py-dev] pytest-timeout 0.2 e-mail thread. > I am interested in OpenStack but you can you detail a bit more of what > you want to achieve? I have attached a rough diagram of what we are building internally (hopefully it will not be filtered out). About a year ago, we attended a presentation on OpenStack (when it was still driven by Anso Labs before they got acquired by Rackspace). We were (and are) currently using a private cloud (VMWare) but contemplated the idea of scaling up to public clouds. Problem we had was that we had to develop custom code for each cloud vendor whether Amazon EC2, Penguin, VMWare, etc. so OpenStack was really a viable path to not lock ourselves in with a given vendor. While a bulk of our tests require embedded devices (in which case virtualization makes little sense) a good chunk can be run on standard workstations (all OS flavors) and in this case it only makes sense to move to a virtual environment. PyTest combined with xdist was the dream come true as we could focus on developing tests meant to run on a single machine and later seamlessly parallelize their execution. There is nothing new here. We were thinking of writing a plugin to xdist (cxdist - c for cloud) that would interface with OpenStack, using the pytest_xdist_setupnodes and pytest_configure_node hooks. In those hooks, the plugin would provision the machine (via OpenStack) and then make xdist believe that it is dealing with physical slaves. Finally at the end of the run, we would teardown the slaves, leaving resources for other tests to run. We have not started yet on this as scalability is not an issue but this is our plan. As the diagram shows, the read boxes are plugins we intend to release to the open source community. Our teams write a lot of "data driven tests" (testing various AV codecs with different configuration parameters using very similar verification procedure) and as a result we make heavy use of funcargs. The pytest_testlink plugin is very similar to the pytest Case Conductor plugin from the Mozilla folks (http://blargon7.com/2012/01/case-conductor-pytest-plugin-proposal/) but interacts with TestLink (teamst.org in which we are currently involved in - improving performance of web services etc.) as opposed to Case Conductor. However in addition to linking a test case ID to a given python test, we have implemented a simple mechanism to generate test cases from what we called Test Description Records. Those are currently stored in excel spreadsheets but will be store in TestLink as test case properties. The object injection is done automatically by the plugin using a custom marker indicating which function argument shall be used for TDR injection as well as correlating TDR with parametrized test functions. The point is that by using the funcarg feature offered by pytest, we can generate tremendous amount of test cases with little effort and this will require scalability and use of virtualization over time. Sorry for the long description, but I wanted to give a complete overview to help comprehend where we are going. Our plan is to eventually release all this to the open source community (when we think the plugins are mature and robust enough). If other people have a need/interest for those plugins (some of them are not illustrated), I would be more than happy to accelerate this process (getting approval from our Open Source Board). Since pytest_cxdist doesn't exist, this could be open source from the get go. Best Laurent

2 3

[py-dev] pytest-quickcheck plugin
by Tetsuya Morimoto March 19, 2012

March 19, 2012

Hi, I've made my first pytest plugin named pytest-quickcheck for generating random data. http://pypi.python.org/pypi/pytest-quickcheck/ Originally, this idea is inspired by Haskell QuickCheck (and paycheck, its Python implementation). http://www.haskell.org/haskellwiki/Introduction_to_QuickCheck http://pypi.python.org/pypi/paycheck/ I think it's useful to make sure of the robustness by generating lots of test-data for data-driven test. However, this plugin is still experimental beta, so I'll welcome any comments or suggestions on. thanks, Tetsuya

1 0

[py-dev] New plugin: pytest-timeout
by Floris Bruynooghe March 9, 2012

March 9, 2012

Hi all, This is to announce the first release of pytest-timeout, a plugin which will interrupt long running (i.e. blocking) tests. This is particularly useful when running the tests on a CI host. When a test is interrupted the stacks of all threads are dumped to stderr, which helps you to locate the reason for the blocking. If the system supports SIGALRM then the test itself is interrupted using pytest.fail() and other tests continue to run, otherwise the stack(s) are dumped to stderr and the process exists immediately. The plugin is available from the cheeseshop: http://pypi.python.org/pypi/pytest-timeout and the code lives at bitbucket: https://bitbucket.org/flub/pytest-timeout/. Feel free to provide any feedback or report issues. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

4 12

[py-dev] XML Parsing Error when esc characters are in traceback
by Anton P March 5, 2012

March 5, 2012

Hello all, I have the following issue with generated junitxml reports: I'm writing test cases which use scapy functions. So I make asserts on some packet objects. As I understand scapy uses esc characters for its own objects representation, and I can see colorful output on the console. But when test is failed, and its traceback went to xml, I can't open generated xml due to the error "XML Parsing Error: not well-formed" In my case the problem with esc charecter 033 (\x1b). I resolved the issue by adding the following code to append_failure method in LogXML class: longrepr = str(report.longrepr).replace('\x1b', '\\033') fail.append(longrepr) I think that it will be good if you add some cleanup functions to avoid such situations. Thank you, -Anton

2 2