[Pytest-commit] Issue #552: Can't mark.xfail a callable test param (hpk42/pytest)

[Tom V]

New issue 552: Can't mark.xfail a callable test param
https://bitbucket.org/hpk42/pytest/issue/552/cant-markxfail-a-callable-test-param

Tom V:

Within a `pytest.mark.parametrize` decorator, marking xfail on a (non-lambda) callable like `pytest.mark.xfail(func)` doesn't work.

These first 3 examples contain marked xfails for the 2nd param, but still show as failed tests when run:

    # xfail applied, but error still shown

    def func1():
        return 1
    def func2(): 
        return 2

    @pytest.mark.parametrize('func_param',
        [func1, pytest.mark.xfail(func2)])
    def test_xfail_param_func(func_param):
        assert func_param() == 1


    class cls1: 
        val = 1
    class cls2: 
        val = 2

    @pytest.mark.parametrize('cls_param',
        [cls1, pytest.mark.xfail(cls2)])
    def test_xfail_param_cls(cls_param):
        assert cls_param.val == 1


    type_cls1 = type('cls1', (), {'val':1})
    type_cls2 = type('cls2', (), {'val':2})
    @pytest.mark.parametrize('type_cls_param',
        [type_cls1, pytest.mark.xfail(type_cls2)])
    def test_xfail_param_type(type_cls_param):
        assert type_cls_param.val == 1

Output:

    $ py.test test_xfail.py

    ===================================================================== test session starts =====================================================================
    platform linux2 -- Python 2.7.6 -- py-1.4.22 -- pytest-2.6.0
    collected 6 items 

    test_xfail.py .F.F.F

    ========================================================================== FAILURES ===========================================================================
    ________________________________________________________________ test_xfail_param_func[func1] _________________________________________________________________

    func = <function func2 at 0x7ff97e62e0c8>

        @pytest.mark.parametrize('func',
            [func1, pytest.mark.xfail(func2)])
        def test_xfail_param_func(func):
    >       assert func() == 1
    E       assert 2 == 1
    E        +  where 2 = <function func2 at 0x7ff97e62e0c8>()

    test_xfail.py:12: AssertionError
    ______________________________________________________________ test_xfail_param_cls[cls_param1] _______________________________________________________________

    cls_param = <class test_xfail.cls2 at 0x7ff97e613460>

        @pytest.mark.parametrize('cls_param',
            [cls1, pytest.mark.xfail(cls2)])
        def test_xfail_param_cls(cls_param):
    >       assert cls_param.val == 1
    E       assert 2 == 1
    E        +  where 2 = <class test_xfail.cls2 at 0x7ff97e613460>.val

    test_xfail.py:21: AssertionError
    ___________________________________________________________ test_xfail_param_type[type_cls_param1] ____________________________________________________________

    type_cls_param = <class 'test_xfail.cls2'>

        @pytest.mark.parametrize('type_cls_param',
            [type_cls1, pytest.mark.xfail(type_cls2)])
        def test_xfail_param_type(type_cls_param):
    >       assert type_cls_param.val == 1
    E       assert 2 == 1
    E        +  where 2 = <class 'test_xfail.cls2'>.val

    test_xfail.py:29: AssertionError
    ============================================================= 3 failed, 3 passed in 0.02 seconds ==============================================================



And some examples where it behaves as expected, with non-callables (and lambdas):

    # xfail applied, as expected, no error shown

    lambda_func1 = lambda :1
    lambda_func2 = lambda :2
    @pytest.mark.parametrize('lambda_param',
        [lambda_func1, pytest.mark.xfail(lambda_func2)])
    def test_xfail_param_lambda(lambda_param):
        assert lambda_param() == 1


    func_dict = {
        'func1': func1,
        'func2': func2,
    }
    @pytest.mark.parametrize('str_param',
        ['func1', pytest.mark.xfail('func2')])
    def test_xfail_param_str(str_param):
        func = func_dict[str_param]
        assert func() == 1


    @pytest.mark.parametrize('int_param',
        [1, pytest.mark.xfail(2)])
    def test_xfail_param_int(int_param):
        assert int_param == 1


    dict1 = {'key': 1}
    dict2 = {'key': 2}
    @pytest.mark.parametrize('dict_param',
        [dict1, pytest.mark.xfail(dict2)])
    def test_xfail_param_dict(dict_param):
        assert dict_param['key'] == 1

And the output, which is what I expect regardless of the type of the params:

    $ py.test test_xfail.py
    ===================================================================== test session starts =====================================================================
    platform linux2 -- Python 2.7.6 -- py-1.4.22 -- pytest-2.6.0
    collected 6 items 

    test_xfail.py .x.x.x

    ============================================================= 3 passed, 3 xfailed in 0.04 seconds =============================================================