
hi all, i just released py.test-1.3.1 [1] which is inlined as svn/pypy/trunk/py and the pypy/test_all.py script is an alias for "py.test". This release particularly refines "expected-to-fail" aka "xfail" semantics: # abort setup or test function, reporting as "expected to fail", or 'x' py.test.xfail() or py.test.xfail(reason) see http://codespeak.net/py/dist/test/plugin/skipping.html for more details. Marking tests as 'xfail' is also good for tests that *sometimes* fail. I just did a grep of "py.test.skip" in pypy/trunk/pypy and there are 468 occurences [2]. Many of these skips seem to be because of implementation issues rather than platform/dependency mismatches and should thus rather use py.test.xfail. Being Skips kind of hides those issues between the rightful skips. The xfail/skip distinction is something that is happening in other parts of the Python world as well and i hope you find it useful as well. best, holger [1] http://codespeak.net/py/dist/announce/release-1.3.1.html [2] the gory list of py.test.skip's in trunk/pypy objspace/std/test/test_proxy_usercreated.py:27: py.test.skip("Impossible to run on appdirect") objspace/std/test/test_longobject.py:27: py.test.skip("XXX broken!") objspace/std/test/test_userobject.py:289: py.test.skip("Cannot run different installers when runappdirect") objspace/std/test/test_smallintobject.py:5: # py.test.skip("WITHSMALLINT is not enabled") objspace/std/test/test_typeobject.py:65: py.test.skip("Not implemented yet") objspace/std/test/test_proxy_internals.py:28: py.test.skip("interp only test") objspace/std/test/test_rangeobject.py:9: py.test.skip("__pypy__.internal_repr() cannot be used to see " objspace/fake/test/test_checkmodule.py:6: py.test.skip('fixme') objspace/flow/test/test_objspace.py:721: py.test.skip("not working") objspace/flow/test/test_objspace.py:728: py.test.skip("not working for now") doc/conftest.py:22: py.test.skip("specify --pypy-doctests to run doctests") config/test/test_makerestdoc.py:11: py.test.skip("don't have docutils") jit/backend/x86/test/test_zrpy_platform.py:15: py.test.skip("tests darwin only stack alignment requirements") jit/backend/x86/test/test_virtualizable.py:8: py.test.skip("Assertion error & llinterp mess") jit/backend/x86/test/test_tlc.py:12: py.test.skip("investigate, maybe") jit/backend/x86/test/test_tlc.py:21: py.test.skip("investigate, maybe") jit/backend/x86/test/conftest.py:7: py.test.skip("x86 directory skipped: cpu is %r" % (cpu,)) jit/backend/x86/test/test_basic.py:38: py.test.skip("issue with ll2ctypes") jit/backend/x86/test/test_basic.py:41: py.test.skip("issue of freeing, probably with ll2ctypes") jit/backend/x86/test/test_ri386_auto_encoding.py:306: py.test.skip("full tests require the GNU 'as' assembler") jit/backend/x86/test/test_ri386_auto_encoding.py:311: py.test.skip("why doesn't 'as' know about CMOVPE/CMOVPO?") jit/backend/x86/test/test_exception.py:11: py.test.skip("Widening to trash") jit/backend/x86/test/test_slist.py:9: py.test.skip("list of voids unsupported by ll2ctypes") jit/backend/llvm/test/conftest.py:4: py.test.skip("llvm backend tests skipped for now") jit/backend/llvm/llvm_rffi.py:7: py.test.skip("Linux only for now") jit/backend/llvm/llvm_rffi.py:27: py.test.skip("llvm (version 2) is required") jit/backend/cli/test/test_runner.py:26: py.test.skip("not supported in non-translated version") jit/backend/cli/test/test_runner.py:37: py.test.skip('fixme! max 32 inputargs so far') jit/backend/cli/test/test_runner.py:43: py.test.skip('fixme!') jit/backend/cli/test/test_runner.py:46: py.test.skip('fixme!') jit/backend/cli/test/conftest.py:4: py.test.skip("CLI backend tests skipped for now") jit/backend/cli/test/test_basic.py:16: py.test.skip("works only after translation") jit/backend/cli/test/test_loop.py:11: py.test.skip("works only after translation") jit/backend/cli/test/test_list.py:11: py.test.skip("works only after translation") jit/backend/cli/test/test_zrpy_send.py:11: py.test.skip('string return values are not supported') jit/backend/cli/test/test_exception.py:11: py.test.skip("works only after translation") jit/backend/cli/test/test_zrpy_basic.py:20: py.test.skip('mono bug?') jit/backend/cli/test/test_zrpy_basic.py:23: py.test.skip('in-progress') jit/backend/cli/test/test_zrpy_basic.py:36: py.test.skip("it seems to fail even with the x86 backend, didn't investigate the problem") jit/backend/cli/test/test_zrpy_slist.py:2: py.test.skip('decide what to do') jit/backend/cli/test/test_zrpy_loop.py:11: py.test.skip('in-progress') jit/backend/test/runner_test.py:931: py.test.skip("requires floats") jit/backend/test/runner_test.py:1012: py.test.skip("requires floats") jit/backend/test/runner_test.py:1825: py.test.skip("implement me") jit/backend/test/runner_test.py:1828: py.test.skip("implement me") jit/backend/test/runner_test.py:1831: py.test.skip("implement me") jit/backend/test/support.py:108: py.test.skip("interp_operations test skipped") jit/backend/test/support.py:119: py.test.skip("use --slow to execute this long-running test") jit/backend/llsupport/test/test_runner.py:12: py.test.skip("llsupport test: cannot compile operations") jit/backend/llgraph/llimpl.py:793: py.test.skip("cond_call_gc_wb not supported") jit/tl/tla/test_tla.py:114: py.test.skip('exercise!') jit/tl/tla/test_tla.py:126: py.test.skip('exercise!') jit/tl/tla/test_tla.py:136: py.test.skip('exercise!') jit/tl/tla/test_tla.py:146: py.test.skip('exercise!') jit/tl/test/test_pypyjit.py:12: py.test.skip("no JIT executable") jit/tl/test/test_tlr.py:15: py.test.skip("?") jit/tl/test/test_tl.py:227: py.test.skip("?") jit/metainterp/test/test_executor.py:254: py.test.skip("requires float support from the backend") jit/metainterp/test/test_send.py:354: py.test.skip("XXX fix me!!!!!!! problem in optimize.py") jit/metainterp/test/test_optimizeopt.py:413: py.test.skip("too bad") jit/metainterp/test/test_optimizeopt.py:1819: py.test.skip("this would fail if we had Fixed again in the specnodes") jit/metainterp/test/test_virtualizable.py:1197: py.test.skip("purely frontend test") jit/metainterp/test/test_tlc.py:46: py.test.skip("buggy interpreter") jit/metainterp/test/test_basic.py:157: py.test.skip("ootype tests skipped for now") jit/metainterp/test/test_basic.py:1442: py.test.skip("test written in a style that " jit/metainterp/test/test_warmspot.py:135: py.test.skip("debug_level is being deprecated") jit/metainterp/test/test_warmspot.py:177: py.test.skip("debug_level is being deprecated") jit/metainterp/test/test_list.py:78: py.test.skip("Constant propagation of length missing") jit/metainterp/test/test_list.py:119: py.test.skip("'[non-null] * n' gives a residual call so far") jit/metainterp/test/test_virtualref.py:40: py.test.skip("purely frontend test") jit/metainterp/test/test_dlist.py:5: py.test.skip("Disabled") jit/metainterp/test/test_dlist.py:142: py.test.skip("I suspect bug somewhere outside of the JIT") jit/metainterp/test/test_slist.py:9: py.test.skip("not yet") jit/metainterp/test/test_del.py:109: py.test.skip("XXX dels are not implemented in the" jit/metainterp/warmspot.py:670: py.test.skip("rewrite_force_virtual: port it to ootype") jit/metainterp/virtualizable.py:23: py.test.skip("ootype: fix virtualizables") rlib/parsing/test/test_pythonparse.py:215: py.test.skip("in progress") rlib/parsing/test/test_deterministic.py:31: py.test.skip("pypy not found on path") rlib/parsing/test/test_deterministic.py:113: #py.test.skip("optimize not working yet") rlib/parsing/test/test_ebnfparse.py:277: #py.test.skip("This needs to be fixed - generated transformer is buggy") rlib/parsing/test/test_ebnfparse.py:443: py.test.skip("fix me somehow") rlib/parsing/test/test_pcre_regtest.py:90: #py.test.skip("Still in progress") rlib/parsing/test/test_regex.py:8: py.test.skip("pypy not found") rlib/rsdl/test/test_video.py:19: py.test.skip("'--view' not specified, " rlib/rsdl/test/test_video.py:54: py.test.skip("interactive test only") rlib/rsdl/test/test_video.py:82: py.test.skip("interactive test only") rlib/rsdl/test/test_video.py:114: py.test.skip("interactive test only") rlib/rsdl/test/test_video.py:148: py.test.skip("interactive test only") rlib/rsdl/test/conftest.py:8: py.test.skip("SDL not installed(?): %s" % (e,)) rlib/test/test_rjvm.py:5: py.test.skip("In Progress...") rlib/test/test_rjvm.py:25: py.test.skip('in progress') rlib/test/test_rgc.py:24: py.test.skip("requires Python 2.5 to call gc.collect() with an arg") rlib/test/test_rgc.py:140: py.test.skip("implement ll_shrink_array for GcStructs or GcArrays that " rlib/test/test_rsocket.py:31: py.test.skip('AF_UNIX not supported.') rlib/test/test_rsocket.py:37: py.test.skip('AF_NETLINK not supported.') rlib/test/test_rsocket.py:108: py.test.skip('No socketpair on Windows') rlib/test/test_rsocket.py:129: py.test.skip('No socketpair on Windows') rlib/test/test_rsocket.py:380: py.test.skip("no inet_pton()") rlib/test/test_rsocket.py:386: py.test.skip("no inet_ntop()") rlib/test/test_rsocket.py:391: py.test.skip('AF_UNIX not supported.') rlib/test/test_libffi.py:203: py.test.skip("Segfaulting test, skip") rlib/test/test_libffi.py:417: py.test.skip("Handle to libc library, Win-only test") rlib/test/test_runicode.py:212: py.test.skip("mbcs encoding is win32-specific") rlib/test/test_runicode.py:233: py.test.skip("Narrow unicode build") rlib/test/test_rope.py:447: py.test.skip("fix me!") rlib/test/test_rope.py:500: py.test.skip("bug in unicode.find that was fixed in 2.5") rlib/test/test_rlocale.py:14: py.test.skip("polish locale unsupported") rlib/test/test_rstackovf.py:28: py.test.skip("not RPython code...") rlib/test/test_rstackovf.py:35: py.test.skip("interpreter is not badly behaved") rlib/test/test_objectmodel.py:136: py.test.skip("xxx no test here") rlib/test/test_rzipfile.py:13: py.test.skip("zlib not installed: %s " % (e, )) tool/pytest/test/test_overview.py:9: py.test.skip("testresult directory not checked out") tool/pytest/htmlreport.py:218: py.test.skip('needs move to own test file') tool/pytest/appsupport.py:252: py.test.skip(msg) tool/pytest/modcheck.py:6: py.test.skip("cannot import %r module" % (name,)) tool/test/test_runsubprocess.py:12: py.test.skip("there is no /bin/echo") tool/test/test_runsubprocess.py:20: py.test.skip("there is no /bin/false") tool/test/test_runsubprocess.py:28: py.test.skip("there is no /bin/cat") tool/bench/test/test_benchmark_report.py:7: py.test.skip(str(e)) tool/package.py:17: if sys.version_info < (2,6): py.test.skip("requires 2.6 so far") rpython/ootypesystem/test/test_ooann.py:290: py.test.skip("Maybe we want this to work") rpython/ootypesystem/test/test_ooclean.py:456: py.test.skip("hash is not preserved during an ootype translation") rpython/ootypesystem/test/test_ootype.py:407: py.test.skip("static types not enforced in ootype") rpython/lltypesystem/test/test_ll2ctypes.py:330: py.test.skip("No gettimeofday on win32") rpython/lltypesystem/test/test_ll2ctypes.py:764: py.test.skip("No fcntl on win32") rpython/lltypesystem/test/test_ll2ctypes.py:1267: py.test.skip("Not supported") rpython/lltypesystem/test/test_llarena.py:217: py.test.skip("cast_adr_to_int() is hard to get consistent") rpython/lltypesystem/test/test_llmemory.py:599: py.test.skip("Fails") rpython/lltypesystem/test/test_lltype.py:212: py.test.skip("test not relevant any more") rpython/lltypesystem/test/test_lltype.py:227: py.test.skip("test not relevant any more") rpython/lltypesystem/test/test_rffi.py:336: py.test.skip("Think how to do it sane") rpython/lltypesystem/test/test_rffi.py:646: py.test.skip('No pipes on windows') rpython/lltypesystem/test/test_rffi.py:703: py.test.skip("Cannot test without ctypes") rpython/lltypesystem/test/test_rffi.py:768: py.test.skip("GenC does not handle char return values correctly") rpython/tool/test/test_rffi_platform.py:13: py.test.skip("this test requires ctypes") rpython/test/test_rdict.py:648: py.test.skip("make dict tests more indepdent from initsize") rpython/test/test_runicode.py:65: py.test.skip("do we want this test to pass?") rpython/test/test_runicode.py:199: py.test.skip("not supported") rpython/test/tool.py:41: py.test.skip("lltypesystem doesn't support %s, yet" % reason) rpython/test/tool.py:43: py.test.skip("ootypesystem doesn't support %s, yet" % reason) rpython/test/test_llann.py:375: py.test.skip("annotation crash") rpython/test/test_rpbc.py:1570: py.test.skip("Should explode or give some warning") rpython/numpy/test/test_array.py:826: py.test.skip('not implemented') rpython/numpy/test/test_array.py:838: py.test.skip('not implemented') rpython/numpy/test/test_array.py:860: py.test.skip("c compilation disabled") rpython/memory/gc/test/test_direct.py:428: py.test.skip("does not support raw_mallocs(sizeof(S)+sizeof(hash))") rpython/memory/gc/markcompact.py:93: import py; py.test.skip("Disabled for now, sorry") rpython/memory/gctransform/test/test_refcounting.py:51: py.test.skip("a probably-illegal test") rpython/memory/test/test_gc.py:644: py.test.skip("Not implemented yet") rpython/memory/test/test_gc.py:715: py.test.skip("Not supported") rpython/memory/test/test_gc.py:764: py.test.skip("Not supported") rpython/memory/test/test_transformed_gc.py:258: py.test.skip("doesn't fit in the model, tested elsewhere too") rpython/memory/test/test_transformed_gc.py:713: py.test.skip("fails for bad reasons in lltype.py :-(") rpython/memory/test/test_transformed_gc.py:848: py.test.skip("this test makes the following test crash. Investigate.") rpython/memory/test/test_transformed_gc.py:1142: py.test.skip("Disabled for now, sorry") rpython/memory/test/test_transformed_gc.py:1444: py.test.skip("not supported") rpython/module/test/test_posix.py:47: import py; py.test.skip("llinterp does not like tuple returns") rpython/module/test/test_posix.py:195: py.test.skip("ootypesystem does not support os.fstat") rpython/module/test/test_posix.py:198: py.test.skip("ootypesystem does not support os.chroot") rpython/module/test/test_posix.py:201: py.test.skip("ootypesystem does not support os.stat") rpython/module/test/test_posix.py:204: py.test.skip("ootypesystem does not support os.chown") rpython/module/test/test_ll_os_stat.py:8: py.test.skip("win32 specific tests") rpython/module/test/test_ll_os.py:40: py.test.skip('Windows specific feature') rpython/module/test/test_ll_os.py:53: py.test.skip('nt specific function') rpython/module/test/test_ll_os.py:83: py.test.skip('posix specific function') rpython/module/test/test_ll_os.py:154: py.test.skip("no ttyname") rpython/module/test/test_ll_termios.py:9: py.test.skip("termios not found") rpython/module/test/test_ll_os_path.py:37: py.test.skip("XXX cannot run os.stat() on the llinterp yet") lib/test2/test_array.py:37: py.test.skip("array.array not picklable before python 2.5") lib/test2/test_array.py:49: py.test.skip("array.array constructor changed in 2.5") lib/test2/test_array.py:78: py.test.skip("requires CPython >= 2.5") lib/test2/test_array.py:86: py.test.skip("no 'u' type code in CPython's struct module") lib/test2/test_array.py:89: py.test.skip("pickle getting confused by the hack in setup_class()") lib/test2/test_grp_extra.py:5: py.test.skip("No grp module on this platform") lib/distributed/test/test_greensock.py:11: py.test.skip("Cannot run this on top of py.py because of PopenGateway") lib/py/_test/pycollect.py:152: py.test.skip("%r is disabled" %(self.obj,)) lib/py/_test/pycollect.py:184: py.test.skip("%r is disabled" %(self.obj,)) lib/py/_test/config.py:221: """ return getvalue() or call py.test.skip if no value exists. """ lib/py/_test/config.py:228: py.test.skip("no %r value found" %(name,)) lib/py/_test/pluginmanager.py:69: py.test.skip("plugin %r is missing" % name) lib/py/_test/pluginmanager.py:140: except py.test.skip.Exception: lib/py/_plugin/pytest_capture.py:266: py.test.skip("capfd funcarg needs os.dup") lib/py/_plugin/pytest_restdoc.py:195: py.test.skip("html file is up to date, use --forcegen to regenerate") lib/py/_plugin/pytest_restdoc.py:309: py.test.skip("%s: %s" %(tryfn, str(e))) lib/py/_plugin/pytest_skipping.py:146: py.test.skip("unsuppored configuration") lib/py/_plugin/pytest_skipping.py:205: py.test.skip(evalskip.getexplanation()) lib/py/_plugin/pytest_skipping.py:210: py.test.skip("xfail") lib/py/_plugin/pytest_pytester.py:274: py.test.skip("no subprocess module") lib/py/_plugin/pytest_pytester.py:358: py.test.skip("need --tools-on-path to run py.test script") lib/py/_plugin/pytest_runner.py:153: elif excinfo.errisinstance(py.test.skip.Exception): lib/py/_plugin/pytest_runner.py:196: if excinfo.errisinstance(py.test.skip.Exception): lib/py/_plugin/pytest_runner.py:398: optionally specified 'minversion' - otherwise call py.test.skip() lib/py/_plugin/pytest_runner.py:405: py.test.skip("could not import %r" %(modname,)) lib/py/_plugin/pytest_runner.py:414: py.test.skip("module %r has __version__ %r, required is: %r" %( lib/py/_plugin/pytest_nose.py:48: # let's substitute the excinfo with a py.test.skip one lib/py/_plugin/pytest_nose.py:49: call2 = call.__class__(lambda: py.test.skip(str(call.excinfo.value)), call.when) lib/py/_plugin/pytest_pdb.py:20: call.excinfo.errisinstance(py.test.skip.Exception): lib/app_test/ctypes_tests/test_win32.py:11: py.test.skip("win32-only tests") lib/app_test/ctypes_tests/test_cast.py:38: py.test.skip("we make copies of strings") lib/app_test/ctypes_tests/test_parameters.py:66: py.test.skip("testing implementation internals") lib/app_test/ctypes_tests/test_parameters.py:86: py.test.skip("testing implementation internals") lib/app_test/ctypes_tests/test_parameters.py:166: py.test.skip("we implement details differently") lib/app_test/ctypes_tests/test_keepalive.py:101: py.test.skip("pypy white-box test") lib/app_test/ctypes_tests/test_struct_fields.py:29: py.test.skip("absent _fields_ semantics not implemented") lib/app_test/ctypes_tests/test_struct_fields.py:36: py.test.skip("subclassing semantics not implemented") lib/app_test/ctypes_tests/test_struct_fields.py:44: py.test.skip("subclassing semantics not implemented") lib/app_test/ctypes_tests/test_keeprefs.py:14: py.test.skip("we make copies of strings") lib/app_test/ctypes_tests/test_keeprefs.py:35: py.test.skip("we make copies of strings") lib/app_test/ctypes_tests/test_callbacks.py:92: py.test.skip("we are less strict about callback return type sanity") lib/app_test/ctypes_tests/test_callbacks.py:140: py.test.skip("callbacks with struct arguments not implemented yet") lib/app_test/ctypes_tests/test_structures.py:164: py.test.skip("custom alignment not supported") lib/app_test/ctypes_tests/test_structures.py:266: py.test.skip("need unicode support on _rawffi level") lib/app_test/ctypes_tests/test_structures.py:287: py.test.skip("not implemented error details") lib/app_test/ctypes_tests/test_structures.py:335: py.test.skip("_abstract_ semantics not implemented") lib/app_test/ctypes_tests/test_structures.py:361: py.test.skip("subclassing semantics not implemented") lib/app_test/ctypes_tests/test_structures.py:482: py.test.skip("mutually dependent lazily defined structures error semantics") lib/app_test/ctypes_tests/test_values.py:35: py.test.skip("tests expect and access cpython dll") lib/app_test/ctypes_tests/test_bitfields.py:8: py.test.skip("bitfields not supported") lib/app_test/ctypes_tests/test_guess_argtypes.py:11: py.test.skip("pypy white-box test") lib/app_test/ctypes_tests/test_guess_argtypes.py:32: py.test.skip("CPython segfaults: see http://bugs.python.org/issue5203") lib/app_test/ctypes_tests/test_repr.py:21: py.test.skip("reprs not implemented") lib/app_test/ctypes_tests/test_repr.py:28: py.test.skip("reprs not implemented") lib/app_test/ctypes_tests/test_funcptr.py:45: py.test.skip("cdecl funcptrs ignoring extra args is not implemented") lib/app_test/ctypes_tests/test_funcptr.py:52: py.test.skip("win32 related") lib/app_test/ctypes_tests/test_funcptr.py:134: py.test.skip("This test needs mmap to make sure the" lib/app_test/ctypes_tests/conftest.py:6: py.test.skip("these tests are meant to be run on top of pypy-c") lib/app_test/ctypes_tests/test_numbers.py:81: py.test.skip("testing implementation internals") lib/app_test/ctypes_tests/test_numbers.py:87: py.test.skip("testing implementation internals") lib/app_test/ctypes_tests/test_init.py:4: py.test.skip("subclassing semantics and implementation details not implemented") lib/app_test/ctypes_tests/test_simplesubclasses.py:29: py.test.skip("subclassing semantics and implementation details not implemented") lib/app_test/ctypes_tests/test_simplesubclasses.py:46: py.test.skip("subclassing semantics and implementation details not implemented") lib/app_test/ctypes_tests/test_varsize_struct.py:7: py.test.skip("resizing not implemented") lib/app_test/ctypes_tests/support.py:15: py.test.skip("white-box tests for pypy _rawffi based ctypes impl") lib/app_test/ctypes_tests/test_functions.py:396: py.test.skip("we are less strict in checking callback parameters") lib/app_test/ctypes_tests/test_stringptr.py:34: py.test.skip("test passes! but modified to avoid getrefcount and detail issues") lib/app_test/ctypes_tests/test_stringptr.py:50: py.test.skip("test passes! but modified to avoid detail issues") lib/app_test/test_marshal_extra.py:76: py.test.skip("this version of CPython doesn't support this object") lib/app_test/test_marshal_extra.py:85: py.test.skip("this version of CPython doesn't support this object") lib/app_test/test_marshal_extra.py:96: py.test.skip("this version of CPython doesn't support this object") lib/app_test/test_marshal_extra.py:111: py.test.skip("this version of CPython doesn't support this object") lib/app_test/test_marshal_extra.py:125: py.test.skip("this version of CPython doesn't support this object") lib/app_test/test_marshal_extra.py:135: py.test.skip("this version of CPython doesn't support this object") lib/app_test/test_locale.py:12: py.test.skip("Locale support on MacOSX is minimal and cannot be tested") lib/app_test/test_locale.py:26: py.test.skip("test locale %s not supported" % cls.tloc) lib/app_test/test_locale.py:32: py.test.skip("XXX fix or kill me") lib/app_test/test_locale.py:52: py.test.skip("XXX fix or kill me") lib/app_test/test_ctypes_support.py:11: py.test.skip("this is expected on top of pypy, we need to fix ctypes in a way that is now in 2.6 in order to make this reliable") lib/app_test/test_pyexpat.py:347: py.test.skip("Not working") lib/app_test/test_defaultdict.py:9: py.test.skip("these tests only run on top of CPython 2.5") lib/app_test/test_pickle_extra.py:6: py.test.skip("test the _pickle_moduledict() addition to pickle.py") lib/app_test/test_dbm_extra.py:6: py.test.skip(e) lib/pyrepl/test/test_functional.py:11: py.test.skip(str(e)) interpreter/test/test_executioncontext.py:250: py.test.skip("test is meant for running with py.test -A") interpreter/test/test_zpy.py:95: py.test.skip("cannot detect process exit code for now") interpreter/test/test_compiler.py:712: ## py.test.skip("unsupported") interpreter/pyparser/test/test_parsestring.py:73: #py.test.skip("crashes in app_codecs, but when cheating using .encode at interp-level passes?!") translator/platform/test/test_distutils.py:10: py.test.skip("Unsupported") translator/platform/test/test_maemo.py:16: py.test.skip("TestMaemo: tests skipped for now") translator/platform/test/test_maemo.py:37: py.test.skip("FIXME") translator/platform/test/test_darwin.py:7: py.test.skip("Darwin only") translator/platform/test/test_darwin.py:52: py.test.skip("i386 only") translator/platform/test/test_darwin.py:76: py.test.skip("i386 only") translator/platform/test/test_darwin.py:94: py.test.skip("i386 only") translator/platform/maemo.py:11: py.test.skip("No scratchbox detected") translator/c/gcc/test/test_thread.py:9: py.test.skip("mingw32 and MSYS are required for asmgcc on Windows") translator/c/gcc/test/conftest.py:7: py.test.skip("x86 directory skipped: cpu is %r" % (cpu,)) translator/c/gcc/test/test_asmgcroot.py:108: py.test.skip("No libffi yet with mingw32") translator/c/gcc/test/test_asmgcroot.py:217: py.test.skip("mingw32 specific test") translator/c/gcc/test/test_asmgcroot.py:220: py.test.skip("mingw32 and MSYS are required for this test") translator/c/gcc/test/test_asmgcroot.py:232: py.test.skip("No libffi yet with mingw32") translator/c/test/test_stackless.py:21: py.test.skip("stackless + refcounting doesn't work any more for now") translator/c/test/test_stackless.py:28: py.test.skip("Boehm GC not present") translator/c/test/test_lltyped.py:253: py.test.skip("we no longer pad our RPython strings with a final NUL") translator/c/test/test_lltyped.py:723: py.test.skip("not easy to test groups too big on 64-bit platforms") translator/c/test/test_newgc.py:1062: py.test.skip("not implemented") translator/c/test/test_newgc.py:1076: py.test.skip("Disabled for now") translator/c/test/test_newgc.py:1079: py.test.skip("not implemented") translator/c/test/test_newgc.py:1082: py.test.skip("not implemented") translator/c/test/test_boehm.py:16: py.test.skip("Boehm GC not present") translator/c/test/test_lladdresses.py:158: py.test.skip("'boehm' may crash") translator/c/test/test_dlltool.py:5: py.test.skip("fix this if needed") translator/c/test/test_standalone.py:108: py.test.skip("instrumentation support is unix only for now") translator/c/test/test_standalone.py:186: py.test.skip("no profopt on win32") translator/c/test/test_standalone.py:486: py.test.skip("not implemented: " translator/c/test/test_standalone.py:567: py.test.skip("implement later, maybe: tracebacks even with ll_assert") translator/c/test/test_standalone.py:608: py.test.skip("TestMaemo: tests skipped for now") translator/c/test/test_standalone.py:617: py.test.skip("Unsupported") translator/c/test/test_standalone.py:620: py.test.skip("Unsupported") translator/c/test/test_extfunc.py:97: py.test.skip("this os has no ftruncate :-(") translator/c/test/test_extfunc.py:115: py.test.skip("this os has no ftruncate :-(") translator/c/test/test_extfunc.py:178: py.test.skip("no WindowsError on this platform") translator/c/test/test_extfunc.py:191: py.test.skip("segfault with tcc :-(") translator/backendopt/test/test_canraise.py:207: py.test.skip("ootype: no explicit stack checks raising RuntimeError") translator/backendopt/test/test_constfold.py:189: py.test.skip("do we want partial folding of getinteriorfield?") translator/backendopt/test/test_mallocv.py:31: py.test.skip(msg) translator/backendopt/test/test_mallocv.py:632: py.test.skip("llptr support not really useful any more") translator/backendopt/test/test_mallocv.py:644: py.test.skip("llptr support not really useful any more") translator/backendopt/test/test_mallocv.py:662: py.test.skip("llptr support not really useful any more") translator/backendopt/test/test_mallocv.py:751: py.test.skip("llptr support not really useful any more") translator/backendopt/test/test_mallocv.py:762: py.test.skip("llptr support not really useful any more") translator/backendopt/test/test_mallocv.py:818: py.test.skip("llptr support not really useful any more") translator/backendopt/test/test_mallocv.py:828: py.test.skip("llptr support not really useful any more") translator/backendopt/test/test_mallocv.py:838: py.test.skip("llptr support not really useful any more") translator/backendopt/test/test_mallocv.py:848: py.test.skip("llptr support not really useful any more") translator/backendopt/test/test_inline.py:60: py.test.skip("ootypesystem doesn't support %s, yet" % reason) translator/backendopt/test/test_malloc.py:19: py.test.skip(msg) translator/backendopt/test/test_malloc.py:220: py.test.skip("fails because of the interior structure changes") translator/backendopt/test/test_malloc.py:233: py.test.skip("fails because of the interior structure changes") translator/backendopt/test/test_malloc.py:287: py.test.skip("fails because of the interior structure changes") translator/backendopt/test/test_malloc.py:317: py.test.skip("fails because of the interior structure changes") translator/backendopt/test/test_malloc.py:372: py.test.skip("fails") translator/backendopt/test/test_malloc.py:436: py.test.skip("oosend prevents malloc removal") translator/backendopt/test/test_malloc.py:439: py.test.skip("oosend prevents malloc removal") translator/jvm/test/test_rarithmetic.py:40: py.test.skip("rpython has no ** on ints") translator/jvm/test/test_rarithmetic.py:45: py.test.skip("divmod not fully supported by rtyper") translator/jvm/test/test_rarithmetic.py:50: # py.test.skip("rpython has no ** on ints") translator/jvm/test/test_rarithmetic.py:52: # py.test.skip("divmod not fully supported by rtyper") translator/jvm/test/test_dict.py:7: py.test.skip("test_resize_during_iteration() doesn't work yet") translator/jvm/test/test_dict.py:10: py.test.skip("fixme: the hashCode method of Records is not good enough") translator/jvm/test/test_dict.py:13: py.test.skip("JVM doesn't support recursive dicts") translator/jvm/test/test_backendopt.py:13: py.test.skip('fixme!') translator/jvm/test/test_backendopt.py:16: py.test.skip('fixme!') translator/jvm/test/test_class.py:7: py.test.skip("JVM doesn't support overridden default value yet") translator/jvm/test/test_class.py:28: py.test.skip('ABSTRACT METHOD FIX: RE-TEST AFTER MERGE') translator/jvm/test/runtest.py:65: py.test.skip("Assembly disabled") translator/jvm/test/runtest.py:68: py.test.skip("Execution disabled") translator/jvm/test/runtest.py:71: # py.test.skip("Compilation disabled") translator/jvm/test/runtest.py:74: # py.test.skip("Execution disabled") translator/jvm/test/runtest.py:81: py.test.skip("Assembly disabled") translator/jvm/test/runtest.py:84: py.test.skip("Execution disabled") translator/jvm/test/runtest.py:122: py.test.skip('Windows --> %s' % reason) translator/jvm/test/runtest.py:126: py.test.skip('PowerPC --> %s' % reason) translator/jvm/test/runtest.py:175: py.test.skip("read_attr not supported on JVM") translator/jvm/test/test_unicode.py:17: py.test.skip("JVM doesn't support unicode for command line arguments") translator/jvm/test/test_unicode.py:24: py.test.skip('fixme!') translator/jvm/test/test_runtest.py:7: py.test.skip('fixme!') translator/jvm/test/test_builtin.py:13: py.test.skip("fails in annotation stage, unrelated to JVM I think") translator/jvm/test/test_builtin.py:16: py.test.skip("fails in annotation stage, unrelated to JVM I think") translator/jvm/test/test_builtin.py:19: py.test.skip("test N/A to jvm: replaced by test_os_dup_oo") translator/jvm/test/test_builtin.py:22: py.test.skip('fixme! how to set environment variables in Java?') translator/jvm/test/test_builtin.py:25: py.test.skip('fixme!') translator/jvm/test/test_builtin.py:28: py.test.skip("so far, debug_llinterpcall is only used on lltypesystem") translator/jvm/test/test_builtin.py:33: py.test.skip('bug in JDK when run headless: ' + translator/jvm/test/test_builtin.py:38: py.test.skip('fixme!') translator/jvm/test/test_string.py:10: py.test.skip("JVM doesn't support unicode for command line arguments") translator/jvm/test/test_string.py:18: py.test.skip("eval has trouble with evaluation of null literals") translator/jvm/test/test_string.py:26: py.test.skip("test fails in JVM specific way") translator/jvm/test/test_float.py:24: py.test.skip("not implemented: single-precision floats") translator/jvm/test/test_list.py:7: py.test.skip("JVM doesn't support recursive lists") translator/jvm/test/test_list.py:10: py.test.skip('fixme!') translator/jvm/test/test_list.py:13: py.test.skip('fixme!') translator/jvm/conftest.py:6: py.test.skip("jvm backend on 64bit unsupported") translator/jvm/genjvm.py:228: py.test.skip("%s is not on your path" % exechelper) translator/cli/test/test_dict.py:7: py.test.skip("CLI doesn't support recursive dicts") translator/cli/test/test_dict.py:10: py.test.skip("CLI doesn't support recursive dicts") translator/cli/test/test_op.py:11: py.test.skip('fixme!') translator/cli/test/runtest.py:220: py.test.skip("Compilation disabled") translator/cli/test/runtest.py:223: py.test.skip("Execution disabled") translator/cli/test/runtest.py:305: py.test.skip('Windows --> %s' % reason) translator/cli/test/runtest.py:309: py.test.skip('PowerPC --> %s' % reason) translator/cli/test/runtest.py:361: py.test.skip('read_attr not supported on gencli tests') translator/cli/test/test_dotnet.py:279: py.test.skip('Mono bug :-(') translator/cli/test/test_dotnet.py:560: py.test.skip("broken test, but this feature is unused so far, so it's not worth fixing it") translator/cli/test/test_dotnet.py:715: py.test.skip(msg) translator/cli/test/test_unicode.py:12: py.test.skip("CLI interpret doesn't support unicode for input arguments") translator/cli/test/test_unicode.py:20: py.test.skip('fixme!') translator/cli/test/test_unicode.py:23: py.test.skip("CLI tests can't have string as input arguments") translator/cli/test/test_builtin.py:8: py.test.skip("CLI doesn't support the os module, yet") translator/cli/test/test_builtin.py:12: py.test.skip("Doesn't work on Windows, yet") translator/cli/test/test_builtin.py:25: py.test.skip("so far, debug_llinterpcall is only used on lltypesystem") translator/cli/test/test_string.py:10: py.test.skip("CLI interpret doesn't support unicode for input arguments") translator/cli/test/test_string.py:18: py.test.skip("CLI doens't support backquotes inside string literals") translator/cli/test/test_string.py:22: py.test.skip("CLI tests can't have string as input arguments") translator/cli/test/test_string.py:27: py.test.skip('fixme!') translator/cli/test/test_float.py:26: py.test.skip("not implemented: single-precision floats") translator/cli/test/test_list.py:8: py.test.skip("CLI doesn't support recursive lists") translator/cli/test/test_list.py:11: py.test.skip('fixme!') translator/cli/test/test_snippet.py:7: py.test.skip('llshl currently broken on CLI') translator/cli/test/test_carbonpython.py:2: py.test.skip("it passes usually, but fails on buildbot, no clue why") translator/cli/test/test_carbonpython.py:111: py.test.skip('This test fails every other day. No clue why :-(') translator/cli/test/test_carbonpython.py:132: py.test.skip('it fails every other day on builbot, no clue why') translator/cli/test/test_carbonpython.py:150: py.test.skip('This test fails every other day. No clue why :-(') translator/cli/test/test_objectmodel.py:7: py.test.skip('r_dict support is still incomplete') translator/cli/sdk.py:8: py.test.skip("%s is not on your path." % helper) translator/cli/support.py:16: py.test.skip('Must use pythonnet for being able to access .NET libraries') translator/tool/test/test_cbuild.py:99: py.test.skip("Need ctypes for that test") translator/tool/test/test_cbuild.py:127: py.test.skip("sdl-config not installed") translator/test/test_exceptiontransform.py:16: py.test.skip("test needs a debug build of Python") translator/sandbox/test/test_sandlib.py:64: py.test.skip("to be updated") translator/sandbox/test/test_sandbox.py:177: py.test.skip("Since this stuff is unimplemented, it won't work anyway " translator/goal/test2/test_targetpypy.py:11: py.test.skip("not working so far") translator/goal/test2/test_app_main.py:66: py.test.skip(str(e)) translator/goal/test2/test_app_main.py:78: py.test.skip( translator/goal/test2/test_app_main.py:270: py.test.skip("this can only work if readline is enabled") translator/goal/test2/test_app_main.py:302: py.test.skip("obscure difference with CPython -- do we care?") translator/oosupport/test/test_treebuilder.py:45: py.test.skip('fixme!') translator/oosupport/test_template/class_.py:68: py.test.skip('cannot return ootype.NULL from translated functions') translator/stackless/test/test_clone.py:7: import py; py.test.skip("to be rewritten with gc_x_clone") translator/stackless/test/test_resume_point.py:254: py.test.skip("please don't write code like this") module/zipimport/test/test_zipimport.py:271: py.test.skip("zlib not available, cannot test compressed zipfiles") module/posix/test/test_posix2.py:37: py.test.skip("no sparse files on default Mac OS X file system") module/posix/test/test_posix2.py:39: py.test.skip("no sparse files on Windows") module/posix/test/test_posix2.py:671: py.test.skip("encoding not good enough") module/posix/test/test_posix2.py:710: py.test.skip("pexpect not found") module/_winreg/test/test_winreg.py:7: py.test.skip("_winreg is a win32 module") module/_locale/test/test_locale.py:38: py.test.skip("necessary locales not installed") module/pypyjit/test/test_pypy_c.py:112: py.test.skip("XXX this is not Windows-friendly") module/pypyjit/test/test_pypy_c.py:489: py.test.skip("exceptions: in-progress") module/pypyjit/test/test_pypy_c.py:511: py.test.skip("exceptions: in-progress") module/pypyjit/test/test_pypy_c.py:621: py.test.skip("meant only for pypy-c") module/pypyjit/test/test_pypy_c.py:632: py.test.skip("pass --pypy!") module/pypyjit/test/test_pypy_c.py:634: py.test.skip("must give a pypy-c with the jit enabled") module/signal/test/test_interp_signal.py:8: py.test.skip("requires os.kill() and os.getpid()") module/signal/test/test_signal.py:8: py.test.skip("requires os.kill() and os.getpid()") module/pyexpat/test/test_build.py:12: py.test.skip("No module expat") module/pyexpat/test/test_build.py:17: py.test.skip("Expat not installed") module/_minimal_curses/test/test_curses.py:13: py.test.skip("Cannot test this here") module/_minimal_curses/test/test_curses.py:34: py.test.skip('pexpect not found') module/_minimal_curses/__init__.py:7: import py; py.test.skip("no _curses module") # no _curses at all module/math/test/test_direct.py:216: py.test.skip("inconsistent behavior before 2.6") module/__pypy__/test/test_special.py:7: py.test.skip("does not make sense on pypy-c") module/clr/test/test_clr.py:9: py.test.skip('Must use pythonnet to access .NET libraries') module/termios/test/test_termios.py:13: py.test.skip("Pexpect not found") module/termios/test/test_termios.py:17: py.test.skip("termios not found") module/readline/test/test_c_readline.py:12: py.test.skip(e) module/readline/test/test_with_pypy.py:13: py.test.skip(e) module/oracle/test/test_connect.py:16: py.test.skip( module/_socket/test/test_sock_app.py:14: mod.skip = py.test.skip module/_socket/test/test_sock_app.py:55: py.test.skip("getservbyname second argument is not optional before python 2.4") module/_socket/test/test_sock_app.py:62: py.test.skip("getservbyport does not exist before python 2.4") module/_socket/test/test_sock_app.py:90: py.test.skip("No socket.fromfd on this platform") module/_socket/test/test_sock_app.py:150: py.test.skip('No socket.inet_pton on this platform') module/_socket/test/test_sock_app.py:166: py.test.skip('No socket.inet_pton on this platform') module/_socket/test/test_sock_app.py:168: py.test.skip("No IPv6 on this platform") module/_socket/test/test_sock_app.py:187: py.test.skip('No socket.inet_pton on this platform') module/_socket/test/test_sock_app.py:189: py.test.skip("No IPv6 on this platform") module/_socket/test/test_sock_app.py:208: py.test.skip("has_ipv6 is always True on PyPy for now") module/_socket/test/test_sock_app.py:219: py.test.skip("Unicode conversion is too slow") module/cpyext/test/test_unicodeobject.py:137: py.test.skip("mcbs encoding only exists on Windows") module/cpyext/test/conftest.py:6: py.test.skip("cannot be run by py.test -A") module/_file/test/test_file.py:172: py.test.skip("likely to deadlock when interpreted by py.py") module/imp/test/test_import.py:772: py.test.skip("unresolved issues with win32 shell quoting rules") module/select/test/test_select.py:217: py.test.skip("select() doesn't work with pipes on win32") module/__builtin__/test/test_classobj.py:774: py.test.skip("can only be run on py.py") module/__builtin__/test/test_classobj.py:794: py.test.skip("can only be run on py.py") module/_stackless/test/test_choicepoint.py:1: import py; py.test.skip("clonable coroutines not really maintained any more") module/_stackless/test/test_clonable.py:1: import py; py.test.skip("clonable coroutines not really maintained any more") module/_stackless/test/test_clonable.py:12: py.test.skip('pure appdirect test (run with -A)') module/_stackless/test/test_clonable.py:18: py.test.skip('no _stackless.clonable') module/_stackless/test/test_interp_clonable.py:4: import py; py.test.skip("clonable coroutines not really maintained any more") module/_stackless/test/test_pickle.py:23: py.test.skip('pure appdirect test (run with -A)') module/zlib/test/test_zlib.py:9: import py; py.test.skip("no zlib module on this host Python") module/zlib/test/test_zlib.py:14: import py; py.test.skip("no zlib C library on this machine") module/_sre/test/test_app_sre.py:291: except py.test.skip.Exception: module/_sre/test/test_app_sre.py:551: except py.test.skip.Exception: annotation/test/test_signature.py:11: py.test.skip("this two annotations should be equal - fix!") annotation/test/test_annrpython.py:2585: py.test.skip("_annenforceargs_ does not work for default arguments") conftest.py:58: py.test.skip(str(e)) conftest.py:67: py.test.skip("cannot runappdirect test: " conftest.py:116: py.test.skip("cannot runappdirect test: " conftest.py:120: py.test.skip("no module __pypy__ on top of CPython") conftest.py:123: py.test.skip("cannot runappdirect this test on top of CPython") conftest.py:127: py.test.skip("cannot runappdirect test: space needs %s = %s, "\ conftest.py:176: py.test.skip("translation test, skipped for appdirect") conftest.py:285: py.test.skip("not running on translated pypy " conftest.py:293: py.test.skip("need translated pypy with: %s, got %s" conftest.py:360: py.test.skip('%s: %s' % (e.__class__.__name__, e)) conftest.py:536: py.test.skip("pexpect not found")

Hi all, On 05/25/2010 10:22 PM, holger krekel wrote:
i just released py.test-1.3.1 [1] which is inlined as svn/pypy/trunk/py and the pypy/test_all.py script is an alias for "py.test". This release particularly refines "expected-to-fail" aka "xfail" semantics:
# abort setup or test function, reporting as "expected to fail", or 'x' py.test.xfail() or py.test.xfail(reason)
see http://codespeak.net/py/dist/test/plugin/skipping.html for more details. Marking tests as 'xfail' is also good for tests that *sometimes* fail.
I just did a grep of "py.test.skip" in pypy/trunk/pypy and there are 468 occurences [2]. Many of these skips seem to be because of implementation issues rather than platform/dependency mismatches and should thus rather use py.test.xfail. Being Skips kind of hides those issues between the rightful skips. The xfail/skip distinction is something that is happening in other parts of the Python world as well and i hope you find it useful as well.
I think another thing is that many of the tests that are now skipped should really be deleted, because they are completely outdated or because it just does not make sense to support them. Cheers, Carl Friedrich

On Thu, May 27, 2010 at 8:34 AM, Carl Friedrich Bolz <cfbolz@gmx.de> wrote:
Hi all,
On 05/25/2010 10:22 PM, holger krekel wrote:
i just released py.test-1.3.1 [1] which is inlined as svn/pypy/trunk/py and the pypy/test_all.py script is an alias for "py.test". This release particularly refines "expected-to-fail" aka "xfail" semantics:
# abort setup or test function, reporting as "expected to fail", or 'x' py.test.xfail() or py.test.xfail(reason)
see http://codespeak.net/py/dist/test/plugin/skipping.html for more details. Marking tests as 'xfail' is also good for tests that *sometimes* fail.
I just did a grep of "py.test.skip" in pypy/trunk/pypy and there are 468 occurences [2]. Many of these skips seem to be because of implementation issues rather than platform/dependency mismatches and should thus rather use py.test.xfail. Being Skips kind of hides those issues between the rightful skips. The xfail/skip distinction is something that is happening in other parts of the Python world as well and i hope you find it useful as well.
I think another thing is that many of the tests that are now skipped should really be deleted, because they are completely outdated or because it just does not make sense to support them.
... but the sheer number of skips means that we don't even want to look at them (which was the xfail part trying to mitigate). Cheers, fijal
participants (3)
-
Carl Friedrich Bolz
-
holger krekel
-
Maciej Fijalkowski