Mailman 3 October 2004 - pytest-dev

[py-dev] utest thoughts
by Ian Bicking Oct. 29, 2004

Oct. 29, 2004

Here's some things I'd like to do with utest. Maybe some of them are possible now. This is kind of a brainstorming list of features, I guess. * Specify tests to run within a module. The only way to select a module that I see now is by filename. Package name would also be nice. Wildcards could also be useful, e.g., utest modulename.'*transaction*'. I think regular expressions are unnecessarily complex. Maybe a wildcard character other than * would be nice, to keep it from conflicting with shell expansion. A setting, or an optional alternative character? Maybe % (like in SQL). * Data-driven tests, where the same code is tested with many different sets of data. Naturally this is often done in a for loop, but it's better if the data turns into multiple tests, each of which are addressable. There's something called a "unit" in there, I think, that relates to this...? But not the same thing as unittest; I think I saw unittest compatibility code as well. Anyway, with unittest I could provide values to the __init__, creating multiple tests that differed only according to data, but then the runner became fairly useless. I'm hoping that be easier with utest. * Specifying an option to the runner that gets passed through to the tests. It seems like the options are fixed now. I'd like to do something like -Ddatabase=mysql. I can do this with environmental variables now, but that's a little crude. It's easiest if it's just generic, like -D for compilers, but of course it would be nicer if there were specific options. Maybe this could be best achieved by specializing utest and distributing my own runner with the project. * I'm not clear how doctest would fit in. I guess I could turn the doctest into a unit test TestCase, then test that. Of course, it would be nice if this was streamlined. I also have fiddled with doctest to use my own comparison functions when testing if we get the expected output. That's not really in the scope of utest -- that should really go in doctest. Anyway, I thought I'd note its existance. * Code coverage tracking. This should be fairly straight-forward to add. The last time I looked around at test runners, Zope3's seemed the best. Well, it would have been better if I could have gotten it to do something. But it *seemed* best. Mining it for features: * Different levels of tests (-a --at-level or --all; default level is 1, which doesn't run all tests). They have lots of tests, so I'm guessing they like to avoid running tests which are unlikely to fail. * A distinction between unit and functional tests (as in acceptance or system tests). This doesn't seem very generic -- these definitions are very loose and not well agreed upon. There's not even any common language for them. I'm not sure how this fits in with level, but some sort of internal categorization of tests seems useful. * A whole build process. I think they run out of the build/ directory that distutils generates. It is a little unclear how paths work out with utest, depending on where you run it from. Setting PYTHONPATH to include your development code seems the easiest way to resolve these issues with utest. I don't have anything with complicated builds, so maybe there's issues I'm unaware of. * A pychecker option. (-c --pychecker) * A pdb option (-D --debug). I was able to add this to utest with fairly small modifications (at least, if I did it correctly). * An option to control garbage collection (-g --gc-threshold). I guess they encounter GC bugs sometimes. * Run tests in a loop (-L --loop). Also for checking memory leaks. I've thought that running loops of tests in separate threads could also be a useful test, for code that actually was supposed to be used with threads. That might be another place for specializing the runner. * Keep bytecode (-k --keepbytocode). Not interesting in itself, but it implies that they don't normally keep bytecode. I expect this is to deal with code where the .py file has been deleted, but the .pyc file is still around. I've wasted time because of that before, so I can imagine its usefulness. * Profiling (-P --profile). Displays top 50 items, by time and # of calls. * Report only first doctest failure (-1 --report-only-first-doctest-failure). * Time the tests and show the slowest 50 tests (-t --top-fifty). I first thought this was just a bad way of doing profiling, but now that I think about it this is to diagnose problems with the tests runnning slowly. That's all the interesting options, I think. There's also options to select which tests you display, but these seem too complex, while still not all that powerful.

3 6

[py-dev] transparent compilation / greenlets?
by hpk＠trillke.net Oct. 20, 2004

Oct. 20, 2004

hi, i just added a "transparent compilation" section (aka "crossing the barriers of compilation") to the future chapter of the current documentation: http://codespeak.net/py/current/doc/future.html and i wonder, does anyone here have an interesting C-module to offer which would be nice to integrate into the py lib? And what about greenlets anyway? :-) I'd like to experiment a bit with them and we could mark them as experimental if we are not sure we want to continue to have them. Hum, i guess it's pretty clear where they would go: py.magic.greenlet :-) This would be a worthwhile test case for getting nice "transparent compilation". cheers, holger

1 0

[py-dev] Re: [py-svn] r7025 - in py/dist/py: path path/extpy path/local path/svn test
by hpk＠trillke.net Oct. 19, 2004

Oct. 19, 2004

Hi Armin, [arigo(a)codespeak.net Tue, Oct 19, 2004 at 03:40:21PM +0200] > New Revision: 7025 > ... > Log: > Get rid of the imp module to import local modules. Instead: > > FSPathBase.getmodule(): uses a new internal method 'self.getcodeobj()' to get > the module's top-level code object, and then (as before) build a module > object, put it in sys.modules, and run the code in the module. > > FSPathBase.getcodeobj(): uses self.read() and dyncode as before. > > LocalPath.getcodeobj(): uses the regular compile() to make the code object, so > that a local module shows up normally in tracebacks and in the co_filename > attributes. This is also where we look for and create .pyc files. In this > scheme, .pyc files are completely ignored for non-local paths. This is > probably not too bad: with svn paths, say, creating the .pyc files would add > them to the repository. > > Plus a few details, including setting up the new module's __path__ so that > relative imports work (at least on local paths). all nice! But i think we should signal more clearly that these methods refer to *python* files. Or do it slightly differently by, for example, providing a py.magic.View and defining the "getpymodule()" and "getpycode()" on a separated out layer to reduce namespace complexity. I am not completly sure though that this is feasible. At least we should rename the methods to getpycode/getpymodule. cheers, holger

1 0

[py-dev] Checkout problem
by David Warnock Oct. 18, 2004

Oct. 18, 2004

Hi, I like the progress and the new docs. So I tried to checkout to get started. [david@amos ~/dev/python]$ svn co http://codespeak.net/svn/py/dist dist-py svn: REPORT request failed on '/svn/!svn/vcc/default' svn: REPORT of '/svn/!svn/vcc/default': 400 Bad Request (http://codespeak.net) [david@amos ~/dev/python]$ I have not used subversion before and anyway should already be in bed so have given up for tonight. Any hints? Thanks Dave -- David Warnock: Sundayta Ltd. http://www.sundayta.com

2 5

[py-dev] automatic documentation online
by hpk＠trillke.net Oct. 17, 2004

Oct. 17, 2004

Hi, at http://codespeak.net/py/current/doc/ you'll find generated html files from the documentation text files at svn/py/dist/doc. This is automatic so whenever a commit happens it takes a few seconds on codespeak to show the latest. Note also, that if you already put something like eval `python ~/projects/py/py/env.py` into your shell boot script then you now have a little command line utility called 'rest.py' which is able to generate html files from ReST files which you specify on the command line. Without any arguments it does it in the current working dir. It will also link the html-docs with the 'style.css' file in the doc directory so that you can see the full documentation on your local side by pointing your browser to the doc-directory. and feel free to fix typos or add/correct stuff but please check with 'rest.py' in your doc directory that there are no ReST-errors. have fun, holger

1 0

[py-dev] Re: [py-svn] r6948 - in py/dist/py: . path/fspy path/pypath path/test test test/report test/report/text
by Laura Creighton Oct. 15, 2004

Oct. 15, 2004

fspy looks like filesystem py to me. I liked the _name_ pypath better. fspython just looks like more typing to me. I think that this module ought to have widespread use indepent of the pypy project, so we should not saddle it with a second-best name because of possible confusion with better named (if more poorly implemented) thing. But I suspect we can brainstorm something better than pypath as well. I think the word 'root' belongs in there someplace? Other opinions? Laura > >I guess that naming it 'py.path.fspython' instead >of the short ..fspy may look nicer ?! >

3 3

[py-dev] utest docs; questions
by Ian Bicking Oct. 14, 2004

Oct. 14, 2004

Reading through the utest docs, I have a few questions: """ Managing test state across test modules, classes and methods ------------------------------------------------------------ Often you want to create some test files, db-connections or other state in order to run tests in a certain environment. With ``py.test`` there are three scopes for which you can provide hooks to manage such state. These hooks will get called by the driver:: """ Where are these functions put? At the module level? If so, why do they receive a module argument? I guess I can imagine:: from common import setup_module So that the setup_module function can work on multiple modules. Is this the intention? Maybe the example below (corrected if necessary) would make this all more clear. """ Another ``PyCollector`` then recurses into the test files and collects functions and methods that have a leading ``test_``. By default, methods are only collected if their class starts with ``Test``. """ It's not clear here how classes and methods work. Is the class instantiated with no arguments, then all the test_* methods run? E.g.: # in test_math.py class TestMath: def test_plus(self): n = self.num for i in range(self.count-1): n = self.num + n assert n == self.num * self.count def setup_class(cls): self.num = 5 self.count = 3 setup_class = classmethod(setup_class) def setup_method(self, method): # Sigh, I can't think of a good example... pass What happens here? E.g.: import test_math # if setup_module existed, call test_math.setup_module(test_math) TestMatch.setup_class() instance = TestMath() instance.setup_method(instance.test_plus) instance.test_plus() # instance.teardown_method(instance.test_plus) # TestMatch.teardown_class() # teardown_module(test_math) Obviously this can be customized by using different Items, but I'm thinking of the default action. -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

2 1

[py-dev] py-svn mailing list
by hpk＠trillke.net Oct. 3, 2004

Oct. 3, 2004

hi, i've just created a py-svn list which receives all updates to the http://codespeak.net/svn/py/dist tree. Please subscribe here: http://codespeak.net/mailman/listinfo/py-svn cheers, holger

1 0

[py-dev] std rename to py done
by hpk＠trillke.net Oct. 3, 2004

Oct. 3, 2004

hello, i finished most of the rename from 'std' to 'py' yesterday night. At least "py.test" (formerly utest) completes the tree without errors. the new "trunk" of the py lib is at http://codespeak.net/svn/py/dist so i left out 'trunk' because i think it makes the url/svn namespace unneccesarily nested. The above URL currently carries the 'py' package along with an (alas, empty :-) doc directory. Instead of writing long emails i intend to write ReST docs. IMPORTANT, please stop committing anything to 'std' which i will set to readonly in short order, i guess. If you _have_ any changes pending then commit them now, i should be able to automatically port these changes to the py lib. Btw, in case you are interested, I have written my own hack (using the parser and tokenizer) for doing the completely automatic refactoring/rename/moves in http://codespeak.net/svn/user/hpk/rename_import.py which expects the target argument to be the resulting directory of a svn cp path/to/your/WC/of/std path/to/new/py operation. You can repeat the rename_import.py as it does a "revert -R" before performing the rename/refactor. This allows for svn diffing the results of the rename. This rename_import.py script can also be 'utest'ed as it contains inline tests. Obviously, rename_import.py requires 'std' not 'py' for apparent reasons but i plan to port and improve the script, and make it available as an example script (it uses the local file and svn working copy path objects as well as in-lined tests). cheers, holger

1 0

[py-dev] background/future plans
by hpk＠trillke.net Oct. 1, 2004

Oct. 1, 2004

Hi everbody, here is another mail to give and discuss some more background, part of which some people already know about, other's don't. First of all, there is a pending renaming from 'std' to 'py' accompanied by some subsequent small renamings. The rationale is - speaking of the 'std' lib is confusing for python-dev and other pythonistas - the name 'py' is pretty indicative and 'py' does support - like stated at EuroPython 2004 - the notion of beeing a "complementary second standard library". - command line tools provided by 'py' will have a prefix of 'py.*' like e.g. 'py.test' so that <tab>-completion works nicely with 'py.<tab>' both inside bash/readline as well as on the interactive python prompt. (i plan to enhance and integrate rlcompleter2 at some point, see http://codespeak.net/rlcompleter2) I'd like to do the renaming this weekend but am not sure yet how to do it. "Bicycle Repair Man" seems to be completly lacking any documentation and i know of know of no other tool to support such a grand scale renaming except writing my own little script. Also 'utest' is probably to become simply 'py.test' being a python namespace and the name of the central script. std-soon-to-be-py uses a strict mechanism to control the export of its names. In std/__init__.py you'll currently find all the namespaces and names listed which are exposed via the import mechanism. This allows the library to be easily refactorable even across major releases while retaining automatically tested backward compatibility. The idea is that - unless something gets deprecated - such "import APIs" are guaranteed across major releases. If it works in X.Y it will work in X.(Y+1). If it is deprecated in X.Y then it may go away in X.(Y+1). The behaviour and the names _indirectly_ reachable through classes are only guaranteed through the tests. So if you wan't to make sure that a certain behaviour lasts across releases you'd better write a test for it and check it in/submit it. Any untested API part could go away. OK, then on with some short introduction to the current py components - already using the proposed naming scheme: py the 'py lib hook' and root namespace. It's always enough to just "import py" and work your way with e.g. 'py.path.svnurl(...)'. Packages and modules will be lazily imported. All namespaces on the way should be clean and should have no garbage. (do a .path.local the local filesystem class providing an object oriented OO API for filesystem accesses .path.svnurl the remote subversion URL class providing listdir(), update(), remove(), ... methods for accesses to the transactional subversion filesystem .path.svnwc a local working copy API offering most of what path.local offers plus commit(), revert(), add() etc.pp. .test.main() the commandline API for starting tests .test.assert_raises the one method besides the plain 'assert' statement that you usually want from a testing module .test.compat.* some unittest.py compatibility stuff like TestCase .test.* various stuff that shouldn't maybe be exposed :-) .magic.dyncode a module which helps with dynamic code generation allowing for better tracebacks and some more convenience .magic.invoke/revoke the mechanism to invoke magic which usually involves (careful) monkey-patching of system libraries like inspect or the builtin namespace .execnet offering an innovative approach to networked applications by allowing them to execute code in each other's process space. Currently, this works for sockets and pipes/child processes, offering a 'SSHGateway' should be no problem as it is basically the same as the "PopenGateway". The small code in 'execnet' is an extract from a hacking week with Armin on a <inhale>multiuser multiplatform multi-line concurrent pygame-based interactive python shell named shpy</inhale>. (also in the codespeak repo). As you probably have noticed there are all kinds of little open ends and food for thought. More than anything, 'the py lib' should allow for a much faster pace and refactoring than the current python lib. Thus I find it most important to agree on mechanisms for making this possible mainly "export" control and tests-are-the-only-behaviour-guaranties at the moment. Again, the testing mechanism plays a central role. cheers, holger

3 2