From memedough at gmail.com Sun Dec 5 10:41:26 2010 From: memedough at gmail.com (meme dough) Date: Sun, 5 Dec 2010 20:41:26 +1100 Subject: [py-dev] pytest-capturelog plugin 0.6 Message-ID: Hi, Thanks to Floris the capturelog plugin now has a funcarg that allows adjusting the log levels and provides access to log records and the log text. :) From holger at merlinux.eu Mon Dec 6 14:46:10 2010 From: holger at merlinux.eu (holger krekel) Date: Mon, 6 Dec 2010 14:46:10 +0100 Subject: [py-dev] pytest-pep8 0.6: configurable PEP8 checking in py.test runs Message-ID: <20101206134610.GO1009@trillke.net> just released an initial version 0.6 of the pytest-pep8 plugin, integrating the ``pep8`` module into py.test runs, allowing full per-project customization and configuration. See http://pypi.python.org/pypi/pytest-pep8 for installation and configuration instructions. best, holger Usage ----------------- install pytest-pep8 via:: easy_install pytest-pep8 # or pip install pytest-pep8 and then type:: py.test --pep8 to activate source code checking. Every file ending in ``.py`` will be discovered and checked, starting from the command line arguments. For example, if you have a file like this:: # content of myfile.py somefunc( 123,456) you can run it with:: $ py.test --pep8 =========================== test session starts ============================ platform linux2 -- Python 2.6.5 -- pytest-2.0.1.dev1 pep8 ignore opts: E202 E221 E222 E241 E301 E302 E401 E501 E701 W293 W391 W601 W602 collecting ... collected 1 items myfile.py F ================================= FAILURES ================================= ________________________________ PEP8-check ________________________________ /tmp/doc-exec-12/myfile.py:2:10: E201 whitespace after '(' somefunc( 123,456) ^ /tmp/doc-exec-12/myfile.py:2:14: E231 missing whitespace after ',' somefunc( 123,456) ^ ========================= 1 failed in 0.01 seconds ========================= Note that in the testing header you see the current list of default "ignores". For the meaning of these error and warning codes, see the error output when running against your files or checkout `pep8.py `_. Configuring PEP8 options per-project --------------------------------------------- Lastly, you may configure PEP8-checking options for your project by adding an ``pep8options`` entry to your ``pytest.ini`` or ``setup.cfg`` file like this:: [pytest] pep8options = +W293 -E200 Running PEP8 checks and no other tests --------------------------------------------- You can also restrict your test run to only perform "pep8" tests and not any other tests by typing:: py.test --pep8 -k pep8 This will only run tests that are marked with the "pep8" keyword which is added for the pep8 test items added by this plugin. Notes ------------- The repository of this plugin is at http://bitbucket.org/hpk42/pytest-pep8 For more info on py.test see http://pytest.org The code is partially based on Ronny Pfannschmidt's pytest-codecheckers plugin. From holger at merlinux.eu Mon Dec 6 15:50:16 2010 From: holger at merlinux.eu (holger krekel) Date: Mon, 6 Dec 2010 15:50:16 +0100 Subject: [py-dev] [TIP] pytest-pep8 0.6: configurable PEP8 checking in py.test runs In-Reply-To: References: <20101206134610.GO1009@trillke.net> Message-ID: <20101206145015.GP1009@trillke.net> On Mon, Dec 06, 2010 at 09:00 -0500, Alfredo Deza wrote: > On Mon, Dec 6, 2010 at 8:46 AM, holger krekel wrote: > > > > > just released an initial version 0.6 of the pytest-pep8 plugin, > > integrating the ``pep8`` module into py.test runs, allowing full > > per-project customization and configuration. > > > > > This looks very useful as py.test plugin. > > Is there any way of getting junit output from it? This would help when using > it with Hudson. > > We already use the junit flag to output test results but wanted to add > something like the pep8 plugin > for correctness :) Heh, actually py.test --pep8 --junitxml=mypath.xml should present all PEP8 errors or warnings in the JunitXML file. If not, i'd consider it a bug which you can report or fix with the http://bitbucket.org/hpk42/pytest-pep8 repo. sidenote: i am not sure about the default behaviour - would it maybe make sense to start out with warning/erroring on everything and leaving it to project-specific configuration which warnings and pep8 errors to ignore? cheers, holger > > > > > See http://pypi.python.org/pypi/pytest-pep8 for installation > > and configuration instructions. > > > > best, > > holger > > > > Usage > > ----------------- > > > > install pytest-pep8 via:: > > > > easy_install pytest-pep8 # or > > pip install pytest-pep8 > > > > and then type:: > > > > py.test --pep8 > > > > to activate source code checking. Every file ending in ``.py`` will be > > discovered and checked, starting from the command line arguments. > > For example, if you have a file like this:: > > > > # content of myfile.py > > > > somefunc( 123,456) > > > > you can run it with:: > > > > $ py.test --pep8 > > =========================== test session starts > > ============================ > > platform linux2 -- Python 2.6.5 -- pytest-2.0.1.dev1 > > pep8 ignore opts: E202 E221 E222 E241 E301 E302 E401 E501 E701 W293 W391 > > W601 W602 > > collecting ... collected 1 items > > > > myfile.py F > > > > ================================= FAILURES > > ================================= > > ________________________________ PEP8-check > > ________________________________ > > /tmp/doc-exec-12/myfile.py:2:10: E201 whitespace after '(' > > somefunc( 123,456) > > ^ > > /tmp/doc-exec-12/myfile.py:2:14: E231 missing whitespace after ',' > > somefunc( 123,456) > > ^ > > > > ========================= 1 failed in 0.01 seconds > > ========================= > > > > Note that in the testing header you see the current list of default > > "ignores". > > For the meaning of these error and warning codes, see the error output > > when running against your files or checkout `pep8.py > > `_. > > > > Configuring PEP8 options per-project > > --------------------------------------------- > > > > Lastly, you may configure PEP8-checking options for your project > > by adding an ``pep8options`` entry to your ``pytest.ini`` > > or ``setup.cfg`` file like this:: > > > > [pytest] > > pep8options = +W293 -E200 > > > > > > Running PEP8 checks and no other tests > > --------------------------------------------- > > > > You can also restrict your test run to only perform "pep8" tests > > and not any other tests by typing:: > > > > py.test --pep8 -k pep8 > > > > This will only run tests that are marked with the "pep8" keyword > > which is added for the pep8 test items added by this plugin. > > > > Notes > > ------------- > > > > The repository of this plugin is at http://bitbucket.org/hpk42/pytest-pep8 > > > > For more info on py.test see http://pytest.org > > > > The code is partially based on Ronny Pfannschmidt's pytest-codecheckers > > plugin. > > > > > > _______________________________________________ > > testing-in-python mailing list > > testing-in-python at lists.idyll.org > > http://lists.idyll.org/listinfo/testing-in-python > > -- From holger at merlinux.eu Mon Dec 6 16:42:30 2010 From: holger at merlinux.eu (holger krekel) Date: Mon, 6 Dec 2010 16:42:30 +0100 Subject: [py-dev] pytest-pep 0.7: check everything, determine your own PEP8 ignorance In-Reply-To: References: <20101206134610.GO1009@trillke.net> <20101206145015.GP1009@trillke.net> Message-ID: <20101206154230.GR1009@trillke.net> Hi Alfredo, thanks for the feedback. i released pytest-pep8 0.7 now which has a [pytest] pep8ignore = OPT1 OPT2 ... option so that you can make up project specific list, starting now with a "check everything" setting. I updated the docs as well, see http://pypi.python.org/pypi/pytest-pep8 cheers, holger On Mon, Dec 06, 2010 at 09:55 -0500, Alfredo Deza wrote: > On Mon, Dec 6, 2010 at 9:50 AM, holger krekel wrote: > > > On Mon, Dec 06, 2010 at 09:00 -0500, Alfredo Deza wrote: > > > On Mon, Dec 6, 2010 at 8:46 AM, holger krekel > > wrote: > > > > > > > > > > > just released an initial version 0.6 of the pytest-pep8 plugin, > > > > integrating the ``pep8`` module into py.test runs, allowing full > > > > per-project customization and configuration. > > > > > > > > > > > This looks very useful as py.test plugin. > > > > > > Is there any way of getting junit output from it? This would help when > > using > > > it with Hudson. > > > > > > We already use the junit flag to output test results but wanted to add > > > something like the pep8 plugin > > > for correctness :) > > > > Heh, actually > > > > py.test --pep8 --junitxml=mypath.xml > > > > should present all PEP8 errors or warnings in the JunitXML file. > > If not, i'd consider it a bug which you can report or fix > > with the http://bitbucket.org/hpk42/pytest-pep8 repo. > > > > Ok great, so no "extras" are involved to make it output to the JunitXML > file. Nice. > > > > > sidenote: i am not sure about the default behaviour - would it > > maybe make sense to start out with warning/erroring on everything > > and leaving it to project-specific configuration which warnings > > and pep8 errors to ignore? > > > > I think it is safe to assume that if you are using this plugin you want > everything > "ON" by default, so you can see what things you are interested in and tweak > (turn off) > as you progress. > > > > > cheers, > > holger > > > > > > > > > > > > > > > See http://pypi.python.org/pypi/pytest-pep8 for installation > > > > and configuration instructions. > > > > > > > > best, > > > > holger > > > > > > > > Usage > > > > ----------------- > > > > > > > > install pytest-pep8 via:: > > > > > > > > easy_install pytest-pep8 # or > > > > pip install pytest-pep8 > > > > > > > > and then type:: > > > > > > > > py.test --pep8 > > > > > > > > to activate source code checking. Every file ending in ``.py`` will be > > > > discovered and checked, starting from the command line arguments. > > > > For example, if you have a file like this:: > > > > > > > > # content of myfile.py > > > > > > > > somefunc( 123,456) > > > > > > > > you can run it with:: > > > > > > > > $ py.test --pep8 > > > > =========================== test session starts > > > > ============================ > > > > platform linux2 -- Python 2.6.5 -- pytest-2.0.1.dev1 > > > > pep8 ignore opts: E202 E221 E222 E241 E301 E302 E401 E501 E701 W293 > > W391 > > > > W601 W602 > > > > collecting ... collected 1 items > > > > > > > > myfile.py F > > > > > > > > ================================= FAILURES > > > > ================================= > > > > ________________________________ PEP8-check > > > > ________________________________ > > > > /tmp/doc-exec-12/myfile.py:2:10: E201 whitespace after '(' > > > > somefunc( 123,456) > > > > ^ > > > > /tmp/doc-exec-12/myfile.py:2:14: E231 missing whitespace after ',' > > > > somefunc( 123,456) > > > > ^ > > > > > > > > ========================= 1 failed in 0.01 seconds > > > > ========================= > > > > > > > > Note that in the testing header you see the current list of default > > > > "ignores". > > > > For the meaning of these error and warning codes, see the error output > > > > when running against your files or checkout `pep8.py > > > > `_. > > > > > > > > Configuring PEP8 options per-project > > > > --------------------------------------------- > > > > > > > > Lastly, you may configure PEP8-checking options for your project > > > > by adding an ``pep8options`` entry to your ``pytest.ini`` > > > > or ``setup.cfg`` file like this:: > > > > > > > > [pytest] > > > > pep8options = +W293 -E200 > > > > > > > > > > > > Running PEP8 checks and no other tests > > > > --------------------------------------------- > > > > > > > > You can also restrict your test run to only perform "pep8" tests > > > > and not any other tests by typing:: > > > > > > > > py.test --pep8 -k pep8 > > > > > > > > This will only run tests that are marked with the "pep8" keyword > > > > which is added for the pep8 test items added by this plugin. > > > > > > > > Notes > > > > ------------- > > > > > > > > The repository of this plugin is at > > http://bitbucket.org/hpk42/pytest-pep8 > > > > > > > > For more info on py.test see http://pytest.org > > > > > > > > The code is partially based on Ronny Pfannschmidt's pytest-codecheckers > > > > plugin. > > > > > > > > > > > > _______________________________________________ > > > > testing-in-python mailing list > > > > testing-in-python at lists.idyll.org > > > > http://lists.idyll.org/listinfo/testing-in-python > > > > > > > > -- > > -- From andy.fundinger at riskfocusinc.com Mon Dec 6 22:20:35 2010 From: andy.fundinger at riskfocusinc.com (Andy Fundinger) Date: Mon, 6 Dec 2010 16:20:35 -0500 (EST) Subject: [py-dev] Upgrading from pytest 1.3.2 to 2.0 - computed conftest options Message-ID: <752301.198.1291670435078.JavaMail.root@ip-208-109-234-141.ip.secureserver.net> I'm trying to upgrade from py.test 1.3.2 to 2.0 but I'm running into issues with my use of the old style conftest.py feature to specify options. I specify options like: option_cov_branch = True option_cov_data_file = os.path.join(os.environ["PROJECT_ROOT"],"testCovFile%d.log"%random.randint(0,100000)) option_tx = ["popen//python=python",]*multiprocessing.cpu_count() the computed ones are a problem as the new .ini files can't take them and the conftest.py file specification seems to be ignored. Is there a particular hook that I can grab and add/change config values early enough to affect all plugins? Regards, Andy Fundinger Risk Focus Inc. 424.243.6227 New York Office: +1 917 725 6006 Fax: +1 917 591 1616 London Office: +44 (0) 207 760 7184 Fax: +44 (0) 207 691 7165 -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Tue Dec 7 12:31:27 2010 From: holger at merlinux.eu (holger krekel) Date: Tue, 7 Dec 2010 12:31:27 +0100 Subject: [py-dev] Upgrading from pytest 1.3.2 to 2.0 - computed conftest options In-Reply-To: <752301.198.1291670435078.JavaMail.root@ip-208-109-234-141.ip.secureserver.net> References: <752301.198.1291670435078.JavaMail.root@ip-208-109-234-141.ip.secureserver.net> Message-ID: <20101207113127.GU1009@trillke.net> Hi Andy, On Mon, Dec 06, 2010 at 16:20 -0500, Andy Fundinger wrote: > I'm trying to upgrade from py.test 1.3.2 to 2.0 but I'm running into > issues with my use of the old style conftest.py feature to specify > options. I specify options like: > > > option_cov_branch = True > > option_cov_data_file = os.path.join(os.environ["PROJECT_ROOT"],"testCovFile%d.log"%random.randint(0,100000)) > > option_tx = ["popen//python=python",]*multiprocessing.cpu_count() > > > the computed ones are a problem as the new .ini files can't take them > and the conftest.py file specification seems to be ignored. Is there a > particular hook that I can grab and add/change config values early > enough to affect all plugins? Indeed a feature regression. I just added a new pytest_cmdline_processargs hook to accomodate the situation. The basic idea is to add command line arguments before option parsing takes place. It is only called on the master. The slaves do not call this hook. It's not released yet but you can see an example in the upcoming docs here: http://pytest.org/2.0.1dev/example/simple.html#dynamically-adding-command-line-options and install the the development version with:: pip install -i http://pypi.testrun.org -U pytest Let me know of any issues. A note already though: i think the pytest-cov plugin does not provide the options you use above. Instead it uses/allows to point to a .coveragerc file. I believe it also cares for coverage data filenames itself. cheers, holger > > > > Regards, > > Andy Fundinger > > Risk Focus Inc. > 424.243.6227 > > > New York Office: +1 917 725 6006 Fax: +1 917 591 1616 > > London Office: +44 (0) 207 760 7184 Fax: +44 (0) 207 691 7165 > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev -- From holger at merlinux.eu Tue Dec 7 12:39:24 2010 From: holger at merlinux.eu (holger krekel) Date: Tue, 7 Dec 2010 12:39:24 +0100 Subject: [py-dev] Upgrading from pytest 1.3.2 to 2.0 - computed conftest options In-Reply-To: <20101207113127.GU1009@trillke.net> References: <752301.198.1291670435078.JavaMail.root@ip-208-109-234-141.ip.secureserver.net> <20101207113127.GU1009@trillke.net> Message-ID: <20101207113924.GV1009@trillke.net> On Tue, Dec 07, 2010 at 12:31 +0100, holger krekel wrote: > I just added a new pytest_cmdline_processargs > hook to accomodate the situation. Actually it's called pytest_cmdline_preparse. And btw, you might want to also upgrade pytest-xdist via pip install -U pytest-xdist -i http://pypi.testrun.org (which will drag along pytest-2.0.1.dev5 i guess) H. From andy.fundinger at riskfocusinc.com Fri Dec 10 15:17:17 2010 From: andy.fundinger at riskfocusinc.com (Andy Fundinger) Date: Fri, 10 Dec 2010 09:17:17 -0500 (EST) Subject: [py-dev] Internal Error in xdist remove Message-ID: <10088290.959.1291990637762.JavaMail.root@ip-208-109-234-141.ip.secureserver.net> Continuing my attempted upgrade to pytest 2.0.0 I'm running into some errors with using xdist for parallel testing, this test is meant to run on two local nodes in parallel and boxed, but somehow it's trying to remove something that isn't there. Any ideas? The full log is below with a few bits removed because they reference my source code, if they're critical I can obscure the data rather than removing them or try to reproduce with non-project code. Andy Fundinger scheduling tests via LoadScheduling INTERNALERROR> config = <_pytest.config.Config object at 0xff22a90> INTERNALERROR> INTERNALERROR> def pytest_cmdline_main(config): INTERNALERROR> """ default command line protocol for initialization, session, INTERNALERROR> running tests and reporting. """ INTERNALERROR> session = Session(config) INTERNALERROR> session.exitstatus = EXIT_OK INTERNALERROR> try: INTERNALERROR> config.pluginmanager.do_configure(config) INTERNALERROR> config.hook.pytest_sessionstart(session=session) INTERNALERROR> config.hook.pytest_collection(session=session) INTERNALERROR> > config.hook.pytest_runtestloop(session=session) INTERNALERROR> INTERNALERROR> config = <_pytest.config.Config object at 0xff22a90> INTERNALERROR> session = INTERNALERROR> INTERNALERROR> /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest-2.0.1.dev5-py2.6.egg/_pytest/main.py:65: INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ INTERNALERROR> INTERNALERROR> self = INTERNALERROR> INTERNALERROR> def __call__(self, **kwargs): INTERNALERROR> methods = self.hookrelay._pm.listattr(self.name) INTERNALERROR> > return self._docall(methods, kwargs) INTERNALERROR> INTERNALERROR> kwargs = {'session': } INTERNALERROR> methods = [, >] INTERNALERROR> self = INTERNALERROR> INTERNALERROR> /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest-2.0.1.dev5-py2.6.egg/_pytest/core.py:390: INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ INTERNALERROR> INTERNALERROR> self = INTERNALERROR> methods = [, >] INTERNALERROR> kwargs = {'session': } INTERNALERROR> INTERNALERROR> def _docall(self, methods, kwargs): INTERNALERROR> self.trace(self.name, kwargs) INTERNALERROR> self.trace.root.indent += 1 INTERNALERROR> mc = MultiCall(methods, kwargs, firstresult=self.firstresult) INTERNALERROR> try: INTERNALERROR> > res = mc.execute() INTERNALERROR> INTERNALERROR> kwargs = {'session': } INTERNALERROR> mc = }> INTERNALERROR> methods = [, >] INTERNALERROR> self = INTERNALERROR> INTERNALERROR> /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest-2.0.1.dev5-py2.6.egg/_pytest/core.py:401: INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ INTERNALERROR> INTERNALERROR> self = }> INTERNALERROR> INTERNALERROR> def execute(self): INTERNALERROR> while self.methods: INTERNALERROR> method = self.methods.pop() INTERNALERROR> kwargs = self.getkwargs(method) INTERNALERROR> > res = method(**kwargs) INTERNALERROR> INTERNALERROR> kwargs = {} INTERNALERROR> method = > INTERNALERROR> self = }> INTERNALERROR> INTERNALERROR> /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest-2.0.1.dev5-py2.6.egg/_pytest/core.py:322: INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ INTERNALERROR> INTERNALERROR> self = INTERNALERROR> INTERNALERROR> def pytest_runtestloop(self): INTERNALERROR> numnodes = len(self.nodemanager.specs) INTERNALERROR> dist = self.config.getvalue("dist") INTERNALERROR> if dist == "load": INTERNALERROR> self.sched = LoadScheduling(numnodes, log=self.log) INTERNALERROR> elif dist == "each": INTERNALERROR> self.sched = EachScheduling(numnodes, log=self.log) INTERNALERROR> else: INTERNALERROR> assert 0, dist INTERNALERROR> self.shouldstop = False INTERNALERROR> self.session_finished = False INTERNALERROR> while not self.session_finished: INTERNALERROR> > self.loop_once() INTERNALERROR> INTERNALERROR> dist = 'load' INTERNALERROR> numnodes = 2 INTERNALERROR> self = INTERNALERROR> INTERNALERROR> /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest_xdist-1.6.dev2-py2.6.egg/xdist/dsession.py:204: INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ INTERNALERROR> INTERNALERROR> self = INTERNALERROR> INTERNALERROR> def loop_once(self): INTERNALERROR> """ process one callback from one of the slaves. """ INTERNALERROR> while 1: INTERNALERROR> try: INTERNALERROR> eventcall = self.queue.get(timeout=2.0) INTERNALERROR> break INTERNALERROR> except queue.Empty: INTERNALERROR> continue INTERNALERROR> callname, kwargs = eventcall INTERNALERROR> assert callname, kwargs INTERNALERROR> method = "slave_" + callname INTERNALERROR> call = getattr(self, method) INTERNALERROR> self.log("calling method: %s(**%s)" % (method, kwargs)) INTERNALERROR> > call(**kwargs) INTERNALERROR> INTERNALERROR> call = > INTERNALERROR> callname = 'testreport' INTERNALERROR> eventcall = ('testreport', {'node': , 'rep': {snip} when='call' outcome='passed'>}) INTERNALERROR> kwargs = {'node': , 'rep': {snip} when='call' outcome='passed'>} INTERNALERROR> method = 'slave_testreport' INTERNALERROR> self = INTERNALERROR> INTERNALERROR> /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest_xdist-1.6.dev2-py2.6.egg/xdist/dsession.py:222: INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ INTERNALERROR> INTERNALERROR> self = , node = INTERNALERROR> rep = INTERNALERROR> INTERNALERROR> def slave_testreport(self, node, rep): INTERNALERROR> if rep.when in ("setup", "call"): INTERNALERROR> > self.sched.remove_item(node, rep.nodeid) INTERNALERROR> INTERNALERROR> node = INTERNALERROR> rep = INTERNALERROR> self = INTERNALERROR> INTERNALERROR> /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest_xdist-1.6.dev2-py2.6.egg/xdist/dsession.py:283: INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ INTERNALERROR> INTERNALERROR> self = , node = {item removed} INTERNALERROR> INTERNALERROR> def remove_item(self, node, item): INTERNALERROR> if item not in self.item2nodes: INTERNALERROR> raise AssertiDN_XSS_NEUTRALIZE_onError(item, self.item2nodes) INTERNALERROR> nodes = self.item2nodes[item] INTERNALERROR> if node in nodes: # the node might have gone down already INTERNALERROR> nodes.remove(node) INTERNALERROR> #if not nodes: INTERNALERROR> # del self.item2nodes[item] INTERNALERROR> pending = self.node2pending[node] INTERNALERROR> > pending.remove(item) INTERNALERROR> E ValueError: list.remove(x): x not in list INTERNALERROR> {item and pending removed} INTERNALERROR> node = INTERNALERROR> nodes = [] INTERNALERROR> self = INTERNALERROR> INTERNALERROR> /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest_xdist-1.6.dev2-py2.6.egg/xdist/dsession.py:103: ValueError Regards, Andy Fundinger Risk Focus Inc. 424.243.6227 New York Office: +1 917 725 6006 Fax: +1 917 591 1616 London Office: +44 (0) 207 760 7184 Fax: +44 (0) 207 691 7165 -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Fri Dec 10 16:44:18 2010 From: holger at merlinux.eu (holger krekel) Date: Fri, 10 Dec 2010 16:44:18 +0100 Subject: [py-dev] Internal Error in xdist remove In-Reply-To: <10088290.959.1291990637762.JavaMail.root@ip-208-109-234-141.ip.secureserver.net> References: <10088290.959.1291990637762.JavaMail.root@ip-208-109-234-141.ip.secureserver.net> Message-ID: <20101210154418.GK13842@trillke.net> Hi Andy, Probably there is some error that his hidden behind this weird traceback. Can't look into it at the moment. Can you come with a repeatable example and maybe file an issue? In any case i am off for three weeks so don't expect an answer from my side, maybe somebody else can help you, though. cheers, holger On Fri, Dec 10, 2010 at 09:17 -0500, Andy Fundinger wrote: > Continuing my attempted upgrade to pytest 2.0.0 I'm running into some errors with using xdist for parallel testing, this test is meant to run on two local nodes in parallel and boxed, but somehow it's trying to remove something that isn't there. Any ideas? The full log is below with a few bits removed because they reference my source code, if they're critical I can obscure the data rather than removing them or try to reproduce with non-project code. > > > Andy Fundinger > > > > scheduling tests via LoadScheduling > INTERNALERROR> config = <_pytest.config.Config object > at 0xff22a90> > INTERNALERROR> > INTERNALERROR> def > pytest_cmdline_main(config): > INTERNALERROR> > """ default command line protocol for initialization, session, > INTERNALERROR> > running tests and reporting. """ > INTERNALERROR> > session = Session(config) > INTERNALERROR> > session.exitstatus = EXIT_OK > INTERNALERROR> > try: > INTERNALERROR> > config.pluginmanager.do_configure(config) > INTERNALERROR> > config.hook.pytest_sessionstart(session=session) > INTERNALERROR> > config.hook.pytest_collection(session=session) > INTERNALERROR> > config.hook.pytest_runtestloop(session=session) > INTERNALERROR> > INTERNALERROR> config = > <_pytest.config.Config object at 0xff22a90> > INTERNALERROR> session = 'gauss_tools'> > INTERNALERROR> > INTERNALERROR> > /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest-2.0.1.dev5-py2.6.egg/_pytest/main.py:65: > INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > INTERNALERROR> > INTERNALERROR> self = 'pytest_runtestloop'> > INTERNALERROR> > INTERNALERROR> def __call__(self, > **kwargs): > INTERNALERROR> > methods = self.hookrelay._pm.listattr(self.name) > INTERNALERROR> > > return self._docall(methods, kwargs) > INTERNALERROR> > INTERNALERROR> kwargs = > {'session': } > INTERNALERROR> methods = [ pytest_runtestloop at 0x1001b758>, DSession.pytest_runtestloop of 0x180b5368>>] > INTERNALERROR> self = > > INTERNALERROR> > INTERNALERROR> > /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest-2.0.1.dev5-py2.6.egg/_pytest/core.py:390: > INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > INTERNALERROR> > INTERNALERROR> self = 'pytest_runtestloop'> > INTERNALERROR> methods = [ at 0x1001b758>, >] > INTERNALERROR> kwargs = {'session': 'gauss_tools'>} > INTERNALERROR> > INTERNALERROR> def _docall(self, > methods, kwargs): > INTERNALERROR> > self.trace(self.name, kwargs) > INTERNALERROR> > self.trace.root.indent += 1 > INTERNALERROR> > mc = MultiCall(methods, kwargs, firstresult=self.firstresult) > INTERNALERROR> > try: > INTERNALERROR> > > res = > mc.execute() > INTERNALERROR> > INTERNALERROR> kwargs = > {'session': } > INTERNALERROR> > mc = meths, kwargs={'session': }> > INTERNALERROR> methods = [ pytest_runtestloop at 0x1001b758>, DSession.pytest_runtestloop of 0x180b5368>>] > INTERNALERROR> self = > > INTERNALERROR> > INTERNALERROR> > /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest-2.0.1.dev5-py2.6.egg/_pytest/core.py:401: > INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > INTERNALERROR> > INTERNALERROR> self = kwargs={'session': }> > INTERNALERROR> > INTERNALERROR> def execute(self): > INTERNALERROR> > while self.methods: > INTERNALERROR> > method = self.methods.pop() > INTERNALERROR> > kwargs = self.getkwargs(method) > INTERNALERROR> > > res = > method(**kwargs) > INTERNALERROR> > INTERNALERROR> kwargs = {} > INTERNALERROR> method = method DSession.pytest_runtestloop of 0x180b5368>> > INTERNALERROR> self = > 'gauss_tools'>}> > INTERNALERROR> > INTERNALERROR> > /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest-2.0.1.dev5-py2.6.egg/_pytest/core.py:322: > INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > INTERNALERROR> > INTERNALERROR> self = instance at 0x180b5368> > INTERNALERROR> > INTERNALERROR> def > pytest_runtestloop(self): > INTERNALERROR> > numnodes = len(self.nodemanager.specs) > INTERNALERROR> > dist = self.config.getvalue("dist") > INTERNALERROR> > if dist == "load": > INTERNALERROR> > self.sched = LoadScheduling(numnodes, log=self.log) > INTERNALERROR> > elif dist == "each": > INTERNALERROR> > self.sched = EachScheduling(numnodes, log=self.log) > INTERNALERROR> > else: > INTERNALERROR> > assert 0, dist > INTERNALERROR> > self.shouldstop = False > INTERNALERROR> > self.session_finished = False > INTERNALERROR> > while not self.session_finished: > INTERNALERROR> > > > self.loop_once() > INTERNALERROR> > INTERNALERROR> dist = > 'load' > INTERNALERROR> numnodes = 2 > INTERNALERROR> self = > > INTERNALERROR> > INTERNALERROR> > /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest_xdist-1.6.dev2-py2.6.egg/xdist/dsession.py:204: > INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > INTERNALERROR> > INTERNALERROR> self = instance at 0x180b5368> > INTERNALERROR> > INTERNALERROR> def > loop_once(self): > INTERNALERROR> > """ process one callback from one of the slaves. > """ > INTERNALERROR> while > 1: > INTERNALERROR> > try: > INTERNALERROR> > eventcall = self.queue.get(timeout=2.0) > INTERNALERROR> > break > INTERNALERROR> > except queue.Empty: > INTERNALERROR> > continue > INTERNALERROR> callname, > kwargs = eventcall > INTERNALERROR> > assert callname, kwargs > INTERNALERROR> > method = "slave_" + callname > INTERNALERROR> > call = getattr(self, method) > INTERNALERROR> > self.log("calling method: %s(**%s)" % (method, kwargs)) > INTERNALERROR> > > call(**kwargs) > INTERNALERROR> > INTERNALERROR> call = > instance at 0x180b5368>> > INTERNALERROR> callname = 'testreport' > INTERNALERROR> eventcall = ('testreport', {'node': > , 'rep': {snip} > > when='call' outcome='passed'>}) > INTERNALERROR> kwargs = {'node': > , 'rep': {snip} > when='call' outcome='passed'>} > INTERNALERROR> method = > 'slave_testreport' > INTERNALERROR> self = > > INTERNALERROR> > INTERNALERROR> > /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest_xdist-1.6.dev2-py2.6.egg/xdist/dsession.py:222: > INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > INTERNALERROR> > INTERNALERROR> self = instance at 0x180b5368>, node = > INTERNALERROR> rep = when='call' outcome='passed'> > INTERNALERROR> > INTERNALERROR> def > slave_testreport(self, node, rep): > INTERNALERROR> if > rep.when in ("setup", "call"): > INTERNALERROR> > > > self.sched.remove_item(node, rep.nodeid) > INTERNALERROR> > INTERNALERROR> node = > > INTERNALERROR> > rep = when='call' outcome='passed'> > INTERNALERROR> self = > > INTERNALERROR> > INTERNALERROR> > /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest_xdist-1.6.dev2-py2.6.egg/xdist/dsession.py:283: > INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > INTERNALERROR> > INTERNALERROR> self = instance at 0x1930f368>, node = > > > {item removed} > > INTERNALERROR> > INTERNALERROR> def > remove_item(self, node, item): > INTERNALERROR> > if item not in self.item2nodes: > INTERNALERROR> > raise AssertiDN_XSS_NEUTRALIZE_onError(item, self.item2nodes) > INTERNALERROR> > nodes = self.item2nodes[item] > INTERNALERROR> > if node in nodes: # the node might have gone down already > INTERNALERROR> > nodes.remove(node) > INTERNALERROR> > #if not nodes: > INTERNALERROR> > # del self.item2nodes[item] > INTERNALERROR> > pending = self.node2pending[node] > INTERNALERROR> > > pending.remove(item) > INTERNALERROR> E > ValueError: list.remove(x): x not in list > INTERNALERROR> > {item and pending removed} > > INTERNALERROR> node = > > > INTERNALERROR> nodes = [] > INTERNALERROR> self = > > > INTERNALERROR> > > > INTERNALERROR> > /home/andy/trunk/ext_linux64/python264/lib/python2.6/site-packages/pytest_xdist-1.6.dev2-py2.6.egg/xdist/dsession.py:103: > ValueError > > > Regards, > > Andy Fundinger > > Risk Focus Inc. > 424.243.6227 > > > New York Office: +1 917 725 6006 Fax: +1 917 591 1616 > > London Office: +44 (0) 207 760 7184 Fax: +44 (0) 207 691 7165 > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev -- From hsoft at hardcoded.net Sat Dec 18 15:51:26 2010 From: hsoft at hardcoded.net (Virgil Dupras) Date: Sat, 18 Dec 2010 15:51:26 +0100 Subject: [py-dev] Advanced monkeypatching? Message-ID: <4C5E2E4C-15CF-45B7-80B1-3F0363D82E66@hardcoded.net> Hi there, Monkeypatching is great and all, but some type of patching is a little bit more complex to do, such as patching the system time and patching os.stat for specific filenames. I have my own "homebrewed" solution for this, but I was wondering if such solution had their place in the monkeypatch module of pytest itself. Does it? Or should a "pytest-monkeypatch++" package be created? The "homebrewed" code I'm talking about is at https://bitbucket.org/hsoft/hsutil/src/43014954c6a8/hsutil/testutil.py#cl-83 ( patch_osstat() and patch_today() ). -- Virgil Dupras From flub at devork.be Sun Dec 19 15:38:01 2010 From: flub at devork.be (Floris Bruynooghe) Date: Sun, 19 Dec 2010 14:38:01 +0000 Subject: [py-dev] Advanced monkeypatching? In-Reply-To: <4C5E2E4C-15CF-45B7-80B1-3F0363D82E66@hardcoded.net> References: <4C5E2E4C-15CF-45B7-80B1-3F0363D82E66@hardcoded.net> Message-ID: Hello On 18 December 2010 14:51, Virgil Dupras wrote: > Hi there, > > Monkeypatching is great and all, but some type of patching is a little bit more complex to do, such as patching the system time and patching os.stat for specific filenames. I have my own "homebrewed" solution for this, but I was wondering if such solution had their place in the monkeypatch module of pytest itself. Does it? Or should a "pytest-monkeypatch++" package be created? > > The "homebrewed" code I'm talking about is at https://bitbucket.org/hsoft/hsutil/src/43014954c6a8/hsutil/testutil.py#cl-83 ( patch_osstat() and patch_today() ). Looking at the patch_osstat() it seems to me that it just fakes os.stat() for one specific path but uses the original in place otherwise. This sounds like it could be just a specialised object used to patch with. I can imagine the monkeypatch plugin to provide a funcarg which provides this functionality, e.g.: def test_foo(monkeypatch, monkey_osstat): monkeypatch.setattr(monkey_ossstat('/tmp/some_file')) ... Though probably with better names ;-). I'm not sure I understand the use case for patch_today() so won't comment on that part. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From hsoft at hardcoded.net Mon Dec 20 09:22:27 2010 From: hsoft at hardcoded.net (Virgil Dupras) Date: Mon, 20 Dec 2010 09:22:27 +0100 Subject: [py-dev] Advanced monkeypatching? In-Reply-To: References: <4C5E2E4C-15CF-45B7-80B1-3F0363D82E66@hardcoded.net> Message-ID: On 2010-12-19, at 3:38 PM, Floris Bruynooghe wrote: > Hello > > On 18 December 2010 14:51, Virgil Dupras wrote: >> Hi there, >> >> Monkeypatching is great and all, but some type of patching is a little bit more complex to do, such as patching the system time and patching os.stat for specific filenames. I have my own "homebrewed" solution for this, but I was wondering if such solution had their place in the monkeypatch module of pytest itself. Does it? Or should a "pytest-monkeypatch++" package be created? >> >> The "homebrewed" code I'm talking about is at https://bitbucket.org/hsoft/hsutil/src/43014954c6a8/hsutil/testutil.py#cl-83 ( patch_osstat() and patch_today() ). > > Looking at the patch_osstat() it seems to me that it just fakes > os.stat() for one specific path but uses the original in place > otherwise. This sounds like it could be just a specialised object > used to patch with. I can imagine the monkeypatch plugin to provide a > funcarg which provides this functionality, e.g.: > > def test_foo(monkeypatch, monkey_osstat): > monkeypatch.setattr(monkey_ossstat('/tmp/some_file')) > ... > > Though probably with better names ;-). > > I'm not sure I understand the use case for patch_today() so won't > comment on that part. > > Regards > Floris > > > -- > Debian GNU/Linux -- The Power of Freedom > www.debian.org | www.gnu.org | www.kernel.org This would kind of work, but it would become complicated to patch os.stat() for two or more files, wouldn't it? As for patch_today(), I created it because when you want to patch the date for a test, you can't just patch datetime because there might be a function somewhere using time.time(), so you'll have inconsistent date patching. However, to patch time.time() to a datetime of your choice, you have to do some arithmetics, hence the patch_today() helper. My question is: Does such helpers have their place in the official pytest package or should it be an external plugin? Regards, Virgil Dupras From Ronny.Pfannschmidt at gmx.de Mon Dec 20 10:41:19 2010 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Mon, 20 Dec 2010 10:41:19 +0100 Subject: [py-dev] Advanced monkeypatching? In-Reply-To: References: <4C5E2E4C-15CF-45B7-80B1-3F0363D82E66@hardcoded.net> Message-ID: <1292838079.8085.2.camel@Klappe2> On Mon, 2010-12-20 at 09:22 +0100, Virgil Dupras wrote: > On 2010-12-19, at 3:38 PM, Floris Bruynooghe wrote: > > > Hello > > > > On 18 December 2010 14:51, Virgil Dupras wrote: > >> Hi there, > >> > >> Monkeypatching is great and all, but some type of patching is a little bit more complex to do, such as patching the system time and patching os.stat for specific filenames. I have my own "homebrewed" solution for this, but I was wondering if such solution had their place in the monkeypatch module of pytest itself. Does it? Or should a "pytest-monkeypatch++" package be created? > >> > >> The "homebrewed" code I'm talking about is at https://bitbucket.org/hsoft/hsutil/src/43014954c6a8/hsutil/testutil.py#cl-83 ( patch_osstat() and patch_today() ). > > > > Looking at the patch_osstat() it seems to me that it just fakes > > os.stat() for one specific path but uses the original in place > > otherwise. This sounds like it could be just a specialised object > > used to patch with. I can imagine the monkeypatch plugin to provide a > > funcarg which provides this functionality, e.g.: > > > > def test_foo(monkeypatch, monkey_osstat): > > monkeypatch.setattr(monkey_ossstat('/tmp/some_file')) > > ... > > > > Though probably with better names ;-). > > > > I'm not sure I understand the use case for patch_today() so won't > > comment on that part. > > > > Regards > > Floris > > > > > > -- > > Debian GNU/Linux -- The Power of Freedom > > www.debian.org | www.gnu.org | www.kernel.org > > This would kind of work, but it would become complicated to patch os.stat() for two or more files, wouldn't it? there could be a monkey_stat object that controlls the set of files/subtrees under control > > As for patch_today(), I created it because when you want to patch the date for a test, you can't just patch datetime because there might be a function somewhere using time.time(), so you'll have inconsistent date patching. However, to patch time.time() to a datetime of your choice, you have to do some arithmetics, hence the patch_today() helper. > > My question is: Does such helpers have their place in the official pytest package or should it be an external plugin? a good starting place would be a external plugin to try different approaches and stabilize a bit since py.test is supposed to be more stable in the 2.x series. regards Ronny -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part URL: From hsoft at hardcoded.net Mon Dec 20 10:56:33 2010 From: hsoft at hardcoded.net (Virgil Dupras) Date: Mon, 20 Dec 2010 10:56:33 +0100 Subject: [py-dev] Advanced monkeypatching? In-Reply-To: <1292838079.8085.2.camel@Klappe2> References: <4C5E2E4C-15CF-45B7-80B1-3F0363D82E66@hardcoded.net> <1292838079.8085.2.camel@Klappe2> Message-ID: <9EE11035-3F4F-4F2A-BE4E-CCE2AD65C080@hardcoded.net> On 2010-12-20, at 10:41 AM, Ronny Pfannschmidt wrote: > On Mon, 2010-12-20 at 09:22 +0100, Virgil Dupras wrote: >> On 2010-12-19, at 3:38 PM, Floris Bruynooghe wrote: >>> >>> Looking at the patch_osstat() it seems to me that it just fakes >>> os.stat() for one specific path but uses the original in place >>> otherwise. This sounds like it could be just a specialised object >>> used to patch with. I can imagine the monkeypatch plugin to provide a >>> funcarg which provides this functionality, e.g.: >>> >>> def test_foo(monkeypatch, monkey_osstat): >>> monkeypatch.setattr(monkey_ossstat('/tmp/some_file')) >>> ... >>> >> >> This would kind of work, but it would become complicated to patch os.stat() for two or more files, wouldn't it? > there could be a monkey_stat object that controlls the set of > files/subtrees under control > Yes, of course, but keeping the proposed API would make it unintuitive for multiple files. Example: monkeypatch.setattr(os, 'stat', monkey_osstat('some_file')) monkeypatch.setattr(os, 'stat', monkey_osstat('some_other_file')) We're not sure what happens there, as os.stat is patched twice. Are we overwriting our old monkeypatch, or is there some magic caching inside monkey_osstat() making it work for both files? That's the kind of question the developer will wonder about. From flub at devork.be Mon Dec 20 14:47:59 2010 From: flub at devork.be (Floris Bruynooghe) Date: Mon, 20 Dec 2010 13:47:59 +0000 Subject: [py-dev] Advanced monkeypatching? In-Reply-To: <9EE11035-3F4F-4F2A-BE4E-CCE2AD65C080@hardcoded.net> References: <4C5E2E4C-15CF-45B7-80B1-3F0363D82E66@hardcoded.net> <1292838079.8085.2.camel@Klappe2> <9EE11035-3F4F-4F2A-BE4E-CCE2AD65C080@hardcoded.net> Message-ID: On 20 December 2010 09:56, Virgil Dupras wrote: > > On 2010-12-20, at 10:41 AM, Ronny Pfannschmidt wrote: >> On Mon, 2010-12-20 at 09:22 +0100, Virgil Dupras wrote: >>> On 2010-12-19, at 3:38 PM, Floris Bruynooghe wrote: >>>> >>>> Looking at the patch_osstat() it seems to me that it just fakes >>>> os.stat() for one specific path but uses the original in place >>>> otherwise. ?This sounds like it could be just a specialised object >>>> used to patch with. ?I can imagine the monkeypatch plugin to provide a >>>> funcarg which provides this functionality, e.g.: >>>> >>>> def test_foo(monkeypatch, monkey_osstat): >>>> ? monkeypatch.setattr(monkey_ossstat('/tmp/some_file')) >>>> ? ... >>>> >>> >>> This would kind of work, but it would become complicated to patch os.stat() for two or more files, wouldn't it? >> there could be a monkey_stat object that controlls the set of >> files/subtrees under control >> > > Yes, of course, but keeping the proposed API would make it unintuitive for multiple files. Example: > > monkeypatch.setattr(os, 'stat', monkey_osstat('some_file')) > monkeypatch.setattr(os, 'stat', monkey_osstat('some_other_file')) > > We're not sure what happens there, as os.stat is patched twice. Are we overwriting our old monkeypatch, or is there some magic caching inside monkey_osstat() making it work for both files? That's the kind of question the developer will wonder about. I was thinking of something slightly different when I said this: def test_foo(monkeypatch, monkey_osstat): stat = monkey_ossstat('file1', 'file2') stat.add_file('file3') monkeypatch.setattr(os, 'stat', stat) stat.add_file('file4', st_size=123) That makes it very clear what happens (at least to me). But as Ronny said it's probably best to try out the various approaches in an external plugin to get past the bikeshedding. If that results in a stable api which is getting used it could be merged with the core. This is what is happening to the capturelog plugin too I think. As for the patch_time() I've never needed it but I understand now why you want it. Doing this via a funcarg which you then use in one of the normal monkeypatch methods will probably be very cumbersome so a funcarg that does the patching directly is probably the way to go (which could eventually become something like "monkeypatch.patch_time(...)" I guess) Regards Floris PS: I speak with no authority to the development of py.test, I'm just another user. -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From Ronny.Pfannschmidt at gmx.de Tue Dec 21 10:54:19 2010 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Tue, 21 Dec 2010 10:54:19 +0100 Subject: [py-dev] Advanced monkeypatching? In-Reply-To: References: <4C5E2E4C-15CF-45B7-80B1-3F0363D82E66@hardcoded.net> <1292838079.8085.2.camel@Klappe2> <9EE11035-3F4F-4F2A-BE4E-CCE2AD65C080@hardcoded.net> Message-ID: <1292925260.8085.6.camel@Klappe2> On Mon, 2010-12-20 at 13:47 +0000, Floris Bruynooghe wrote: > On 20 December 2010 09:56, Virgil Dupras wrote: > > > > On 2010-12-20, at 10:41 AM, Ronny Pfannschmidt wrote: > >> On Mon, 2010-12-20 at 09:22 +0100, Virgil Dupras wrote: > >>> On 2010-12-19, at 3:38 PM, Floris Bruynooghe wrote: > >>>> > >>>> Looking at the patch_osstat() it seems to me that it just fakes > >>>> os.stat() for one specific path but uses the original in place > >>>> otherwise. This sounds like it could be just a specialised object > >>>> used to patch with. I can imagine the monkeypatch plugin to provide a > >>>> funcarg which provides this functionality, e.g.: > >>>> > >>>> def test_foo(monkeypatch, monkey_osstat): > >>>> monkeypatch.setattr(monkey_ossstat('/tmp/some_file')) > >>>> ... > >>>> > >>> > >>> This would kind of work, but it would become complicated to patch os.stat() for two or more files, wouldn't it? > >> there could be a monkey_stat object that controlls the set of > >> files/subtrees under control > >> > > > > Yes, of course, but keeping the proposed API would make it unintuitive for multiple files. Example: > > > > monkeypatch.setattr(os, 'stat', monkey_osstat('some_file')) > > monkeypatch.setattr(os, 'stat', monkey_osstat('some_other_file')) > > > > We're not sure what happens there, as os.stat is patched twice. Are we overwriting our old monkeypatch, or is there some magic caching inside monkey_osstat() making it work for both files? That's the kind of question the developer will wonder about. > > I was thinking of something slightly different when I said this: > > def test_foo(monkeypatch, monkey_osstat): > stat = monkey_ossstat('file1', 'file2') > stat.add_file('file3') > monkeypatch.setattr(os, 'stat', stat) > stat.add_file('file4', st_size=123) i was thinking more along the lines of:: def test_foo(osstat_patch): stat.add_file(...) #setup single files stat.add_callback(root, somefunction) # hand off a whole tree i suppose better function names are possible also im wondering if this should integrate with py.path more deeply (having a virtual view on the whole fs seems neat) > > That makes it very clear what happens (at least to me). > > But as Ronny said it's probably best to try out the various approaches > in an external plugin to get past the bikeshedding. If that results > in a stable api which is getting used it could be merged with the > core. This is what is happening to the capturelog plugin too I think. > > As for the patch_time() I've never needed it but I understand now why > you want it. Doing this via a funcarg which you then use in one of > the normal monkeypatch methods will probably be very cumbersome so a > funcarg that does the patching directly is probably the way to go > (which could eventually become something like > "monkeypatch.patch_time(...)" I guess) > > Regards > Floris > > PS: I speak with no authority to the development of py.test, I'm just > another user. > > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part URL: From flub at devork.be Tue Dec 21 11:45:46 2010 From: flub at devork.be (Floris Bruynooghe) Date: Tue, 21 Dec 2010 10:45:46 +0000 Subject: [py-dev] Advanced monkeypatching? In-Reply-To: <1292925260.8085.6.camel@Klappe2> References: <4C5E2E4C-15CF-45B7-80B1-3F0363D82E66@hardcoded.net> <1292838079.8085.2.camel@Klappe2> <9EE11035-3F4F-4F2A-BE4E-CCE2AD65C080@hardcoded.net> <1292925260.8085.6.camel@Klappe2> Message-ID: On 21 December 2010 09:54, Ronny Pfannschmidt wrote: > On Mon, 2010-12-20 at 13:47 +0000, Floris Bruynooghe wrote: >> On 20 December 2010 09:56, Virgil Dupras wrote: >> > >> > On 2010-12-20, at 10:41 AM, Ronny Pfannschmidt wrote: >> >> On Mon, 2010-12-20 at 09:22 +0100, Virgil Dupras wrote: >> >>> On 2010-12-19, at 3:38 PM, Floris Bruynooghe wrote: >> >>>> >> >>>> Looking at the patch_osstat() it seems to me that it just fakes >> >>>> os.stat() for one specific path but uses the original in place >> >>>> otherwise. ?This sounds like it could be just a specialised object >> >>>> used to patch with. ?I can imagine the monkeypatch plugin to provide a >> >>>> funcarg which provides this functionality, e.g.: >> >>>> >> >>>> def test_foo(monkeypatch, monkey_osstat): >> >>>> ? monkeypatch.setattr(monkey_ossstat('/tmp/some_file')) >> >>>> ? ... >> >>>> >> >>> >> >>> This would kind of work, but it would become complicated to patch os.stat() for two or more files, wouldn't it? >> >> there could be a monkey_stat object that controlls the set of >> >> files/subtrees under control >> >> >> > >> > Yes, of course, but keeping the proposed API would make it unintuitive for multiple files. Example: >> > >> > monkeypatch.setattr(os, 'stat', monkey_osstat('some_file')) >> > monkeypatch.setattr(os, 'stat', monkey_osstat('some_other_file')) >> > >> > We're not sure what happens there, as os.stat is patched twice. Are we overwriting our old monkeypatch, or is there some magic caching inside monkey_osstat() making it work for both files? That's the kind of question the developer will wonder about. >> >> I was thinking of something slightly different when I said this: >> >> def test_foo(monkeypatch, monkey_osstat): >> ? ? stat = monkey_ossstat('file1', 'file2') >> ? ? stat.add_file('file3') >> ? ? monkeypatch.setattr(os, 'stat', stat) >> ? ? stat.add_file('file4', st_size=123) > > i was thinking more along the lines of:: > > ? ?def test_foo(osstat_patch): > ? ? ? ?stat.add_file(...) #setup single files > ? ? ? ?stat.add_callback(root, somefunction) # hand off a whole tree > > i suppose better function names are possible > also im wondering if this should integrate with py.path more deeply > (having a virtual view on the whole fs seems neat) That is a nice idea, I have at least one test where I create a simple directory structure using (the old) py.test.ensuretemp() to test if some sorting on different os.stat() fields works in a function. This is an annoying case since I need to do time.sleep() in between creating the files as some platforms don't have a high enough resolution on file timestamps. So being able to "set" stat attributes on py.path items would certainly be an interesting area to explore. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From ralf at brainbot.com Mon Dec 27 20:12:33 2010 From: ralf at brainbot.com (Ralf Schmitt) Date: Mon, 27 Dec 2010 20:12:33 +0100 Subject: [py-dev] use of md5 checksum in config.py/getconftestmodules looks fishy Message-ID: <87lj3b6ndq.fsf@muni.brainbot.com> _pytest/config.py contains the following code: ,---- | ... | conftestpath = path.join("conftest.py") | if conftestpath.check(file=1): | key = conftestpath.computehash() | # XXX logging about conftest loading | if key not in self._md5cache: | clist.append(self.importconftest(conftestpath)) | self._md5cache[key] = conftestpath | else: | # use some kind of logging | print ("WARN: not loading %s" % conftestpath) `---- If I put multiple conftest.py files with the same content into different subdirecties py.test will only read one of them. I try to make py.test ignore multiple subdirectories by putting the same conftest.py file into those subdirectories. py.test currently only ignores one of those. Regards -- Ralf