Mailman 3 June 2005 - pytest-dev

[py-dev] greenlet.main_greenlet
by Tom Locke Aug. 27, 2005

Aug. 27, 2005

Hi, I had a need to be able to test if the current greenlet was the main greenlet. I figured this might be something that crops up now and again, so I added a module attribute `main_greeenlet` (also copied to the type object), so I can just test for: greenlet.getcurrent() == greenlet.main_greenlet There's two small changes to greenlet.c (see diff) Tom Index: greenlet.c =================================================================== --- greenlet.c (revision 12401) +++ greenlet.c (working copy) @@ -758,7 +758,7 @@ }; static char* copy_on_greentype[] = { - "getcurrent", "error", "GreenletExit", NULL + "getcurrent", "error", "GreenletExit", "main_greenlet", NULL }; void initgreenlet(void) @@ -787,6 +787,8 @@ if (ts_current == NULL) return; + Py_INCREF(ts_current); + PyModule_AddObject(m, "main_greenlet", (PyObject*)ts_current); Py_INCREF(&PyGreen_Type); PyModule_AddObject(m, "greenlet", (PyObject*) &PyGreen_Type); Py_INCREF(PyExc_GreenletError);

2 1

[py-dev] py.test and expected failures
by Giovanni Bajo June 24, 2005

June 24, 2005

Hello, Is there any plan to add "expected failure" support to py.test? Something like: @py.test.xfail def test_foo(): .... py.test would then silently skip over the failure, and will instead cry as soon as the test starts working again ("unexpected success"). Thanks! -- Giovanni Bajo

1 0

[py-dev] What's next for py.log?
by Grig Gheorghiu June 21, 2005

June 21, 2005

Hi, Holger Do you think the core consumer and producer code will remain as is for py.log? It seems to work pretty well. I might still sub-class the Producer class for my multiple-keyword needs -- but that can easily be done in my application code, without changing py.log itself. Are you thinking about adding more consumer objects such as syslog, email, Win Events? If so, I can give some of them a shot. I'm also thinking about advertising py.log a little on my blog. Or should I refrain until the py release? Let me know what you think. Grig

2 4

[py-dev] distributed computing with py.execnet
by Carl Friedrich Bolz June 21, 2005

June 21, 2005

Hi Py-Dev! I experimented a bit with py.execnet today and just want to share the result of the experiment. I wrote a little proof of concept for distributing a big computational task between several computers (think cluster) without having to install anything on them. There are several processes involved: a server that establishes connections via execnet gateways and sends them their task via the channels. Then the server listens at a socket. If a node has finished its task, it sends the server a notification via the socket and then the server fetches the result via the appropriate channel. Then the node gets a new task via the channel. I used the search for Mersenne primes with the Lucas-Lehmer-test as the computational task (which one wouldn't implement in Python, of course). The whole implementation, server and primality test is less than 130 lines of code (see below). There are still some things missing (like a clean shutdown for example) but the basic concept works very well. execnet rocks! Regards, Carl Friedrich #-----------server.py-------------- import py import SocketServer ip = "192.168.10.1" # ip of the server nodes = ["cfbolz@linux", "jana@albert"] port = 50007 chunk_size = 50 source = py.code.Source(py.path.local().join("task.py").read()) log = py.log.Producer("server") py.log.setconsumer("server", None) class TaskDistributor(object): def __init__(self, nodes, chunk_size=1): self.i = 1 self.nodes = nodes self.chunk_size = chunk_size self.channels = [] for node in nodes: print node channel = py.execnet.SshGateway(node).remote_exec(source) self.channels.append(channel) assert channel.receive() == "online" channel.send(len(self.channels) - 1) channel.send((ip, port)) channel.send(range(self.i, self.i + self.chunk_size)) channel.send(len(nodes) * 3) assert channel.receive() == "ok" log.init("sent task %s" % (self.i,)) self.i += self.chunk_size def new_task(self, number): channel = self.channels[number] results = channel.receive() channel.send("ok") channel.send(range(self.i, self.i + self.chunk_size)) log.new_job(self.i) self.i += self.chunk_size assert channel.receive() == "ok" for result in results: if result[1]: print "%s = 2**%s - 1 is a prime! result from node %s" % (2 ** result[0] - 1, result[0], number) if not result[1]: print "2**%s - 1 is not prime" % (self.i, ) class RequestHandler(SocketServer.BaseRequestHandler): def handle(self): try: number = int(self.request.recv(10)) log.request("from", number) self.request.send("ok") RequestHandler.distributor.new_task(number) except ValueError: self.request.send("not a number") log.request("closing socket") self.request.close() if __name__ == '__main__': RequestHandler.distributor = TaskDistributor(nodes, 100) server = SocketServer.TCPServer((ip, port), RequestHandler) print server server.serve_forever() #-----------task.py-------------- import operator import random import socket import time import sys try: import psyco psyco.full() except ImportError: pass def fermattest(p, tests=30): if p == 1: return False for i in xrange(tests): a = long(random.random() * (p - 1) + 1) if not pow(a, p, p) == a: return False return True def M(p): return 2 ** p - 1 def lucastest(p): curr = 4 m = M(p) for i in range(p - 2): curr = (curr ** 2 - 2) % m if curr == 0: return True return False def process_task(task_data): results = [] for p in task_data: print p if not fermattest(p): results.append((p, False)) else: results.append((p, lucastest(p))) return results channel.send("online") mynumber = str(channel.receive()) socketaddress = channel.receive() task_data = channel.receive() sleep = channel.receive() channel.send("ok") f = file("/tmp/out.txt", "w", 0) print >> f, "sleeping" time.sleep(sleep) # give the server time to establish all connections print >> f, "starting to work" while 1: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) result = process_task(task_data) print >>f, result, socketaddress print >>f, "connecting", s try: s.connect(socketaddress) except Exception, e: print >>f, "error!", e f.close() raise print >>f, "connected2" s.send(mynumber) print >>f, "sent number" assert s.recv(2) == "ok" s.close() channel.send(result) print >>f, "sent result" assert channel.receive() == "ok" task_data = channel.receive() print >>f, "new task data %s" % task_data channel.send("ok")

2 3

[py-dev] py.log and multiple keywords
by Grig Gheorghiu June 20, 2005

June 20, 2005

Holger and others interested in this topic, I was thinking that in many situations it would be helpful to be able to specify multiple keywords that would each point to a different consumer and that would be all 'kick in' on a call to the producer. Basically, I'd want this test to pass: def test_multiple_keywords(self): log = py.log.Producer('console logfile db') py.log.setconsumer('console', py.log.STDOUT) logfile = tempdir.join('mylogfile.out') py.log.setconsumer('logfile', open(str(logfile), 'w', buffering=1)) dbfile = tempdir.join('mydbfile.out') py.log.setconsumer('db', open(str(dbfile), 'w', buffering=1)) res, out, err = callcapture(log, "hello") assert out.strip() == "[console:logfile:db] hello" assert logfile.readlines() == ['[console:logfile:db] hello\n'] assert dbfile.readlines() == ['[console:logfile:db] hello\n'] Right now the last 2 assert statements fail, because _getconsumer returns as soon as it finds a match for the first keyword, so only the 'console' consumer gets a chance to consume the message. I'm thinking that the _getconsumer function could be a generator that yields a message-processing function for each keyword it finds in its registry. What do you think? Grig

2 6

[py-dev] Re: [py-svn] r13515 - in py/dist/py: . log log/testing
by Grig Gheorghiu June 17, 2005

June 17, 2005

--- hpk(a)codespeak.net wrote: > Author: hpk > Date: Fri Jun 17 07:44:36 2005 > New Revision: 13515 > > Modified: > py/dist/py/__init__.py > py/dist/py/log/consumer.py > py/dist/py/log/producer.py > py/dist/py/log/testing/test_log.py > Log: > issue3 in-progress > > refactored logging API > > - no severity levels offered by default anymore > > - no logging module used at the moment > > - simplified API > > - there now is a 'py.log.default' defaultlogger > > Hi, Holger I've looked at the refactored version of the log functionality and for some reason it doesn't seem as intuitive to use as the old one. Now, as an end user, you need to know the distinction between a 'producer' and a 'consumer', which might be hard to grasp, at least at first sight. You also need to use the Producer class directly, as it is not hidden anymore. My preference would be to hide names such as producers and consumers and expose more user-friendly names such as logs and loggers. Other than that, I like the fact that the keyword-based matching offers a lot of flexibility. I can do for example: py.log.db = py.log.Producer('db') py.log.setconsumer('db', function_to_insert_in_db) py.log.db("hello world") Have you have any use for multiple keywords? Just curious, as I haven't seen the need for it yet. About severity levels: I guess in the current version you can just set the consumer for that particular keyword to None in a configuration file, and that would disable logging for that keyword. That's how you would turn on and off debugging messages for example, am I right? Grig

1 0

[py-dev] Odd import issue with tests for compatibility modules.
by David Stanek June 17, 2005

June 17, 2005

The support for Python compatibility modules has been merged into trunk. I pulled the modules (doctest, optparse and testwrap) along with their tests from Python-r241. I was able to modify almost all of the tests to make sure they are testing the compat module and not the library module However, I am having an issue with py/compat/testing/doctest2.py. The original has a testing function that imports itself and then passtest that to a test.test_support.run_doctest method. To mimic that I tried passing the run_doctests method the following: * sys.modules['__main__'] * test_doctest2 after a from py.__.compat.testing import test_doctest2 These do not work...well, all test are successful, but 2 are not being run. And they just happen to be the two that contain unicode characters. If I remove the characters the test will execute successfully. The test is not failing it is just not being run. To debug I added a line to print out the __doc__ of the module, which is one of the tests that is not being executed. The string did NOT include the doctest code. So this seems to be an import thing and not something to do with unicode and/or doctest. Any ideas? I have to take a break from this so that I can get a fresh perspective. David

3 4

Re: [py-dev] refactoring py.log ...
by Grig Gheorghiu June 16, 2005

June 16, 2005

Forgot to CC the list again... Hi, Holger --- holger krekel <hpk(a)trillke.net> wrote: > Hi Grig, > > On Tue, Jun 14, 2005 at 16:10 -0700, Grig Gheorghiu wrote: > > --- holger krekel <hpk(a)trillke.net> wrote: > > > Doesn't the logging module offer the underlying output/formatting > > > logic in a more raw form, maybe? Or else, isn't the logic needed > for > > > File, Stdout and Stderr trivial to implement? We then probably > > > don't need to bother with the logging module at all. > > > > I'll look into using more low-level functions/classes from the > logging > > module. As far as implementing its functionality from scratch: at > least > > for File/Stdout/Stderr it wouldn't be very hard to do, but then > we'd > > need to worry about thread safety etc., which are issues that the > > logging module takes care for you. Plus the whole mechanism of > > registering various handlers for various severity levels is pretty > > nice, although again it wouldn't be that hard to mimic. > > Yes, i now even think that reusing the logging module at API level > might not be such a good idea. Reusing some code and ideas by > copy-and-paste may just be enough. It's no dark magic that is > changing everyday. > > Moreover, I am really beginning to come from the usage > side and i feel that i don't really want severity-distinctions > statically everywhere in the source code although it may seem > natural at first. (see my example in an earlier mail). > > I guess i am at the verge of wanting to follow a more radical > keyword/tag based approach. Would you mind if i refactor > and simplify things accordingly? It's possible that we > introduce some default severity levels but i'd like to > postpone it until there is a real need. I don't mind at all if you refactor and simplify things. I haven't done much work on it since my last commit. You're probably right about not needing severity levels right away. I agree with you that we should look at things from an 'end-user' point of view and thus make the calls to the py.log functionality as simple and natural as possible. > > > Btw, I don't know how the logging's email backend works but there > > > probably is some "commit()" like function that actually > > > triggers sending a mail, or not? I probably don't want an email > > > message on each logging entry :-) > > > > My understanding is that it DOES send an email per log entry :-(, > but I > > haven't tinkered with the SMTP logging backend that much yet. > > This may point to an interesting problem. Logging backends/consumers > are not really stateless. For example, traditionally in Unix > the sysloggers have two modes of writing to files, one that > commits ('fsync()') and one that doesn't. The latter is > faster but log entries may get lost if there is a crash (even > a minute old entries). Or does the logging module offer this > distinction? Anyway, it's just a few lines of code and it's > ok if we do it ourselves (and test it accordingly). > > Sending batched log entries is a use case where we > probably could do something simple, e.g. > > l = [] > py.log.set_logger('default', l.append) > call_program() # logs various stuff > for msg in l: > if 'error' in msg.keywords: > # send an email with a selection of messages > > IOW, we don't even neccessarily need a custom Email logger > and introduce names and API design for it. Yeah, it would be nice to have the 'batch email' functionality. I don't think the logging module offers any distinction between single email messages and batch mode. > > I'll be playing with it, as well as the Syslog and WinEvent > > stuff, and I'll let you know what I find. > > that would be useful. Haven't had time to do this yet.... Grig

1 0

[py-dev] refactoring py.log ...
by holger krekel June 16, 2005

June 16, 2005

Hi Grig, On Tue, Jun 14, 2005 at 16:10 -0700, Grig Gheorghiu wrote: > --- holger krekel <hpk(a)trillke.net> wrote: > > Doesn't the logging module offer the underlying output/formatting > > logic in a more raw form, maybe? Or else, isn't the logic needed for > > File, Stdout and Stderr trivial to implement? We then probably > > don't need to bother with the logging module at all. > > I'll look into using more low-level functions/classes from the logging > module. As far as implementing its functionality from scratch: at least > for File/Stdout/Stderr it wouldn't be very hard to do, but then we'd > need to worry about thread safety etc., which are issues that the > logging module takes care for you. Plus the whole mechanism of > registering various handlers for various severity levels is pretty > nice, although again it wouldn't be that hard to mimic. Yes, i now even think that reusing the logging module at API level might not be such a good idea. Reusing some code and ideas by copy-and-paste may just be enough. It's no dark magic that is changing everyday. Moreover, I am really beginning to come from the usage side and i feel that i don't really want severity-distinctions statically everywhere in the source code although it may seem natural at first. (see my example in an earlier mail). I guess i am at the verge of wanting to follow a more radical keyword/tag based approach. Would you mind if i refactor and simplify things accordingly? It's possible that we introduce some default severity levels but i'd like to postpone it until there is a real need. > > Btw, I don't know how the logging's email backend works but there > > probably is some "commit()" like function that actually > > triggers sending a mail, or not? I probably don't want an email > > message on each logging entry :-) > > My understanding is that it DOES send an email per log entry :-(, but I > haven't tinkered with the SMTP logging backend that much yet. This may point to an interesting problem. Logging backends/consumers are not really stateless. For example, traditionally in Unix the sysloggers have two modes of writing to files, one that commits ('fsync()') and one that doesn't. The latter is faster but log entries may get lost if there is a crash (even a minute old entries). Or does the logging module offer this distinction? Anyway, it's just a few lines of code and it's ok if we do it ourselves (and test it accordingly). Sending batched log entries is a use case where we probably could do something simple, e.g. l = [] py.log.set_logger('default', l.append) call_program() # logs various stuff for msg in l: if 'error' in msg.keywords: # send an email with a selection of messages IOW, we don't even neccessarily need a custom Email logger and introduce names and API design for it. > I'll be playing with it, as well as the Syslog and WinEvent > stuff, and I'll let you know what I find. that would be useful. cheers, holger

1 0

Re: [py-dev] Adding logdebug option to defaultconftest.py
by Grig Gheorghiu June 14, 2005

June 14, 2005

Forgot to copy the py-dev list.... --- holger krekel <hpk(a)trillke.net> wrote: > On Tue, Jun 14, 2005 at 15:56 -0700, Grig Gheorghiu wrote: > > --- holger krekel <hpk(a)trillke.net> wrote: > > > I am starting to think about using > > > py.log here and there and it seems like i am actually going > > > for something like: > > > > > > class Something: > > > log = py.log.Producer('something') > > > > > > def method(self): > > > ... > > > self.log.adhoccategory("processing", x) > > > > > > this shows by default on stdout unless you set a catch-all > > > consumer or a specific one for e.g. 'something'. > > > > > > the "debugging" aspect then becomes a matter of defining > > > which logging information i care for (which could be changed > > > interactively or per command line switches as you suggested) > > > Do that make sense to you? > > > > Yes, it makes sense to me. I guess I was thinking more in terms of > > 'consumers' (which interface more heavily with the logging module), > > whereas you're approaching it in terms of 'agnostic' producers. So > I > > think you're right, it doesn't make sense to burden the set_logger > > function with yet another parameter. > > right, that's at least the experimental approach. We'll see how > it plays out. > > > Maybe the logging level should only be a parameter to the specific > > loggers defined in consumer.py: File, Stdout, Stderr, Email, etc. > Right > > now it's arbitrarily set to logging.DEBUG. > > Doesn't the logging module offer the underlying output/formatting > logic in a more raw form, maybe? Or else, isn't the logic needed for > File, Stdout and Stderr trivial to implement? We then probably > don't need to bother with the logging module at all. I'll look into using more low-level functions/classes from the logging module. As far as implementing its functionality from scratch: at least for File/Stdout/Stderr it wouldn't be very hard to do, but then we'd need to worry about thread safety etc., which are issues that the logging module takes care for you. Plus the whole mechanism of registering various handlers for various severity levels is pretty nice, although again it wouldn't be that hard to mimic. > Btw, I don't know how the logging's email backend works but there > probably is some "commit()" like function that actually > triggers sending a mail, or not? I probably don't want an email > message on each logging entry :-) My understanding is that it DOES send an email per log entry :-(, but I haven't tinkered with the SMTP logging backend that much yet. I'll be playing with it, as well as the Syslog and WinEvent stuff, and I'll let you know what I find. Grig

1 0