Mailman 3 July 2006 - pypy-dev

Fix for #145 (-m cmdline option)
by Leonardo Soto July 26, 2006

July 26, 2006

Hi, I've tried to upload the attached patch on roundup, but it gives me a "Required issue properties title, priority not supplied" error. It could be a permission error, but the message don't suggest that. So the patch is here. Comments and corrections are welcomed. -- Leonardo Soto M.

1 0

Registering Pypy on Freshmeat
by Jacob Hallén July 26, 2006

July 26, 2006

It just struck me that maybe we should register the Pypy project on Freshmeat. It would increase the visibility of the project as well as increase the visibility of Python in the more general OpenSource world. What do people think? Jacob

1 0

Re: [pypy-dev] [pypy-svn] r30565 - in pypy/dist/pypy/tool/build: . bin builds test
by Ben.Young＠risk.sungard.com July 26, 2006

July 26, 2006

Hi Guido, Cool tool! Is this something I would be able to run just in the evenings/weekends etc? Will it pick up that fact that I don't have boehm automatically? Cheers, Ben pypy-svn-bounces(a)codespeak.net wrote on 26/07/2006 12:18:51: > Author: guido > Date: Wed Jul 26 13:18:25 2006 > New Revision: 30565 > > Added: > pypy/dist/pypy/tool/build/ (props changed) > pypy/dist/pypy/tool/build/README.txt (contents, props changed) > pypy/dist/pypy/tool/… [View More]build/__init__.py (contents, props changed) > pypy/dist/pypy/tool/build/bin/ (props changed) > pypy/dist/pypy/tool/build/bin/client (contents, props changed) > pypy/dist/pypy/tool/build/bin/path.py (contents, props changed) > pypy/dist/pypy/tool/build/bin/server (contents, props changed) > pypy/dist/pypy/tool/build/bin/startcompile (contents, props changed) > pypy/dist/pypy/tool/build/builds/ > pypy/dist/pypy/tool/build/client.py (contents, props changed) > pypy/dist/pypy/tool/build/config.py (contents, props changed) > pypy/dist/pypy/tool/build/conftest.py (contents, props changed) > pypy/dist/pypy/tool/build/execnetconference.py (contents, props changed) > pypy/dist/pypy/tool/build/server.py (contents, props changed) > pypy/dist/pypy/tool/build/test/ (props changed) > pypy/dist/pypy/tool/build/test/fake.py (contents, props changed) > pypy/dist/pypy/tool/build/test/path.py (contents, props changed) > pypy/dist/pypy/tool/build/test/test.zip (contents, props changed) > pypy/dist/pypy/tool/build/test/test_client.py (contents, props changed) > pypy/dist/pypy/tool/build/test/test_pypybuilder.py (contents, > props changed) > pypy/dist/pypy/tool/build/test/test_request_storage.py > (contents, props changed) > pypy/dist/pypy/tool/build/test/test_server.py (contents, props changed) > Log: > Added 'pypybuilder', a tool to build a 'build farm' from > participating clients, > the clients register to a server with information about what they can build, > then the server waits for build requests and dispatches to clients. Clients > send back a build when they're done, on which the server sends out emails to > whoever is waiting for the build. When a build is already available, the > requestor is provided with a path (will be URL in the future) to the build, > if no client is available for a certain request the request is queued until > there is one. > Worked on this from https://merlinux.de/svn/user/guido/pypybuilder before > checking in here. > > > Added: pypy/dist/pypy/tool/build/README.txt > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/README.txt Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,59 @@ > +============ > +PyPyBuilder > +============ > + > +What is this? > +============= > + > +PyPyBuilder is an application that allows people to build PyPy instances on > +demand. If you have a nice idle machine connected to the Internet, and don't > +mind us 'borrowing' it every once in a while, you can start up the client > +script (in bin/client) and have the server send compile jobs to your machine. > +If someone requests a build of PyPy that is not already available on the PyPy > +website, and your machine is capable of making such a build, the > server may ask > +your machine to create it. If enough people participate, with diverse enough > +machines, an ad-hoc 'build farm' is created this way. > + > +Components > +========== > + > +The application consists of 3 main components: a server component, > a client and > +a small component to start compile jobs, which we'll call 'startcompile' for > +now. > + > +The server waits for clients to register, and for compile job requests. When > +clients register, they pass the server information about what > compilations they > +can handle (system info). Then when the 'startcompile' component requests a > +compilation job, the server first checks whether a binary is > already available, > +and if so returns that. > + > +If there isn't one, the server walks through a list of connected > clients to see > +if there is one that can handle the job, and if so tells it to perform it. If > +there's no client to handle the job, it gets queued until there is. > + > +Once a build is available, the server will send an email to all > email addresses > +(it could be more than one person asked for the same build at the same time!) > +passed to it by 'startcompile'. > + > +Installation > +============ > + > +Client > +------ > + > +Installing the system should not be required: just run '. > /bin/client' to start > +the client. Note that it depends on the `py lib`_. > + > +Server > +------ > + > +Also for the server there's no real setup required, and again there's a > +dependency on the `py lib`_. > + > +.. _`py lib`: http://codespeak.net/py > + > +More info > +========= > + > +For more information, bug reports, patches, etc., please send an email to > +guido(a)merlinux.de. > > Added: pypy/dist/pypy/tool/build/__init__.py > ============================================================================== > > Added: pypy/dist/pypy/tool/build/bin/client > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/bin/client Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,58 @@ > +#!/usr/bin/python > + > +BUFSIZE = 1024 > + > +import path > +import sys > +import random > +from pypy.tool.build import config > + > +# XXX using random values for testing > +modules = ['_stackless', '_socket'] > + > +""" > +random.shuffle(modules) > +sysinfo = { > + 'maxint': random.choice((sys.maxint, (2 ** 63 - 1))), > + 'use_modules': modules[:random.randrange(len(modules) + 1)], > + 'byteorder': random.choice(('little', 'big')), > +} > +""" > + > +sysinfo = { > + 'maxint': sys.maxint, > + 'use_modules': ['_stackless', '_socket'], > + 'byteorder': sys.byteorder, > +} > + > +if __name__ == '__main__': > + from py.execnet import SshGateway > + from pypy.tool.build.client import init > + gw = SshGateway(config.server) > + channel = init(gw, sysinfo, path=config.path, port=config.port) > + print channel.receive() # welcome message > + try: > + while 1: > + data = channel.receive() > + if not isinstance(data, dict): # needs more checks here > + raise ValueError( > + 'received wrong unexpected data of type %s' % > (type(data),) > + ) > + info = data > + # XXX we should compile here, using data dict for info > + print 'compilation requested for info %r, now faking > that' % (info,) > + import time; time.sleep(10) > + > + # write the zip to the server in chunks to server > + # XXX we're still faking this > + zipfp = (path.packagedir / 'test/test.zip').open() > + while True: > + chunk = zipfp.read(BUFSIZE) > + if not chunk: > + break > + channel.send(chunk) > + channel.send(None) # tell the server we're done > + print 'done with compilation, waiting for next' > + finally: > + channel.close() > + gw.exit() > > Added: pypy/dist/pypy/tool/build/bin/path.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/bin/path.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,5 @@ > +import py > + > +packagedir = py.magic.autopath().dirpath().dirpath() > +rootpath = packagedir.dirpath().dirpath().dirpath() > +py.std.sys.path.append(str(rootpath)) > > Added: pypy/dist/pypy/tool/build/bin/server > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/bin/server Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,27 @@ > +#!/usr/bin/python > + > +import path > +from pypy.tool.build import config > + > +from py.execnet import SshGateway > + > +if __name__ == '__main__': > + from py.execnet import SshGateway > + from pypy.tool.build.server import init > + > + gw = SshGateway(config.server) > + channel = init(gw, port=config.port, path=config.path, > + projectname=config.projectname, > + buildpath=str(config.buildpath), > + mailhost=config.mailhost, > + mailport=config.mailport, > + mailfrom=config.mailfrom) > + > + try: > + while 1: > + data = channel.receive() > + assert isinstance(data, str) > + print data > + finally: > + channel.close() > + gw.exit() > > Added: pypy/dist/pypy/tool/build/bin/startcompile > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/bin/startcompile Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,57 @@ > +#!/usr/bin/python > + > +import path > +import sys > +import random > +from pypy.tool.build import config > + > +initcode = """ > + import sys > + sys.path += %r > + > + from pypy.tool.build import ppbserver > + channel.send(ppbserver.compile(%r, %r)) > + channel.close() > +""" > +def init(gw, sysinfo, email, port=12321): > + from pypy.tool.build import execnetconference > + > + conference = execnetconference.conference(gw, port, False) > + channel = conference.remote_exec(initcode % (config.path, > email, sysinfo)) > + return channel > + > +if __name__ == '__main__': > + from py.execnet import SshGateway > + > + from optparse import OptionParser > + optparser = OptionParser('%prog [options] email') > + for args, kwargs in config.options: > + optparser.add_option(*args, **kwargs) > + optparser.add_option('-r', '--revision', dest='revision', > default='trunk', > + help='SVN revision (defaults to "trunk")') > + > + (options, args) = optparser.parse_args() > + > + if not args or len(args) != 1: > + optparser.error('please provide an email address') > + > + sysinfo = dict([(attr, getattr(options, attr)) for attr in > dir(options) if > + not attr.startswith('_') and > + not callable(getattr(options, attr))]) > + > + print 'going to start compile job with info:' > + for k, v in sysinfo.items(): > + print '%s: %r' % (k, v) > + print > + > + gw = SshGateway(config.server) > + channel = init(gw, sysinfo, args[0], port=config.port) > + ispath, data = channel.receive() > + if ispath: > + print ('a suitable result is already available, you can find it ' > + 'at "%s"' % (data,)) > + else: > + print data > + print 'you will be mailed once it\'s ready' > + channel.close() > + gw.exit() > > Added: pypy/dist/pypy/tool/build/client.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/client.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,71 @@ > +import time > +import thread > + > +class PPBClient(object): > + def __init__(self, channel, sysinfo, testing=False): > + self.channel = channel > + self.sysinfo = sysinfo > + self.busy_on = None > + self.testing = testing > + > + from pypy.tool.build import ppbserver > + self.server = ppbserver > + self.server.register(self) > + > + def sit_and_wait(self): > + """connect to the host and wait for commands""" > + self.channel.waitclose() > + self.channel.close() > + > + def compile(self, info): > + """send a compile job to the client side > + > + this waits until the client is done, and assumes the client sends > + back the whole binary as a single string (XXX this > should change ;) > + """ > + self.busy_on = info > + self.channel.send(info) > + thread.start_new_thread(self.wait_until_done, (info,)) > + > + def wait_until_done(self, info): > + buildpath = self.server.get_new_buildpath(info) > + > + fp = buildpath.zipfile.open('w') > + if not self.testing: > + try: > + while True: > + try: > + chunk = self.channel.receive() > + except EOFError: > + # stop compilation, client has disconnected > + return > + if chunk is None: > + break > + fp.write(chunk) > + finally: > + fp.close() > + > + self.server.compilation_done(info, buildpath) > + self.busy_on = None > + > +initcode = """ > + import sys > + sys.path += %r > + > + from pypy.tool.build.client import PPBClient > + > + try: > + client = PPBClient(channel, %r, %r) > + client.sit_and_wait() > + finally: > + channel.close() > +""" > +def init(gw, sysinfo, path=None, port=12321, testing=False): > + from pypy.tool.build import execnetconference > + > + if path is None: > + path = [] > + > + conference = execnetconference.conference(gw, port, False) > + channel = conference.remote_exec(initcode % (path, sysinfo, testing)) > + return channel > > Added: pypy/dist/pypy/tool/build/config.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/config.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,45 @@ > +import py > + > +# general settings, used by both server and client > +server = 'johnnydebris.net' > +port = 12321 > +path = ['/home/johnny/temp/pypy-dist'] > + > +# option definitions for the startcompile script > +# for now we have them here, we should probably use pypy's config instead > +# though... > +import sys > +def _use_modules_callback(option, opt_str, value, parser): > + parser.values.use_modules = [m.strip() for m in value.split(',') > + if m.strip()] > + > +def _maxint_callback(option, opt_str, value, parser): > + parser.values.maxint = 2 ** (int(value) - 1) - 1 > + > +options = [ > + (('-m', '--use-modules'), {'action': 'callback', 'type': 'string', > + 'callback': _use_modules_callback, > + 'dest': 'use_modules', 'default': [], > + 'help': 'select the modules you > want to use'}), > + (('-i', '--maxint'), {'action': 'callback', 'callback': _maxint_callback, > + 'default': sys.maxint, 'dest': 'maxint', > + 'type': 'string', > + 'help': ('size of an int in bits (32/64, ' > + 'defaults to sys.maxint)')}), > + (('-b', '--byteorder'), {'action': 'store', > + 'dest': 'byteorder', 'default': > sys.byteorder, > + 'nargs': 1, > + 'help': ('byte order (little/big, defaults ' > + 'to sys.byteorder)')}), > +] > + > +# settings for the server > +projectname = 'pypy' > +buildpath = '/home/johnny/temp/pypy-dist/pypy/tool/build/builds' > +mailhost = '127.0.0.1' > +mailport = 25 > +mailfrom = 'johnny(a)johnnydebris.net' > + > +# settings for the tests > +testpath = [str(py.magic.autopath().dirpath().dirpath())] > + > > Added: pypy/dist/pypy/tool/build/conftest.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/conftest.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,20 @@ > +import py > +from py.__.documentation.conftest import Directory as Dir, DoctestText, \ > + ReSTChecker > +mypath = py.magic.autopath().dirpath() > + > +Option = py.test.Config.Option > +option = py.test.Config.addoptions("pypybuilder test options", > + Option('', '--functional', > + action="store_true", dest="functional", default=False, > + help="run pypybuilder functional tests" > + ), > +) > + > +py.test.pypybuilder_option = option > + > +class Directory(Dir): > + def run(self): > + if self.fspath == mypath: > + return ['README.txt', 'test'] > + return super(Directory, self).run() > > Added: pypy/dist/pypy/tool/build/execnetconference.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/execnetconference.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,126 @@ > +""" > +An extension to py.execnet to allow multiple programs to exchange information > +via a common server. The idea is that all programs first open a gateway to > +the same server (e.g. an SshGateway), and then call the conference() function > +with a local TCP port number. The first program must pass is_server=True and > +the next ones is_server=False: the first program's remote gateway is used as > +shared server for the next ones. > + > +For all programs, the conference() call returns a new gateway that is > +connected to the Python process of this shared server. Information can > +be exchanged by passing data around within this Python process. > +""" > +import py > +from py.__.execnet.register import InstallableGateway > + > + > +def conference(gateway, port, is_server='auto'): > + if is_server: # True or 'auto' > + channel = gateway.remote_exec(r""" > + import thread > + from socket import * > + s = socket(AF_INET, SOCK_STREAM) > + port = channel.receive() > + try: > + s.bind(('', port)) > + s.listen(5) > + except error: > + channel.send(0) > + else: > + channel.send(1) > + > + def readall(s, n): > + result = '' > + while len(result) < n: > + t = s.read(n-len(result)) > + if not t: > + raise EOFError > + result += t > + return result > + > + def handle_connexion(clientsock, address): > + clientfile = clientsock.makefile('r+b',0) > + source = clientfile.readline().rstrip() > + clientfile.close() > + g = {'clientsock' : clientsock, 'address' : address} > + source = eval(source) > + if source: > + g = {'clientsock' : clientsock, 'address' : address} > + co = compile(source+'\n', source, 'exec') > + exec co in g > + > + while True: > + conn, addr = s.accept() > + if addr[0] == '127.0.0.1': # else connexion refused > + thread.start_new_thread(handle_connexion, > (conn, addr)) > + del conn > + """) > + channel.send(port) > + ok = channel.receive() > + if ok: > + return gateway > + if is_server == 'auto': > + pass # fall-through and try as a client > + else: > + raise IOError("cannot listen on port %d (already in > use?)" % port) > + > + if 1: # client > + channel = gateway.remote_exec(r""" > + import thread > + from socket import * > + s = socket(AF_INET, SOCK_STREAM) > + port = channel.receive() > + s.connect(('', port)) > + channel.send(1) > + def receiver(s, channel): > + while True: > + data = s.recv(4096) > + #print >> open('LOG','a'), 'backward', repr(data) > + channel.send(data) > + if not data: break > + thread.start_new_thread(receiver, (s, channel)) > + try: > + for data in channel: > + #print >> open('LOG','a'), 'forward', repr(data) > + s.sendall(data) > + finally: > + s.shutdown(1) > + """) > + channel.send(port) > + ok = channel.receive() > + assert ok > + return InstallableGateway(ConferenceChannelIO(channel)) > + > + > +class ConferenceChannelIO: > + server_stmt = """ > +io = SocketIO(clientsock) > +""" > + > + error = (EOFError,) > + > + def __init__(self, channel): > + self.channel = channel > + self.buffer = '' > + > + def read(self, numbytes): > + #print >> open('LOG', 'a'), 'read %d bytes' % numbytes > + while len(self.buffer) < numbytes: > + t = self.channel.receive() > + if not t: > + #print >> open('LOG', 'a'), 'EOFError' > + raise EOFError > + self.buffer += t > + buf, self.buffer = self.buffer[:numbytes], self.buffer[numbytes:] > + #print >> open('LOG', 'a'), '--->', repr(buf) > + return buf > + > + def write(self, data): > + #print >> open('LOG', 'a'), 'write(%r)' % (data,) > + self.channel.send(data) > + > + def close_read(self): > + pass > + > + def close_write(self): > + self.channel.close() > > Added: pypy/dist/pypy/tool/build/server.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/server.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,332 @@ > +import random > +import time > +import thread > +import smtplib > +import py > + > +def issubdict(d1, d2): > + """sees whether a dict is a 'subset' of another dict > + > + dictvalues can be immutable data types and list and dicts of > + immutable data types and lists and ... (recursive) > + """ > + for k, v in d1.iteritems(): > + if not k in d2: > + return False > + d2v = d2[k] > + if isinstance(v, dict): > + if not issubdict(v, d2v): > + return False > + elif isinstance(v, list): > + if not set(v).issubset(set(d2v)): > + return False > + elif v != d2v: > + return False > + return True > + > +# XXX note that all this should be made thread-safe at some point (meaning it > +# currently isn't)!! > + > +class RequestStorage(object): > + """simple registry that manages information""" > + def __init__(self, info_to_path=[]): > + self._id_to_info = {} # id -> info dict > + self._id_to_emails = {} # id -> requestor email address > + self._id_to_path = {} # id -> filepath > + > + self._last_id = 0 > + self._id_lock = thread.allocate_lock() > + > + self._build_initial(info_to_path) > + > + def request(self, email, info): > + """place a request > + > + this either returns a path to the binary (if it's available > + already) or an id for the info > + """ > + self._normalize(info) > + infoid = self.get_info_id(info) > + path = self._id_to_path.get(infoid) > + if path is not None: > + return path > + self._id_to_emails.setdefault(infoid, []).append(email) > + > + def get_info_id(self, info): > + """retrieve or create an id for an info dict""" > + self._id_lock.acquire() > + try: > + self._normalize(info) > + for k, v in self._id_to_info.iteritems(): > + if v == info: > + return k > + self._last_id += 1 > + id = self._last_id > + self._id_to_info[id] = info > + return id > + finally: > + self._id_lock.release() > + > + def add_build(self, info, path): > + """store the data for a build and make it available > + > + returns a list of email addresses for the people that should be > + warned > + """ > + self._normalize(info) > + infoid = self.get_info_id(info) > + emails = self._id_to_emails.pop(infoid) > + self._id_to_path[infoid] = path > + return emails > + > + def _build_initial(self, info_to_path): > + """fill the dicts with info about files that are already built""" > + for info, path in info_to_path: > + id = self.get_info_id(info) > + self._id_to_path[id] = path > + > + def _normalize(self, info): > + for k, v in info.iteritems(): > + if isinstance(v, list): > + v.sort() > + > +from py.__.path.local.local import LocalPath > +class BuildPath(LocalPath): > + def _info(self): > + info = getattr(self, '_info_value', {}) > + if info: > + return info > + infopath = self / 'info.txt' > + if not infopath.check(): > + return {} > + for line in infopath.readlines(): > + line = line.strip() > + if not line: > + continue > + chunks = line.split(':') > + key = chunks.pop(0) > + value = ':'.join(chunks) > + info[key] = eval(value) > + self._info_value = info > + return info > + > + def _set_info(self, info): > + self._info_value = info > + infopath = self / 'info.txt' > + infopath.ensure() > + fp = infopath.open('w') > + try: > + for key, value in info.iteritems(): > + fp.write('%s: %r\n' % (key, value)) > + finally: > + fp.close() > + > + info = property(_info, _set_info) > + > + def _zipfile(self): > + return py.path.local(self / 'data.zip') > + > + def _set_zipfile(self, iterable): > + # XXX not in use right now... > + fp = self._zipfile().open('w') > + try: > + for chunk in iterable: > + fp.write(chunk) > + finally: > + fp.close() > + > + zipfile = property(_zipfile, _set_zipfile) > + > +class PPBServer(object): > + retry_interval = 10 > + > + def __init__(self, projname, channel, builddir, mailhost=None, > + mailport=None, mailfrom=None): > + self._projname = projname > + self._channel = channel > + self._builddir = builddir > + self._mailhost = mailhost > + self._mailport = mailport > + self._mailfrom = mailfrom > + > + self._buildpath = py.path.local(builddir) > + self._clients = [] > + info_to_path = [(p.info, str(p)) for p in > + self._get_buildpaths(builddir)] > + self._requeststorage = RequestStorage(info_to_path) > + self._queued = [] > + > + self._queuelock = thread.allocate_lock() > + self._namelock = thread.allocate_lock() > + > + def register(self, client): > + self._clients.append(client) > + self._channel.send('registered %s with info %r' % ( > + client, client.sysinfo)) > + client.channel.send('welcome') > + > + def compile(self, requester_email, info): > + """start a compilation > + > + returns a tuple (ispath, data) > + > + if there's already a build available for info, this will return > + a tuple (True, path), if not, this will return (False, message), > + where message describes what is happening with the request (is > + a build made rightaway, or is there no client available?) > + > + in any case, if the first item of the tuple returned is False, > + an email will be sent once the build is available > + """ > + path = self._requeststorage.request(requester_email, info) > + if path is not None: > + self._channel.send('already a build for this info available') > + return (True, path) > + for client in self._clients: > + if client.busy_on == info: > + self._channel.send('build for %r currently in progress' % > + (info,)) > + return (False, 'this build is already in progress') > + # we don't have a build for this yet, find a client to compile it > + if self.run(info): > + return (False, 'found a suitable client, going to build') > + else: > + self._queuelock.acquire() > + try: > + self._queued.append(info) > + finally: > + self._queuelock.release() > + return (False, 'no suitable client found; your request > is queued') > + > + def run(self, info): > + """find a suitable client and run the job if possible""" > + # XXX shuffle should be replaced by something smarter obviously ;) > + clients = self._clients[:] > + random.shuffle(clients) > + rev = info.pop('revision', 'trunk') > + for client in clients: > + # popping out revision here, going to add later... the client > + # should be able to retrieve source code for any revision (so > + # it doesn't need to match a revision field in client.sysinfo) > + if client.busy_on or not issubdict(info, client.sysinfo): > + continue > + else: > + info['revision'] = rev > + self._channel.send( > + 'going to send compile job with info %r to %s' % ( > + info, client > + ) > + ) > + client.compile(info) > + return True > + info['revision'] = rev > + self._channel.send( > + 'no suitable client available for compilation with info %r' % ( > + info, > + ) > + ) > + > + def serve_forever(self): > + """this keeps the script from dying, and re-tries jobs""" > + self._channel.send('going to serve') > + while 1: > + time.sleep(self.retry_interval) > + self._cleanup_clients() > + self._try_queued() > + > + def get_new_buildpath(self, info): > + path = BuildPath(str(self._buildpath / self._create_filename())) > + path.info = info > + return path > + > + def compilation_done(self, info, path): > + """client is done with compiling and sends data""" > + self._channel.send('compilation done for %r, written to %s' % ( > + info, path)) > + emails = self._requeststorage.add_build(info, path) > + for emailaddr in emails: > + self._send_email(emailaddr, info, path) > + > + def _cleanup_clients(self): > + self._queuelock.acquire() > + try: > + clients = self._clients[:] > + for client in clients: > + if client.channel.isclosed(): > + if client.busy_on: > + self._queued.append(client.busy_on) > + self._clients.remove(client) > + finally: > + self._queuelock.release() > + > + def _try_queued(self): > + self._queuelock.acquire() > + try: > + toremove = [] > + for info in self._queued: > + if self.run(info): > + toremove.append(info) > + for info in toremove: > + self._queued.remove(info) > + finally: > + self._queuelock.release() > + > + def _get_buildpaths(self, dirpath): > + for p in py.path.local(dirpath).listdir(): > + yield BuildPath(str(p)) > + > + _i = 0 > + def _create_filename(self): > + self._namelock.acquire() > + try: > + today = time.strftime('%Y%m%d') > + buildnames = [p.basename for p in > + py.path.local(self._buildpath).listdir()] > + while True: > + name = '%s-%s-%s' % (self._projname, today, self._i) > + self._i += 1 > + if name not in buildnames: > + return name > + finally: > + self._namelock.release() > + > + def _send_email(self, addr, info, path): > + self._channel.send('going to send email to %s' % (addr,)) > + if self._mailhost is not None: > + msg = '\r\n'.join([ > + 'From: %s' % (self._mailfrom,), > + 'To: %s' % (addr,), > + 'Subject: %s compilation done' % (self._projname,), > + '', > + 'The compilation you requested is done. You can find it at', > + str(path), > + ]) > + server = smtplib.SMTP(self._mailhost, self._mailport) > + server.set_debuglevel(0) > + server.sendmail(self._mailfrom, addr, msg) > + server.quit() > + > +initcode = """ > + import sys > + sys.path += %r > + > + try: > + from pypy.tool.build.server import PPBServer > + server = PPBServer(%r, channel, %r, %r, %r, %r) > + > + # make the server available to clients as pypy.tool.build.ppbserver > + from pypy.tool import build > + build.ppbserver = server > + > + server.serve_forever() > + finally: > + channel.close() > +""" > +def init(gw, port=12321, path=[], projectname='pypy', buildpath=None, > + mailhost=None, mailport=25, mailfrom=None): > + from pypy.tool.build import execnetconference > + conference = execnetconference.conference(gw, port, True) > + channel = conference.remote_exec(initcode % (path, projectname,buildpath, > + mailhost, mailport, > + mailfrom)) > + return channel > > Added: pypy/dist/pypy/tool/build/test/fake.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/test/fake.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,54 @@ > +from pypy.tool.build.server import BuildPath > + > +class FakeChannel(object): > + def __init__(self): > + self._buffer = [] > + > + def send(self, item): > + self._buffer.append(item) > + > + def receive(self): > + return self._buffer.pop(0) > + > + def close(self): > + pass > + > + def waitclose(self): > + pass > + > +class FakeClient(object): > + def __init__(self, info): > + self.channel = FakeChannel() > + self.sysinfo = info > + self.busy_on = None > + > + def compile(self, info): > + info.pop('revision') > + for k, v in info.items(): > + self.channel.send('%s: %r' % (k, v)) > + self.channel.send(None) > + self.busy_on = info > + > +class FakeServer(object): > + def __init__(self, builddirpath): > + builddirpath.ensure(dir=True) > + self._channel = FakeChannel() > + self._builddirpath = builddirpath > + self._clients = [] > + self._done = [] > + > + def register(self, client): > + self._clients.append(client) > + > + def compilation_done(self, info, data): > + self._done.append((info, data)) > + > + i = 0 > + def get_new_buildpath(self, info): > + name = 'build-%s' % (self.i,) > + self.i += 1 > + bp = BuildPath(str(self._builddirpath / name)) > + bp.info = info > + bp.ensure(dir=1) > + return bp > + > > Added: pypy/dist/pypy/tool/build/test/path.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/test/path.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,6 @@ > +import py > + > +testpath = py.magic.autopath().dirpath() > +packagepath = testpath.dirpath() > +rootpath = packagepath.dirpath().dirpath().dirpath() > +py.std.sys.path.append(str(rootpath)) > > Added: pypy/dist/pypy/tool/build/test/test.zip > ============================================================================== > Binary file. No diff available. > > Added: pypy/dist/pypy/tool/build/test/test_client.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/test/test_client.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,43 @@ > +import path > +from pypy.tool.build import client > +import py > +import time > +from fake import FakeChannel, FakeServer > + > +class ClientForTests(client.PPBClient): > + def __init__(self, *args, **kwargs): > + super(ClientForTests, self).__init__(*args, **kwargs) > + self._done = [] > + > +def setup_module(mod): > + mod.temp = temp = py.test.ensuretemp('pypybuilder-client') > + mod.svr = svr = FakeServer(temp) > + > + import pypy.tool.build > + pypy.tool.build.ppbserver = svr > + > + mod.c1c = c1c = FakeChannel() > + mod.c1 = c1 = ClientForTests(c1c, {'foo': 1, 'bar': [1,2]}) > + svr.register(c1) > + > + mod.c2c = c2c = FakeChannel() > + mod.c2 = c2 = ClientForTests(c2c, {'foo': 2, 'bar': [2,3]}) > + svr.register(c2) > + > +def test_compile(): > + info = {'foo': 1} > + c1.compile(info) > + c1.channel.receive() > + c1.channel.send('foo bar') > + c1.channel.send(None) > + > + # meanwhile the client starts a thread that waits until there's data > + # available on its own channel, with our FakeChannel it has > data rightaway, > + # though (the channel out and in are the same, and we just sent 'info' > + # over the out one) > + time.sleep(1) > + > + done = svr._done.pop() > + > + assert done[0] == info > + assert done[1] == (temp / 'build-0') > > Added: pypy/dist/pypy/tool/build/test/test_pypybuilder.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/test/test_pypybuilder.py Wed Jul 26 > 13:18:25 2006 > @@ -0,0 +1,137 @@ > +import path > +from pypy.tool.build import client, server, execnetconference > +from pypy.tool.build import config > +import py > + > +# some functional tests (although some of the rest aren't strictly > +# unit tests either), to run use --functional as an arg to py.test > +def test_functional_1(): > + if not py.test.pypybuilder_option.functional: > + py.test.skip('skipping functional test, use --functional to run it') > + > + # XXX this one is a bit messy, it's a quick functional test forthe whole > + # system, but for instance contains time.sleep()s to make sure > all threads > + # get the time to perform tasks and such... > + > + sleep_interval = 0.3 > + > + # first initialize a server > + sgw = py.execnet.PopenGateway() > + temppath = py.test.ensuretemp('pypybuilder-functional') > + sc = server.init(sgw, port=config.port, path=config.testpath, > + buildpath=str(temppath)) > + > + # give the server some time to wake up > + py.std.time.sleep(sleep_interval) > + > + # then two clients, both with different system info > + sysinfo1 = { > + 'foo': 1, > + 'bar': [1,2], > + } > + cgw1 = py.execnet.PopenGateway() > + cc1 = client.init(cgw1, sysinfo1, port=config.port, testing=True) > + > + sysinfo2 = { > + 'foo': 2, > + 'bar': [1], > + } > + cgw2 = py.execnet.PopenGateway() > + cc2 = client.init(cgw2, sysinfo2, port=config.port, testing=True) > + > + # give the clients some time to register themselves > + py.std.time.sleep(sleep_interval) > + > + # now we're going to send some compile jobs > + code = """ > + import sys > + sys.path += %r > + > + from pypy.tool.build import ppbserver > + channel.send(ppbserver.compile(%r, %r)) > + channel.close() > + """ > + compgw = py.execnet.PopenGateway() > + compconf = execnetconference.conference(compgw, config.port) > + > + # this one should fail because there's no client found for foo = 3 > + compc = compconf.remote_exec(code % (config.testpath, 'foo1(a)bar.com', > + {'foo': 3})) > + > + # sorry... > + py.std.time.sleep(sleep_interval) > + > + ret = compc.receive() > + assert not ret[0] > + assert ret[1].find('no suitable client found') > -1 > + > + # this one should be handled by client 1 > + compc = compconf.remote_exec(code % (config.testpath, 'foo2(a)bar.com', > + {'foo': 1, 'bar': [1]})) > + > + # and another one > + py.std.time.sleep(sleep_interval) > + > + ret = compc.receive() > + assert not ret[0] > + assert ret[1].find('found a suitable client') > -1 > + > + # the messages may take a bit to arrive, too > + py.std.time.sleep(sleep_interval) > + > + # client 1 should by now have received the info to build for > + cc1.receive() # 'welcome' > + ret = cc1.receive() > + assert ret == {'foo': 1, 'bar': [1], 'revision': 'trunk'} > + > + # this should have created a package in the temp dir > + assert len(temppath.listdir()) == 1 > + > + # now we're going to satisfy the first request by adding a new client > + sysinfo3 = {'foo': 3} > + cgw3 = py.execnet.PopenGateway() > + cc3 = client.init(cgw3, sysinfo3, port=config.port, testing=True) > + > + # again a bit of waiting may be desired > + py.std.time.sleep(sleep_interval) > + > + # _try_queued() should check whether there are new clients available for > + # queued jobs > + code = """ > + import sys, time > + sys.path += %r > + > + from pypy.tool.build import ppbserver > + ppbserver._try_queued() > + # give the server some time, the clients 'compile' in threads > + time.sleep(%s) > + channel.send(ppbserver._requeststorage._id_to_emails) > + channel.close() > + """ > + compgw2 = py.execnet.PopenGateway() > + compconf2 = execnetconference.conference(compgw2, config.port) > + > + compc2 = compconf2.remote_exec(code % (config.testpath, sleep_interval)) > + > + > + # we check whether all emails are now sent, since after adding the third > + # client, and calling _try_queued(), both jobs should have beenprocessed > + ret = compc2.receive() > + assert ret.values() == [] > + > + # this should also have created another package in the temp dir > + assert len(temppath.listdir()) == 2 > + > + # some cleanup (this should all be in nested try/finallys, blegh) > + cc1.close() > + cc2.close() > + cc3.close() > + compc.close() > + compc2.close() > + sc.close() > + > + cgw1.exit() > + cgw2.exit() > + compgw.exit() > + compgw2.exit() > + sgw.exit() > > Added: pypy/dist/pypy/tool/build/test/test_request_storage.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/test/test_request_storage.py Wed Jul > 26 13:18:25 2006 > @@ -0,0 +1,64 @@ > +import path > +import py > +from pypy.tool.build.server import RequestStorage > + > +def test_request_storage(): > + s = RequestStorage() > + > + assert s._id_to_info == {} > + assert s._id_to_emails == {} > + assert s._id_to_path == {} > + > + info = {'foo': 1} > + infoid = s.get_info_id(info) > + > + path = s.request('foo(a)bar.com', info) > + assert path is None > + assert s._id_to_info == {infoid: info} > + assert s._id_to_emails == {infoid: ['foo(a)bar.com']} > + assert s._id_to_path == {} > + > + path = s.request('bar(a)bar.com', info) > + assert path is None > + assert s._id_to_info == {infoid: info} > + assert s._id_to_emails == {infoid: ['foo(a)bar.com', 'bar(a)bar.com']} > + assert s._id_to_path == {} > + > + emails = s.add_build(info, 'foobar') > + assert emails == ['foo(a)bar.com', 'bar(a)bar.com'] > + assert s._id_to_info == {infoid: info} > + assert s._id_to_emails == {} > + assert s._id_to_path == {infoid: 'foobar'} > + > + info2 = {'foo': 2, 'bar': [1,2]} > + infoid2 = s.get_info_id(info2) > + > + path = s.request('foo(a)baz.com', info2) > + assert path is None > + assert s._id_to_info == {infoid: info, infoid2: info2} > + assert s._id_to_emails == {infoid2: ['foo(a)baz.com']} > + assert s._id_to_path == {infoid: 'foobar'} > + > + emails = s.add_build(info2, 'foobaz') > + assert emails == ['foo(a)baz.com'] > + assert s._id_to_info == {infoid: info, infoid2: info2} > + assert s._id_to_emails == {} > + assert s._id_to_path == {infoid: 'foobar', infoid2: 'foobaz'} > + > + path = s.request('foo(a)qux.com', info) > + assert path == 'foobar' > + > +def test__build_initial(): > + s = RequestStorage([({'foo': 1}, 'foo'), ({'foo': 2}, 'bar'),]) > + > + id1 = s.get_info_id({'foo': 1}) > + id2 = s.get_info_id({'foo': 2}) > + > + assert s._id_to_info == {id1: {'foo': 1}, id2: {'foo': 2}} > + assert s._id_to_emails == {} > + assert s._id_to_path == {id1: 'foo', id2: 'bar'} > + > +def test__normalize(): > + s = RequestStorage() > + assert (s._normalize({'foo': ['bar', 'baz']}) == > + s._normalize({'foo': ['baz', 'bar']})) > > Added: pypy/dist/pypy/tool/build/test/test_server.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/test/test_server.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,132 @@ > +import path > +from pypy.tool.build import server > +import py > +from fake import FakeChannel, FakeClient > +from pypy.tool.build.server import RequestStorage > +from pypy.tool.build.server import BuildPath > +import time > + > +def setup_module(mod): > + mod.temppath = temppath = py.test.ensuretemp('pypybuilder-server') > + mod.svr = server.PPBServer('pypytest', FakeChannel(), str(temppath)) > + > + mod.c1 = FakeClient({'foo': 1, 'bar': [1,2]}) > + mod.svr.register(mod.c1) > + > + mod.c2 = FakeClient({'foo': 2, 'bar': [2,3]}) > + mod.svr.register(mod.c2) > + > +def test_server_issubdict(): > + from pypy.tool.build.server import issubdict > + assert issubdict({'foo': 1, 'bar': 2}, {'foo': 1, 'bar': 2, 'baz': 3}) > + assert not issubdict({'foo': 1, 'bar': 2}, {'foo': 1, 'baz': 3}) > + assert not issubdict({'foo': 1, 'bar': 3}, {'foo': 1, 'bar': 2,'baz': 3}) > + assert issubdict({'foo': [1,2]}, {'foo': [1,2,3]}) > + assert not issubdict({'foo': [1,2,3]}, {'foo': [1,2]}) > + assert issubdict({'foo': 1L}, {'foo': 1}) > + assert issubdict({}, {'foo': 1}) > + assert issubdict({'foo': [1,2]}, {'foo': [1,2,3,4], 'bar': [1,2]}) > + > +# XXX: note that the order of the tests matters! the first test reads the > +# information from the channels that was set by the setup_module() function, > +# the rest assumes this information is already read... > + > +def test_register(): > + assert len(svr._clients) == 2 > + assert svr._clients[0] == c1 > + assert svr._clients[1] == c2 > + > + assert c1.channel.receive() == 'welcome' > + assert c2.channel.receive() == 'welcome' > + py.test.raises(IndexError, "c1.channel.receive()") > + > + assert svr._channel.receive().find('registered') > -1 > + assert svr._channel.receive().find('registered') > -1 > + py.test.raises(IndexError, 'svr._channel.receive()') > + > +def test_compile(): > + # XXX this relies on the output not changing... quite scary > + info = {'foo': 1} > + ret = svr.compile('test(a)domain.com', info) > + assert not ret[0] > + assert ret[1].find('found a suitable client') > -1 > + assert svr._channel.receive().find('going to send compile job') > -1 > + assert c1.channel.receive() == 'foo: 1' > + assert c1.channel.receive() is None > + py.test.raises(IndexError, "c2.channel.receive()") > + > + svr.compile('test(a)domain.com', {'foo': 3}) > + assert svr._channel.receive().find('no suitable client available') > -1 > + > + info = {'bar': [3]} > + ret = svr.compile('test(a)domain.com', info) > + assert svr._channel.receive().find('going to send') > -1 > + assert c2.channel.receive() == 'bar: [3]' > + assert c2.channel.receive() is None > + py.test.raises(IndexError, "c1.channel.receive()") > + > + info = {'foo': 1} > + ret = svr.compile('test(a)domain.com', info) > + assert not ret[0] > + assert ret[1].find('this build is already') > -1 > + assert svr._channel.receive().find('currently in progress') > -1 > + > + c1.busy_on = None > + bp = BuildPath(str(temppath / 'foo')) > + svr.compilation_done(info, bp) > + ret = svr.compile('test(a)domain.com', info) > + assert ret[0] > + assert isinstance(ret[1], BuildPath) > + assert ret[1] == bp > + assert svr._channel.receive().find('compilation done for') > -1 > + for i in range(2): > + assert svr._channel.receive().find('going to send email to') > -1 > + assert svr._channel.receive().find('already a build for this info') > -1 > + > +def test_buildpath(): > + tempdir = py.test.ensuretemp('pypybuilder-buildpath') > + # grmbl... local.__new__ checks for class equality :( > + bp = BuildPath(str(tempdir / 'test1')) > + assert not bp.check() > + assert bp.info == {} > + > + bp.info = {'foo': 1, 'bar': [1,2]} > + assert bp.info == {'foo': 1, 'bar': [1,2]} > + assert (sorted((bp / 'info.txt').readlines()) == > + ['bar: [1, 2]\n', 'foo: 1\n']) > + > + assert isinstance(bp.zipfile, py.path.local) > + bp.zipfile = ['foo', 'bar', 'baz'] > + assert bp.zipfile.read() == 'foobarbaz' > + > +def test__create_filename(): > + svr._i = 0 # reset counter > + today = time.strftime('%Y%m%d') > + name1 = svr._create_filename() > + assert name1 == 'pypytest-%s-0' % (today,) > + assert svr._create_filename() == ('pypytest-%s-1' % (today,)) > + bp = BuildPath(str(temppath / ('pypytest-%s-2' % (today,)))) > + try: > + bp.ensure() > + assert svr._create_filename() == 'pypytest-%s-3'% (today,) > + finally: > + bp.remove() > + > +def test_get_new_buildpath(): > + svr._i = 0 > + today = time.strftime('%Y%m%d') > + > + path1 = svr.get_new_buildpath({'foo': 'bar'}) > + try: > + assert isinstance(path1, BuildPath) > + assert path1.info == {'foo': 'bar'} > + assert path1.basename == 'pypytest-%s-0' % (today,) > + > + try: > + path2 = svr.get_new_buildpath({'foo': 'baz'}) > + assert path2.info == {'foo': 'baz'} > + assert path2.basename == 'pypytest-%s-1' % (today,) > + finally: > + path2.remove() > + finally: > + path1.remove() > _______________________________________________ > pypy-svn mailing list > pypy-svn(a)codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > [View Less]

2 3

Re: [pypy-dev] about bz2 and more
by Armin Rigo July 26, 2006

July 26, 2006

Hi Lawrence, On Wed, Jul 26, 2006 at 10:50:40AM +0200, Lawrence Oluyede wrote: > Bz2 is kinda a beast module doing heavy operations with function > pointers, buffering and so on. The problem I encountered lies in the > initialization of the bz2 file object. It has support for buffering, > universal newlines and more. All this stuff is exposed in CPython via > file()/open() factory functions. Those are not supported at RPython > level and I can't write the class at app-level. … [View More]

1 0

JS backend updates
by Maciek Fijalkowski July 24, 2006

July 24, 2006

There were some updates recently about JS backend. Recently: - Added some decorator which might be usefull at declaring server-side of AJAX communication - Polished a little bit backend so now it is able to run richards and pystone as soon as you cut off zeroing assumption out of ll_alloc_and_set - Added some documentation - Bugfixes and some helper functions (ie StringBuilder) so now "aaa".upper() works as expected Coming work: - Hiding browser differencies - more examples & docs - … [View More]

2 1

Example of .NET filehandles
by Ben.Young＠risk.sungard.com July 24, 2006

July 24, 2006

Hi Antonio, Here's the example I mentioned. I don't think it's particularly safe, so it may be better to keep a map of ints to filestreams that you have opened, and use that if possible, and fall back to this if you can't. Cheers, Ben using System; using System.Collections.Generic; using System.Text; namespace posix { class Program { static void Main(string[] args) { System.IO.FileStream stream1 = new System.IO.FileStream (args[0], System.IO.FileMode.… [View More]

1 0

Newb questions about ext module writing
by Scott Dial July 24, 2006

July 24, 2006

Greetings everyone, I'm relatively new to PyPy. I've been aware of it for a while, but never really got into poking around at all. But recently I decided this is a project that I would like to get involved in. I think at the time I planned on doing something big, like addding a JVM backend or something like that.. maybe I'll get back to that eventually. I realized I was fairly clueless with the codebase and decided to jump on writing a module as a type of introductory project. The … [View More]

4 7

20th Level IP Communicator?
by Michael Spencer July 22, 2006

July 22, 2006

http://gigaom.com/2006/07/15/sipphone-brings-voice-to-nokia-770-tablet/#com… I thought you'd like this comment: "looks like i’m about to fail my saving throw against “cool gadget temptation”."

1 1

.NET build failures
by Ben.Young＠risk.sungard.com July 21, 2006

July 21, 2006

Hi Antocuni, I ran the .NET build last night and it got a lot further. The errors I get now are: [translation:ERROR] ***** FAILURE ***** [translation:ERROR] [translation:ERROR] c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(66076) : error -- Duplicate label: '__check_block_5' [translation:ERROR] c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(372039) : error -- Duplicate label: '__check_block_5' [translation:ERROR] c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(… [View More]

2 2

PyPy sprint Limerick 21-27th of August
by Beatrice During July 21, 2006

July 21, 2006

Hi there The PyPy team goes sprinting in Limerick/Ireland - are you interested in participating? Ireland PyPy sprint 21th-27th August 2006 ==================================================== The next PyPy sprint will happen in the nice city of Limerick in Ireland from 21st till 27th August. (Most people intend to arrive 20th August). The main focus of the sprint will be on JIT compiler works, various optimization works, porting extension modules, infrastructure works like a build tool … [View More]

1 0