[pypy-dev] [pypy-svn] r30565 - in pypy/dist/pypy/tool/build: . bin builds test
Ben.Young at risk.sungard.com
Ben.Young at risk.sungard.com
Wed Jul 26 13:34:29 CEST 2006
Hi Guido,
Cool tool! Is this something I would be able to run just in the
evenings/weekends etc? Will it pick up that fact that I don't have boehm
automatically?
Cheers,
Ben
pypy-svn-bounces at codespeak.net wrote on 26/07/2006 12:18:51:
> Author: guido
> Date: Wed Jul 26 13:18:25 2006
> New Revision: 30565
>
> Added:
> pypy/dist/pypy/tool/build/ (props changed)
> pypy/dist/pypy/tool/build/README.txt (contents, props changed)
> pypy/dist/pypy/tool/build/__init__.py (contents, props changed)
> pypy/dist/pypy/tool/build/bin/ (props changed)
> pypy/dist/pypy/tool/build/bin/client (contents, props changed)
> pypy/dist/pypy/tool/build/bin/path.py (contents, props changed)
> pypy/dist/pypy/tool/build/bin/server (contents, props changed)
> pypy/dist/pypy/tool/build/bin/startcompile (contents, props
changed)
> pypy/dist/pypy/tool/build/builds/
> pypy/dist/pypy/tool/build/client.py (contents, props changed)
> pypy/dist/pypy/tool/build/config.py (contents, props changed)
> pypy/dist/pypy/tool/build/conftest.py (contents, props changed)
> pypy/dist/pypy/tool/build/execnetconference.py (contents, props
changed)
> pypy/dist/pypy/tool/build/server.py (contents, props changed)
> pypy/dist/pypy/tool/build/test/ (props changed)
> pypy/dist/pypy/tool/build/test/fake.py (contents, props changed)
> pypy/dist/pypy/tool/build/test/path.py (contents, props changed)
> pypy/dist/pypy/tool/build/test/test.zip (contents, props changed)
> pypy/dist/pypy/tool/build/test/test_client.py (contents, props
changed)
> pypy/dist/pypy/tool/build/test/test_pypybuilder.py (contents,
> props changed)
> pypy/dist/pypy/tool/build/test/test_request_storage.py
> (contents, props changed)
> pypy/dist/pypy/tool/build/test/test_server.py (contents, props
changed)
> Log:
> Added 'pypybuilder', a tool to build a 'build farm' from
> participating clients,
> the clients register to a server with information about what they can
build,
> then the server waits for build requests and dispatches to clients.
Clients
> send back a build when they're done, on which the server sends out
emails to
> whoever is waiting for the build. When a build is already available, the
> requestor is provided with a path (will be URL in the future) to the
build,
> if no client is available for a certain request the request is queued
until
> there is one.
> Worked on this from https://merlinux.de/svn/user/guido/pypybuilder
before
> checking in here.
>
>
> Added: pypy/dist/pypy/tool/build/README.txt
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/README.txt Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,59 @@
> +============
> +PyPyBuilder
> +============
> +
> +What is this?
> +=============
> +
> +PyPyBuilder is an application that allows people to build PyPy
instances on
> +demand. If you have a nice idle machine connected to the Internet, and
don't
> +mind us 'borrowing' it every once in a while, you can start up the
client
> +script (in bin/client) and have the server send compile jobs to your
machine.
> +If someone requests a build of PyPy that is not already available on
the PyPy
> +website, and your machine is capable of making such a build, the
> server may ask
> +your machine to create it. If enough people participate, with diverse
enough
> +machines, an ad-hoc 'build farm' is created this way.
> +
> +Components
> +==========
> +
> +The application consists of 3 main components: a server component,
> a client and
> +a small component to start compile jobs, which we'll call
'startcompile' for
> +now.
> +
> +The server waits for clients to register, and for compile job requests.
When
> +clients register, they pass the server information about what
> compilations they
> +can handle (system info). Then when the 'startcompile' component
requests a
> +compilation job, the server first checks whether a binary is
> already available,
> +and if so returns that.
> +
> +If there isn't one, the server walks through a list of connected
> clients to see
> +if there is one that can handle the job, and if so tells it to perform
it. If
> +there's no client to handle the job, it gets queued until there is.
> +
> +Once a build is available, the server will send an email to all
> email addresses
> +(it could be more than one person asked for the same build at the same
time!)
> +passed to it by 'startcompile'.
> +
> +Installation
> +============
> +
> +Client
> +------
> +
> +Installing the system should not be required: just run '.
> /bin/client' to start
> +the client. Note that it depends on the `py lib`_.
> +
> +Server
> +------
> +
> +Also for the server there's no real setup required, and again there's a
> +dependency on the `py lib`_.
> +
> +.. _`py lib`: http://codespeak.net/py
> +
> +More info
> +=========
> +
> +For more information, bug reports, patches, etc., please send an email
to
> +guido at merlinux.de.
>
> Added: pypy/dist/pypy/tool/build/__init__.py
>
==============================================================================
>
> Added: pypy/dist/pypy/tool/build/bin/client
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/bin/client Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,58 @@
> +#!/usr/bin/python
> +
> +BUFSIZE = 1024
> +
> +import path
> +import sys
> +import random
> +from pypy.tool.build import config
> +
> +# XXX using random values for testing
> +modules = ['_stackless', '_socket']
> +
> +"""
> +random.shuffle(modules)
> +sysinfo = {
> + 'maxint': random.choice((sys.maxint, (2 ** 63 - 1))),
> + 'use_modules': modules[:random.randrange(len(modules) + 1)],
> + 'byteorder': random.choice(('little', 'big')),
> +}
> +"""
> +
> +sysinfo = {
> + 'maxint': sys.maxint,
> + 'use_modules': ['_stackless', '_socket'],
> + 'byteorder': sys.byteorder,
> +}
> +
> +if __name__ == '__main__':
> + from py.execnet import SshGateway
> + from pypy.tool.build.client import init
> + gw = SshGateway(config.server)
> + channel = init(gw, sysinfo, path=config.path, port=config.port)
> + print channel.receive() # welcome message
> + try:
> + while 1:
> + data = channel.receive()
> + if not isinstance(data, dict): # needs more checks here
> + raise ValueError(
> + 'received wrong unexpected data of type %s' %
> (type(data),)
> + )
> + info = data
> + # XXX we should compile here, using data dict for info
> + print 'compilation requested for info %r, now faking
> that' % (info,)
> + import time; time.sleep(10)
> +
> + # write the zip to the server in chunks to server
> + # XXX we're still faking this
> + zipfp = (path.packagedir / 'test/test.zip').open()
> + while True:
> + chunk = zipfp.read(BUFSIZE)
> + if not chunk:
> + break
> + channel.send(chunk)
> + channel.send(None) # tell the server we're done
> + print 'done with compilation, waiting for next'
> + finally:
> + channel.close()
> + gw.exit()
>
> Added: pypy/dist/pypy/tool/build/bin/path.py
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/bin/path.py Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,5 @@
> +import py
> +
> +packagedir = py.magic.autopath().dirpath().dirpath()
> +rootpath = packagedir.dirpath().dirpath().dirpath()
> +py.std.sys.path.append(str(rootpath))
>
> Added: pypy/dist/pypy/tool/build/bin/server
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/bin/server Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,27 @@
> +#!/usr/bin/python
> +
> +import path
> +from pypy.tool.build import config
> +
> +from py.execnet import SshGateway
> +
> +if __name__ == '__main__':
> + from py.execnet import SshGateway
> + from pypy.tool.build.server import init
> +
> + gw = SshGateway(config.server)
> + channel = init(gw, port=config.port, path=config.path,
> + projectname=config.projectname,
> + buildpath=str(config.buildpath),
> + mailhost=config.mailhost,
> + mailport=config.mailport,
> + mailfrom=config.mailfrom)
> +
> + try:
> + while 1:
> + data = channel.receive()
> + assert isinstance(data, str)
> + print data
> + finally:
> + channel.close()
> + gw.exit()
>
> Added: pypy/dist/pypy/tool/build/bin/startcompile
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/bin/startcompile Wed Jul 26 13:18:25
2006
> @@ -0,0 +1,57 @@
> +#!/usr/bin/python
> +
> +import path
> +import sys
> +import random
> +from pypy.tool.build import config
> +
> +initcode = """
> + import sys
> + sys.path += %r
> +
> + from pypy.tool.build import ppbserver
> + channel.send(ppbserver.compile(%r, %r))
> + channel.close()
> +"""
> +def init(gw, sysinfo, email, port=12321):
> + from pypy.tool.build import execnetconference
> +
> + conference = execnetconference.conference(gw, port, False)
> + channel = conference.remote_exec(initcode % (config.path,
> email, sysinfo))
> + return channel
> +
> +if __name__ == '__main__':
> + from py.execnet import SshGateway
> +
> + from optparse import OptionParser
> + optparser = OptionParser('%prog [options] email')
> + for args, kwargs in config.options:
> + optparser.add_option(*args, **kwargs)
> + optparser.add_option('-r', '--revision', dest='revision',
> default='trunk',
> + help='SVN revision (defaults to "trunk")')
> +
> + (options, args) = optparser.parse_args()
> +
> + if not args or len(args) != 1:
> + optparser.error('please provide an email address')
> +
> + sysinfo = dict([(attr, getattr(options, attr)) for attr in
> dir(options) if
> + not attr.startswith('_') and
> + not callable(getattr(options, attr))])
> +
> + print 'going to start compile job with info:'
> + for k, v in sysinfo.items():
> + print '%s: %r' % (k, v)
> + print
> +
> + gw = SshGateway(config.server)
> + channel = init(gw, sysinfo, args[0], port=config.port)
> + ispath, data = channel.receive()
> + if ispath:
> + print ('a suitable result is already available, you can find it
'
> + 'at "%s"' % (data,))
> + else:
> + print data
> + print 'you will be mailed once it\'s ready'
> + channel.close()
> + gw.exit()
>
> Added: pypy/dist/pypy/tool/build/client.py
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/client.py Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,71 @@
> +import time
> +import thread
> +
> +class PPBClient(object):
> + def __init__(self, channel, sysinfo, testing=False):
> + self.channel = channel
> + self.sysinfo = sysinfo
> + self.busy_on = None
> + self.testing = testing
> +
> + from pypy.tool.build import ppbserver
> + self.server = ppbserver
> + self.server.register(self)
> +
> + def sit_and_wait(self):
> + """connect to the host and wait for commands"""
> + self.channel.waitclose()
> + self.channel.close()
> +
> + def compile(self, info):
> + """send a compile job to the client side
> +
> + this waits until the client is done, and assumes the client
sends
> + back the whole binary as a single string (XXX this
> should change ;)
> + """
> + self.busy_on = info
> + self.channel.send(info)
> + thread.start_new_thread(self.wait_until_done, (info,))
> +
> + def wait_until_done(self, info):
> + buildpath = self.server.get_new_buildpath(info)
> +
> + fp = buildpath.zipfile.open('w')
> + if not self.testing:
> + try:
> + while True:
> + try:
> + chunk = self.channel.receive()
> + except EOFError:
> + # stop compilation, client has disconnected
> + return
> + if chunk is None:
> + break
> + fp.write(chunk)
> + finally:
> + fp.close()
> +
> + self.server.compilation_done(info, buildpath)
> + self.busy_on = None
> +
> +initcode = """
> + import sys
> + sys.path += %r
> +
> + from pypy.tool.build.client import PPBClient
> +
> + try:
> + client = PPBClient(channel, %r, %r)
> + client.sit_and_wait()
> + finally:
> + channel.close()
> +"""
> +def init(gw, sysinfo, path=None, port=12321, testing=False):
> + from pypy.tool.build import execnetconference
> +
> + if path is None:
> + path = []
> +
> + conference = execnetconference.conference(gw, port, False)
> + channel = conference.remote_exec(initcode % (path, sysinfo,
testing))
> + return channel
>
> Added: pypy/dist/pypy/tool/build/config.py
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/config.py Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,45 @@
> +import py
> +
> +# general settings, used by both server and client
> +server = 'johnnydebris.net'
> +port = 12321
> +path = ['/home/johnny/temp/pypy-dist']
> +
> +# option definitions for the startcompile script
> +# for now we have them here, we should probably use pypy's config
instead
> +# though...
> +import sys
> +def _use_modules_callback(option, opt_str, value, parser):
> + parser.values.use_modules = [m.strip() for m in value.split(',')
> + if m.strip()]
> +
> +def _maxint_callback(option, opt_str, value, parser):
> + parser.values.maxint = 2 ** (int(value) - 1) - 1
> +
> +options = [
> + (('-m', '--use-modules'), {'action': 'callback', 'type': 'string',
> + 'callback': _use_modules_callback,
> + 'dest': 'use_modules', 'default': [],
> + 'help': 'select the modules you
> want to use'}),
> + (('-i', '--maxint'), {'action': 'callback', 'callback':
_maxint_callback,
> + 'default': sys.maxint, 'dest':
'maxint',
> + 'type': 'string',
> + 'help': ('size of an int in bits
(32/64, '
> + 'defaults to
sys.maxint)')}),
> + (('-b', '--byteorder'), {'action': 'store',
> + 'dest': 'byteorder', 'default':
> sys.byteorder,
> + 'nargs': 1,
> + 'help': ('byte order (little/big,
defaults '
> + 'to sys.byteorder)')}),
> +]
> +
> +# settings for the server
> +projectname = 'pypy'
> +buildpath = '/home/johnny/temp/pypy-dist/pypy/tool/build/builds'
> +mailhost = '127.0.0.1'
> +mailport = 25
> +mailfrom = 'johnny at johnnydebris.net'
> +
> +# settings for the tests
> +testpath = [str(py.magic.autopath().dirpath().dirpath())]
> +
>
> Added: pypy/dist/pypy/tool/build/conftest.py
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/conftest.py Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,20 @@
> +import py
> +from py.__.documentation.conftest import Directory as Dir, DoctestText,
\
> + ReSTChecker
> +mypath = py.magic.autopath().dirpath()
> +
> +Option = py.test.Config.Option
> +option = py.test.Config.addoptions("pypybuilder test options",
> + Option('', '--functional',
> + action="store_true", dest="functional", default=False,
> + help="run pypybuilder functional tests"
> + ),
> +)
> +
> +py.test.pypybuilder_option = option
> +
> +class Directory(Dir):
> + def run(self):
> + if self.fspath == mypath:
> + return ['README.txt', 'test']
> + return super(Directory, self).run()
>
> Added: pypy/dist/pypy/tool/build/execnetconference.py
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/execnetconference.py Wed Jul 26 13:18:25
2006
> @@ -0,0 +1,126 @@
> +"""
> +An extension to py.execnet to allow multiple programs to exchange
information
> +via a common server. The idea is that all programs first open a
gateway to
> +the same server (e.g. an SshGateway), and then call the conference()
function
> +with a local TCP port number. The first program must pass
is_server=True and
> +the next ones is_server=False: the first program's remote gateway is
used as
> +shared server for the next ones.
> +
> +For all programs, the conference() call returns a new gateway that is
> +connected to the Python process of this shared server. Information can
> +be exchanged by passing data around within this Python process.
> +"""
> +import py
> +from py.__.execnet.register import InstallableGateway
> +
> +
> +def conference(gateway, port, is_server='auto'):
> + if is_server: # True or 'auto'
> + channel = gateway.remote_exec(r"""
> + import thread
> + from socket import *
> + s = socket(AF_INET, SOCK_STREAM)
> + port = channel.receive()
> + try:
> + s.bind(('', port))
> + s.listen(5)
> + except error:
> + channel.send(0)
> + else:
> + channel.send(1)
> +
> + def readall(s, n):
> + result = ''
> + while len(result) < n:
> + t = s.read(n-len(result))
> + if not t:
> + raise EOFError
> + result += t
> + return result
> +
> + def handle_connexion(clientsock, address):
> + clientfile = clientsock.makefile('r+b',0)
> + source = clientfile.readline().rstrip()
> + clientfile.close()
> + g = {'clientsock' : clientsock, 'address' :
address}
> + source = eval(source)
> + if source:
> + g = {'clientsock' : clientsock, 'address' :
address}
> + co = compile(source+'\n', source, 'exec')
> + exec co in g
> +
> + while True:
> + conn, addr = s.accept()
> + if addr[0] == '127.0.0.1': # else connexion
refused
> + thread.start_new_thread(handle_connexion,
> (conn, addr))
> + del conn
> + """)
> + channel.send(port)
> + ok = channel.receive()
> + if ok:
> + return gateway
> + if is_server == 'auto':
> + pass # fall-through and try as a client
> + else:
> + raise IOError("cannot listen on port %d (already in
> use?)" % port)
> +
> + if 1: # client
> + channel = gateway.remote_exec(r"""
> + import thread
> + from socket import *
> + s = socket(AF_INET, SOCK_STREAM)
> + port = channel.receive()
> + s.connect(('', port))
> + channel.send(1)
> + def receiver(s, channel):
> + while True:
> + data = s.recv(4096)
> + #print >> open('LOG','a'), 'backward', repr(data)
> + channel.send(data)
> + if not data: break
> + thread.start_new_thread(receiver, (s, channel))
> + try:
> + for data in channel:
> + #print >> open('LOG','a'), 'forward', repr(data)
> + s.sendall(data)
> + finally:
> + s.shutdown(1)
> + """)
> + channel.send(port)
> + ok = channel.receive()
> + assert ok
> + return InstallableGateway(ConferenceChannelIO(channel))
> +
> +
> +class ConferenceChannelIO:
> + server_stmt = """
> +io = SocketIO(clientsock)
> +"""
> +
> + error = (EOFError,)
> +
> + def __init__(self, channel):
> + self.channel = channel
> + self.buffer = ''
> +
> + def read(self, numbytes):
> + #print >> open('LOG', 'a'), 'read %d bytes' % numbytes
> + while len(self.buffer) < numbytes:
> + t = self.channel.receive()
> + if not t:
> + #print >> open('LOG', 'a'), 'EOFError'
> + raise EOFError
> + self.buffer += t
> + buf, self.buffer = self.buffer[:numbytes],
self.buffer[numbytes:]
> + #print >> open('LOG', 'a'), '--->', repr(buf)
> + return buf
> +
> + def write(self, data):
> + #print >> open('LOG', 'a'), 'write(%r)' % (data,)
> + self.channel.send(data)
> +
> + def close_read(self):
> + pass
> +
> + def close_write(self):
> + self.channel.close()
>
> Added: pypy/dist/pypy/tool/build/server.py
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/server.py Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,332 @@
> +import random
> +import time
> +import thread
> +import smtplib
> +import py
> +
> +def issubdict(d1, d2):
> + """sees whether a dict is a 'subset' of another dict
> +
> + dictvalues can be immutable data types and list and dicts of
> + immutable data types and lists and ... (recursive)
> + """
> + for k, v in d1.iteritems():
> + if not k in d2:
> + return False
> + d2v = d2[k]
> + if isinstance(v, dict):
> + if not issubdict(v, d2v):
> + return False
> + elif isinstance(v, list):
> + if not set(v).issubset(set(d2v)):
> + return False
> + elif v != d2v:
> + return False
> + return True
> +
> +# XXX note that all this should be made thread-safe at some point
(meaning it
> +# currently isn't)!!
> +
> +class RequestStorage(object):
> + """simple registry that manages information"""
> + def __init__(self, info_to_path=[]):
> + self._id_to_info = {} # id -> info dict
> + self._id_to_emails = {} # id -> requestor email address
> + self._id_to_path = {} # id -> filepath
> +
> + self._last_id = 0
> + self._id_lock = thread.allocate_lock()
> +
> + self._build_initial(info_to_path)
> +
> + def request(self, email, info):
> + """place a request
> +
> + this either returns a path to the binary (if it's available
> + already) or an id for the info
> + """
> + self._normalize(info)
> + infoid = self.get_info_id(info)
> + path = self._id_to_path.get(infoid)
> + if path is not None:
> + return path
> + self._id_to_emails.setdefault(infoid, []).append(email)
> +
> + def get_info_id(self, info):
> + """retrieve or create an id for an info dict"""
> + self._id_lock.acquire()
> + try:
> + self._normalize(info)
> + for k, v in self._id_to_info.iteritems():
> + if v == info:
> + return k
> + self._last_id += 1
> + id = self._last_id
> + self._id_to_info[id] = info
> + return id
> + finally:
> + self._id_lock.release()
> +
> + def add_build(self, info, path):
> + """store the data for a build and make it available
> +
> + returns a list of email addresses for the people that
should be
> + warned
> + """
> + self._normalize(info)
> + infoid = self.get_info_id(info)
> + emails = self._id_to_emails.pop(infoid)
> + self._id_to_path[infoid] = path
> + return emails
> +
> + def _build_initial(self, info_to_path):
> + """fill the dicts with info about files that are already
built"""
> + for info, path in info_to_path:
> + id = self.get_info_id(info)
> + self._id_to_path[id] = path
> +
> + def _normalize(self, info):
> + for k, v in info.iteritems():
> + if isinstance(v, list):
> + v.sort()
> +
> +from py.__.path.local.local import LocalPath
> +class BuildPath(LocalPath):
> + def _info(self):
> + info = getattr(self, '_info_value', {})
> + if info:
> + return info
> + infopath = self / 'info.txt'
> + if not infopath.check():
> + return {}
> + for line in infopath.readlines():
> + line = line.strip()
> + if not line:
> + continue
> + chunks = line.split(':')
> + key = chunks.pop(0)
> + value = ':'.join(chunks)
> + info[key] = eval(value)
> + self._info_value = info
> + return info
> +
> + def _set_info(self, info):
> + self._info_value = info
> + infopath = self / 'info.txt'
> + infopath.ensure()
> + fp = infopath.open('w')
> + try:
> + for key, value in info.iteritems():
> + fp.write('%s: %r\n' % (key, value))
> + finally:
> + fp.close()
> +
> + info = property(_info, _set_info)
> +
> + def _zipfile(self):
> + return py.path.local(self / 'data.zip')
> +
> + def _set_zipfile(self, iterable):
> + # XXX not in use right now...
> + fp = self._zipfile().open('w')
> + try:
> + for chunk in iterable:
> + fp.write(chunk)
> + finally:
> + fp.close()
> +
> + zipfile = property(_zipfile, _set_zipfile)
> +
> +class PPBServer(object):
> + retry_interval = 10
> +
> + def __init__(self, projname, channel, builddir, mailhost=None,
> + mailport=None, mailfrom=None):
> + self._projname = projname
> + self._channel = channel
> + self._builddir = builddir
> + self._mailhost = mailhost
> + self._mailport = mailport
> + self._mailfrom = mailfrom
> +
> + self._buildpath = py.path.local(builddir)
> + self._clients = []
> + info_to_path = [(p.info, str(p)) for p in
> + self._get_buildpaths(builddir)]
> + self._requeststorage = RequestStorage(info_to_path)
> + self._queued = []
> +
> + self._queuelock = thread.allocate_lock()
> + self._namelock = thread.allocate_lock()
> +
> + def register(self, client):
> + self._clients.append(client)
> + self._channel.send('registered %s with info %r' % (
> + client, client.sysinfo))
> + client.channel.send('welcome')
> +
> + def compile(self, requester_email, info):
> + """start a compilation
> +
> + returns a tuple (ispath, data)
> +
> + if there's already a build available for info, this will
return
> + a tuple (True, path), if not, this will return (False,
message),
> + where message describes what is happening with the request
(is
> + a build made rightaway, or is there no client available?)
> +
> + in any case, if the first item of the tuple returned is
False,
> + an email will be sent once the build is available
> + """
> + path = self._requeststorage.request(requester_email, info)
> + if path is not None:
> + self._channel.send('already a build for this info
available')
> + return (True, path)
> + for client in self._clients:
> + if client.busy_on == info:
> + self._channel.send('build for %r currently in progress'
%
> + (info,))
> + return (False, 'this build is already in progress')
> + # we don't have a build for this yet, find a client to compile
it
> + if self.run(info):
> + return (False, 'found a suitable client, going to build')
> + else:
> + self._queuelock.acquire()
> + try:
> + self._queued.append(info)
> + finally:
> + self._queuelock.release()
> + return (False, 'no suitable client found; your request
> is queued')
> +
> + def run(self, info):
> + """find a suitable client and run the job if possible"""
> + # XXX shuffle should be replaced by something smarter obviously
;)
> + clients = self._clients[:]
> + random.shuffle(clients)
> + rev = info.pop('revision', 'trunk')
> + for client in clients:
> + # popping out revision here, going to add later... the
client
> + # should be able to retrieve source code for any revision
(so
> + # it doesn't need to match a revision field in
client.sysinfo)
> + if client.busy_on or not issubdict(info, client.sysinfo):
> + continue
> + else:
> + info['revision'] = rev
> + self._channel.send(
> + 'going to send compile job with info %r to %s' % (
> + info, client
> + )
> + )
> + client.compile(info)
> + return True
> + info['revision'] = rev
> + self._channel.send(
> + 'no suitable client available for compilation with info %r'
% (
> + info,
> + )
> + )
> +
> + def serve_forever(self):
> + """this keeps the script from dying, and re-tries jobs"""
> + self._channel.send('going to serve')
> + while 1:
> + time.sleep(self.retry_interval)
> + self._cleanup_clients()
> + self._try_queued()
> +
> + def get_new_buildpath(self, info):
> + path = BuildPath(str(self._buildpath /
self._create_filename()))
> + path.info = info
> + return path
> +
> + def compilation_done(self, info, path):
> + """client is done with compiling and sends data"""
> + self._channel.send('compilation done for %r, written to %s' % (
> + info,
path))
> + emails = self._requeststorage.add_build(info, path)
> + for emailaddr in emails:
> + self._send_email(emailaddr, info, path)
> +
> + def _cleanup_clients(self):
> + self._queuelock.acquire()
> + try:
> + clients = self._clients[:]
> + for client in clients:
> + if client.channel.isclosed():
> + if client.busy_on:
> + self._queued.append(client.busy_on)
> + self._clients.remove(client)
> + finally:
> + self._queuelock.release()
> +
> + def _try_queued(self):
> + self._queuelock.acquire()
> + try:
> + toremove = []
> + for info in self._queued:
> + if self.run(info):
> + toremove.append(info)
> + for info in toremove:
> + self._queued.remove(info)
> + finally:
> + self._queuelock.release()
> +
> + def _get_buildpaths(self, dirpath):
> + for p in py.path.local(dirpath).listdir():
> + yield BuildPath(str(p))
> +
> + _i = 0
> + def _create_filename(self):
> + self._namelock.acquire()
> + try:
> + today = time.strftime('%Y%m%d')
> + buildnames = [p.basename for p in
> + py.path.local(self._buildpath).listdir()]
> + while True:
> + name = '%s-%s-%s' % (self._projname, today, self._i)
> + self._i += 1
> + if name not in buildnames:
> + return name
> + finally:
> + self._namelock.release()
> +
> + def _send_email(self, addr, info, path):
> + self._channel.send('going to send email to %s' % (addr,))
> + if self._mailhost is not None:
> + msg = '\r\n'.join([
> + 'From: %s' % (self._mailfrom,),
> + 'To: %s' % (addr,),
> + 'Subject: %s compilation done' % (self._projname,),
> + '',
> + 'The compilation you requested is done. You can find it
at',
> + str(path),
> + ])
> + server = smtplib.SMTP(self._mailhost, self._mailport)
> + server.set_debuglevel(0)
> + server.sendmail(self._mailfrom, addr, msg)
> + server.quit()
> +
> +initcode = """
> + import sys
> + sys.path += %r
> +
> + try:
> + from pypy.tool.build.server import PPBServer
> + server = PPBServer(%r, channel, %r, %r, %r, %r)
> +
> + # make the server available to clients as
pypy.tool.build.ppbserver
> + from pypy.tool import build
> + build.ppbserver = server
> +
> + server.serve_forever()
> + finally:
> + channel.close()
> +"""
> +def init(gw, port=12321, path=[], projectname='pypy', buildpath=None,
> + mailhost=None, mailport=25, mailfrom=None):
> + from pypy.tool.build import execnetconference
> + conference = execnetconference.conference(gw, port, True)
> + channel = conference.remote_exec(initcode % (path,
projectname,buildpath,
> + mailhost, mailport,
> + mailfrom))
> + return channel
>
> Added: pypy/dist/pypy/tool/build/test/fake.py
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/test/fake.py Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,54 @@
> +from pypy.tool.build.server import BuildPath
> +
> +class FakeChannel(object):
> + def __init__(self):
> + self._buffer = []
> +
> + def send(self, item):
> + self._buffer.append(item)
> +
> + def receive(self):
> + return self._buffer.pop(0)
> +
> + def close(self):
> + pass
> +
> + def waitclose(self):
> + pass
> +
> +class FakeClient(object):
> + def __init__(self, info):
> + self.channel = FakeChannel()
> + self.sysinfo = info
> + self.busy_on = None
> +
> + def compile(self, info):
> + info.pop('revision')
> + for k, v in info.items():
> + self.channel.send('%s: %r' % (k, v))
> + self.channel.send(None)
> + self.busy_on = info
> +
> +class FakeServer(object):
> + def __init__(self, builddirpath):
> + builddirpath.ensure(dir=True)
> + self._channel = FakeChannel()
> + self._builddirpath = builddirpath
> + self._clients = []
> + self._done = []
> +
> + def register(self, client):
> + self._clients.append(client)
> +
> + def compilation_done(self, info, data):
> + self._done.append((info, data))
> +
> + i = 0
> + def get_new_buildpath(self, info):
> + name = 'build-%s' % (self.i,)
> + self.i += 1
> + bp = BuildPath(str(self._builddirpath / name))
> + bp.info = info
> + bp.ensure(dir=1)
> + return bp
> +
>
> Added: pypy/dist/pypy/tool/build/test/path.py
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/test/path.py Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,6 @@
> +import py
> +
> +testpath = py.magic.autopath().dirpath()
> +packagepath = testpath.dirpath()
> +rootpath = packagepath.dirpath().dirpath().dirpath()
> +py.std.sys.path.append(str(rootpath))
>
> Added: pypy/dist/pypy/tool/build/test/test.zip
>
==============================================================================
> Binary file. No diff available.
>
> Added: pypy/dist/pypy/tool/build/test/test_client.py
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/test/test_client.py Wed Jul 26 13:18:25
2006
> @@ -0,0 +1,43 @@
> +import path
> +from pypy.tool.build import client
> +import py
> +import time
> +from fake import FakeChannel, FakeServer
> +
> +class ClientForTests(client.PPBClient):
> + def __init__(self, *args, **kwargs):
> + super(ClientForTests, self).__init__(*args, **kwargs)
> + self._done = []
> +
> +def setup_module(mod):
> + mod.temp = temp = py.test.ensuretemp('pypybuilder-client')
> + mod.svr = svr = FakeServer(temp)
> +
> + import pypy.tool.build
> + pypy.tool.build.ppbserver = svr
> +
> + mod.c1c = c1c = FakeChannel()
> + mod.c1 = c1 = ClientForTests(c1c, {'foo': 1, 'bar': [1,2]})
> + svr.register(c1)
> +
> + mod.c2c = c2c = FakeChannel()
> + mod.c2 = c2 = ClientForTests(c2c, {'foo': 2, 'bar': [2,3]})
> + svr.register(c2)
> +
> +def test_compile():
> + info = {'foo': 1}
> + c1.compile(info)
> + c1.channel.receive()
> + c1.channel.send('foo bar')
> + c1.channel.send(None)
> +
> + # meanwhile the client starts a thread that waits until there's
data
> + # available on its own channel, with our FakeChannel it has
> data rightaway,
> + # though (the channel out and in are the same, and we just sent
'info'
> + # over the out one)
> + time.sleep(1)
> +
> + done = svr._done.pop()
> +
> + assert done[0] == info
> + assert done[1] == (temp / 'build-0')
>
> Added: pypy/dist/pypy/tool/build/test/test_pypybuilder.py
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/test/test_pypybuilder.py Wed Jul 26
> 13:18:25 2006
> @@ -0,0 +1,137 @@
> +import path
> +from pypy.tool.build import client, server, execnetconference
> +from pypy.tool.build import config
> +import py
> +
> +# some functional tests (although some of the rest aren't strictly
> +# unit tests either), to run use --functional as an arg to py.test
> +def test_functional_1():
> + if not py.test.pypybuilder_option.functional:
> + py.test.skip('skipping functional test, use --functional to run
it')
> +
> + # XXX this one is a bit messy, it's a quick functional test forthe
whole
> + # system, but for instance contains time.sleep()s to make sure
> all threads
> + # get the time to perform tasks and such...
> +
> + sleep_interval = 0.3
> +
> + # first initialize a server
> + sgw = py.execnet.PopenGateway()
> + temppath = py.test.ensuretemp('pypybuilder-functional')
> + sc = server.init(sgw, port=config.port, path=config.testpath,
> + buildpath=str(temppath))
> +
> + # give the server some time to wake up
> + py.std.time.sleep(sleep_interval)
> +
> + # then two clients, both with different system info
> + sysinfo1 = {
> + 'foo': 1,
> + 'bar': [1,2],
> + }
> + cgw1 = py.execnet.PopenGateway()
> + cc1 = client.init(cgw1, sysinfo1, port=config.port, testing=True)
> +
> + sysinfo2 = {
> + 'foo': 2,
> + 'bar': [1],
> + }
> + cgw2 = py.execnet.PopenGateway()
> + cc2 = client.init(cgw2, sysinfo2, port=config.port, testing=True)
> +
> + # give the clients some time to register themselves
> + py.std.time.sleep(sleep_interval)
> +
> + # now we're going to send some compile jobs
> + code = """
> + import sys
> + sys.path += %r
> +
> + from pypy.tool.build import ppbserver
> + channel.send(ppbserver.compile(%r, %r))
> + channel.close()
> + """
> + compgw = py.execnet.PopenGateway()
> + compconf = execnetconference.conference(compgw, config.port)
> +
> + # this one should fail because there's no client found for foo = 3
> + compc = compconf.remote_exec(code % (config.testpath,
'foo1 at bar.com',
> + {'foo': 3}))
> +
> + # sorry...
> + py.std.time.sleep(sleep_interval)
> +
> + ret = compc.receive()
> + assert not ret[0]
> + assert ret[1].find('no suitable client found') > -1
> +
> + # this one should be handled by client 1
> + compc = compconf.remote_exec(code % (config.testpath,
'foo2 at bar.com',
> + {'foo': 1, 'bar': [1]}))
> +
> + # and another one
> + py.std.time.sleep(sleep_interval)
> +
> + ret = compc.receive()
> + assert not ret[0]
> + assert ret[1].find('found a suitable client') > -1
> +
> + # the messages may take a bit to arrive, too
> + py.std.time.sleep(sleep_interval)
> +
> + # client 1 should by now have received the info to build for
> + cc1.receive() # 'welcome'
> + ret = cc1.receive()
> + assert ret == {'foo': 1, 'bar': [1], 'revision': 'trunk'}
> +
> + # this should have created a package in the temp dir
> + assert len(temppath.listdir()) == 1
> +
> + # now we're going to satisfy the first request by adding a new
client
> + sysinfo3 = {'foo': 3}
> + cgw3 = py.execnet.PopenGateway()
> + cc3 = client.init(cgw3, sysinfo3, port=config.port, testing=True)
> +
> + # again a bit of waiting may be desired
> + py.std.time.sleep(sleep_interval)
> +
> + # _try_queued() should check whether there are new clients
available for
> + # queued jobs
> + code = """
> + import sys, time
> + sys.path += %r
> +
> + from pypy.tool.build import ppbserver
> + ppbserver._try_queued()
> + # give the server some time, the clients 'compile' in threads
> + time.sleep(%s)
> + channel.send(ppbserver._requeststorage._id_to_emails)
> + channel.close()
> + """
> + compgw2 = py.execnet.PopenGateway()
> + compconf2 = execnetconference.conference(compgw2, config.port)
> +
> + compc2 = compconf2.remote_exec(code % (config.testpath,
sleep_interval))
> +
> +
> + # we check whether all emails are now sent, since after adding the
third
> + # client, and calling _try_queued(), both jobs should have
beenprocessed
> + ret = compc2.receive()
> + assert ret.values() == []
> +
> + # this should also have created another package in the temp dir
> + assert len(temppath.listdir()) == 2
> +
> + # some cleanup (this should all be in nested try/finallys, blegh)
> + cc1.close()
> + cc2.close()
> + cc3.close()
> + compc.close()
> + compc2.close()
> + sc.close()
> +
> + cgw1.exit()
> + cgw2.exit()
> + compgw.exit()
> + compgw2.exit()
> + sgw.exit()
>
> Added: pypy/dist/pypy/tool/build/test/test_request_storage.py
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/test/test_request_storage.py Wed Jul
> 26 13:18:25 2006
> @@ -0,0 +1,64 @@
> +import path
> +import py
> +from pypy.tool.build.server import RequestStorage
> +
> +def test_request_storage():
> + s = RequestStorage()
> +
> + assert s._id_to_info == {}
> + assert s._id_to_emails == {}
> + assert s._id_to_path == {}
> +
> + info = {'foo': 1}
> + infoid = s.get_info_id(info)
> +
> + path = s.request('foo at bar.com', info)
> + assert path is None
> + assert s._id_to_info == {infoid: info}
> + assert s._id_to_emails == {infoid: ['foo at bar.com']}
> + assert s._id_to_path == {}
> +
> + path = s.request('bar at bar.com', info)
> + assert path is None
> + assert s._id_to_info == {infoid: info}
> + assert s._id_to_emails == {infoid: ['foo at bar.com', 'bar at bar.com']}
> + assert s._id_to_path == {}
> +
> + emails = s.add_build(info, 'foobar')
> + assert emails == ['foo at bar.com', 'bar at bar.com']
> + assert s._id_to_info == {infoid: info}
> + assert s._id_to_emails == {}
> + assert s._id_to_path == {infoid: 'foobar'}
> +
> + info2 = {'foo': 2, 'bar': [1,2]}
> + infoid2 = s.get_info_id(info2)
> +
> + path = s.request('foo at baz.com', info2)
> + assert path is None
> + assert s._id_to_info == {infoid: info, infoid2: info2}
> + assert s._id_to_emails == {infoid2: ['foo at baz.com']}
> + assert s._id_to_path == {infoid: 'foobar'}
> +
> + emails = s.add_build(info2, 'foobaz')
> + assert emails == ['foo at baz.com']
> + assert s._id_to_info == {infoid: info, infoid2: info2}
> + assert s._id_to_emails == {}
> + assert s._id_to_path == {infoid: 'foobar', infoid2: 'foobaz'}
> +
> + path = s.request('foo at qux.com', info)
> + assert path == 'foobar'
> +
> +def test__build_initial():
> + s = RequestStorage([({'foo': 1}, 'foo'), ({'foo': 2}, 'bar'),])
> +
> + id1 = s.get_info_id({'foo': 1})
> + id2 = s.get_info_id({'foo': 2})
> +
> + assert s._id_to_info == {id1: {'foo': 1}, id2: {'foo': 2}}
> + assert s._id_to_emails == {}
> + assert s._id_to_path == {id1: 'foo', id2: 'bar'}
> +
> +def test__normalize():
> + s = RequestStorage()
> + assert (s._normalize({'foo': ['bar', 'baz']}) ==
> + s._normalize({'foo': ['baz', 'bar']}))
>
> Added: pypy/dist/pypy/tool/build/test/test_server.py
>
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/test/test_server.py Wed Jul 26 13:18:25
2006
> @@ -0,0 +1,132 @@
> +import path
> +from pypy.tool.build import server
> +import py
> +from fake import FakeChannel, FakeClient
> +from pypy.tool.build.server import RequestStorage
> +from pypy.tool.build.server import BuildPath
> +import time
> +
> +def setup_module(mod):
> + mod.temppath = temppath = py.test.ensuretemp('pypybuilder-server')
> + mod.svr = server.PPBServer('pypytest', FakeChannel(),
str(temppath))
> +
> + mod.c1 = FakeClient({'foo': 1, 'bar': [1,2]})
> + mod.svr.register(mod.c1)
> +
> + mod.c2 = FakeClient({'foo': 2, 'bar': [2,3]})
> + mod.svr.register(mod.c2)
> +
> +def test_server_issubdict():
> + from pypy.tool.build.server import issubdict
> + assert issubdict({'foo': 1, 'bar': 2}, {'foo': 1, 'bar': 2, 'baz':
3})
> + assert not issubdict({'foo': 1, 'bar': 2}, {'foo': 1, 'baz': 3})
> + assert not issubdict({'foo': 1, 'bar': 3}, {'foo': 1, 'bar':
2,'baz': 3})
> + assert issubdict({'foo': [1,2]}, {'foo': [1,2,3]})
> + assert not issubdict({'foo': [1,2,3]}, {'foo': [1,2]})
> + assert issubdict({'foo': 1L}, {'foo': 1})
> + assert issubdict({}, {'foo': 1})
> + assert issubdict({'foo': [1,2]}, {'foo': [1,2,3,4], 'bar': [1,2]})
> +
> +# XXX: note that the order of the tests matters! the first test reads
the
> +# information from the channels that was set by the setup_module()
function,
> +# the rest assumes this information is already read...
> +
> +def test_register():
> + assert len(svr._clients) == 2
> + assert svr._clients[0] == c1
> + assert svr._clients[1] == c2
> +
> + assert c1.channel.receive() == 'welcome'
> + assert c2.channel.receive() == 'welcome'
> + py.test.raises(IndexError, "c1.channel.receive()")
> +
> + assert svr._channel.receive().find('registered') > -1
> + assert svr._channel.receive().find('registered') > -1
> + py.test.raises(IndexError, 'svr._channel.receive()')
> +
> +def test_compile():
> + # XXX this relies on the output not changing... quite scary
> + info = {'foo': 1}
> + ret = svr.compile('test at domain.com', info)
> + assert not ret[0]
> + assert ret[1].find('found a suitable client') > -1
> + assert svr._channel.receive().find('going to send compile job') >
-1
> + assert c1.channel.receive() == 'foo: 1'
> + assert c1.channel.receive() is None
> + py.test.raises(IndexError, "c2.channel.receive()")
> +
> + svr.compile('test at domain.com', {'foo': 3})
> + assert svr._channel.receive().find('no suitable client available')
> -1
> +
> + info = {'bar': [3]}
> + ret = svr.compile('test at domain.com', info)
> + assert svr._channel.receive().find('going to send') > -1
> + assert c2.channel.receive() == 'bar: [3]'
> + assert c2.channel.receive() is None
> + py.test.raises(IndexError, "c1.channel.receive()")
> +
> + info = {'foo': 1}
> + ret = svr.compile('test at domain.com', info)
> + assert not ret[0]
> + assert ret[1].find('this build is already') > -1
> + assert svr._channel.receive().find('currently in progress') > -1
> +
> + c1.busy_on = None
> + bp = BuildPath(str(temppath / 'foo'))
> + svr.compilation_done(info, bp)
> + ret = svr.compile('test at domain.com', info)
> + assert ret[0]
> + assert isinstance(ret[1], BuildPath)
> + assert ret[1] == bp
> + assert svr._channel.receive().find('compilation done for') > -1
> + for i in range(2):
> + assert svr._channel.receive().find('going to send email to') >
-1
> + assert svr._channel.receive().find('already a build for this info')
> -1
> +
> +def test_buildpath():
> + tempdir = py.test.ensuretemp('pypybuilder-buildpath')
> + # grmbl... local.__new__ checks for class equality :(
> + bp = BuildPath(str(tempdir / 'test1'))
> + assert not bp.check()
> + assert bp.info == {}
> +
> + bp.info = {'foo': 1, 'bar': [1,2]}
> + assert bp.info == {'foo': 1, 'bar': [1,2]}
> + assert (sorted((bp / 'info.txt').readlines()) ==
> + ['bar: [1, 2]\n', 'foo: 1\n'])
> +
> + assert isinstance(bp.zipfile, py.path.local)
> + bp.zipfile = ['foo', 'bar', 'baz']
> + assert bp.zipfile.read() == 'foobarbaz'
> +
> +def test__create_filename():
> + svr._i = 0 # reset counter
> + today = time.strftime('%Y%m%d')
> + name1 = svr._create_filename()
> + assert name1 == 'pypytest-%s-0' % (today,)
> + assert svr._create_filename() == ('pypytest-%s-1' % (today,))
> + bp = BuildPath(str(temppath / ('pypytest-%s-2' % (today,))))
> + try:
> + bp.ensure()
> + assert svr._create_filename() == 'pypytest-%s-3'% (today,)
> + finally:
> + bp.remove()
> +
> +def test_get_new_buildpath():
> + svr._i = 0
> + today = time.strftime('%Y%m%d')
> +
> + path1 = svr.get_new_buildpath({'foo': 'bar'})
> + try:
> + assert isinstance(path1, BuildPath)
> + assert path1.info == {'foo': 'bar'}
> + assert path1.basename == 'pypytest-%s-0' % (today,)
> +
> + try:
> + path2 = svr.get_new_buildpath({'foo': 'baz'})
> + assert path2.info == {'foo': 'baz'}
> + assert path2.basename == 'pypytest-%s-1' % (today,)
> + finally:
> + path2.remove()
> + finally:
> + path1.remove()
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
>
More information about the Pypy-dev
mailing list