make module for Python?
François Pinard
pinard at iro.umontreal.ca
Tue May 8 10:17:12 EDT 2001
[Roman Suzi]
> I have not found it anywhere on the Web, but probably it has different
> name, so it's better to ask. I want a make-utility functionality (maybe
> not all, but basic) to be available from Python. Is there anything
> for this?
Other people wrote about the contest, so I will not repeat that information
here. About at that time, I needed a `make' replacement for myself,
and wrote one, which I include below. I use it regularly, and happily!
> I'd liked to have make dependence info not only for files, but for
> arbitrary class instances. Isn't it cool?
Of course, if by any chance you improve what follows, please share back your
modifications with me! :-)
-------------- next part --------------
#!/usr/bin/env python
# Copyright ? 2000 Progiciels Bourbeau-Pinard inc.
# Fran?ois Pinard <pinard at iro.umontreal.ca>, 2000.
"""\
Make-like facilities.
This module offers an equivalent for Make style rules, goals, dependencies
and actions. It does not implement implicit rules nor macro capabilities,
yet most needs in this area are well supplemented by the power of Python.
import make
maker = make.Maker()
The Maker constructor accepts a few options, which also may be changed by
later calls to `maker.set_options()'. Options initially all default to 0.
Option `logall' asks for all goal reports to be sent to stdout, default is
to send only goals in error. Option `verbose' asynchronously write partial
results on stderr as soon as possible, yet lines never mangle each other,
and titles are added as appropriate to indicate to which goal lines pertain.
`agent' announces a maximal number of parallel processes, 0 is read as 100.
`ignore' is set to ignore all exit statuses from processes. `dryrun'
only show commands that would be executed, without actually executing them.
Subclass `make.Rule' for each of your rules having different actions,
and override the `action' method in your subclass. If many rules happen
to share the same actions, instantiate your own rule class many times:
class MyRule(make.Rule):
def action(self, *arguments):
self.do(SYSTEM_COMMAND_1)
self.do(SYSTEM_COMMAND_2)
...
MyRule(GOAL_NAME_1, maker, [REQUIRE_NAMES_1...], ACTION_ARGUMENTS_1...)
MyRule(GOAL_NAME_2, maker, [REQUIRE_NAMES_2...], ACTION_ARGUMENTS_2...)
...
Requirements and dependencies, as well as arguments for the action methods,
are specified while creating an instance of your rule, this also registers
the goal. Goals and requires are all given as strings. If many rules
share the same GOAL_NAME, their require lists get logically concatenated,
as well as their actions.
Actions may use:
self.do(SYSTEM_COMMAND)
self.warn(DIAGNOSTIC)
self.fail(DIAGNOSTIC)
Options `verbose', `ignore' and `dryrun' of the `make.Maker()' call may be
overriden for the duration of a `self.do' call, by giving them on that call.
`self.do' also accepts a `filter' argument, see the code for details.
Once rules are set, one launches execution for a given a set of goal strings:
success = maker.run(GOAL_NAME_1, GOAL_NAME_2, ...)
For this function to succeed, all given goals should succeed. A goal
succeeds if all its requires succeed, and if all attempted actions are
successfully run. A require succeeds if it names a successful goal;
otherwise, it succeeds if it names an existing file or directory.
Success is stamped with a time. Failed goals or requires have no stamp.
If a goal or a require names a file or a directory, its stamp is the
`mtime' of that file or directory. Otherwise, a goal gets stamped with
the current time.
If a goal names an existing file or directory for which `mtime' is greater
or equal to the stamp of all requires of this rule, then the actions for
this rule are declared successful without being attempted.
Any failed require or failed system command turns subsequent `self.do'
for that rule into no-operations and inhibits pending actions for this goal.
Justificative comments:
For years, I've been growing Makefiles for synchronising my projects
between machines, rebuilding Web sites, and doing various other things;
and resorted to parallel Make builds to save my time, as I do these things
on many remote machines at once. The output of parallel processes get all
mangled at times, to the point of becoming hard to decipher and sort out.
Also, there are also limits on what one can cleanly do with Makefiles.
So, for some while, I have not been satisfied, and decided to try rewriting
part of all this in Python. It went surprisingly well! At the hearth,
I wrote a simple Make-alike Python module for my needs, later used by some
other Python programs. It is fairly straightforward and simple to use,
so let me share it with you. I quickly looked around, and most Make-alike
projects for Python are far more ambitious than this little thing.
"""
# FIXME: Detect dependency cycles, which would cause thread deadlocks.
import os, stat, string, sys, time, threading
class Maker:
def __init__(self, logall=0, verbose=0, agents=0, ignore=0, dryrun=0):
self.registry = {}
self.writer = Writer()
self.agents = None
self.set_options(logall=logall, verbose=verbose,
agents=agents, ignore=ignore, dryrun=dryrun)
self.top_name = '_Top_Maker_'
def set_options(self, logall=None, verbose=None,
agents=None, ignore=None, dryrun=None):
if logall is not None:
self.logall = logall
if verbose is not None:
self.verbose = verbose
if agents is not None:
if agents <= 0:
agents = 100
if self.agents is None:
self.process = threading.Semaphore(agents)
self.agents = agents
while agents > self.agents:
self.process.release()
self.agents = self.agents + 1
while agents < self.agents:
self.process.acquire()
self.agents = self.agents - 1
if ignore is not None:
self.ignore = ignore
if dryrun is not None:
self.dryrun = dryrun
def run(self, *requires):
stamp = Rule(self.top_name, self, requires).get_stamp()
del self.registry[self.top_name]
return stamp is not None
class Rule(threading.Thread):
def __init__(self, name, maker, requires, *arguments):
if not maker.registry.has_key(name):
maker.registry[name] = self
threading.Thread.__init__(self)
self.head = None
self.setName(name)
self.maker = maker
self.requires = requires or []
self.calls = [(self.action, arguments)]
self.lock = threading.Lock()
self.started = 0
self.stamp = -1 # a very low number :-)
else:
self.head = head = maker.registry[name]
head.requires = head.requires + (requires or [])
head.calls.append((self.action, arguments))
def launch(self):
assert self.head is None
if not self.started:
self.lock.acquire()
if not self.started:
self.start()
self.started = 1
self.lock.release()
def get_stamp(self):
assert self.head is None
self.launch()
self.join()
return self.stamp
def run(self):
assert self.head is None
registry = self.maker.registry
if self.requires:
for name in self.requires:
if registry.has_key(name):
registry[name].launch()
for name in self.requires:
if registry.has_key(name):
stamp = registry[name].get_stamp()
else:
stamp = get_file_stamp(name)
if stamp is None:
self.fail("%s: Was not remade." % name)
elif self.stamp is not None and stamp > self.stamp:
self.stamp = stamp
if self.stamp is not None:
name = self.getName()
if name == self.maker.top_name:
stamp = None
else:
stamp = get_file_stamp(name)
if stamp is None:
inhibit = 0
else:
inhibit = stamp >= self.stamp
self.stamp = stamp
if not inhibit:
for action, arguments in self.calls:
apply(action, arguments)
if self.stamp is None:
break
else:
if stamp is None:
stamp = int(time.time())
if stamp < self.stamp:
self.fail("Clock skew detected.")
else:
self.stamp = stamp
self.calls = None
self.maker.writer.flush(self.maker.logall or self.stamp is None)
def action(self, *arguments): return 1
def do(self, command, logall=None, verbose=None,
ignore=None, dryrun=None, filter=None):
head = self.head or self
if head.stamp is not None:
head.maker.process.acquire()
if logall is None:
logall = head.maker.logall
if verbose is None:
verbose = head.maker.verbose
if ignore is None:
ignore = head.maker.ignore
if dryrun is None:
dryrun = head.maker.dryrun
if filter is None:
filter = self.filter
head.warn(command)
if dryrun:
status = None
else:
program, arguments = string.split(command, None, 1)
file = os.popen(string.join([program, '2>&1', arguments]))
if not filter(file, head.maker.writer.write, verbose=verbose):
head.fail("Filter failed: %s" % command)
status = file.close()
if status is not None:
if ignore:
head.warn("Exit %d (ignored): %s" % (status >> 8, command))
else:
head.fail("Exit %d: %s" % (status >> 8, command))
head.maker.process.release()
def filter(self, file, write, verbose=0):
line = file.readline()
while line:
write(line, verbose=verbose)
line = file.readline()
return 1
def warn(self, text):
head = self.head or self
verbose = head.maker.verbose
head.maker.writer.write('... %s\n' % text, verbose=verbose)
def fail(self, text):
head = self.head or self
# We want errors on the terminal, even if not verbose. However,
# consider the error will show in the goal report, sent on stdout.
verbose = head.maker.verbose or not sys.stdout.isatty()
head.maker.writer.write('*** %s\n' % text, verbose=verbose)
head.stamp = None
class Writer:
def __init__(self):
self.lock = threading.Lock()
self.tag = None
self.streams = {}
def write(self, text, verbose=0):
tag = threading.currentThread().getName()
lines = self.streams.get(tag)
if lines:
lines.append(text)
else:
self.streams[tag] = [text]
if verbose:
self.lock.acquire()
write = sys.stderr.write
if tag != self.tag:
label = '[%s]' % tag
spacer = ' ' * (79 - len(label))
write('%s%s\n' % (spacer, label))
self.tag = tag
write(text)
self.lock.release()
def seen_lines(self):
tag = threading.currentThread().getName()
return self.streams.get(tag, [])
def flush(self, verbose=0):
tag = threading.currentThread().getName()
lines = self.streams.get(tag)
if lines:
del self.streams[tag]
if verbose:
self.lock.acquire()
file = sys.stdout
label = '[%s]' % tag
spacer = ' ' * (79 - len(label))
file.write('=' * 79 + '\n')
file.write('%s%s\n' % (spacer, label))
file.writelines(lines)
self.tag = None
self.lock.release()
def get_file_stamp(name):
try:
stamp = os.stat(name)[stat.ST_MTIME]
except OSError:
stamp = None
return stamp
-------------- next part --------------
--
Fran?ois Pinard http://www.iro.umontreal.ca/~pinard
More information about the Python-list
mailing list