Special tokens:
@ -- escapes any character following it (i.e., @c is always translated
to c)
[ -- a short tag opener. Closed with a matching ]
::(newline) -- beginning a a long tag. everything until the
indent level returns to that of the line which started the long
tag is part of the contents.
(newline)(newline) -- new paragraph.
The syntax of short tags is '[' tagname ' ' contents ']' where contents is
any valid snippet of docstring, with the exception of a long tag.
The syntax of long tags is
tagname (attr '=' value)*'::'
contents
Where contents is any valid snippet of docstring.
List of tags, and where they are valid:
code (both as long and short) -- anywhere
example (long, no attributes) -- anywhere
arg, return (attributes: name, type, default (optional) ) --
function docstring, never inside another long tag.
rest-arg, kw-arg (long, no attributes) --
function docstring, never inside another long tag.
data (attributes: name, type) --
class/module doc-string, never inside another long tag.
exception (attributes: name) --
class/module doc-string, never inside another long tag.
member (attributes: name, type) --
class doc-string, never inside another long tag.
function, module, class, exception, var, method, keyword, member,
pytype, file, url, arg (short tag) -- anywhere
(these are for general markup syntax, so I've probably forgotten
some):
list (long tag) -- anywhere
item (long tag) -- anywhere
emph (short tag) -- anywhere
(Note: there are some long tags with the same name as short tags. This
poses no problems: the tags are different tags!)
This was brief on purpose: I don't believe hardly anyone will actually
read the spec: most people will just start writing doc-strings based on what
they have seen. So, here is an example of a marked up module. I want to thank
Gordon McMillan for providing a well-documented module: I just modified the
syntax.
(search for ======= if you want to skip the example module and get to the
description of the intermediary format)
"""Utilities for comparing files and directories."""
import os
import stat
import statcache
_cache = {}
BUFSIZE=8*1024
def cmp(f1, f2, shallow=1,use_statcache=0):
"""Compare two files.
arg name=f1 type=string::
First file name
arg name=f2 type=string::
Second file name
arg name=shallow type=bool default=1::
Just check stat signature (do not read the files).
arg name=use_statcache type=bool default=0::
Do not stat() each file directly: go through
the statcache module for more efficiency.
return type=bool::
1 if the files are the same, 0 otherwise.
This function uses a cache for past comparisons and the results,
with a cache invalidation mechanism relying on stale signatures.
Of course, if [arg use_statcache] is true, this mechanism is defeated,
and the cache will never grow stale.
"""
stat_function = (os.stat, statcache.stat)[use_statcache]
s1, s2 = _sig(stat_function(f1)), _sig(stat_function(f2))
if s1[0]!=stat.S_IFREG or s2[0]!=stat.S_IFREG: return 0
if shallow and s1 == s2: return 1
if s1[1]!=s2[1]: return 0
result = _cache.get((f1, f2))
if result and (s1, s2)==result[:2]:
return result[2]
outcome = _do_cmp(f1, f2)
_cache[f1, f2] = s1, s2, outcome
return outcome
def _sig(st):
return (stat.S_IFMT(st[stat.ST_MODE]),
st[stat.ST_SIZE],
st[stat.ST_MTIME])
def _do_cmp(f1, f2):
bufsize = BUFSIZE
fp1 , fp2 = open(f1, 'rb'), open(f2, 'rb')
while 1:
b1, b2 = fp1.read(bufsize), fp2.read(bufsize)
if b1!=b2: return 0
if not b1: return 1
# Directory comparison class.
#
class dircmp:
"""A class that manages the comparison of 2 directories.
High level usage:
list::
item::
[code x = dircmp(dir1, dir2)]
item::
[code x.report()] -> prints a report on the differences between
dir1 and dir2
item::
[code x.report_partial_closure()] -> prints report on
differences between dir1 and dir2, and reports on common
immediate subdirectories.
item::
[code x.report_full_closure()] -> like report_partial_closure,
but fully recursive.
member name=left_list type=list::
files in [var dir1], except for the ones in [var hide] and
[var ignore] lists.
member name=right_list type=list::
files in [var dir2], except for the ones in [var hide] and
[var ignore] lists.
member name=common type=list::
names in both [var dir1] and [var dir2]
member name=left_only type=list::
names in [var dir1] but not in [var dir2]
member name=right_only type=list::
names in [var dir2] but not in [var dir1]
member name=common_dirs type=list::
subdirectories in both [var dir1] and [var dir2].
member name=common_files type=list::
files in both [var dir1] and [var dir2].
member name=common_funny type=list::
names in both [var dir1] and [var dir2] where the type differs between
[var dir1] and [var dir2], or the name is not [function stat]-able.
member name=same_files type=list::
list of identical files.
member name=diff_files type=list::
list of filenames which differ.
member name=funny_files type=list::
list of files which could not be compared.
member name=funny_files type=list::
list of files which could not be compared.
member name=subdirs type=dictionary::
values are [class dircmp] objects, keyed by names in
[member common_dirs].
"""
def __init__(self, a, b, ignore=None, hide=None): # Initialize
'''\
initialize an directory comparison
arg name=a type=string::
first directory
arg name=b type=string::
second directory
arg name=ignore type=list or None::
list of directories to ignore when comparing. None means to
ignore [code ['RCS', 'CVS', 'tags']].
arg name=hide type=list or None::
list of directories to hide when comparing. None means to hide
defaults to [code [os.curdir, os.pardir]].
'''
self.left = a
self.right = b
if hide is None:
self.hide = [os.curdir, os.pardir] # Names never to be shown
else:
self.hide = hide
if ignore is None:
self.ignore = ['RCS', 'CVS', 'tags'] # Names ignored in comparison
else:
self.ignore = ignore
def phase0(self): # Compare everything except common subdirectories
self.left_list = _filter(os.listdir(self.left),
self.hide+self.ignore)
self.right_list = _filter(os.listdir(self.right),
self.hide+self.ignore)
self.left_list.sort()
self.right_list.sort()
__p4_attrs = ('subdirs',)
__p3_attrs = ('same_files', 'diff_files', 'funny_files')
__p2_attrs = ('common_dirs', 'common_files', 'common_funny')
__p1_attrs = ('common', 'left_only', 'right_only')
__p0_attrs = ('left_list', 'right_list')
def __getattr__(self, attr):
if attr in self.__p4_attrs:
self.phase4()
elif attr in self.__p3_attrs:
self.phase3()
elif attr in self.__p2_attrs:
self.phase2()
elif attr in self.__p1_attrs:
self.phase1()
elif attr in self.__p0_attrs:
self.phase0()
else:
raise AttributeError, attr
return getattr(self, attr)
def phase1(self): # Compute common names
a_only, b_only = [], []
common = {}
b = {}
for fnm in self.right_list:
b[fnm] = 1
for x in self.left_list:
if b.get(x, 0):
common[x] = 1
else:
a_only.append(x)
for x in self.right_list:
if common.get(x, 0):
pass
else:
b_only.append(x)
self.common = common.keys()
self.left_only = a_only
self.right_only = b_only
def phase2(self): # Distinguish files, directories, funnies
self.common_dirs = []
self.common_files = []
self.common_funny = []
for x in self.common:
a_path = os.path.join(self.left, x)
b_path = os.path.join(self.right, x)
ok = 1
try:
a_stat = statcache.stat(a_path)
except os.error, why:
# print 'Can\'t stat', a_path, ':', why[1]
ok = 0
try:
b_stat = statcache.stat(b_path)
except os.error, why:
# print 'Can\'t stat', b_path, ':', why[1]
ok = 0
if ok:
a_type = stat.S_IFMT(a_stat[stat.ST_MODE])
b_type = stat.S_IFMT(b_stat[stat.ST_MODE])
if a_type <> b_type:
self.common_funny.append(x)
elif stat.S_ISDIR(a_type):
self.common_dirs.append(x)
elif stat.S_ISREG(a_type):
self.common_files.append(x)
else:
self.common_funny.append(x)
else:
self.common_funny.append(x)
def phase3(self): # Find out differences between common files
xx = cmpfiles(self.left, self.right, self.common_files)
self.same_files, self.diff_files, self.funny_files = xx
def phase4(self): # Find out differences between common subdirectories
# A new dircmp object is created for each common subdirectory,
# these are stored in a dictionary indexed by filename.
# The hide and ignore properties are inherited from the parent
self.subdirs = {}
for x in self.common_dirs:
a_x = os.path.join(self.left, x)
b_x = os.path.join(self.right, x)
self.subdirs[x] = dircmp(a_x, b_x, self.ignore, self.hide)
def phase4_closure(self): # Recursively call phase4() on subdirectories
self.phase4()
for x in self.subdirs.keys():
self.subdirs[x].phase4_closure()
def report(self):
'''\
Print a report on the differences between [member a] and [member b].
Output format is purposely lousy
'''
print 'diff', self.left, self.right
if self.left_only:
self.left_only.sort()
print 'Only in', self.left, ':', self.left_only
if self.right_only:
self.right_only.sort()
print 'Only in', self.right, ':', self.right_only
if self.same_files:
self.same_files.sort()
print 'Identical files :', self.same_files
if self.diff_files:
self.diff_files.sort()
print 'Differing files :', self.diff_files
if self.funny_files:
self.funny_files.sort()
print 'Trouble with common files :', self.funny_files
if self.common_dirs:
self.common_dirs.sort()
print 'Common subdirectories :', self.common_dirs
if self.common_funny:
self.common_funny.sort()
print 'Common funny cases :', self.common_funny
def report_partial_closure(self):
'''Print reports on [arg self] and on [member subdirs]'''
self.report()
for x in self.subdirs.keys():
print
self.subdirs[x].report()
def report_full_closure(self):
'''Report on [var self] and [member subdirs] recursively.'''
self.report()
for x in self.subdirs.keys():
print
self.subdirs[x].report_full_closure()
# Compare common files in two directories.
# Return:
# - files that compare equal
# - files that compare different
# - funny cases (can't stat etc.)
#
def cmpfiles(a, b, common):
"""Compare common files in two directories.
arg name=a type=string::
name of first directory.
arg name=b type=string::
name of second directory.
arg name=common type=list::
names of common files to be compared
return type=tuple::
list::
item::
files that compare equal
item::
files that are different
item::
filenames that aren't regular files.
"""
res = ([], [], [])
for x in common:
res[_cmp(os.path.join(a, x), os.path.join(b, x))].append(x)
return res
# Compare two files.
# Return:
# 0 for equal
# 1 for different
# 2 for funny cases (can't stat, etc.)
#
def _cmp(a, b):
try:
return not abs(cmp(a, b))
except os.error:
return 2
# Return a copy with items that occur in skip removed.
#
def _filter(list, skip):
result = []
for item in list:
if item not in skip: result.append(item)
return result
# Demonstration and testing.
#
def demo():
import sys
import getopt
options, args = getopt.getopt(sys.argv[1:], 'r')
if len(args) <> 2: raise getopt.error, 'need exactly two args'
dd = dircmp(args[0], args[1])
if ('-r', '') in options:
dd.report_full_closure()
else:
dd.report()
if __name__ == '__main__':
demo()
=======
The intermediary format is XML.
I have not yet written a DTD, but this is a general sketch --
The root element is "module"
Inside "module", you can have "description", "class", "function", "exception"
and "data".
Inside "class", you can have "description", "class", "function", "exception",
"function", "data" and "member"
Inside "function", and "description", "arg", "kw-arg", "rest-arg".
and "return" elements.
Inside "exception", there is a "description".
Inside "data", there is a "description".
Inside "member", there is a "description".
Inside "arg", "kw-arg", "rest-arg" and "return" there is a "description"
The following elements have a "type" attribute:
"member", "data", "arg", "return".
The following elements have a "name" attribute:
"class", "function", "exception", "data", "member", "arg".
Inside a "description" there are "p" elements.
Inside "p" element, there is PCDATA, "code" element, "example" element,
"module-ref", "class-ref", "exception-ref", "member-ref", "data-ref",
"function-ref", "arg-ref", "var", "keyword", "pytype", "file", "url", "emph"
and "list".
Inside the *-ref elements, "code", "example", "var", "keyword", "pytype",
"file" and "url" there is PCDATA.
Inside "list" there are "item" elements.
The content model of "item" and "emph" is the same as that for "p".
=============
The OOL format will be similar to the intermediary format, with two important
changes:
1. It will have "link" elements, which allow it to pull data out of a
module doc-strings. These link elements should be powerful enough so that
for most modules, a canonical OOL file will suffice.
2. It will have some SGML minimization to make writing it less painful.
Please comment!
--
Moshe Zadka <mzadka(a)geocities.com>.
INTERNET: Learn what you know.
Share what you don't.
