Mailman 3 October 2002 - Python-Dev

Re: [Python-Dev] metaclass insanity
by Michael Hudson 31 Oct '02

31 Oct '02

Kevin Jacobs <jacobs(a)penguin.theopalgroup.com> writes: > On 30 Oct 2002, Michael Hudson wrote: > > For moderately nefarious reasons[1] I've being trying to write a > > metaclass whose instances have writable __bases__. This in itself > > isn't so hard, but having assigments to __bases__ "do the right thing" > > has eluded me, basically because I can't seem to affect the mro. > > The mro is an internal data structure of new-style classes, so redefining > mro() doesn't change the values used. Yeah, I noticed that eventually. > Here is my (non-working version) that > attempts to re-assign the class of an object, although it fails on a layout > violation with Python 2.2.2. > > def base_getter(cls): > return cls.__my_bases__ > > def base_setter(cls,bases): > if not bases: > bases = (object,) > metaclass = getattr(cls, '__metaclass__', type) > new_cls = metaclass(cls.__name__, bases, dict(cls.__dict__)) > cls.__class__ = new_cls > > class MetaBase(type): > __bases__ = property(base_getter,base_setter) > > def __new__(cls, name, bases, ns): > ns['__my_bases__'] = tuple(bases) > return super(MetaBase, cls).__new__(cls, name, bases, ns) > > class Foo(object): > __metaclass__ = MetaBase > class Baz(object): pass > > Foo.__bases__ = Foo.__bases__ + (Baz,) I don't think this has a hope of working does it? This approach would need to rebind "Foo" in the last line, no? > Which results in: > TypeError: __class__ assignment: 'Foo' object layout differs from 'MetaBase' > > I haven't looked into why this is being flagged as a layout error, though > my first instinct is to say that the check is too conservative in this case. > I'll think about it more and dig into the code. I think the error message gives it away: in "cls.__class__ = new_cls", cls.__class__ is MetaBase, new_cls is the new Foo. I guess one way of doing this would be to reimplement mro resolution and so on in Python, which would be annoying and inefficient. Hmm. Cheers, M. -- [3] Modem speeds being what they are, large .avi files were generally downloaded to the shell server instead[4]. [4] Where they were usually found by the technical staff, and burned to CD. -- Carlfish, asr

2 1

Keyword module cruft question
by Patrick K. O'Brien 30 Oct '02

30 Oct '02

I'm working on an article about Python introspection and came across something I consider a blemish. I thought I'd ask about it here and see how you folks feel about it. BTW, I'm looking at Python 2.2.2 on Linux. If you look at the keyword module, you'll see that it has a keyword attribute: >>> import keyword >>> dir(keyword) ['__all__', '__builtins__', '__doc__', '__file__', '__name__', 'iskeyword', 'keyword', 'kwdict', 'kwlist', 'main'] >>> keyword.keyword 'yield' >>> Basically, the keyword attribute is just the last value from this process: kwdict = {} for keyword in kwlist: kwdict[keyword] = 1 So it is a bit of extraneous cruft, I believe. No big deal, and nothing that a del keyword couldn't fix. But it does look sloppy when you are writing about introspection and you see an attribute that has no real value. I wish it weren't there, as I'd rather not have to explain it. But it got me thinking whether other modules in the standard library have this issue, and whether this is something that should be cleaned up. Thoughts? -- Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." -----------------------------------------------

4 6

metaclass insanity
by Michael Hudson 30 Oct '02

30 Oct '02

For moderately nefarious reasons[1] I've being trying to write a metaclass whose instances have writable __bases__. This in itself isn't so hard, but having assigments to __bases__ "do the right thing" has eluded me, basically because I can't seem to affect the mro. Here's what I have so far: def base_getter(klass): return klass.__priv_bases__ def base_setter(klass, v): klass.__priv_bases__ = v class MetaMutableBases(type): def __new__(self, name, bases, ns): t = type.__new__(self, name, bases, ns) t.__priv_bases__ = type.__dict__["__bases__"].__get__(self) return t __bases__ = property(base_getter, base_setter) def mro(self): if hasattr(self, "__priv_bases__"): return type("noname", self.__priv_bases__, {}).__mro__ else: return type.mro(self) I've tried variations on the same, but haven't got anything working yet. Anyone have any ideas? I think my brain's exploded. And I realise this is in some sense more appropriate for c.l.py, but I think my odds of a good answer are vastly higher here. Cheers, M. [1] in http://groups.google.com/groups?selm=m2r8e99l0b.fsf%40python.net I posted a metaclass that attempts to automatically upgrade instances of its instances on redefinition. Ideally, this would also frob its subclasses so they now inherited from redefined class. I'm aware this can go wrong in various ways, but I think it would be pretty handy in a large range of situations. -- This is not to say C++ = bad, Lisp = good. It's to say C++ = bad irrespective of everything else. -- Alain Picard, comp.lang.lisp

2 1

Re: Incorporating bsddb3
by Andrew Kuchling 30 Oct '02

30 Oct '02

GvR wrote: >Now I'm confused. I thought this was Robun Dunn's bsddb3 module? Is >there another? It's the same one. Robin Dunn's original module used SWIG to wrap the BerkeleyDB API, and I offered to rewrite the interface as a hand-written extension module, which is why my name turns up. (Adding my name to the copyright seemed unjustified to me at the time, though; writing the extension was mostly a trivial process of rearranging Robin's SWIG definitions with some Emacs/elisp work.) --amk (www.amk.ca) JULIET: O Romeo, Romeo, wherefore art thou Romeo? -- _Romeo and Juliet_, II, ii

1 0

Patch to use dict subclasses in eval(), exec
by Jeff Epler 30 Oct '02

30 Oct '02

Using dict subclasses in eval() and exec ---------------------------------------- With the attached patch, you can now use a dict subclass which implements __getitem__ as the global or builtin namespace for eval() and exec. If you do not use the feature, the performance impact is low (<2%). The fast case adds only one test+branch to each eval_code() call (to set 'fastglobals' and 'slowbuiltins'), each lookup in globals (to check 'fastglobals'), and each lookup in builtins (to check 'slowbuiltins'). If you do use the feature, the performance impact is unfortunately quite substantial (400%). If you use subclasses of dict with eval()/exec, but do not define __getitem__, the performance penalty is modest (~10%). The smaller penalty of using a dict subclass without __getitem__ could probably be erased if the 'fastglobals' test can check that __getitem__ is not defined Included are a patch and a test suite. Note that the patch causes a failure in test_descr which is testing that __getitem__ is *not* called on a dictionary subclass. Pystone timings on a 1GHz Duron system running RedHat Linux 8.0 (gcc 3.2) follow. CVS +patch +subclass +__getattr__ 1 2.74 2.76 3 14.05 2 2.74 2.78 3.02 14.07 3 2.74 2.78 3.03 14.07 4 2.76 2.78 3.04 14.08 5 2.76 2.79 3.04 14.1 avg 2.748 2.778 3.026 14.074 % 100 101.1 110.1 512.2 * 2.3a0 CVS (28 Oct 2002) Pystone(1.1) time for 50000 passes = 2.74 This machine benchmarks at 18248.2 pystones/second Pystone(1.1) time for 50000 passes = 2.74 This machine benchmarks at 18248.2 pystones/second Pystone(1.1) time for 50000 passes = 2.74 This machine benchmarks at 18248.2 pystones/second Pystone(1.1) time for 50000 passes = 2.76 This machine benchmarks at 18115.9 pystones/second Pystone(1.1) time for 50000 passes = 2.76 This machine benchmarks at 18115.9 pystones/second * 2.3a0 CVS + patch Pystone(1.1) time for 50000 passes = 2.78 This machine benchmarks at 17985.6 pystones/second Pystone(1.1) time for 50000 passes = 2.78 This machine benchmarks at 17985.6 pystones/second Pystone(1.1) time for 50000 passes = 2.79 This machine benchmarks at 17921.1 pystones/second Pystone(1.1) time for 50000 passes = 2.79 This machine benchmarks at 17921.1 pystones/second Pystone(1.1) time for 50000 passes = 2.76 This machine benchmarks at 18115.9 pystones/second * 2.3a0 CVS + patch + dict subclass w/o __getitem__ (commandline: ./python -c 'class D(dict): pass d = D(globals()); execfile("Lib/test/pystone.py", d)') Pystone(1.1) time for 50000 passes = 3.03 This machine benchmarks at 16501.7 pystones/second Pystone(1.1) time for 50000 passes = 3.04 This machine benchmarks at 16447.4 pystones/second Pystone(1.1) time for 50000 passes = 3 This machine benchmarks at 16666.7 pystones/second Pystone(1.1) time for 50000 passes = 3.04 This machine benchmarks at 16447.4 pystones/second Pystone(1.1) time for 50000 passes = 3.02 This machine benchmarks at 16556.3 pystones/second * 2.3a0 CVS + patch + dict subclass w/__getitem__ (commandline: ./python -c 'class D(dict): __getitem__ = lambda s, i: dict.__getitem__(s, i) d = D(globals()); execfile("Lib/test/pystone.py", d)') Pystone(1.1) time for 50000 passes = 14.05 This machine benchmarks at 3558.72 pystones/second Pystone(1.1) time for 50000 passes = 14.08 This machine benchmarks at 3551.14 pystones/second Pystone(1.1) time for 50000 passes = 14.07 This machine benchmarks at 3553.66 pystones/second Pystone(1.1) time for 50000 passes = 14.07 This machine benchmarks at 3553.66 pystones/second Pystone(1.1) time for 50000 passes = 14.1 This machine benchmarks at 3546.1 pystones/second

8 19

Valgrind on 2.2.2
by Neal Norwitz 29 Oct '02

29 Oct '02

I ran valgrind 1.0.3 on Python 2.2.2. Well, almost 2.2.2, it was from 12 Oct. After 90+ minutes and over 512 MB of RAM, there are no new/major issues to report. The complete valgrind log (231k) can be found here: http://www.metaslash.com/py/valgrind-2_2_2.txt I've cleaned up the log (200k) to remove some of the uninteresting stuff: http://www.metaslash.com/py/valgrind-clean-2_2_2.txt There are a few small memory leaks. I think most of the leaks are related to threads. The report contains memory still in use as well as leaks. To find memory leaks, search for ' lost ' without quotes. It's quite possible some of the memory still in use is due to reference leaks. Note: one common leak is from readline. Some of the read errors are from GLIBC. I had to skip test_pty, I think it caused the tests to hang. test_commands and test_popen2 fail when running valgrind. test_popen2 is due to valgrind output going to stdout/stderr. Neal

5 11

Using defaultdict as globals/locals for eval()
by Geert Jansen 28 Oct '02

28 Oct '02

Dear Python developers, I have a question about "dict" subclasses and eval() that I would kindly like to ask. The problem I have is the following: For a web content system in Python, I am using Python's eval() function to run code embedded in templates. Currently, the code failes with a NameError if a variable is referenced inside the code that is not defined in the locals or globals dictionary. Unfortunately, this is a rather common situation: one often uses variables (like error codes) that aren't set all the time. Intializing these variables is a tedious job so a default value would suit me nicely. I created a derived class of the standard `dict' that fills in a default value when a key is not found. This is exactly the same as Guido describes in his "descintro" paper. I tried to use this dictionary as the "globals" parameter with eval(). As Guido already describes in his paper, this doesn't work. Quoting Guido: "The interpreter uses an internal function to access the dictionary, which bypasses our __getitem__() override. I admit that this can be a problem (although it is only a problem in this context, when a dict subclass is used as a locals/globals dictionary); it remains to be seen if I can fix this without compromising performance in the common case." Is there a solution to this problem in sight? Or altenately, is there a way I can find out which variables are used inside a compiled code block so I can initialize non-specified variables? I have a vague memory that the nested scopes feature has to determine at compile time which variables are being used in a code block. Thanks for your time! Greetings, Geert Jansen

5 5

__getitem__ in user defined classes
by Raymond Hettinger 28 Oct '02

28 Oct '02

In Py2.3, __getitem__ conveniently supports slices for builtin sequences: 'abcde'.__getitem__(slice(2,4)) For user defined classes to emulate this behavior, they need to test the index argument to see whether it is a slice and then loop over the slice indices like this: class SquaresToTen: """Acts like a list of squares but computes only when needed""" def __len__(self): return 11 def __getitem__(self, index): if isinstance(index, slice): return [x**2 for x in range(index.start, index.stop, index.step)] else: return index**2 print SquaresToTen()[2] print SquaresToTen()[7:1:-2] This could be simplified somewhat by making slices iterable so that the __getitem__ definition looks more like this: def __getitem__(self, index): if isinstance(index, slice): return [x**2 for x in index] else: return index**2 Raymond Hettinger

4 3

"conflicting types" in python CVS build
by Alex Coventry 27 Oct '02

27 Oct '02

> > hope this helps. > > Not really. Can you please copy line 169 of pgenmain.c? I can't see a > problem. at the time, lines 167-169 were char * PyOS_Readline(char *prompt) { which conflicted with the prototype in pythonrun.h. but it looks like a change which fixed that was checked in this morning. sorry for the confusion. alex.

2 1

Weekly Python Bug/Patch Summary
by Skip Montanaro 27 Oct '02

27 Oct '02

Bug/Patch Summary ----------------- 318 open / 2981 total bugs (no change) 96 open / 1743 total patches (+4) New Bugs -------- email incompatibility upgrading to 2.2.2 (2002-10-20) http://python.org/sf/626119 missing DECREF's in embedding example (2002-10-21) http://python.org/sf/626275 Support RFC 2111 in email package (2002-10-21) http://python.org/sf/626452 urllib2 doesn't do HTTP-EQUIV & Refresh (2002-10-21) http://python.org/sf/626543 strptime() always returns 0 in dst field (2002-10-21) http://python.org/sf/626570 Build error using make VPATH feature (2002-10-22) http://python.org/sf/626926 canvas.create_box() crashes Tk thread (2002-10-22) http://python.org/sf/626936 KeyPress bindings don't work (2002-10-23) http://python.org/sf/627798 curses library not found during make (2002-10-23) http://python.org/sf/627864 Set constructor fails with NameError (2002-10-24) http://python.org/sf/628246 pydoc.Helper.topics not based on docs (2002-10-24) http://python.org/sf/628258 Deprecate PyNumber_Check (2002-10-25) http://python.org/sf/628842 pickle won't dump instances after reload (2002-10-25) http://python.org/sf/628925 Race condition in asyncore poll (2002-10-26) http://python.org/sf/629097 build does not respect --prefix (2002-10-27) http://python.org/sf/629345 New Patches ----------- db4 include not found (2002-08-02) http://python.org/sf/590377 Support Unicode normalization (2002-10-21) http://python.org/sf/626485 Support Hangul Syllable names (2002-10-21) http://python.org/sf/626548 Bytecode copy bug in freeze (2002-10-24) http://python.org/sf/627900 xmlrpclib: Optional 'nil' support (2002-10-24) http://python.org/sf/628208 Experimental Inno Setup Win32 installer (2002-10-24) http://python.org/sf/628301 Detect BLT on more systems (2002-10-26) http://python.org/sf/629126 install lib-dynload .so files mode 555 (2002-10-26) http://python.org/sf/629278 Closed Bugs ----------- mac module documentation inaccuracy. (2002-01-17) http://python.org/sf/505150 ConfigParser bug/limitation (2002-02-01) http://python.org/sf/511737 ConfigParser cleanup, items() [needs tests] (2002-04-17) http://python.org/sf/545096 Nested anchors in Ref Man index.html (2002-04-20) http://python.org/sf/546579 Wrong description for PyErr_Restore (2002-07-03) http://python.org/sf/577000 PyString_AsString underdocumented (2002-09-08) http://python.org/sf/606463 packages doc in the tutorial (2002-09-29) http://python.org/sf/616211 PythonLauncher Dialogs do not work (2002-10-06) http://python.org/sf/619245 test_builtin hangs on MacOSX 10.2.1 (2002-10-06) http://python.org/sf/619400 sizeof(struct stat) changes (2002-10-10) http://python.org/sf/621523 urllib.pathname2url(), .url2pathname() (2002-10-14) http://python.org/sf/623196 Typo at line 351 of email/Charset.py (2002-10-18) http://python.org/sf/625509 IDE starts in / on OSX (2002-10-19) http://python.org/sf/625734 Closed Patches -------------- readline /dev/tty problem (2002-02-04) http://python.org/sf/512981 Installation database patch (2002-05-29) http://python.org/sf/562100 add unescape method to xml.sax.saxutils (2002-09-23) http://python.org/sf/613256 autoconf-2.5X tweaks for configure.in (2002-10-15) http://python.org/sf/623771

1 0