Mailman 3 January 2007 - Python-Dev

Happy Birthday, Guido!
by Aahz 31 Jan '07

31 Jan '07

Thanks again for giving me something fun to do with my life. ;-) -- Aahz (aahz(a)pythoncraft.com) <*> http://www.pythoncraft.com/ "I disrespectfully agree." --SJM

6 6

Scoping [Patience, please]
by lingwitt＠gmail.com 31 Jan '07

31 Jan '07

This email is broken up into multiple sections: (1) Introduction (2) Problem (3) Probably Poor Solution (4) Tangent (Utter Tomfoolery) #### Introduction #### Hello, This is my first post to the list, and I apologize for 4 reasons: (1) I haven't been lurking for more than a few days. (2) I am by no means well versed in python. (3) I know nothing of language design. (4) This isn't a very good introduction. Nonetheless, I have a habit of beginning with institutions that are on the verge of sweeping changes, and python seems no different. To curb the future troubles against which this unfortunate fate of mine often forces me, I have been pouring over PEPs (scanning really, because it's far past my bed time). I came across the discussion of scoping, and while it seems to be a very badly beaten and decaying horse, I ask that you'd be kind enough to include me in this disgruntled discourse. #### Problem #### See http://www.python.org/dev/peps/pep-3104/ Consider the following: >>> x = 10 >>> def a(): ... x = 3 ... def b(): ... global x ... x = 4 ... print x ... ... b() ... print x ... >>> a() 4 3 >>> print x 4 The keyword 'global' is only good for the module scope. #### Probably Poor Solution #### See http://www.python.org/dev/peps/pep-3104/ There seems to be a predilection towards limited and/or general means of outer scope referencing, such as (1) prefixing variables with dots. (2) scope override declarations. However, I don't understand the virtue of this generality (of course, I haven't thought much about it either, so please lend me your expertise). Most of the time, it would seem a person should know exactly which outer variable he is after; he knows exactly which scope. Fortunately, the scopes in question are always named. Can't we just reference what's already there? >>> x = 10 >>> def a(): ... x = 3 ... def b(): ... a.x = 4 ... print a.x ... ... b() ... print x ... >>> a() 4 4 >>> print x 10 Of course, the module itself is unnamed, and it would be completely consistent to use the single prefixed dot to reference it: >>> x = 10 >>> def a(): ... x = 3 ... def b(): ... a.x = 4 ... print a.x ... ... .x = 5 ... b() ... print x ... >>> a() 4 4 >>> print x 5 However, I would think a placeholder is easier to spot: >>> x = 10 >>> def a(): ... x = 3 ... def b(): ... a.x = 4 ... print a.x ... ... @.x = 5 ... b() ... print x ... >>> a() 4 4 >>> print x 5 Of course, what happens if 'a' is a variable name? >>> x = 10 >>> def a(): ... x = 3 ... def b(): ... a = 15 ... a.x = 4 #? ... print a.x ... ... @.x = 5 ... b() ... print x ... This would never happen in practice. Why bother considering it? However, one could get around it with a fuller qualification: >>> x = 10 >>> def a(): ... x = 3 ... def b(): ... a = 15 ... @.a.x = 4 ... print a.x ... ... @.x = 5 ... b() ... print x ... >>> a() 4 4 >>> print x 5 In any case, those are just some thoughts off of the top of my (tired) head. It seems to me that it unifies a lot of scoping ideas. everything can be accessed with a qualified name. Sorry if this is nonsensical. #### Tangent (Utter Tomfoolery) #### More interestingly, one sees (if one squints) the distinction between modules, classes, and functions blur: >>> def new_b(): ... print "tee hee" >>> a.b = new_b() >>>a() tee hee 3 >>> print x 5 or perhaps: >>> # Interestingly, the following code is valid already ... >>> def Person(name, age, location): ... def print_record(): ... print('The last person created had the name ' + Person.name_last) ... print("However, I don't know any of the details") ... ... def instance(): ... def print_record(): ... print('Name: ' + name) ... print('age: ' + str(age)) ... print('location: ' + location) ... ... if name == 'lingwitt': ... name = age = location = 'classified' ... ... Person.name_last = name ... ... return instance ... >>>p1 = Person('spam', 80, 'eggsville') >>>Person.name_last spam >>># Now the code becomes invalid ... >>>Person.print_record() The last person created had the name spam However, I don't know any of the details >>>p1.print_record() name: spam age: 80 location: eggsville It would appear that modules, classes, and functions are scopes with different magic associated with them. #### Warning!!! This is an unlisted section! Sillyness Ensues #### It would appear that a language could be built around scopes; One could define a scope and then use that same scope as a template for other scopes (actually, it sounds kind of like prototyping languages); this seems to encompass the behavior of modules, classes, and functions. Seems like all the "kinds" of scopes could be unified. I'll stop here, because now I'm worried that I've made myself look silly. Thanks for your time. Yours truly, Herr Witten P.S. Be gentle.

2 2

2.5.1 release postponed
by Neal Norwitz 30 Jan '07

30 Jan '07

As you may have noticed we missed the schedule for getting 2.5.1 out. Unfortunately several of us are having scheduling problems with making this release. Since there don't seem to be major problems with the current 2.5 release, our current plan is to defer release until April when everyone should have more time to do it right. n

1 0

Problem with signals in a single threaded application
by Ulisses Furquim 29 Jan '07

29 Jan '07

Hi, I'm aware of the problems with signals in a multithreaded application, but I was using signals in a single-threaded application and noticed something that seemed wrong. Some signals were apparently being lost, but when another signal came in the python handler for that "lost" signal was being called. The problem seems to be inside the signal module. The global variable is_tripped is incremented every time a signal arrives. Then, inside PyErr_CheckSignals() (the pending call that calls all python handlers for signals that arrived) we can return immediately if is_tripped is zero. If is_tripped is different than zero, we loop through all signals calling the registered python handlers and after that we zero is_tripped. This seems to be ok, but what happens if a signal arrives after we've returned from its handler (or even after we've checked if that signal arrived) and before we zero is_tripped? I guess we can have a situation where is_tripped is zero but some Handlers[i].tripped are not. In fact, I've inserted some debugging output and could see that this actually happens and then I've written the following test program to reproduce the problem. #!/usr/bin/env python2.5 import sys import os import time import signal def alarm_handler(*args): sys.stderr.write('alarmmmmmmmmmmm!\n') def sigio_handler(*args): sys.stderr.write('Entering SIGIO handler\n') os.kill(os.getpid(), signal.SIGALRM) sys.stderr.write('Leaving SIGIO handler\n') signal.signal(signal.SIGIO, sigio_handler) signal.signal(signal.SIGALRM, alarm_handler) os.kill(os.getpid(), signal.SIGIO) ini = time.time() while True : if time.time() - ini > 3.0: sys.stderr.write('Loop!\n') ini = time.time() When we run this program, the handler for the SIGALRM isn't called after we return from the SIGIO handler. We return to our main loop and print 'Loop!' every 3 seconds aprox. and the SIGALRM handler is called only when another signal arrives (like when we hit Ctrl-C). I've read some threads about signals in the archives and I was under the impression signals should work reliably on single-threaded applications. Am I right? I've thought about a way to fix this, but I don't know what is the current plan for signals support in python, so what can be done? Best regards, -- Ulisses

5 12

Re: [Python-Dev] Unipath package
by glyph＠divmod.com 29 Jan '07

29 Jan '07

On 28 Jan, 06:30 pm, sluggoster(a)gmail.com wrote: >The discussion has been hampered by the lack of released code. Only >Orendorff's class has been widely used in production systems. The >others have either never been used or only by their authors; they >haven't made it to the Cheeseshop. Unipath is merely to say "Here's >another way to do it; see if it works for you." Certainly the Path >methods need more testing and use in the real world before they'd be >ready for the stdlib. The FSPath methods are more experimental so I'd >say they need a year of use before they can be considered sufficiently >stable. Mike is mistaken here; Twisted has a module, twisted.python.filepath, which has been used extensively in production as well as by other projects. You can see such a project here (just the first hit on google code search): http://www.google.com/codesearch?hl=en&q=+twisted+python+filepath+show:3PfU… and the source to the module itself here: http://twistedmatrix.com/trac/browser/trunk/twisted/python/filepath.py Twisted's filepath implementation also provides a zipfile implementation of the same interface, so that, for example, you can copy a directory in the filesystem and put it into a zipfile with the same function. We plan to eventually also include tarfile and in-memory implementations. Of course, I am FilePath's author, so I have a certain bias; but I think it would be well suited to the standard library, and I would be interested to hear any feedback on it, especially that which would make it unsuitable for inclusion.

1 0

Unipath package
by Mike Orr 29 Jan '07

29 Jan '07

I finally finished my path package (Unipath) and put it in the Cheeseshop. http://sluggo.scrapping.cc/python/unipath/ There's a Path class for pathname calculations, and a FSPath subclass for filesystem calls. I'm hoping Path -- or something resembling it -- will find its way into os.path in Python 2.6 or 3.0. FSPath is full of convenience methods so it may not be everybody's cup of tea, but perhaps something similar can go into Python in the farther future Unipath is an early alpha release so the API may change as it gets more real-world use. There's an extensive unittest suite, which passes on Python 2.5 and 2.4.4 on Linux. Windows and Macintosh testers are needed. Following are highlights from the python-3000 discussion and deviations from it: - Path subclasses unicode, or str if the platform can't handle Unicode pathnames. - FSPath subclasses Path. This allows you to do "from unipath import FSPath as Path" and pretend there's only one class. I find this much more convenient in applications, so you don't have to cast objects back and forth between the classes. Also, it just became infeasable not to let them inherit, because so many methods call other methods. - Nevertheless, you can use Path alone and rest assured it will never touch the filesystem. You can even use Path objects with os.* functions if you are so heretically inclined. If Path is accepted into the stdlib, FSPath will inherit it from there. - I tried splitting FSPath into several mixins according to type of operation, but the need for methods to call other methods in a different category sabotaged that too. So FSPath proudly has about fifty methods. (Path has 10 public methods and 4 properties.) - The dirname property is called .parent. The basename property is .name. The extension property is .ext. The name without extension is .stem. - .components() returns a list of directory components. The first component is "/", a Windows drive root, a UNC share, or "" for a relative path. .split_root() returns the root and the rest. - PosixPath, NTPath, and MacPath are Path subclasses using a specific path library. This allows you to express non-native paths and convert paths. Passing a relative foreign path to a *Path constructor converts it to the destination type. Passing an absolute foreign path is an error, because there's no sane way to interpret "C:\\" on Posix or "/" on Windows. I'm skeptical whether this non-native support is really worth it, because .norm() and .norm_case() already convert slashes to backslashes (which is what Talin really wanted to do -- have Posix paths in a config file and automatically convert them to the native format). And if Python is going to drop Mac OS 9 soon then MacPath is obsolete. So maybe this non-native path code will prove less than useful and will be deleted. On the other hand, if someone is burning to write a zippath or ftppath library, you can use it with this. - Setting Path.auto_norm to true will automatically normalize all paths on construction. I'm not sure if this should be the default, because normalizing may produce the wrong path (e.g., if it contains a symlink). You can always pass norm=True or norm=False to the constructor to enable/disable it. - p.child("subdir", "grandkid") is Glyph's favorite "safe join" method that prevents creating a path pointing above 'p'. You can use it as a .joinpath if you don't mind this restriction. The constructor allows multiple positional arguments, which are joined using os.path.join(). - Listing is p.listdir(pattern=None, filter=None, names_only=False). This returns a non-recursive list of paths. If 'names_only' is true it returns the same as os.listdir(). p.walk(pattern=None, filter=None, top_down=True) is the recursive counterpart, yielding paths. In both cases 'pattern' is a glob pattern, and 'filter' is one of the constants (FILES, DIRS, LINKS, FILES_NO_LINKS, DIRS_NO_LINKS, DEAD_LINKS) or a custom boolean function. I spent a lot of time going back and forth between Orendorff's six listing methods and Raphael's one, and finally decided on this, partly because I'm not satisfied with how either Orendorff's class or Raphael's or os.walk() handle symlinks -- sometimes you want to ignore the links and then iterate them separately. - .read_file(mode) and .write_file(content, mode) are a compromise between Orendorff's seven methods and purists' desire for zero methods. - .mkdir(), .rmdir(), .rmtree(), .copy(), .copy_stat(), .copy_tree(), and .move() are more fancy than their stdlib/Orendorff counterparts. They silently succeed if the operation is already done, and have arguments to smartly create/delete intermediate directories, etc. - Two extra functions are in 'unipath.tools'. 'dict2dir' creates a directory hierarchy modeled after a dict. 'dump_path' displays an ASCII tree of a directory hierarchy, with file sizes and symlink targets showing. Enjoy! and please provide feedback. --Mike Orr <sluggoster(a)gmail.com>

6 11

Re: [Python-Dev] Python's C interface for types
by Nick Maclaren 27 Jan '07

27 Jan '07

=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin(a)v.loewis.de> wrote: > > [not sure what "And so it goes" means in English] I apologise. I try to restrain myself from using excessive idiom, but sometimes I forget. It means "That is how things are, and there is and will be more of the same." > It may be a bit problematic to implement, but I think a clean > specification is possible. If a and b are numbers, and a==b, > then hash(a)==hash(b). I'm not sure whether "approximately 5.0" > equals 5 or not: if it does, it should hash the same as 5, > if it doesn't, it may or may not hash the same (whatever is > easier to implement). > For 0: hash(+0.0)==hash(-0.0)==hash(0)=hash(0L)=0 Unfortunately, that assumes that equality is transitive. With the advanced floating-point models, it may not be. For example, if you want to avoid the loss of error information, exact infinity and approximate infinity (the result of overflow) have different semantics. Similarly with infinitesimals. Even at present, Python's float (Decimal probably more so) doesn't allow you to do some things that are quite reasonable. For example, let us say that I am implementing a special function and want to distinguish -0.0 and +0.0. Why can't I use a dictionary? >>> a = float("+0.0") >>> b = float("-0.0") >>> print a, b 0.0 -0.0 >>> c = {a: "+0.0", b: "-0.0"} >>> print c[a], c[b] -0.0 -0.0 Well, we all know why. But it is not what some quite reasonable programmers will expect. And Decimal (with its cohorts and variant precisions) has this problem quite badly - as do I. No, I don't have an answer. You are damned if you do, and damned if you don't. It is an insoluble problem, and CURRENTLY doesn't justify two hashing mechanisms (i.e. ANY difference and EQUALITY difference). Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [Python-Dev] Problem with signals in a single threaded application
by Nick Maclaren 27 Jan '07

27 Jan '07

I apologise for going off-topic, but this is an explanation of why I said that signal handling is not reliable. The only relevance to Python is that Python should avoid relying on signals if possible, and try to be a little defensive if not. Signals will USUALLY do what is expected, but not always :-( Anything further by Email, please. Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: > > > This one looks like an oversight in Python code, and so is a bug, > > but it is important to note that signals do NOT work reliably under > > any Unix or Microsoft system. > > That's a rather pessimistic way of putting it. In my > experience, signals in Unix mostly do what they're > meant to do quite reliably -- it's just a matter of > understanding what they're meant to do. Yes, it is pessimistic, but I am afraid that my experience is that it is so :-( That doesn't deny your point that they MOSTLY do 'work', but car drivers MOSTLY don't need to wear seat belts, either. I am talking about high-RAS objectives, and ones where very rare failure modes can become common (e.g. HPC and other specialist uses). More commonly, there are plain bugs in the implementations which are sanctioned by the standards (Linux is relatively disdainful of such legalistic games). Because they say that everything is undefined behaviour, many vendors' support mechanisms will refuse to accept bug reports unless you push like hell. And, as some are DIABOLICALLY difficult to explain, let alone demonstrate, they can remain lurking for years or decades. > There may be bugs in certain systems that cause > signals to get lost under obscure circumstances, but > that's no reason for Python to make the situation > worse by introducing bugs of its own. 100% agreed. > > Two related signals received between two 'checkpoints' (i.e. when > > the signal is tested and cleared). You may only get one of them, > > and 'related' does not mean 'the same'. > > I wasn't aware that this could happen between > different signals. If it can, there must be some > rationale as to why the second signal is considered > redundant. Otherwise there's a bug in either the > design or the implementation. Nope. There is often a clash between POSIX and the hardware, or a cause where a 'superior' signal overrides an 'inferior' one. I have seen SIGKILL flush some other signals, for example. And, on some systems, SIGFPE may be divided into the basic hardware exceptions. If you catch SIGFPE as such, all of those may be cleared. I don't think that many (any?) current systems do that. And it is actually specified to occur for the SISSTOP, SIGTSTP, SIGTTIN, SIGTTOU, SIGCONT group. > > A second signal received while the first is being 'handled' by the > > operating system or language run-time system. > > That one sounds odd to me. I would expect a signal > received during the execution of a handler to be > flagged and cause the handler to be called again > after it returns. But then I'm used to the BSD > signal model, which is relatively sane. It's nothing to do with the BSD model, which may be saner but still isn't 100% reliable, but occurs at a lower layer. At the VERY lowest level, when a genuine hardware event causes an interrupt, the FLIH (first-level interrupt handler) runs in God mode (EVERYTHING disabled) until it classifies what is going on. This is a ubiquitous misdesign of modern hardware, but that is off-topic. Hardware 'signals' from other CPUs/devices may well get lost if they occur in that window. And there are other, but less extreme, causes at higher levels in the operating system. Unix and Microsoft do NOT have a reliable signal delivery model, where the sender of a signal checks if the recipient has got it and retries if not. Some operating systems do - but I don't think that BSD does. > > A signal sent while the operating system is doing certain things to > > the application (including, sometimes, when it is swapped out or > > deep in I/O.) > > That sounds like an outright bug. I can't think > of any earthly reason why the handler shouldn't > be called eventually, if it remains installed and > the process lives long enough. See above. It gets lost at a low level. That is why you can cause serious time drift on an "IBM PC" (most modern ones) by hammering the video card or generating streams of floating-point fixups. Most people don't notice, because xntp or equivalent fixes it up. And there are worse problems. I could start on cross-CPU TLB and ECC handling on large shared memory systems. I managed to get an Origin in a state where it wouldn't even power down from the power-off button, and I had to flip breakers, due to THAT one! I have reason to believe that all largish SMP systems have similar problems. Again, it is possible to design an operating system to avoid those issues, but we are talking about mainstream ones, and they don't. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [Python-Dev] Python's C interface for types
by Nick Maclaren 27 Jan '07

27 Jan '07

Having looked into the answers a bit more deeply, I am afraid that I am still a bit puzzled. 1) As I understand it, PyMem_Malloc won't cause trouble, but won't be automatically freed, either, as it doesn't return a new reference. I don't think that immediately following it by PyCObject_FromVoidPtr (which is what I do) helps with that. What I need is some standard type that allows me to allocate an anonymous block of memory; yes, I can define such a type, but that seems excessive. Is there one? 2) _PyLong_FromByteArray and _PyLong_AsByteArray aren't in the API and have no comments. Does that mean that they are unstable, in the sense that they may change behaviour in new versions of Python? And will they be there in 3.0? Thanks for any help, again. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

3 2

Object creation hook
by Kristján V. Jónsson 27 Jan '07

27 Jan '07

Hello there. I am trying to insert a hook into python enabling a callback for all just-created objects. The intention is to debug and find memory leaks, e.g. by having the hook function insert the object into a WeakKeyDictionary. I have already added a method to "object" to set such a hook, and "object_new" now calls it upon completion, but this is far from covering all places. Initially, I thought object_init were the place, but almost no classes call object.__init__ from their __init__ method. Then there is the separate case of old-style classes. Any suggestions on how to do a global object creation hook in python? Kristján

6 7