Mailman 3 September 1999 - Python-Dev

Methods and friends
by M.-A. Lemburg 17 Sep '99

17 Sep '99

I was just looking into classobject.c again after probably more than a year and wondered... has anyone tried optimizing the method lookup and binding process recently ? It seems that for every method call the following happens: 1. the method is looked up in the instance dict; this fails 2. the method is looked up in the class dict; this succeeds and returns a function 3. the class then turns the function into a new unbound method 4. the instance sees the unbound method and creates a new bound method (deleting the unbound method) and returns it Two possible improvements: · let the instance use a special class_getattr_ex function that does not do the extra unbound method step (3.) or · make methods mutable and have step 4. insert the instance object into the method object to turn it into a bound method Has anyone tried this ? Does it produce anything noticeable w/r to method call speed ? BTW, why does the new module check for instance objects for the INSTANCE argument ? Methods can handle any object in this slot, just as they handle any callable object as "function". Could be put to some use when wrapping types into classes or vice versa... which is what motivated the above ;-) Just curious, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 105 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

2 2

Towards a Python based import scheme
by M.-A. Lemburg 16 Sep '99

16 Sep '99

Back again... To get a little more constructive I've started hacking away on Greg Stein's imputil.py to make it work with my DateTime package. The DateTime packages does a lot of from...import... and intra-package imports, plus it loads a shared lib as extension. The original version of imputil I fetched from Greg's page did work out of the box (from...import... hassles) and obviously did not support in-package shared libs. I've added both features so that the test script in DateTime can run successfully. Things that remain are: · the win32 registry stuff (needs C code) · the Mac fork stuff (needs C code) · a working __path__ implementation (is anyone using this attribute which only is available in packages ?) · probably a whole bunch of other quirks · some speedups (there currently are too many stat()s) Please give it a try: http://starship.skyport.net/~lemburg/imputil.py in color: http://starship.skyport.net/~lemburg/imputil.py.html -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

6 9

relative package imports, version numbering, yadda yadda
by Skip Montanaro 14 Sep '99

14 Sep '99

I haven't been paying real close attention to the relative package import thread on python-dev, but some way or other I strongly believe something has to be done to improve the situation I now find myself in or more and more people are going to start getting bitten by the same sort of problem. Allow me to 'splain. I use a web server based on Zope's ZServer (which is itself based on medusa) sitting behind an Apache web server as a long-running process to handle what many people traditionally used CGI for. It's a big performance boost over CGI. Unfortunately, I have recently been experiencing frequent server hangs. So far I've been unable to figure out what the cause is. I do notice occasional tracebacks like: Unhandled exception in thread: Traceback (innermost last): File "ZServer/PubCore/ZServerPublisher.py", line 97, in __init__ File "/home/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish self.stdout.close() File "/home/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close self._channel.push(CallbackProducer(self._channel.done)) File "/home/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push if send: self.initiate_send() File "/usr/lib/python1.5/asynchat.py", line 199, in initiate_send _push_mode = 0 File "/usr/lib/python1.5/asynchat.py", line 191, in refill_buffer self.ac_in_buffer = '' File "/usr/lib/python1.5/asynchat.py", line 253, in pop # this could maybe be made faster with a computed regex? IndexError: list index out of range and also notice that the server can pile up a huge number of connections in the ESTABLISHED state, at which point the whole mess grinds to a halt with not much computation or network traffic happening. A separate shell script uses netstat to detect when a large number of sockets have piled up and restarts the server. Brutal, but crudely effective. (When in doubt, treat the symptoms...) Today it just dawned on me looking at the above traceback that ZServer is getting the wrong version of asynchat.py (and presumably of asyncore.py as well). It's using the version delivered with Python 1.5.2 distribution instead of the version that was delivered with the version of Zope I'm using (1.something). So I started looking around at the versions and dates of various copies of asynchat.py. Here's what I found: source version number:date owner Python 1.5.2 1.2:1999/06/18 guido Python CVS 1.2:1999/06/18 guido Zope 1.? 1.7:1999/04/09 amos Zope 2.0 1.9:1999/07/19 amos Medusa 990902 2.24:1999/07/07 rushing What's apparently been happening is that people have picked up asyncore and asynchat at various time and stuck them in their own CVS repositories without somehow freezing the Id string of the version they originally got from Sam Rushing. It's not clear what the differences are until you compare the actual files. It turns out that the Zope 2.0 and Medusa versions have no content differences, only wildly different version numbers. The Medusa and Python CVS versions only have one difference: if index > 0: # don't bother reporting the empty string (source of subtle bugs) self.collect_incoming_data (self.ac_in_buffer[:index]) which *may* be what's causing my problems (note the IndexError in my traceback). Somewhere along the way I think we need to apply more restraint to our packaging and numbering of Python modules. I don't know what form that restraint will eventually take, but at this point I'm willing to replace Greg Ward's s/must/should/ with s/should/must/. a-nightmare-indeed!-i don't think-this-is-what-Sam-meant-ly y'rs Skip Montanaro | http://www.mojam.com/ skip(a)mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented...

4 3

Memory usage in Python
by Ivan Porres 13 Sep '99

13 Sep '99

Dear all, I've been reading this list for a while but this is the first time I post a message. I am interested on using Python as a scripting language for small embedded systems. One of the limiting factors on a embedded system is the memory. Memory cost money, uses space and drains power. We have released a big patch to selectively disable some features of a Python interpreter at compile time. With the patch, it is possible to disable, i.e., complex numbers or file objects, in order to reduce the memory foot print of the Python interpreter. The patch is available at http://www.abo.fi/~iporres/python Now, I am interested on studying and reducing the memory consumption of Python programs. That is all the memory allocated by the interpreter in order to run a python program. I would like to know if somebody else has been working on the same direction. Is there any document explaining how Python uses the memory? I would be nice to know which structures are most used, their size, how often are accessed, etc... I have the impression that most people is concerned about optimising Python for speed, but I still think that some Python users will also benefit from size optimisations. Ivan -- Ivan Porres Paltor Turku Centre for Computer Science Åbo Akademi, Department of Computer Science Phone: +358-2-2154033 Lemminkäinengatan 14A FIN-20520 Turku - Finland http://www.abo.fi/~iporres

2 1

Catching "return" and "return expr" at compile time
by Skip Montanaro 12 Sep '99

12 Sep '99

Attached is a context diff against the latest version of Python/compile.c that checks at compile time for functions that both return expressions or execute return statements with no expression (or equivalently, fall off the end of the function). I figured I'd post it here to get a little friendly feedback and bug discovery before shooting it off to c.l.py. I modified compile.c instead of some preexisting PyLint script because I don't know what's popular out there. On the other hand, I'm sure most people who would be interested in this sort of thing have access to the C source... The basic idea is that each straight line chunk of code is terminated one of four ways: 1. return with no expression 2. return an expression 3. raise an exception 4. fall off the end of the chunk Falling off the end of the function is obviously treated like return with no expression. (This is, after all, what motivated me to do this. ;-) This information is recorded in a new bit vector added to the struct compiling object that's carried around during the compilation. Compound statements simply aggregate the bit vectors for their various clauses in ways appropriate to their semantics. At the end of a function's compilation, the set of return bits computed up to that point tells you whether or not to spit out a warning. Note that it does nothing to recognize constant expressions. The following function will generate a warning: def f(): i = 0 while 1: i = i + 1 if i > 10: return i even though the only way to return from the function is the return statement. To get the above to shut up the compiler you'd have to do something like class CantGetHere: pass def f(): i = 0 while 1: i = i + 1 if i > 10: return i raise CantGetHere Raise statements are treated as a valid way to "return" from a function. Registering them as separate styles of returns serves effectively to turn off the "no return" bit for a block of code. Raise is compatible with either form of return, though they aren't compatible with each other. The code is run whenever a module is compiled. I didn't bother to add a new flag to the python interpreter to enable/disable warnings during compilation, though a -w switch for Python has been mentioned before. I ran the modified byte code compiler over a silly test module as well as executing ./python Lib/test/regrtest.py ./python Lib/compileall.py It uncovered some interesting programming practices and one item I think is an actual bug. In Lib/gzip.py, GzipFile._read returns EOFError at one point instead of raising it. At other points in the method it raises EOFError. There are no other return statements in the function. (I haven't taken the time/had the nerve to run it against my own Python code yet. ;-) I'm firmly of the opinion that more subtle bugs exist in the way people write functions that return and raise values than in the code that calls those functions, contrary to a few vocal folks on c.l.py who may believe otherwise. Enjoy, Skip Montanaro | http://www.mojam.com/ skip(a)mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented...

9 25

FYI: Techniques for scientific C++
by David Ascher 12 Sep '99

12 Sep '99

In case you haven't seen enough C++ papers, you might find this paper interesting. It's a good way to learn more about templates without trying to actually use them, and it's a very good way to decide to stay away from high-power C++ techniques like expression templates. Yeeagh! (I especially like the program shown in section 1.11.1). --david Date: Thu, 26 Aug 1999 21:59:06 -0500 (EST) From: Todd Veldhuizen <tveldhui(a)extreme.indiana.edu> Subject: OON: Techniques for scientific C++ I've updated and expanded my slides on "Techniques for Scientific C++" into a document. It's available in HTML and Postscript from: HTML: http://extreme.indiana.edu/~tveldhui/papers/techniques/ PS: http://extreme.indiana.edu/~tveldhui/papers/techniques/techniques.ps Here's the table of contents. 1.1: About this document 1.2: C++ Compilers 1.2.1: Placating sub-standard compilers 1.2.2: The compiler landscape 1.2.3: C++-specific optimization 1.3: Compile times 1.3.1: Headers 1.3.2: Prelinking 1.3.3: The program database approach -- Visual Age C++ 1.3.4: Quadratic/Cubic template algorithms 1.4: Static Polymorphism 1.4.1: Are virtual functions evil? 1.4.2: Solution A: simple engines 1.4.3: Solution B: the Barton and Nackman Trick 1.5: Callback inlining techniques 1.5.1: Callbacks: the typical C++ approach 1.5.1.1: Expression templates 1.5.1.2: STL-style function objects 1.5.1.3: Pointer-to-function as a template parameter 1.6: Managing code bloat 1.6.1: Avoid kitchen-sink template parameters 1.6.2: Put function bodies outside template classes 1.6.3: Inlining levels 1.7: Containers 1.7.1: STL-style containers 1.7.2: Data/View containers 1.8: Aliasing and restrict 1.9: Traits 1.9.1: An example: average() 1.9.2: Type promotion example 1.10: Expression templates 1.10.1: Performance implications of pairwise evaluation 1.10.2: Recursive templates 1.10.3: Expression templates: building parse trees 1.10.4: A minimal implementation 1.10.5: Refinements 1.10.6: Pointers to more information 1.10.7: References 1.11: Template metaprograms 1.11.1: Template metaprograms: some history 1.11.2: The need for specialized algorithms 1.11.3: Using template metaprograms to specialize algorithms 1.12: Comma overloading 1.12.1: An example 1.13: Interfacing with Fortran codes 1.14: Some thoughts on performance tuning 1.14.1: General suggestions 1.14.2: Know what your compiler can do 1.14.3: Data structures and algorithms 1.14.4: Efficient use of the memory hierarchy Cheers, Todd - -- Todd Veldhuizen tveldhui(a)acm.org Indiana Univ. Comp. Sci. http://extreme.indiana.edu/~tveldhui/ - --------------------- Object Oriented Numerics List --------------------------

2 1

jpythonc: unreachable Python becomes unreachable Java (fwd)
by David Ascher 11 Sep '99

11 Sep '99

Greg Ward notes on the JPython list: > Looks like I've found a definite bug in jpythonc -- unreachable Python > code compiles to unreachable Java code. Here's a simple example: > [..] > Yikes... does this mean that jpythonc will have to analyze Python code > for reachability? Is this even possible? Guess the answer for now is, > "Don't do that!". This is another sort of warning that a checker could do in the core which could catch some errors. Not sure that it makes sense to try and find all the cases that an arbitrary Java compiler could find, but presumably Greg's original code which led him to the jpythonc bug find had a plain-old python bug in it? Did Aaron's pylint find any nonreachable code? --david

1 0

Where does lint-like functionality belong?
by Skip Montanaro 10 Sep '99

10 Sep '99

Okay, if Tim's assertions are correct (and I have no reason to suspect them), a separate pylint will be doomed, so the only reasonable thing to do is to place it in the core where it will be rarin' to go all the time. Perl's experience with -w seems to suggest that it's best to always enable whatever warnings you can as well. (More and more I see people using gcc's -Wall flag as well.) Now, my return consistency stuff was easy enough to write in C for two reasons. One, I'm fairly comfortable with the compile.c code. Two, adding my checks required no extra memory management overhead. Consider a few other checks you might conceivably add to the byte code compiler: * tab nanny stuff (already enabled with -t, right?) * variables set but not used * variables used before set If all of this sort of stuff is added to the compiler proper, I predict a couple major problems will surface: * The complexity of the code will increase significantly, making it harder to maintain and enhance * Fewer and fewer people will be able to approach the code, making it less likely that new checks are added * Future extensions like pluggable virtual machines will be harder to add because their byte code compilers will be harder to integrate into the core In addition, more global checks probably won't be possible (reasoning about code across module boundaries for instance) because the compiler's view of the world is fairly narrow. I think lint-like tools should be implemented in Python (possibly with the support of an extension module for performance-critical sections) which is then called from the compiler proper under appropriate circumstances (warnings enabled, all necessary modules importable, etc). I believe the code would be much more easily maintained and extended. You'd be able to swap in a new byte code compiler without risking the loss of your checking code. Skip

1 0

More Python command-line features
by Jim Fulton 08 Sep '99

08 Sep '99

While we're talking about Python command-line features, I have a couple of requests that would make life alot easier for us. 1. I'd like the options given before the script name to be accessable to Python scripts. For example, in: python -O foo.py bar spam I'd like to have a sys variable that told me that the arguments ['-O'] were included before the arguments ['foo.py', 'bar', 'spam'] (aka sys.argv). This is needed if I want to fork/exec (or span or whatever) Python with the same option. 2. I'd like to be able to supply environment variables to Python on the command line, as in: python -O -e PYTHONHOME=/Zope foo.py bar spam This would be very helpful in environment-variable-challenged environments like windows 9x, and might be handy elswhere as well. If people agree that these would be good ideas, I'd ve happy to supply a patch. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

4 3