Mailman 3 September 1999 - Python-Dev

Alternative Approach to Relative Imports
by M.-A. Lemburg Oct. 4, 1999

Oct. 4, 1999

Hi everybody, I'm currently busy converting my code to use mx.DateTime instead of just DateTime and must say that this is a real pain. Now instead of reviving the __ discussion I'd like to turn to what could be a workable compromise (I think Jim proposed something along these lines already and there was similar code in ni.py). The proposal is really only an addition to the lookup scheme used by the importer. No additional syntax is involved and best of all, it is backward compatible... The … [View More]

8 16

Naming and Binding of Objects (was: Relative Package Imports)
by Vladimir Marangozov Oct. 3, 1999

Oct. 3, 1999

The relative import thread has hit me with full force. I'm not sure I understand all of what has been said, but since I know this is mainly a naming issue, I want to point you to one of the best CS papers I have had in my hands. It's not really my style to recommend references, but this one should be of major interest to this audience (and Python respectively): Saltzer J., "Naming and Binding of Objects", in Bayer R., "Operating Systems - An Advanced Course", pp. 99-208, LNCS 60, 1978. It's … [View More]

3 2

getopt helper?
by Mark Hammond Oct. 3, 1999

Oct. 3, 1999

A little thought I have had a few times now, so it is time to pass it on for comment. Basically _every_ time I use getopt, I write code like this: import getopt try: opts, args = getopt.getopt(sys.argv[1:], "slc:d:e:") except getopt.error, why: print why print usage % (os.path.basename(sys.argv[0],)) sys.exit(1) Every single time. I have never used getopt without this code. How about we put this functionality into getopt itself? It could be triggered either by adding a new "… [View More]

8 15

Re: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) (fwd)
by David Ascher Sept. 30, 1999

Sept. 30, 1999

---------- Forwarded message ---------- On Sat, 18 Sep 1999, Vladimir Marangozov wrote: > > Saltzer J., "Naming and Binding of Objects", in Bayer R., "Operating > Systems - An Advanced Course", pp. 99-208, LNCS 60, 1978. > > It's not available online (it was written on a typewriter), so I'd be > happy to send a hard copy of it to anyone who raises a hand in private > mail (or cannot find LNCS 60). I then asked him for a copy stating: > If you send it to me, I can OCR … [View More]

3 3

Tackling circular dependencies in 2.0?
by Skip Montanaro Sept. 22, 1999

Sept. 22, 1999

I think in Python 2.0 it would be nice to have some way to reclaim circular dependencies without the programmer explicitly having to do something like implement a destroy() method and requiring other programmers to (remember to) call it. I forget what the current state of affairs is w.r.t. future memory management in Python. Not knowing anything much about memory management, would it be possible to have a sort of mixed ref count/garbage collection system where you only use the gc stuff as a … [View More]

5 10

Re: Path hacking
by James C. Ahlstrom Sept. 22, 1999

Sept. 22, 1999

"Guido van Rossum" wrote: > --> The solution: Ah, finally a specific proposal... > Suppose your application (as a whole, not the individual top-level > script) is called Spam -- this may well also be the name of your > top-level package. Then start each top-level script with the single > line > > import Spam_path > > before importing anything else. This should not be necessary if you use the name "sitecustomize" instead of "… [View More]Spam_path" right? The file sitecustomize.py is automatically imported. Actually all this sounds like site.py all over again. > Your installer, once it knows the absolute pathname of your > application's root directory, crafts a file Spam_path.py which > contains code that inserts the right absolute pathname into sys.path. I don't think this is necessary either. The sys module is available. So sitecustomize.py can say: import sys mydir = sys.path[0] if not mydir: import os mydir = os.getcwd() sys.path = [mydir] # To be really extreme about it # Note: inserting mydir as sys.path[0] should be redundant but is not > Your installer then installs a copy of this file (or a symbolic link > to it) *in each bin directory where it installs top-level Python > scripts*. > > Because the script's directory is first on the default path, the Spam > scripts will pick up Spam_path without any help from $PYTHONPATH. Hmmm. Is this really true? Nothing else, for example the registry, can change sys.path[0]? Ever? Please say yes. > I know this doesn't resolve the relative import thread (how's that > going by the way? :-) but Barry & Fred & I agree that this is the best > solution to the problem stated in Barry's message to which I am > following up here. This is a good idea, but there are a few problems. It depends on sys.path[0] being the directory of the Python file being executed as the main program. I guess I never really trusted this before. I think if this is the case it should never be ''. A relative path or no path on the command line (the __main__ program) should be replaced by the full path in the sys module setup. Then the "mydir = os.getcwd()" above is not necessary. And inserting mydir as sys.path[0] is truly redundant should the current directory change (as it certainly will). This is currently a problem with sys.path[0] which should be fixed no matter what else happens. The files exceptions.py and site.py must be in all the bin directories as well as sitecustomize.py because they are automatically imported in Py_Initialize(). The above doesn't work when you start the Python command interpreter (no main). I know, its a minor point. It seems to me this totally solves Jim Fulton's and Marc's problem and makes "__" unnecessary. You just install zope and mx in zopedir, perform the above, and presto you have a new private name space where you can control all your names. But there must be some problem here I haven't thought of. I still worry that this is not powerful enough. Greg Stein has volunteered to re-write import.c in Python (using imputil.py) and this is a Great Idea. Lots of Python could probably be written in itself. I would like to try writing the main program in Python and eliminating the special freeze main program. Once you start on this road (and I think it is a good road) you have Python code which is more truly part of the binary interpreter than a library. Proposal: Use a special PYTHONPATH on startup to find "special" Python files which are really part of the interpreter. There are three directories Python knows about. Namely sys.path[0] (once it is fixed), sys.executable and sys.dllfullpath, the directory of python15.dll or other shared library (once it is added to sys). How about prepending the single directory sys.executable to sys.path during Py_Initialize()? And demanding that modules like the new Greg_importer.py[c], exceptions.py[c] and site.py[c] be placed there. Actually I would prefer sys.dllfullpath if it exists, since that is where the interpreter is, and I am trying to associate these special internal Python files exactly with their correct Python interpreter. Alternative Proposal: Py_Initialize() first imports its files from sys.executable + '/' + PyInternal.pyl (again I prefer sys.dllfullpath). PyInternal.pyl is a Python library file (like a Java Jar file) which would contain modules like exceptions, etc. The PyInternal.pyl file has the standard Python library file format (whatever that turns out to be). It is not an error if this file is absent. Jim Ahlstrom [View Less]

9 27

wet signature form clarification
by Guido van Rossum Sept. 20, 1999

Sept. 20, 1999

Several of you have asked me a clarification of the "contribution description" required on the wet signature form; given that you are all very prolific contributors with a weak memory, it would be nice if you could write something like "everything I've contributed until Sept. 18, 1999" in the space there rather than dig out all the specific module names. I got a call from CNRI's lawyer today, and she agreed that this was fine. Boy am I pleased today! I think she called it a "group … [View More]

1 0

Preview of some distutils news
by Greg Ward Sept. 20, 1999

Sept. 20, 1999

Judging by the recent traffic on python-dev, lots of people here are interested in the problems of distributing and installing Python modules. So I thought y'all might want to know that I've finally got some interesting news to report on the distutils-sig, and will shortly be reporting it, soliciting feedback, testers, etc. If you're not already on that list, this might be a good time to join. (Oh, the news? Distutils can now compile and install NumPy, PIL, and mxDateTime (just because I … [View More]

1 0

Relative Package Imports
by M.-A. Lemburg Sept. 20, 1999

Sept. 20, 1999

Hi everybody, I've spent the last two hours trying to get relative package imports to work because I need them for my extension packages which will soon all move under a new top-level package name to overcome the conflicts with PIL and Zope. Here are the results... Demo Package Structure: (see the attached demopkg.zip) [a] [b] bc.py ab.py With the attached patch you can do the following: # Pretty useless... import a.b.__.ab # From inside bc.py: from __ import … [View More]ab # At top-level (also useless, but shows how this situation is handled): import __.sys # __ is bound to None since we are at top-level; sys is still # being loaded though. Of course, common usage will be of the form: form __.__ import submodule_at_higher_level Please tell me what you think and give it a try. It's a first try and may have some design errors. Especially the way head and tail are treated in Python/import.c:import_module_ex may cause trouble -- I need help here. Note: The patch is against the CVS version. If you run Python in verbose mode, the patch will produce some verbose output of what it's doing. Enjoy, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 113 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ --- /home/lemburg/orig/Python/Python/import.c Fri Apr 9 19:00:51 1999 +++ Python/import.c Fri Sep 10 20:51:02 1999 @@ -1572,10 +1572,14 @@ load_next(mod, altmod, p_name, buf, p_bu char *dot = strchr(name, '.'); int len; char *p; PyObject *result; + if (Py_VerboseFlag) + printf("# load_next: (1) name='%s', buf='%.*s'\n", + name,*p_buflen,buf); + if (dot == NULL) { *p_name = NULL; len = strlen(name); } else { @@ -1586,10 +1590,39 @@ load_next(mod, altmod, p_name, buf, p_bu PyErr_SetString(PyExc_ValueError, "Empty module name"); return NULL; } + /* Handle "__" indicator telling the import mechanism to + continue the search one level higher in the package + hierarchy */ + if (strncmp(name,"__",len) == 0) { + PyObject *modules = PyImport_GetModuleDict(); + + /* Strip the final dotted name from buf */ + dot = strrchr(buf, '.'); + if (dot == NULL) + *p_buflen = 0; + else + *p_buflen = dot - buf; + buf[*p_buflen] = '\0'; + + /* Fetch the parent module or revert to a top-level search */ + if (*p_buflen > 0) { + mod = PyDict_GetItemString(modules,buf); + if (mod == NULL) { + PyErr_SetString(PyExc_SystemError, + "Parent module missing"); + return NULL; + } + } + else + mod = Py_None; + Py_INCREF(mod); + return mod; + } + p = buf + *p_buflen; if (p != buf) *p++ = '.'; if (p+len-buf >= MAXPATHLEN) { PyErr_SetString(PyExc_ValueError, @@ -1597,10 +1630,14 @@ load_next(mod, altmod, p_name, buf, p_bu return NULL; } strncpy(p, name, len); p[len] = '\0'; *p_buflen = p+len-buf; + + if (Py_VerboseFlag) + printf("# load_next: (2) modname='%s', fullname=buf='%s'\n", + p,buf); result = import_submodule(mod, p, buf); if (result == Py_None && altmod != mod) { Py_DECREF(result); /* Here, altmod must be None and mod must not be None */ [View Less]

13 81

License cleanup
by Guido van Rossum Sept. 18, 1999

Sept. 18, 1999

After more than four years of living with an out-of-dated license for Python, CNRI has finally agreed to clean up Python's copyright status. I expect that this won't have any real effect before Python 1.6 is released, but I am required to start preparing for the transition now. We will use a new license (a clone of the JPython license) and we will require that all contributors explicitly allow us the use of their contribution: either a few email paragraphs in an email message, or a longer form … [View More]

14 30

Methods and friends
by M.-A. Lemburg Sept. 17, 1999

Sept. 17, 1999

I was just looking into classobject.c again after probably more than a year and wondered... has anyone tried optimizing the method lookup and binding process recently ? It seems that for every method call the following happens: 1. the method is looked up in the instance dict; this fails 2. the method is looked up in the class dict; this succeeds and returns a function 3. the class then turns the function into a new unbound method 4. the instance sees the unbound method and creates a new … [View More]

2 2

Towards a Python based import scheme
by M.-A. Lemburg Sept. 16, 1999

Sept. 16, 1999

Back again... To get a little more constructive I've started hacking away on Greg Stein's imputil.py to make it work with my DateTime package. The DateTime packages does a lot of from...import... and intra-package imports, plus it loads a shared lib as extension. The original version of imputil I fetched from Greg's page did work out of the box (from...import... hassles) and obviously did not support in-package shared libs. I've added both features so that the test script in DateTime can run … [View More]

6 9

relative package imports, version numbering, yadda yadda
by Skip Montanaro Sept. 14, 1999

Sept. 14, 1999

I haven't been paying real close attention to the relative package import thread on python-dev, but some way or other I strongly believe something has to be done to improve the situation I now find myself in or more and more people are going to start getting bitten by the same sort of problem. Allow me to 'splain. I use a web server based on Zope's ZServer (which is itself based on medusa) sitting behind an Apache web server as a long-running process to handle what many people traditionally … [View More]used CGI for. It's a big performance boost over CGI. Unfortunately, I have recently been experiencing frequent server hangs. So far I've been unable to figure out what the cause is. I do notice occasional tracebacks like: Unhandled exception in thread: Traceback (innermost last): File "ZServer/PubCore/ZServerPublisher.py", line 97, in __init__ File "/home/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish self.stdout.close() File "/home/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close self._channel.push(CallbackProducer(self._channel.done)) File "/home/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push if send: self.initiate_send() File "/usr/lib/python1.5/asynchat.py", line 199, in initiate_send _push_mode = 0 File "/usr/lib/python1.5/asynchat.py", line 191, in refill_buffer self.ac_in_buffer = '' File "/usr/lib/python1.5/asynchat.py", line 253, in pop # this could maybe be made faster with a computed regex? IndexError: list index out of range and also notice that the server can pile up a huge number of connections in the ESTABLISHED state, at which point the whole mess grinds to a halt with not much computation or network traffic happening. A separate shell script uses netstat to detect when a large number of sockets have piled up and restarts the server. Brutal, but crudely effective. (When in doubt, treat the symptoms...) Today it just dawned on me looking at the above traceback that ZServer is getting the wrong version of asynchat.py (and presumably of asyncore.py as well). It's using the version delivered with Python 1.5.2 distribution instead of the version that was delivered with the version of Zope I'm using (1.something). So I started looking around at the versions and dates of various copies of asynchat.py. Here's what I found: source version number:date owner Python 1.5.2 1.2:1999/06/18 guido Python CVS 1.2:1999/06/18 guido Zope 1.? 1.7:1999/04/09 amos Zope 2.0 1.9:1999/07/19 amos Medusa 990902 2.24:1999/07/07 rushing What's apparently been happening is that people have picked up asyncore and asynchat at various time and stuck them in their own CVS repositories without somehow freezing the Id string of the version they originally got from Sam Rushing. It's not clear what the differences are until you compare the actual files. It turns out that the Zope 2.0 and Medusa versions have no content differences, only wildly different version numbers. The Medusa and Python CVS versions only have one difference: if index > 0: # don't bother reporting the empty string (source of subtle bugs) self.collect_incoming_data (self.ac_in_buffer[:index]) which *may* be what's causing my problems (note the IndexError in my traceback). Somewhere along the way I think we need to apply more restraint to our packaging and numbering of Python modules. I don't know what form that restraint will eventually take, but at this point I'm willing to replace Greg Ward's s/must/should/ with s/should/must/. a-nightmare-indeed!-i don't think-this-is-what-Sam-meant-ly y'rs Skip Montanaro | http://www.mojam.com/ skip(a)mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... [View Less]

4 3

Memory usage in Python
by Ivan Porres Sept. 13, 1999

Sept. 13, 1999

Dear all, I've been reading this list for a while but this is the first time I post a message. I am interested on using Python as a scripting language for small embedded systems. One of the limiting factors on a embedded system is the memory. Memory cost money, uses space and drains power. We have released a big patch to selectively disable some features of a Python interpreter at compile time. With the patch, it is possible to disable, i.e., complex numbers or file objects, in order to … [View More]

2 1

Catching "return" and "return expr" at compile time
by Skip Montanaro Sept. 12, 1999

Sept. 12, 1999

Attached is a context diff against the latest version of Python/compile.c that checks at compile time for functions that both return expressions or execute return statements with no expression (or equivalently, fall off the end of the function). I figured I'd post it here to get a little friendly feedback and bug discovery before shooting it off to c.l.py. I modified compile.c instead of some preexisting PyLint script because I don't know what's popular out there. On the other hand, I'm sure … [View More]most people who would be interested in this sort of thing have access to the C source... The basic idea is that each straight line chunk of code is terminated one of four ways: 1. return with no expression 2. return an expression 3. raise an exception 4. fall off the end of the chunk Falling off the end of the function is obviously treated like return with no expression. (This is, after all, what motivated me to do this. ;-) This information is recorded in a new bit vector added to the struct compiling object that's carried around during the compilation. Compound statements simply aggregate the bit vectors for their various clauses in ways appropriate to their semantics. At the end of a function's compilation, the set of return bits computed up to that point tells you whether or not to spit out a warning. Note that it does nothing to recognize constant expressions. The following function will generate a warning: def f(): i = 0 while 1: i = i + 1 if i > 10: return i even though the only way to return from the function is the return statement. To get the above to shut up the compiler you'd have to do something like class CantGetHere: pass def f(): i = 0 while 1: i = i + 1 if i > 10: return i raise CantGetHere Raise statements are treated as a valid way to "return" from a function. Registering them as separate styles of returns serves effectively to turn off the "no return" bit for a block of code. Raise is compatible with either form of return, though they aren't compatible with each other. The code is run whenever a module is compiled. I didn't bother to add a new flag to the python interpreter to enable/disable warnings during compilation, though a -w switch for Python has been mentioned before. I ran the modified byte code compiler over a silly test module as well as executing ./python Lib/test/regrtest.py ./python Lib/compileall.py It uncovered some interesting programming practices and one item I think is an actual bug. In Lib/gzip.py, GzipFile._read returns EOFError at one point instead of raising it. At other points in the method it raises EOFError. There are no other return statements in the function. (I haven't taken the time/had the nerve to run it against my own Python code yet. ;-) I'm firmly of the opinion that more subtle bugs exist in the way people write functions that return and raise values than in the code that calls those functions, contrary to a few vocal folks on c.l.py who may believe otherwise. Enjoy, Skip Montanaro | http://www.mojam.com/ skip(a)mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... [View Less]

9 25

FYI: Techniques for scientific C++
by David Ascher Sept. 11, 1999

Sept. 11, 1999

In case you haven't seen enough C++ papers, you might find this paper interesting. It's a good way to learn more about templates without trying to actually use them, and it's a very good way to decide to stay away from high-power C++ techniques like expression templates. Yeeagh! (I especially like the program shown in section 1.11.1). --david Date: Thu, 26 Aug 1999 21:59:06 -0500 (EST) From: Todd Veldhuizen <tveldhui(a)extreme.indiana.edu> Subject: OON: Techniques for scientific C++ … [View More]I've updated and expanded my slides on "Techniques for Scientific C++" into a document. It's available in HTML and Postscript from: HTML: http://extreme.indiana.edu/~tveldhui/papers/techniques/ PS: http://extreme.indiana.edu/~tveldhui/papers/techniques/techniques.ps Here's the table of contents. 1.1: About this document 1.2: C++ Compilers 1.2.1: Placating sub-standard compilers 1.2.2: The compiler landscape 1.2.3: C++-specific optimization 1.3: Compile times 1.3.1: Headers 1.3.2: Prelinking 1.3.3: The program database approach -- Visual Age C++ 1.3.4: Quadratic/Cubic template algorithms 1.4: Static Polymorphism 1.4.1: Are virtual functions evil? 1.4.2: Solution A: simple engines 1.4.3: Solution B: the Barton and Nackman Trick 1.5: Callback inlining techniques 1.5.1: Callbacks: the typical C++ approach 1.5.1.1: Expression templates 1.5.1.2: STL-style function objects 1.5.1.3: Pointer-to-function as a template parameter 1.6: Managing code bloat 1.6.1: Avoid kitchen-sink template parameters 1.6.2: Put function bodies outside template classes 1.6.3: Inlining levels 1.7: Containers 1.7.1: STL-style containers 1.7.2: Data/View containers 1.8: Aliasing and restrict 1.9: Traits 1.9.1: An example: average() 1.9.2: Type promotion example 1.10: Expression templates 1.10.1: Performance implications of pairwise evaluation 1.10.2: Recursive templates 1.10.3: Expression templates: building parse trees 1.10.4: A minimal implementation 1.10.5: Refinements 1.10.6: Pointers to more information 1.10.7: References 1.11: Template metaprograms 1.11.1: Template metaprograms: some history 1.11.2: The need for specialized algorithms 1.11.3: Using template metaprograms to specialize algorithms 1.12: Comma overloading 1.12.1: An example 1.13: Interfacing with Fortran codes 1.14: Some thoughts on performance tuning 1.14.1: General suggestions 1.14.2: Know what your compiler can do 1.14.3: Data structures and algorithms 1.14.4: Efficient use of the memory hierarchy Cheers, Todd - -- Todd Veldhuizen tveldhui(a)acm.org Indiana Univ. Comp. Sci. http://extreme.indiana.edu/~tveldhui/ - --------------------- Object Oriented Numerics List -------------------------- [View Less]

2 1

jpythonc: unreachable Python becomes unreachable Java (fwd)
by David Ascher Sept. 10, 1999

Sept. 10, 1999

Greg Ward notes on the JPython list: > Looks like I've found a definite bug in jpythonc -- unreachable Python > code compiles to unreachable Java code. Here's a simple example: > [..] > Yikes... does this mean that jpythonc will have to analyze Python code > for reachability? Is this even possible? Guess the answer for now is, > "Don't do that!". This is another sort of warning that a checker could do in the core which could catch some errors. Not sure that it makes … [View More]

1 0

Where does lint-like functionality belong?
by Skip Montanaro Sept. 10, 1999

Sept. 10, 1999

Okay, if Tim's assertions are correct (and I have no reason to suspect them), a separate pylint will be doomed, so the only reasonable thing to do is to place it in the core where it will be rarin' to go all the time. Perl's experience with -w seems to suggest that it's best to always enable whatever warnings you can as well. (More and more I see people using gcc's -Wall flag as well.) Now, my return consistency stuff was easy enough to write in C for two reasons. One, I'm fairly comfortable … [View More]

1 0

More Python command-line features
by Jim Fulton Sept. 8, 1999

Sept. 8, 1999

While we're talking about Python command-line features, I have a couple of requests that would make life alot easier for us. 1. I'd like the options given before the script name to be accessable to Python scripts. For example, in: python -O foo.py bar spam I'd like to have a sys variable that told me that the arguments ['-O'] were included before the arguments ['foo.py', 'bar', 'spam'] (aka sys.argv). This is needed if I want to fork/exec (or span or whatever) Python … [View More]

4 3