Mailman 3 March 2004 - Python-Dev

Re: [Python-Dev] [ python-Patches-876206 ] scary frame speed hacks
by Christian Tismer 03 Mar '04

03 Mar '04

Armin Rigo wrote: > Hello, > > In summary, we need to: > > - cache possibly more than one frame per code object; > - free old frames for code objects that haven't been run for a while. > > At this point I need more thinking. If someone comes up with an elegant way > to do both (as opposed, say, to abusing a lot of fields to put PyFrameObject > structures into two linked lists at the same time), he'd be welcome :-) What's bad about my proposal? In short: No frame cache in the code object. Have a small array of chains of cached frames. Index this by the address of the code object modulo some prime which is the array size. -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

1 0

python installation in solaris machine
by V.Selvam 02 Mar '04

02 Mar '04

Hi, Im newb to python. I tried to install Python in my sun Solaris machine(SPARC). The following steps I did 1. Downloaded the python source from python.org download page. 2. unzipped the archival. 3. run the configure 4. then I tried to run the 'make' command, Im getting the error message saying that 'fatal error' and come out from installation. Error is given below (Please look at the attached file for details.) /********************************************************************************************************************************* gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DHAVE_CONFIG_H -c ./Modules/symtablemodule.c -o Modules/symtablemodule.o gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DHAVE_CONFIG_H -c ./Modules/xxsubtype.c -o Modules/xxsubtype.o if test -f buildno; then \ expr `cat buildno` + 1 >buildno1; \ mv -f buildno1 buildno; \ else echo 1 >buildno; fi gcc -c -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DHAVE_CONFIG_H -DBUILD=`cat buildno` -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c rm -f libpython2.2.a ar cr libpython2.2.a Modules/getbuildinfo.o sh: ar: not found *** Error code 1 make: Fatal error: Command failed for target `libpython2.2.a' /*********************************************************************************************************************************** Any one can help me in that ?? Please look at the attached file for details. Thanks in advance Regards V.Selvam

5 4

Boundary checks on arguments to time.strftime()
by Brett C. 02 Mar '04

02 Mar '04

Bug #897625 (http://python.org/sf/897625) deals with giving time.strftime() date information that is outside of proper bounds (e.g., using a negative number for the day of the week). This can lead to a crash since it seems some strftime implementations just index into an array for values for text and thus a negative value can lead to bad things happening. I would like to raise a ValueError if the argument is out of range. Problem is that this will break code. I could just force all negative values to all values outside the proper bounds to a reasonable value, but that seems to go against the path of least surprise. That is question 1. Question 2 is what to really check. This really is only a concern for month and day of the week since everything else is just a number and doesn't have some name representation. I could check all 9 values, though, or just these two. Question 3 is whether to extend this to time.asctime() . I have talked to Tim about this and his thoughts are to just deal with time.strftime() and leave everything else alone. That's fine with me, but there is the same possibility of having problems with asctime(). But then again, checking value for asctime() would potentially break even more code. So, thoughts? -Brett

5 6

Idea for a fast calling convention
by Raymond Hettinger 02 Mar '04

02 Mar '04

I've been working on ways to speedup parameter passing for function calls. It's somewhat expensive to pull parameters off of the stack, assemble them into a new tuple, have the function disassemble the tuple, and ultimately free the tuple. METH_NOARGS and METH_O show nice speed-ups by skipping the tuple packing and unpacking. Since they are handled as special cases, that approach doesn't handle the general case with multiple or optional arguments. My idea is for a new flag, METH_STACK, that generalizes (and potentially replaces) METH_NOARGS AND METH_O. When CALL_FUNCTION sees the flag, it dispatches (*meth)(self, &stack, numargs). On the receiving end, the arguments get retrieved with analogues to PyArg_ParseTuple() and PyArg_UnpackTuple() which access the parameters directly and do not need a tuple as a carrier. Raymond Hettinger

9 15

Re: [Python-Dev] Idea for a fast calling convention
by Christian Tismer 01 Mar '04

01 Mar '04

Armin Rigo wrote: > Hi Christian! > > On Sat, Feb 28, 2004 at 07:12:54PM +0100, Christian Tismer wrote: > >>p.s.: I believe some automatic source analysis and rewrite might >>pay off in other areas as well. Grepping through the sources, >>there are still very many similar patterns of PyArg_ParseTupleXXX >>calls, which could be replaced by less general, optimized versions. > > > I tried that some time ago, when python-dev first talked about METH_O. I did > a script that looks for these PyArg_ParseTuple(args, "xxx", ...) calls and > replace them with some in-line equivalent, e.g. "PyTuple_GET_SIZE(args) == 2 > && PyTuple_GET_ITEM(args, 0)->ob_type == etc. && etc.". > > It didn't pay out. The METH_O optimization was a much better one. It seems > that PyArg_ParseTuple() isn't just called often enough in practice now. > > BTW this remark also applies to the proposals for METH_STACK to avoid the > tuple creation altogether. What amount of time is really spent there? Maybe a big profiling run could clarify if and where something can be saved. It is also a relative measure: If the function that I call takes up lots of computation time, I won't care too much about parameter processing. On the other hand, a candidate like len() is so cheap to compute that it really would pay off (but that one is already taken, sure). For METH_STACK, I'm not sure. Does it raise complication, or does it make things even clearer? From my Stackless POV, throwing parameters into the callee and not having to care about the arg tuple is much more "continuation-like". I *would* have a problem with the current style if the callee wouldn't increfs its parameters, anyway, so I'm allowed to drop the arg tuple early. Fortunately here is some redundance of refcounts. I guess it has a little improvements not only by saving tuple creation but also that many decrefs, probably not soo much. A consequent implementation through the whole source seems to be quite a lot of effort. Maybe a relative measure for some common cases can give some figures? ciao - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

1 0

How to debug pyexpat SIGSEGV with GDB?
by Brad Clements 01 Mar '04

01 Mar '04

Sorry if this is OT. I've hit a repeatable segfault in pyexpat on RH Linux 9 with Python 2.3.3 (I'm trying rss2email). It seems that XML_parse is returning an error, but when XML_GetCurrentLineNumber is called positionPtr is not valid. I catch this in GDB and have looked at the stack (see below), but when I get back up the stack into PyCFunction_Call I don't know what to do. Ideally, I want to find out the Python source file and line number that is currently being executed, then look at the Python source to figure out exactly which pyexpat call is being made just before the call to get_parse_result. It seems strange that (it appears) that Python code is calling get_parse_result directly. So, how can I figure out where in the Python source the function call is coming from using gdb? I'm sure it involves "print" and some casts.. I couldn't find a howto on python.org -- Python 2.3.3 (#1, Dec 22 2003, 14:01:09) [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 Starting program: /usr/local/bin/python2.3 rss2email.py feeds.dat run --no-send [New Thread 1074948352 (LWP 2379)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1074948352 (LWP 2379)] normal_updatePosition (enc=0x407211c0, ptr=0x40785000 <Address 0x40785000 out of bounds>, end=0x823ac09 "öteborg, Sweden, 7-9 June 2004. To help us serve the communit y in the best way possible, we need your input on what you think we should do in the way of tutorials.\"]\n\n\n\n\n\n, end=0x823ac09 "öteborg, Sweden, 7-9 June 2004. To help us serve the communit y in the best way possible, we need your input on what you think we should do in the way of tutorials.\"]\n\n\n\n\n\n<p "..., pos=0x8238954) at xmltok_impl.c:1745 #1 0x40701804 in XML_GetCurrentLineNumber (parser=0x82387c0) at /usr/local/src/Python-2.3.3/Modules/expat/xmlparse.c:1605 #2 0x406ff800 in set_error (self=0x4067e8ec, code=XML_ERROR_INVALID_TOKEN) at /usr/local/src/Python-2.3.3/Modules/pyexpat.c:124 #3 0x406fe324 in xmlparse_Parse (self=0x4067e8ec, args=0x4064456c) at /usr/local/src/Python-2.3.3/Modules/pyexpat.c:888 #4 0x080ed614 in PyCFunction_Call (func=0x4067f38c, arg=0x4064456c, kw=0x40785000) at ../Python-2.3.3/Objects/methodobject.c:108 #5 0x080a4de7 in call_function (pp_stack=0xbfffe91c, oparg=1080551660) at ../Python-2.3.3/Python/ceval.c:3439 (gdb) frame 0 #0 normal_updatePosition (enc=0x407211c0, ptr=0x40785000 <Address 0x40785000 out of bounds>, end=0x823ac09 "öteborg, Sweden, 7-9 June 2004." pos=0x8238954) at xmltok_impl.c:1745 1745 switch (BYTE_TYPE(enc, ptr)) { (gdb) list 1740 const char *ptr, 1741 const char *end, 1742 POSITION *pos) 1743 { 1744 while (ptr != end) { 1745 switch (BYTE_TYPE(enc, ptr)) { 1746 #define LEAD_CASE(n) \ 1747 case BT_LEAD ## n: \ 1748 ptr += n; \ 1749 break; (gdb) (gdb) frame 1 #1 0x40701804 in XML_GetCurrentLineNumber (parser=0x82387c0) at /usr/local/src/Python-2.3.3/Modules/expat/xmlparse.c:1605 1605 XmlUpdatePosition(encoding, positionPtr, eventPtr, &position); (gdb) list 1600 1601 int XMLCALL 1602 XML_GetCurrentLineNumber(XML_Parser parser) 1603 { 1604 if (eventPtr) { 1605 XmlUpdatePosition(encoding, positionPtr, eventPtr, &position); 1606 positionPtr = eventPtr; 1607 } 1608 return position.lineNumber + 1; 1609 } (gdb) frame 2 #2 0x406ff800 in set_error (self=0x4067e8ec, code=XML_ERROR_INVALID_TOKEN) at /usr/local/src/Python-2.3.3/Modules/pyexpat.c:124 124 int lineno = XML_GetErrorLineNumber(parser); (gdb) list 119 set_error(xmlparseobject *self, enum XML_Error code) 120 { 121 PyObject *err; 122 char buffer[256]; 123 XML_Parser parser = self->itself; 124 int lineno = XML_GetErrorLineNumber(parser); 125 int column = XML_GetErrorColumnNumber(parser); 126 127 /* There is no risk of overflowing this buffer, since 128 even for 64-bit integers, there is sufficient space. */ (gdb) print *parser $2 = {m_userData = 0x2, m_handlerArg = 0x40720ac0, m_buffer = 0x82387c0 "ìèg@ìèg@\b@v@TA\005\b$D\005\b´E\005\b\b@v@=Dw@\b@x@5\004\001", m_mem = {malloc_fcn = 0x1, realloc_fcn = 0, free_fcn = 0}, m_bufferPtr = 0x0, m_bufferEnd = 0x1 <Address 0x1 out of bounds>, m_bufferLim = 0x0, m_parseEndByteIndex = 8192, m_parseEndPtr = 0x0, m_dataBuf = 0x0, m_dataBufEnd = 0x81e66b0 "<\224f@¬\222f@\204\222f@ì\223f@ü\222f@\024\224f@<TA@,ÿe@", m_startElementHandler = 0x1, m_endElementHandler = 0x810a7a0 <PyString_Type>, m_characterDataHandler = 0x23, m_processingInstructionHandler = 0xffffffff, m_commentHandler = 0, m_startCdataSectionHandler = 0x7273752f, m_endCdataSectionHandler = 0x636f6c2f, m_defaultHandler = 0x6c2f6c61, m_startDoctypeDeclHandler = 0x702f6269, m_endDoctypeDeclHandler = 0x6f687479, m_unparsedEntityDeclHandler = 0x332e326e, m_notationDeclHandler = 0x6165772f, m_startNamespaceDeclHandler = 0x6665726b, m_endNamespaceDeclHandler = 0x79702e, m_notStandaloneHandler = 0, m_externalEntityRefHandler = 0, m_externalEntityRefHandlerArg = 0x1, m_skippedEntityHandler = 0x8117660 <PyCode_Type>, m_unknownEncodingHandler = 0x2, m_elementDeclHandler = 0x2, m_attlistDeclHandler = 0x2, m_entityDeclHandler = 0x43, m_xmlDeclHandler = 0x4067c3a0, m_encoding = 0x40677fac, m_initEncoding = { initEnc = {scanners = {0x4067d3bc, 0x40643e0c, 0x4012802c, 0x4012802c}, literalScanners = {0x4067e920, 0x4067f180}, sameName = 0x122, nameMatchesAscii = 0x4067f1a0, nameLength = 0x1, skipS = 0x810a7a0 <PyString_Type>, getAtts = 0x23, charRefNumber = 0xffffffff, predefinedEntityName = 0, updatePosition = 0x7273752f, isPublicId = 0x636f6c2f, utf8Convert = 0x6c2f6c61, utf16Convert = 0x702f6269, minBytesPerChar = 1869116537, isUtf8 = 110 'n', isUtf16 = 50 '2'}, encPtr = 0x6165772f}, m_internalEncoding = 0x6665726b, m_protocolEncodingName = 0x79702e <Address 0x79702e out of bounds>, m_ns = 0 '\0', m_ns_triplets = 0 '\0', m_unknownEncodingMem = 0x0, m_unknownEncodingData = 0x1, m_unknownEncodingHandlerData = 0x8117660, m_unknownEncodingRelease = 0x1, m_prologState = {handler = 0x4, level = 2, role_none = 67, includeLevel = 1080537744, documentEntity = 1080310604, inEntityValue = 1080546284}, m_processor = 0x40678e14, m_errorCode = 1074954284, m_eventPtr = 0x4012802c "}\v", m_eventEndPtr = 0x4067e9a0 "\001", m_positionPtr = 0x4067f220 "\002", m_openInternalEntities = 0x125, m_defaultExpandInternalEntities = 48 '0', m_tagLevel = 1080552288, m_declEntity = 0x810a7a0, m_doctypeName = 0x23 <Address 0x23 out of bounds>, m_doctypeSysid = 0xffffffff <Address 0xffffffff out of bounds>, m_doctypePubid = 0x0, m_declAttributeType = 0x7273752f <Address 0x7273752f out of bounds>, m_declNotationName = 0x636f6c2f <Address 0x636f6c2f out of bounds>, m_declNotationPublicId = 0x6c2f6c61 <Address 0x6c2f6c61 out of bounds>, m_declElementType = 0x702f6269, m_declAttributeId = 0x6f687479, m_declAttributeIsCdata = 110 'n', m_declAttributeIsId = 50 '2', m_dtd = 0x6165772f, m_curBase = 0x6665726b <Address 0x6665726b out of bounds>, m_tagStack = 0x79702e, m_freeTagList = 0x0, m_inheritedBindings = 0x0, m_freeBindingList = 0x4067ea20, m_attsSize = 135362144, m_nSpecifiedAtts = 0, m_idAttIndex = 0, m_atts = 0x1, m_nsAtts = 0x42, m_nsAttsVersion = 1080542056, m_nsAttsPower = 236 'ì', m_position = { lineNumber = 1080528484, columnNumber = 1074954284}, m_tempPool = { blocks = 0x4012802c, freeBlocks = 0x4012802c, end = 0x4067ea20 "`ëg@ §\020\b#", ptr = 0x4067d410 " \020g@@\215g@ýÿÿÿ\001", start = 0x121 <Address 0x121 out of bounds>, mem = 0x4067f1c0}, m_temp2Pool = {blocks = 0x1, freeBlocks = 0x810a7a0, end = 0x22 <Address 0x22 out of bounds>, ptr = 0x5e84331 <Address 0x5e84331 out of bounds>, start = 0x0, mem = 0x70747468}, m_groupConnector = 0x6f2f2f3a <Address 0x6f2f2f3a out of bounds>, m_groupSize = 1852402798, m_namespaceSeparator = 101 'e', m_parentParser = 0x746f6266, m_isParamEntity = 46 '.', m_useForeignDTD = 111 'o', m_paramEntityParsing = 1852795427} (gdb) (gdb) frame 3 #3 0x406fe324 in xmlparse_Parse (self=0x4067e8ec, args=0x4064456c) at /usr/local/src/Python-2.3.3/Modules/pyexpat.c:888 888 return set_error(self, XML_GetErrorCode(self->itself)); (gdb) list 883 { 884 if (PyErr_Occurred()) { 885 return NULL; 886 } 887 if (rv == 0) { 888 return set_error(self, XML_GetErrorCode(self->itself)); 889 } 890 if (flush_character_buffer(self) < 0) { 891 return NULL; 892 } (gdb) frame 4 #4 0x080ed614 in PyCFunction_Call (func=0x4067f38c, arg=0x4064456c, kw=0x40785000) at ../Python-2.3.3/Objects/methodobject.c:108 108 return (*meth)(self, arg); (gdb) list 103 size = PyTuple_GET_SIZE(arg); 104 if (size == 1) 105 arg = PyTuple_GET_ITEM(arg, 0); 106 else if (size == 0) 107 arg = NULL; 108 return (*meth)(self, arg); 109 } 110 break; 111 default: 112 PyErr_BadInternalCall(); -- Brad Clements, bkc(a)murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax http://www.wecanstopspam.org/ AOL-IM: BKClements

6 7

Optional separatorargument for file.writelines() and StringIO.writelines()
by Raymond Hettinger 01 Mar '04

01 Mar '04

Currently, writelines() does not add trailing line separators.. This is fine when working with readlines() but a PITA in other situations. If we added an optional separator argument, it would be easier to add newlines and we would gain some of the flexibility of str.join() at full C speed. Raymond Hettinger

8 10

other uses for "as" [was Re: new syntax for wrapping (PEP 318)]
by Doug Holton 01 Mar '04

01 Mar '04

> Skip> If you are going to add [...] as a function modifier syntax, > Skip> I would argue that it should allow the full power of list > Skip> (or generator) comprehensions. > > Agreed. And why limit it to lists - why not any expression that > evaluates to a list? Or maybe any sequence? Which is one reason > that a bare "[...]" doesn't seem sufficient. The "as" syntax is easier to understand and more readable, but you'd like to have the full power of lists as well (plus make the parsing easier). When there is just a single item though, the list syntax seems overkill. I can already tell if this feature is added it will likely stick with the list syntax instead of the "as" keyword because that is the easier solution to code. But here is an "as" solution that doesn't require extra coding for now. Have the parser check for a single item OR a list of items. You could add an "as" keyword but essentially ignore it in this case. This would mean though that this would not work: def myfunc(x, y) as synchronized, classmethod: ... You'd have to use the list syntax for multiple modifiers. If there is a decision to add the "as" keyword, here are some possible future expansions of its use. What it does is make the use of modifiers and type checking more readable. In general "X as Y" would mean --> Y(X) and "X as [Y, Z]" would mean --> Z(Y(X)) for example: X = temp_val as int means --> x = int(temp_val) thisobject = obj as MyType means --> thisobject = MyType(obj) Then there are Visual Basic -like params (let's not start a war about it): def func1(x as int, y as float): ... instead of: def func1(x, y): try: x2 = int(x) y2 = float(y) ...

5 6

RE: [Python-Dev] Re: new syntax for wrapping (PEP 318)
by Michael Chermside 01 Mar '04

01 Mar '04

Jim Jewett writes: [I'm not going to quote him... details at http://mail.python.org/pipermail/python-dev/2004-February/042685.html ] Jim, thank you for putting this so clearly. I want to second just about everything that you said. Since you've gone into it in great detail, I'll try the brief, bullet-points version: The "def name [modifiers]:" is unpythonic because: * It re-uses the "[]" for a completely new purpose, unrelated to lists or indexing. * It relies heavily on punctuation (the placement of the ":"). * It doesn't "read like english". * There's no good way for a newbie to look it up. All of these would be addressed by using a keyword. The only objection seems to be "we couldn't find a word we all liked". But that's hardly a good reason to go with pure punctuation. Me, personally, I like "as". It works great for wrappers like classmethod, and is a sufficiently generic word that it wouldn't offend my sense of grammer when used with other kinds of wrappers. But like Jim I'm not strongly in favor of any particular keyword, just against the use of pure punctuation. -- Michael Chermside

15 28