Mailman 3 May 2004 - Python-Dev

RE: [Python-Dev] Is core dump always a bug? Advice requested
by Michel Pelletier 14 May '04

14 May '04

> > If arbitrary branching patterns are allowed, then in order to ensure > > absence of stack overflow/underflow, etc., it will be necessary to do > > things like proving that certain loops can't execute more than a > > certain number of times. This smells halting-problem-complete to me. > > I think it would only need to check that any path through the loop has a no > net effect on stack depth -- before branching back to the beginning of the > loop, it must pop everything it pushed. The ASM Java bytecode generation API has a similar flow control algorithm to determine the maximum stack size for java bytecode. http://asm.objectweb.org/ It uses the visitor pattern to visit classes, methods and instructions during bytecode generation that construct a control flow graph that is used at class generation time to calculate the maximum local variables and stack depth for methods (see Label.java and CodeWriter.java in their WebCVS). his information is embedded into the class, and is used at JVM verification time when the bytecode is reanalyzed to ensure that the numbers match the actual behavior of the bytecode. I've seen a lot of concern about bytecode hacking crashing the interpreter, but the Java VM Spec goes further to state that there is a risk of exploitation. I stared at ceval for a while to see if I could pick up such a hole. It hurt. I hear Phillip Eby when he says that anyone sending bytecode over the wire agent-fasion "darn well better" write their own bytecode verifier. But in a world of massive paralleism and dynamic computing, that might not seem like such a bad feature to offer stock. Would there be an interest in at least a PEP to consider a bytecode verifier? I'd be willing to take it on, but my experience is limited to JVM bytecode and verification. -Michel

7 10

.pth files in current or script directory
by Delaney, Timothy C (Timothy) 14 May '04

14 May '04

A discussion (last week?) on c.l.py prompted a thought about .pth files. Feel free to shoot it down ;) The discussion in question concerned someone using a version control system, and needing to change PYTHONPATH depending on which branch they were in. My immediately two thoughts were: 1. Use a build system (e.g. make) to set PYTHONPATH appropriately. This is they type of thing we do. 2. Use a .pth file in the directory the code is run from (or possibly, the same directory as the script). Before posting the second, I thought I'd better try it out - fortunately for me :) Putting a .pth in the current or script directory did nothing. I'm wondering if anyone sees utility in being able to put a .pth file in those locations and have them used in the same way as if they were in the python executable directory. For example, if you have a package 'p', with a 'test' subpackage (containing scripts), it would be very simple to stick a 'test.pth' file into the 'p.test' package, with the sole contents of 'test.pth' being '..'. This would add the (absolute path of the) parent directory to the root paths for module import. I feel there's some kind of link here with PEP 328, but I'm not quite sure where it fits. Tim Delaney

3 2

RE: [Python-Dev] Is core dump always a bug? Advice requested
by Robert Brewer 13 May '04

13 May '04

Michel Pelletier wrote: > I am also working on a short list of static and structural > constraints for > bytecode, this is what I have so far (the ones i'm unsure about are > questions): A couple gotchas when useing new.code(): 1. co_names, co_varnames, co_freevars, and co_cellvars take tuples of strings; passing unicode strings crashes the interpreter. co_consts, however, is unicode-safe. 2. Some co_flags should be checked to be consistent with other args; for example, shouldn't CO_NOFREE be set if co_freevars is empty? Robert Brewer MIS Amor Ministries fumanchu(a)amor.org

1 0

RE: [Python-Dev] Is core dump always a bug? Advice requested
by Michel Pelletier 13 May '04

13 May '04

On Thursday 13 May 2004 04:43, python-dev-request(a)python.org wrote: > From: "Phillip J. Eby" <pje(a)telecommunity.com> > As I think Tim has already mentioned, all you do is track the stack depth > for each instruction, and ensure it's consistent. If there is an > inconsistency in the predicted stack level for an instruction, as compared > with any of the instructions that jump to it (no matter where they jump > from), you have a problem. > > Or, to look at it another way: <snip code example> Thanks Phillip, this code is a great starting point. I was trying to reproduce the algorithm given in the JVM Spec section 4.9, but it is seriously complicated by strict type issue which have no meaning here. Also, as Armin pointed out, there are a couple instances of instructions that would be difficult to analize without some changes to their design (END_FINALLY and MAKE_CLOSURE). But he suggests relatively simple solutions to both. I am also working on a short list of static and structural constraints for bytecode, this is what I have so far (the ones i'm unsure about are questions): Static Constraints on Bytecode Instructions 1. The bytecode string must not be empty (len(co_code) > 0). 2. Is there maximum co_code length? 3. The first instruction in the bytecode string begins at index 0. 4. Only valid bytecodes with correct number of operands can be in the bytecode string. 5. The index of the next instruction equals the index of the current instruction plus the length of the current instruction and all its operands. 6. The last byte of the last instruction must be len(co_code) - 1. 7. The last instruction must be RETURN_VALUE? 8. Exception handlers verification? Maybe the compiler could generate a table of handlers, their handled types, and their instruction ranges for verification. The run-time benefit would be that an exception handler could be quickly dispatched instead of falling through each except block in sequence looking for the matching handler as it looks to work now (not that it's a big performance bottleneck...) Static Constraints on Bytecode Instruction Operands 1. The target of a jump instruction must be within the code boundaries and must not fall between an instruction and its operands. 2. The operand of a LOAD_CONST instruction must be an valid index into the co_consts. 3. Can a similar bounds check be made at verfication time for LOAD_FAST? 4. Can index into co_names, co_varnames, co_freevars, co_cellvars be verfied? Structural Constraints between Bytecode Instructions 1. Each instruction must only be executed with the appropriate number of arguments in the operand stack, regardless of the execution path that leads to its invocation. 2. If an instruction can be executed along several different execution paths, the value stack must have the same depth prior to the execution of the instruction, regardless of the path taken. 3. At no point during execution can the value stack grow to a depth greater than that implied by co_stacksize. > This algorithm should readily handle all forward and backward jumps, while > ensuring that no stack growth or shrinkage can take place, because every > instruction must have a single known stack depth, no matter how many > forward or backward jumps it takes to get there. > > Notice that the algorithm will "follow" every branch, but will not retrace > the code at the target location once the stack depth is seen to be > consistent. It also tackles underflow and "falling off the end" of the > bytecode, while simultaneously computing the maximum stack depth > used. Each instruction is fetched and processed for its depth effects and > other characteristics exactly once, so the timing is O(n). And, it has a > nice place to put opcode-specific operand validation for opcodes that use > locals, freevars, names, or constants. Yea! As Tim points out below there are some subtleties, but this is great. > Anyway, I think we can safely halt discussion of the halting problem > now. :) And, since I've now reduced this from an interesting theoretical > idea to talk about, into something resembling work (God forbid), anybody > who wants to actually *implement* this crazy idea is now free to do so, > without even having to read the paper Tim linked to. :) Well I did read it, and I suggest anyone else also read chapter 4 of the Java VM Spec. -Michel

2 1

cookielib module in 2.4?
by John J Lee 13 May '04

13 May '04

I have a module that I'd like to get included in Python 2.4. It adds transparent HTTP cookie handling to urllib2 (and urllib too, I guess, if somebody wants to write the little bit of code I imagine is required). I'm prepared to maintain the thing, of course. It has been distributed for 2 years in a 1.5.2-compatible form, as the ClientCookie package, at wwwsearch.sf.net. I've just made a branch in which I have removed 1.5.2 (and various other) cruft, along with some superfluous features, and turned it into a single-file module (plus a small patch to urllib2). ATM it's in a subversion branch here: http://codespeak.net/svn/user/jjlee/wwwsearch/ClientCookie/branch/python-2.… I've renamed it 'cookielib'. That's partly to fit in with the Python standard library's naming convention for network client modules, partly so that it's clear to people whether they're importing the old 1.5.2-compatible version or the 2.4 standard library version. Actually, the most frequent use case does not involve importing the module: import urllib2 opener = urllib2.build_opener(urllib2.HTTPCookieProcessor()) response = opener.open("http://example.com/") So, assuming the module is likely to be accepted (given the answers to my post here last year), I'd be happy if anybody here could 1. indicate if I'm roughly on the right track (especially welcome if you've had a look at the subversion branch above), 2. give me a rough idea how long I have to sort out the remaining issues and submit a patch The major remaining issue is documentation, which is not yet in LaTeX format (nor in Python std. lib. style) -- see my next post. There are also two or three niggles which I'd like to discuss with Jeremy Hylton, Moshe Zadka, or anybody else who knows urllib2. Finally, there's the fact that the module contains untested thread-synchronisation code (I'm pretty thread-ignorant). Perhaps that's an academic issue ATM, though: I failed recently to eliminate deadlocks in a fairly simple bit of code using urllib2 + threading, even when I verified there were no locks allocated by any module other than threading itself... TIA John

4 6

IPv6 support in Win binary again
by Mat Ford 13 May '04

13 May '04

There was a thread back in October 2003 on the subject of including IPv6 support in the Windows binary: http://mail.python.org/pipermail/python-dev/2003-October/038623.html Can anyone give me a status update on this? Is support there now? If not, is there a predictable timeframe in which it can be expected to appear? Thanks in advance, Mat --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.678 / Virus Database: 440 - Release Date: 06/05/2004

4 3

Re: [Python-Dev] Is core dump always a bug? Advice requested
by Jewett, Jim J 13 May '04

13 May '04

Martin v. L?wis: > Therefore, byte code verification usually puts additional > constraints on byte code, rejecting some correct code as > unverifiable. An alternative would be to accept unverifiable code. You won't stop all problems, but you can still catch the most common mistakes. For instance, the example that started this thread was apparently an ASCII string of uncomiled source code. Just checking that all instructions are valid bytecodes might have caught this. Also checking that arguments are possible (no jumping beyond the code) would probably have caught it. Verifying stack depth and such might be overkill for a first pass sanity check. -jJ

1 0

RELEASED Python 2.3.4, release candidate 1
by Anthony Baxter 13 May '04

13 May '04

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team and the Python community, I'm happy to announce the release of Python 2.3.4 (release candidate 1). Python 2.3.4 is a bug-fix release. See the release notes at the website (also available as Misc/NEWS in the source distribution) for details of the bugs squished in this release. Assuming no major problems crop up, a final release of Python 2.3.4 will follow late next week. For more information on Python 2.3.4, including download links for various platforms, release notes, and known issues, please see: ~ http://www.python.org/2.3.4 Highlights of this new release include: ~ - Bug fixes. According to the release notes, more than 20 bugs ~ have been fixed, including a couple of bugs that could cause ~ Python to crash. These were discovered by Zope3. Highlights of the previous major Python release (2.3) are available from the Python 2.3 page, at ~ http://www.python.org/2.3/highlights.html Enjoy the new release, Anthony Anthony Baxter anthony(a)python.org Python Release Manager (on behalf of the entire python-dev team) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFAo3yTDt3F8mpFyBYRAicdAJwOngKitELJ25u3oCW+iQcc581wbQCgh1Ah f/Ci2omZVG7p63xY9cfZiSw= =e3Rv -----END PGP SIGNATURE-----

1 0

Is core dump always a bug? Advice requested
by Terry Reedy 13 May '04

13 May '04

The following was posted in the last 25 hours on c.l.p: "Xaver Hinterhuber" <xaver_hinterhuber(a)web.de> wrote in message news:c7o6al$i1j$1@online.de... > codeObject = new.code( > 0, # argcount > 0, # nlocals > 0, # stacksize > 0, # flags > codeString, # code > (), # consts > (), # names > (), # varnames > 'content', # filename > 'content', # name > 3, # first line number > codeString # lnotab > ) > f = new.function(codeObject, dict, 'f') > f() > > Everything runs fine, until the function is called with f(). > When python tries to execute f(), the core dump happens. > I don't have any clue why python core dumps. > The codeString is nothing complex, its a one-liner. ... but, as too typical of new posters, not posted ;-) My question: I understand that core dumping by the Python interpreter is normally considered a bug -- one that should be reported on SF -- and fixed. But does that principle apply to this sort of use (or probably abuse) of the new module. In other words, does the 'no core dump' contract include a 'byte code compiled by the interpreter' proviso. Or is ceval intended to be robust against random strings masquerading as valid byte code (which would seem require an InvalidByteCode exception). Or in still other words, is there a bug that I should encourage reporting of, or is 'don't do that' the extent of a proper response? Terry J. Reedy

15 30

Build a single file module-foo.html from Python LaTeX source?
by John J Lee 12 May '04

12 May '04

Is there some magic make command that generates the module-foo.html for just a single libfoo.tex file? Doesn't necessarily have to generate 100%-identical output to the full run. I want to write Python-LaTeX-format docs for a module module I hope to get into Python 2.4, and document urllib2's handlers better. It's nice to be able to see the HTML output as you edit, without having to process the whole std. lib. manual every time. John

1 0