Mailman 3 July 2008 - Python-Dev

No beta2 tonight
by Barry Warsaw July 25, 2008

July 25, 2008

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 We have green buildbots, yay! Thanks everyone for that. However, we still have three release blocker issues that I am not comfortable deferring. 3088 test_multiprocessing hangs intermittently on POSIX platforms 3375 _multiprocessing.so build problems 874900 threading module can deadlock after fork My biggest concern is 3375 and 3088, the latter has a dependency on 874900. 3375 is very strange since a subsequent 'make' completes just fine. … [View More]

19 43

crash in 3.0b2 exception code
by Stefan Behnel July 24, 2008

July 24, 2008

Hi, I get a crash in one of lxml's doctests in 3.0b2, and it looks like it's coming from plain Python code (as opposed to Cython code). Just a quick check before I start digging into this, has anyone seen this before? Stefan [...] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1209997120 (LWP 15431)] 0x080f3b89 in BaseException_str (self=0x8d747d4) at Objects/exceptions.c:88 88 switch (PyTuple_GET_SIZE(self->args)) { (gdb) bt #0 0x080f3b89 in … [View More]

2 1

Re: [Python-Dev] Python VM
by Jakob Sievers July 22, 2008

July 22, 2008

I added a page to wiki.python.org: http://wiki.python.org/moin/CPythonVmInternals This incorporates most of Martin v. Löwis's additions (except for a few bits which I need to look into more). In any case, thanks for the feedback! Cheers, -jakob

1 0

Python VM
by Jakob Sievers July 22, 2008

July 22, 2008

Hi, I've been reading the Python VM sources over the last few afternoons and I took some notes, which I thought I'd share (and if anyone familiar with the VM internals could have a quick look at them, I'd really appreciate it). Cheers, -jakob Unless otherwise noted, the source file in question is Python/ceval.c. Control Flow ============ The calling sequence is: main() (in python.c) -> Py_Main() (main.c) -> PyRun_FooFlags() (pythonrun.c) -> run_bar() (pythonrun.c) -> … [View More]PyEval_EvalCode() (ceval.c) -> PyEval_EvalCodeEx() (ceval.c) -> PyEval_EvalFrameEx() (ceval.c). EvalCodeEx() does some initialization (creating a new execution frame, argument processing, and some generator-specific stuff I haven't looked at yet) before calling EvalFrameEx() which contains the main interpreter loop. Threads ======= PyEval_InitThreads() initializes the GIL (interpreter_lock) and sets main_thread to the (threading package dependent) ID of the current thread. Thread switching is done using PyThreadState_Swap(), which sets _PyThreadState_Current (both defined in pystate.c) and PyThreadState_GET() (an alias for _PyThreadState_Current) (pystate.h). Async Callbacks =============== Asynchronous callbacks can be registered by adding the function to be called to pendingcalls[] (see Py_AddPendingCall()). The state of this queue is communicated to the main loop via things_to_do. State ===== The global state is recorded in a (per-process?) PyInterpreterState struct and a per-thread PyThreadState struct. Each execution frame's state is contained in that frame's PyFrameObject (which includes the instruction stream, the environment (globals, locals, builtins, etc.), the value stack and so forth). EvalFrameEx()'s local variables are initialized from this frame object. Instruction Stream ================== The instruction stream looks as follows (c.f. assemble_emit() in compile.c): A byte stream where each instruction consists of either 1) a single byte opcode: OP 2) a single byte opcode plus a two-byte immediate argument: OP LO HI 3) a special opcode followed by the real opcode followed by a four byte argument: EXTENDED_ARG OP BYTE2 BYTE1 BYTE4 BYTE3 Opcode Prediction ================= One nice trick used to speed up opcode dispatch is the following: Using the macros PREDICT() and PREDICTED() it is sometimes possible to jump directly to the code implementing the next instruction rather than having to go through the whole loop preamble, e.g. case FOO: // ... PREDICT(BAR); continue; PREDICTED(BAR); case BAR: // ... expands to case FOO: // ... if (*next_instr == BAR) goto PRED_BAR; continue; PRED_BAR: next_instr++; case BAR: // ... Main Loop ========= Variables and macros used in EvalFrameEx() ------------------------------------------ The value stack: PyObject **stack_pointer; The instruction stream: unsigned char *next_instr; NEXTOP(), NEXTARG(), PEEKARG(), JUMPTO(), and JUMPBY() simply fiddle with next_instr. Likewise for TOP(), SET_SECOND(), PUSH(), POP(), etc. and stack_pointer. Current opcode plus argument: int opcode; int oparg; Error status: enum why_code why; // no, exn, exn re-raised, return, break, continue, yield int err; // non-zero is error The environment: PyObject *names; PyObject *consts; and PyObject **fastlocals; which is accessed via GETLOCAL() and SETLOCAL(). Finally, there are some more PyObject *'s (v, w, u, and so forth, used as temporary variables) as well as PyObject *retval; Basic structure --------------- EvalFrameEx() { why = WHY_NOT; err = 0; for (;;) { <------------------+---+ // do periodic tasks | | | | fast_next_opcode: | | opcode = NEXTOP(); | | if (HAS_ARG(opcode)) | | oparg = NEXTARG(); | | | | dispatch_opcode: | | switch(opcode) { | | | | continue; -------------------+ | | break; ----------------------+ | | | // Also, opcode prediction | | // jumps around inside the | | // switch statement | | | | } <-----------------------+ | | on_error: | // no error: continue -----------+ // otherwise why == WHY_EXCEPTION after this fast_block_end: // unwind stacks if there was an error } // more unwinding fast_yield: // reset current thread's exception info exit_eval_frame: // set thread's execution frame to previous execution frame return retval; } Periodic Tasks -------------- By checking and decrementing _Py_Ticker, the main loop executes certain tasks once every _Py_CheckInterval iterations (in fact Py_AddPendingCall() sets _Py_Ticker to zero, ensuring that pending calls are executed right after the next instruction which doesn't jump to fast_next_opcode): - If there are things_to_do, Py_MakePendingCalls() is called. - The GIL is releases and re-acquired, giving other threads a chance to run. Instruction implementation -------------------------- Some general notes: - Successful instructions either goto fast_next_opcode or continue. - Unsuccessful instructions break out of the switch. - The value stack holds only PyObject *'s. - Instructions must take care to Py_INCREF() and Py_DECREF() the reference counts of PyObject's whose addresses have been pushed onto/popped off the value stack. - Objects are transferred onto the value stack by GETITEM()'ing them from consts or names, or by GETLOCAL()'ing them using oparg as an offset into fastlocals (c.f. LOAD_* instructions). - err is used as a general `error occurred' flag, both inside the code implementing an opcode and `globally' for the entire loop. Nested blocks ------------- Nested loop and try blocks are handled as follows: Each frame maintains a block stack; when entering a nested block, a SETUP_* instruction adds a PyTryBlock to the PyFrameObject's f_blockstack and registers that block's type (instruction which created the block), handler (offset into the instruction stream) and level (value stack level before the nested block was entered). When such blocks are exited normally (e.g. last iteration of a loop), the final POP_BLOCK instruction restores the value stack to the state it was in before the block. If a block is exited abnormally (e.g. a break instruction), the code following fast_block_end unwinds the value stack and jumps to the block's handler. Certain instructions, e.g. RETURN_VALUE, cause the entire block stack to be unwound (leading to multiple unwinds of the value stack). See also the comments in compile.c (compiler_try_finally()) which include nice ASCII art diagrams (incidentally, there's a typo on line 2382: s/en/an/). Error handling -------------- Internal errors (bad oparg, say) generally result in why being set to WHY_EXCEPTION and breaking out of the switch (if the code implementing the instruction doesn't set why, the code after on_error will). The code following fast_block_end will jump to the correct exception handler and set the global exception related variables (exception information is stored in the current execution frame and the thread state. See set_exc_info() and reset_exc_info()). // eof [View Less]

4 3

import error in python2.6
by Bristow Thankachan July 22, 2008

July 22, 2008

Hi all, During the porting of Zope2, I am stuck with import errors in many modules. The same code works well in python2.5 and 2.4. Can anybody give the details of this import error in python2.6 and how to get the error corrected? with regards Bristow

5 4

Subversion 1.5 and better merge support
by Jesus Cea July 22, 2008

July 22, 2008

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Subversion 1.5 solves the "repeated merge" problem. At last!!. Somebody is considering upgrading python svn to 1.5, and allowing only commits coming from 1.5 clients?. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/… [View More]

3 2

Syntax error in python2.6
by Bristow Thankachan July 21, 2008

July 21, 2008

Hi everybody, During the porting of Zope2 to Python2.6, I am stuck with a syntax error in the module AccessControl, which is given below. def reorder(s, with=None, without=()): ^ SyntaxError: invalid syntax in line 56 of /home/zope/ztrunk26/lib/python/RestrictedPython/Utilities.py. The same code when run in python2.4 and python2.5 didn't give any syntax errors. Can anybody suggest the reason for this syntax error in python2.6.

7 8

import error in python2.6
by Bristow Thankachan July 21, 2008

July 21, 2008

Hi all, I asked about the import error in python2.6 while trying to port zope2. The error message is given below. File "/home/zope/ztrunk26/lib/python/AccessControl/ImplC.py", line 30, in <module> from ImplPython import RestrictedDTML, SecurityManager, ZopeSecurityPolicy ImportError: No module named ImplPython It works well in python2.4 and 2.5. Please give your suggestions. with regards Bristow

2 1

Search broken on the python dev docs?
by Gregory P. Smith July 21, 2008

July 21, 2008

http://docs.python.org/dev/ the search box worked for earlier releases but has been broken and returns nothing useful of late. If I enter simple terms like 'time' or 'os' or 'os.walk' what is returned is pathetic. how does this work? is an index corrupt or not being regenerated? -gps

2 1

Fuzzing bugs: most bugs are closed
by Victor Stinner July 21, 2008

July 21, 2008

Hi, I filled 14 issues about bugs found by fuzzing (see my other email "Play with fuzzing" for more informations). Most bugs are now closed, cool :-) Last bugs: == Trivial open bugs == segfault on locale.gettext(None) - http://bugs.python.org/issue3302 - attached patch is trivial: fix the PyArg_ParseTuple() to block None value, and reject empty domain string for bindtextdomain() (to avoid strange error "OSError(0): success") invalid ref count on locale.strcoll() error - http://bugs.… [View More]python.org/issue3303 - attached patch is trivial: add "if (rel1)" _multiprocessing.Connection() doesn't check handle - http://bugs.python.org/issue3321 - _multiprocessing.Connection(fd) doesn't check that fd is a valid file handle and so may crash on poll (the "evil" FD_SET() call) - my patch add "|| fstat(handle, &statbuf)" to make sure that the file descriptor is valid == Complex open bugs == block operation on closed socket/pipe for multiprocessing - http://bugs.python.org/issue3311 - close() method sets the file handle to -1 but most methods don't check the handle and so may fail or crash. Especially poll() calls FD_SET((SOCKET)conn->handle, &rfds); with handle=-1 => crash. - my patch creates a new MP error: "return MP_CLOSED_FILE;", used if handle is INVALID_HANDLE_VALUE to block operations (send, receive, poll) on closed files for socket and pipe. bugs in scanstring_str() and scanstring_unicode() of _json module - http://bugs.python.org/issue3322 - scanstring() function crashs if second argument is a big negative integer. There is no attached patch because I don't understand this function enough to fix it correctly, but I suggest to raise a ValueError if end is too small/big invalid object destruction in re.finditer() - or "PyObject_DEL inconsistency if pydebug option is used" - http://bugs.python.org/issue3299 - It's the most complex bug, I prefer to write a new email :-) == Need backport / port to python 3.0 == invalid call to PyMem_Free() in fileio_init() - http://bugs.python.org/issue3304 - patch applied in Python 2.6 (trunk) but not in Python 3000: "i'm assuming that'll be merged into py3k automagically." wrote Gregory P. Smith missing lock release in BZ2File_iternext() - http://bugs.python.org/issue3309 - patch applied in Python 2.6 but "Needs backporting to release25-maint." wrote Gregory P. Smith When all bugs will be closed, I will restart a fuzzing Python ;-) But I also tried with my patches and I was unable to find new bugs, great! Victor [View Less]

2 2