[Marek Baczek Baczyński]

Doesn't it leak memory when Ctrl+C'd (on Windows at least?)

Not really. "Leak" is reserved for cases where memory is unaccounted for. In this case, the memory is consumed by the ever-growing list:

...

...

x = [1] x.extend(-y for y in x) Traceback (most recent call last): File "<stdin>", line 1, in ? File "<stdin>", line 1, in <generator expression> KeyboardInterrupt len(x) 67090195 x[:10] [1, -1, 1, -1, 1, -1, 1, -1, 1, -1]

At that point, doing

...

...

del x[:]

reclaimed a few hundred megabytes.

[George Yoshida]

...

It does not always go into an infinite loop. I was bitten by this:

...

...

...

x = [] x.extend(-y for y in x) Segmentation fault

[Bob Ippolito]

No algorithm that requires infinite memory will run for an infinite amount of time on a finite computer. Of course it should raise an exception instead of segfaulting though.. could it be blowing the stack?

No, its stack use is bounded (and small) no matter how long it runs. On Windows it eventually raises MemoryError. My guess is that George is using Linux. "It's a feature" that the Linux malloc() can lie (== malloc(n) can return a non-NULL value p even if you're going to get a segfault if you try to write to p+i for some i in range(n)). Linus likens this to airlines over-selling seats, based on the likelihood that someone will miss their flight. Argue with him <wink>. When malloc() claims to return memory that can't actually be used, there's not much Python can do about that (other than blow up when trying to use it).

On 25-sep-04, at 6:41, Tim Peters wrote:

[George Yoshida]

...

...

It does not always go into an infinite loop. I was bitten by this:

...

...

...

x = [] x.extend(-y for y in x) Segmentation fault

[Bob Ippolito]

...

No algorithm that requires infinite memory will run for an infinite amount of time on a finite computer. Of course it should raise an exception instead of segfaulting though.. could it be blowing the stack?

No, its stack use is bounded (and small) no matter how long it runs.

I get a bus error on OSX (although with a slightly out of date python2.4 from CVS). Why should this loop at all? x is the empty list, and the generator comprehension should therefore end up with an empty sequence. It's not like your initial example where the list was non-empty to at the start. It crashes because of an Py_INCREF(item) at line 2727 in listobject.c where item is NULL: 2722 assert(PyList_Check(seq)); 2723 2724 if (it->it_index < PyList_GET_SIZE(seq)) { 2725 item = PyList_GET_ITEM(seq, it->it_index); 2726 ++it->it_index; 2727 Py_INCREF(item); 2728 return item; 2729 } 2730 2731 Py_DECREF(seq); BWT. seq is null as well.

[Armin Rigo, on

...

...

x = [] x.extend(-y for y in x) Segmentation fault ]

The segfault is immediate. And the example is different, as Ronald pointed out: the list 'x' is empty!

Good eye! I overlooked that too.

Uh oh. We have a real bug in listextend(): the list being extended is in a semi-invalid state when it's calling tp_iternext() on the 2nd iterable. This might call back Python code, which can inspect the list. The above example does just that. Crash.

"Semi-invalid" means that all invariants are respected but the final items in the list are NULL. Reading them crashes. And I'm not even talking about the nasty things you can do if you modify the list while it's being extended :-)

Yup. The code doesn't check for C int overflow of m+n either.

The safest solution would be to use a regular app1() to add each item as the iterable produce them instead of optimizing this case. I'm not sure we need the high-flying optimization of listextend() in this case (this is the case where the iterable we extend the list with is neither a list nor a tuple). I believe that the speed of app1() would be acceptable, given the fixed bug and the overall decrease of code complexity (though that should be measured).

I think it's easy to fix. "The usual rule" applies: you can't assume anything about a mutable object after potentially calling back into Python. So trying to save info in "i", "m", or "n" across loop iterations can't work, and the list can never be left in an insane state ("semi" or not) at any time user code may get invoked. But since we have both "num allocated" and "num used" members in the list struct now, it's easy to use those instead of trying to carry info in locals. Patch attached. Anyone object? Of course in the example at the start of this msg, it leaves x empty.

I think it's easy to fix. "The usual rule" applies: you can't assume anything about a mutable object after potentially calling back into Python. So trying to save info in "i", "m", or "n" across loop iterations can't work, and the list can never be left in an insane state ("semi" or not) at any time user code may get invoked. But since we have both "num allocated" and "num used" members in the list struct now, it's easy to use those instead of trying to carry info in locals.

FWIW, I've searched the codebase and found no other variants on this problem. None of the other update/extend methods try to remember self data between iterations. Other calls to list_resize immediately fill-in the NULLS before calling arbitrary Python code. And, other places that use the over-allocation trick, map() for example, are working with a brand new list or tuple that has not been exposed to the rest of the application. One situation did look suspect. _PySequence_IterSearch() remembers an index/count across calls to PyIter_Next() -- it looks like the worst that could happen is the index or count would be wrong, but no crashers.

Patch attached. Anyone object? Of course in the example at the start of this msg, it leaves x empty.

Looks good. Reads well. Solves the problem. The timings are still fast. The test suite runs w/o exception. Please apply. Raymond

[Bob Ippolito]

...

...

...

...

...

x = [] x.extend(-y for y in x) Segmentation fault

I get a MemoryError. To help with get a comprehensive view when I look at this more closely tomorrow, can you try out variations on the theme with other mutables: myset.update deque.extend dict.update dict.fromkeys array.extend Raymond

Quoting Raymond Hettinger <python@rcn.com>:

To help with get a comprehensive view when I look at this more closely tomorrow, can you try out variations on the theme with other mutables:

myset.update deque.extend dict.update dict.fromkeys array.extend

Returning to Tim's original infinite loop, the behaviour is interestingly variable. List and array go into the infinite loop. Deque and dictionary both detect that the loop variable has been mutated and throw a specific exception. Set throws the same exception as dictionary does (presumably, the main container inside 'set' is a dictionary) Details of behaviour: Python 2.4a3 (#16, Sep 21 2004, 17:33:57) [GCC 3.4.1 20040702 (Red Hat Linux 3.4.1-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. X>> x = [1] X>> x.extend(-y for y in x) Traceback (most recent call last): File "<stdin>", line 1, in ? File "<stdin>", line 1, in <generator expression> KeyboardInterrupt X>> len(x) 73727215 X>> x = set([1]) X>> x set([1]) X>> x.update(-y for y in x) Traceback (most recent call last): File "<stdin>", line 1, in ? File "<stdin>", line 1, in <generator expression> RuntimeError: dictionary changed size during iteration X>> x set([1, -1]) X>> from collections import deque X>> x = deque([1]) X>> x.extend(-y for y in x) Traceback (most recent call last): File "<stdin>", line 1, in ? File "<stdin>", line 1, in <generator expression> RuntimeError: deque changed size during iteration X>> x deque([1, -1]) X>> from array import array X>> x = array('b', '1') X>> x.extend(-y for y in x) Traceback (most recent call last): File "<stdin>", line 1, in ? File "<stdin>", line 1, in <generator expression> KeyboardInterrupt X>> len(x) 6327343 X>> x = dict.fromkeys([1]) X>> x {1: None} X>> x.update((-y, None) for y in x) Traceback (most recent call last): File "<stdin>", line 1, in ? File "<stdin>", line 1, in <generator expression> RuntimeError: dictionary changed size during iteration X>> x {1: None, -1: None} X>> x.fromkeys(-y for y in x) {-1: None}

Quoting Tim Peters <tim.peters@gmail.com>:

That's because they never suffered from list's ill-advised documentation effectively blessing mutation while iterating <0.5 wink>.

Ah. Interesting to know. So catching this is recommended when it's feasible?

...

Set throws the same exception as dictionary does (presumably, the main container inside 'set' is a dictionary)

Details of behaviour:

The last one is extremely surprising:

And it never actually happened, either. It's a transcription error on my part. I made a mistake when testing the dict.update version (I wrote "-y for y in x", instead of "(-y, None) for y in x"). When deleting that from the transcript, I also accidentally deleted the x.fromkeys() example. When I added that example back in, I put it in the wrong spot (after the x.update example, instead of before it). So, no, dict.update isn't randomly eating dictionary entries. Sorry 'bout the false alarm. . . Cheers, Nick.

[Tim]

...

That's because they never suffered from list's ill-advised documentation effectively blessing mutation while iterating <0.5 wink>.

[Nick]

Ah. Interesting to know. So catching this is recommended when it's feasible?

According to me, but perhaps not according to all. You can work very hard to provide predictable semantics for mutation while iterating, by defining cursor objects that somehow retain sensible guarantees even if the object they point into mutates. In effect, "the current index" is a cursor in this respect when iterating over a list, and the semantics are that "the current index", on each iteration, goes up by one, and is an offset from the start of whatever state the list happens to have at that time. So, e.g., this behavior is guaranteed:

...

...

x = range(10) for elt in x: ... x.remove(elt) x [1, 3, 5, 7, 9]

"Guaranteed" doesn't necessarily mean unsurprising, or even useful, though. I do have uses for this behavior, but I'd be happy to give them up. The "natural" behavior of dicts when mutating while iterating is effectively unexplainable -- it "does whatever it does", based on internal details of the hashed distribution of keys into buckets, and even on the history of insertions (which affects hash collision resolution). I'm glad Python gripes about that now (it didn't always). It would also be possible, but difficult, to implement "sane" iteration+mutation semantics for dicts. A dict cursor object would need to be aware of which objects had and hadn't already been passed out by the iteration, and would even need to be robust against the dict reorganizing itself completely when it changes size. It's a lot easier all around to say "if you have to, iterate over a snapshot of the keys". In some cases, we're reduced to saying that with no way to catch violations. ZODB's BTrees are a good example here. People routinely get in trouble by mutating them while iterating over them, but the implementation is such that it would be very difficult to detect such a thing.

Ilya> a number of standard python modules come with a command line Ilya> interfaces, e.g. pydoc.py, pdb.py , unittest.py, timeit.py, uu.py Ilya> But it appears that there is no convenient out-of-the-box way to Ilya> invoke these tools from command line... Ilya> Basically one either has to write wrappers or to Ilya> invoke them like this: python /usr/lib/python2.3/pdb.py Ilya> Neither approach is convenient... Ilya> Am I missing something obvious? Search for "Scripts to install" in the setup.py file that comes with the Python distribution. If there are other scripts you'd like to see installed, just submit a patch for setup.py. Skip

Ilya Sandler wrote:

A problem:

a number of standard python modules come with a command line interfaces, e.g. pydoc.py, pdb.py , unittest.py, timeit.py, uu.py But it appears that there is no convenient out-of-the-box way to invoke these tools from command line...

Basically one either has to write wrappers or to invoke them like this: python /usr/lib/python2.3/pdb.py

Neither approach is convenient...

Am I missing something obvious? If not, then would the following make sense?

When a script specified from command line is not found and the script name does not end with py, treat the script as a module name and execute that module as __main__

So python pdb would be equivalent to python /usr/lib/python2.3/pdb.py

A possible variation of the same idea would be to have an explicit command line option -m (or -M). More typing, but less magic...

+1 on the -m command-line variation, with the following change: I'd like Python to import the module and then run it's main function. I've been meaning to suggest smething like this myself. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Anthony Baxter wrote:

...

+1 on the -m command-line variation, with the following change:

I'd like Python to import the module and then run it's main function.

I've been meaning to suggest smething like this myself.

I'd prefer it import the module, with __name__ == "__main__", because it's compatible with what we do now for a module that's also a script. But I like the idea, nonetheless.

Question: should python -m foo.bar.baz work? I'd say "yes".

Me too. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Ilya Sandler wrote:

A problem:

a number of standard python modules come with a command line interfaces, e.g. pydoc.py, pdb.py , unittest.py, timeit.py, uu.py But it appears that there is no convenient out-of-the-box way to invoke these tools from command line...

Basically one either has to write wrappers or to invoke them like this: python /usr/lib/python2.3/pdb.py

Neither approach is convenient...

Am I missing something obvious? If not, then would the following make sense?

When a script specified from command line is not found and the script name does not end with py, treat the script as a module name and execute that module as __main__

So python pdb would be equivalent to python /usr/lib/python2.3/pdb.py

A possible variation of the same idea would be to have an explicit command line option -m (or -M). More typing, but less magic...

There is already has been some discussion about importing from command line: http://mail.python.org/pipermail/python-dev/2003-December/041240.html I suggested the following: 1. python -p package Equivalent to: import package 2. python -p package.zip Equivalent to: import sys sys.path.insert(0, "package.zip") import package -- Dmitry Vasiliev (dima at hlabs.spb.ru) http://hlabs.spb.ru