[Python-Dev] string_join overrides TypeError exception thrown in generator

[Stephen Thorne]

Hi,

An interesting problem was pointed out to me, which I have distilled
to this testcase:
def gen():
     raise TypeError, "I am a TypeError"
     yield 1

def one(): return ''.join( x for x in gen() )
def two(): return ''.join([x for x in gen()])

for x in one, two:
    try:
         x()
    except TypeError, e:
         print e

Expected output is:
"""
I am a TypeError
I am a TypeError
"""

Actual output is:
"""
sequence expected, generator found
I am a TypeError
"""

Upon looking at the implementation of 'string_join' in
stringobject.c[1], It's quite obvious what's gone wrong, an exception
has been triggered in PySequence_Fast, and string_join overrides that
exception, assuming that the only TypeErrors thrown by PySequence_Fast
are caused by 'orig' being a value that was an invalid sequence type,
ignoring the possibility that a TypeError could be thrown by
exhausting a generator.

	seq = PySequence_Fast(orig, "");
	if (seq == NULL) {
		if (PyErr_ExceptionMatches(PyExc_TypeError))
			PyErr_Format(PyExc_TypeError,
				     "sequence expected, %.80s found",
				     orig->ob_type->tp_name);
		return NULL;
	}

I can't see an obvious solution, but perhaps generators should get
special treatment regardless. Reading over this code it looks like the
generator is exhausted all at once, instead of incrementally..
-- 
Stephen Thorne
Development Engineer

[1] http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Objects/stringobject.c?rev=2.231&view=markup