[Python-Dev] string_join overrides TypeError exception thrown in generator

Stephen Thorne stephen.thorne at gmail.com
Mon Aug 15 08:40:22 CEST 2005


An interesting problem was pointed out to me, which I have distilled
to this testcase:
def gen():
     raise TypeError, "I am a TypeError"
     yield 1

def one(): return ''.join( x for x in gen() )
def two(): return ''.join([x for x in gen()])

for x in one, two:
    except TypeError, e:
         print e

Expected output is:
I am a TypeError
I am a TypeError

Actual output is:
sequence expected, generator found
I am a TypeError

Upon looking at the implementation of 'string_join' in
stringobject.c[1], It's quite obvious what's gone wrong, an exception
has been triggered in PySequence_Fast, and string_join overrides that
exception, assuming that the only TypeErrors thrown by PySequence_Fast
are caused by 'orig' being a value that was an invalid sequence type,
ignoring the possibility that a TypeError could be thrown by
exhausting a generator.

	seq = PySequence_Fast(orig, "");
	if (seq == NULL) {
		if (PyErr_ExceptionMatches(PyExc_TypeError))
				     "sequence expected, %.80s found",
		return NULL;

I can't see an obvious solution, but perhaps generators should get
special treatment regardless. Reading over this code it looks like the
generator is exhausted all at once, instead of incrementally..
Stephen Thorne
Development Engineer

[1] http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Objects/stringobject.c?rev=2.231&view=markup

More information about the Python-Dev mailing list