Iterating generator from C (PostgreSQL's pl/python RETUN SETOF/RECORD iterator support broken on RedHat buggy libs)

Hannu Krosing hannu at
Mon May 15 22:33:10 CEST 2006

Sorry for cross-posting, but this IS a cross-platform issue.

Christian Tismer tismer at  wrote:
> Sven Suursoho wrote:
> >>>  Is there any way to rewrite following program to handle returned 
> >>> generator  without hitting this bug?
> The only way I can think of getting around this is to make
> sure that there is always a running python frame.
> This would be possible if you can control how the
> extension is called.

What would be the easiest way to hold a "always running" python frame ?

The actual calls to iterator come through a pl/python framework, which can hold some 
state - like some inter-call dictionaries - so keeping also a simple outer frame 
there may be possible.

What danger (apart from general uglyness) may lurk there in keeping this frame ?

> > Unfortunately, this is not an option because I can't control used 
> > environment: I'm trying to improve PostgreSQL's stored procedure 
> > language PL/Python and this software can be used everywhere.
> Then there is no other way than to report the bug, get the
> fix back-ported and nagging the PL/Python maintainers
> to update things after the fix.

Unfortunately there is no 'after the fix', as the fix must happen outside 
postgresql/plpython development and should also run on oldish systems.

The reason we want to get some workaround for that bug is the need 
to overcome resistance from core postgreSQL developers to inclusion of our 
plpython enchancements to postgreSQLs bundled plpython due to one specific use 
of our generic enchancement (using a built-in generator, usually a function with yield) 
on buggy RedHat's bundled causing crashes.

Also, PL/Python has been in minimal maintenance mode for many years, with 
basically only immediate bugfixing going on. 

We at Skype (that is currently Sven Suursoho) are the first ones doing 
active development on it after Andrew Bosma dropped development many years 
ago once he got just the very basic stuff working.

> Also a test should be added which is probably missing since a while.

Test where ? In build process, so that RedHat will spot it and 
fix their RPM builds ? 

As for testing in actual pl/python build environment, we had objections from 
leading postgresql Tom Lane that even if we do test it at build time, 
a determined DBA may substitute a buggy later and still crash her DB instance.

Do you have any ideas, how to test for buggy asserts in python runtime 
environment without actually triggering the assert ?

Then we could introduce some (uglish) special handling for generators in 
pl/pythons iterator.

> I'd put a warning somewhere that generators are broken in
> debug mode, file an issue as well, but avoid trying to hack
> around this. It would make the bug even more resistent :-)

We have been trying to advocate such approach, but so far with modest results :(

Hannu Krosing
Database Architect
Skype Technologies OÜ
Akadeemia tee 21 F, Tallinn, 12618, Estonia

Skype me:  callto:hkrosing
Get Skype for free:

More information about the Python-list mailing list