Psycopg and threads problem

[Alban Hertroys]

Michael Fuhr wrote:
> Alban Hertroys <alban at magproductions.nl> writes:
> 
>>I'm using psycopg to insert records in a number of threads. After the 
>>threads finish, another thread runs to collect the inserted data. Now, 
>>my problem is that psycopg let's my threads end before the inserts 
>>actually took place, resulting in my collecting thread finding no 
>>records... They are inserted after it checks.
> 
> 
> I just wrote a small test program and was unable to duplicate your
> problem -- the inserting threads ran, then the collecting thread
> saw the data.
> 
> Without seeing your code we can only guess at what's wrong.  Please
> post the simplest program that reproduces the problem.

That's going to be difficult, as it is a rather complex system of 
interwoven threads with XML parsers and such. I tried making a test case 
myself, but after an hour of programming I wasn't near the actual situation.
It probably doesn't help that I don't have any previous experience with 
either Python or thread programming and that I _need_ a vacation...

> What versions of Python, psycopg, and PostgreSQL are you using?
> What operating system and version are you running on?

Python 2.3
Psycopg 1.0
PostgreSQL 7.4.2

Keep in mind that these are Debian Linux versions, and with their 
tendency to "touch" things to break it better... ;)

>>Is there a way to tell psycopg or python to wait until the insert took 
>>place?
> 
> In my test program, the inserting threads called conn.commit() and
> then exited.  I then joined all of the inserting threads before
> starting the collecting thread.  You could also use a condition
> variable to signal when the inserts have been committed.

At the moment, I still use commit (it has to go eventually, I need to be 
able to rollback), but the threads don't end at that point. Instead, 
they wait until they can insert the next record.

This may be the cause of my problem. It's all rather complex; I may be 
missing something in my own code (of which the base was done by someone 
else, who can't be reached for a while, to make matters worse).

Alban.