[Tutor] glibc error while Python script runs - Solved

[Bernard Lebel]

On 1/19/06, Kent Johnson <kent37 at tds.net> wrote:
> Hi Bernard,
>
> I'm glad you got it working but kind of surprised at what you had to do.
> You shouldn't have to have a single thread to access the database. In
> your original desing were you sharing a connection between threads? That
> could cause trouble. But if each connection has its own thread and you
> are using transactions and isolation levels appropriately, they
> shouldn't stomp on each other.

I'm not sure what you mean by transaction and isolation levels. I get
some sort of an idea, but could you elaborate on the idea?

I have not considered having one connection per thread, and have not
tested the idea. The way it works right now is that there is only one
connection, as a class instance attribute. All transaction go through
that connection.


>
> There are some good recipes for queuing up tasks and getting callbacks
> that might be simpler than your dictionary solution. For example see
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/435883
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/203871
>
> You could also use a second queue to return results to the main thread.

Thanks I'll check these out.

Bernard



>
> Kent
>
> Bernard Lebel wrote:
> > Hello,
> >
> > For the record, in case anyone would like to know what happened with
> > this issue...
> >
> >
> > It seems that the problem was related to the way I managed MySQL
> > connections. There was the main thread that would query every 5
> > seconds the database to see if a someone wanted to abort the job
> > running on the render node.
> >
> > However, the actual rendering was managed by a child thread, who would
> > perform many MySQL transactions, depending on the output produced by
> > the rendering software.
> >
> > So there were two separate threads doing MySQL queries. I noticed by
> > reading my debug logs that some of these queries would become
> > interlaced, that is, one query would try to update while another would
> > try to select-fetch. In turned out that this would severe the
> > connection with the database, and soon after Python would crash
> > altogether.
> >
> > So the solution was to start some sort of queue server in a separate
> > thread. This queue would consist of a list, and each time the program
> > would want to perform a MySQL operation, it would add it to the queue.
> > Then, the caller would enter a while loop that would wait for the
> > server to carry the operation.
> >
> > The server would run a while loop, that check the queue once every
> > iteration to see if there is something to do. When it finds something,
> > it would carry the entire transaction, then put the result in a
> > dictionary, using an ID number assigned when the transaction was
> > submitted to the queue.
> >
> > The "submitter" would then pick up the result, and return it to the
> > original caller.
> > The result is that no two queries would take place at the same time.
> >
> > Once I did that, my program stopped crashing, and actually has been
> > running very smooth since then. In fact my progam is now very very
> > close to completion.
> >
> >
> >
> > Kent: I took your suggestion and changed the program design and
> > splitted the code into separate modules. It was tricky to "re-wire"
> > everything together (it took me at least 4 days), but past that, I'm
> > glad I did it.
> >
> >
> > Thanks everyone for the help, it has been a great learning experience!
> >
> >
> > Bernard
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> http://mail.python.org/mailman/listinfo/tutor
>