Multiple modules with database access + general app design?

Magnus Lycka lycka at
Thu Jan 19 18:00:37 CET 2006

Robin Haswell wrote:
> Can anyone give me advice on making this all a bit more transparent? I
> guess I really would like a method to bring all these files in to the same
> scope to make everything seem to be all one application, even though
> everything is broken up in to different files.

This is very much a deliberate design decision in Python.
I haven't used PHP, but in e.g. C, the #include directive
means that you pollute your namespace with all sorts of
strange names from all the third party libraries you are
using, and this doesn't scale well. As your application
grows, you'll get mysterious bugs due to strange name clashes,
removing some module you no-longer need means that your app
won't build since the include file you no longer include in
turn included another file that you should have included but
didn't etc. In Python, explicit is better than implicit (type
"import this" at the Python prompt) and while this causes some
extra typing it helps with code maintenance. You can always
see where a name in your current namespace comes from (unless
you use "from xxx import *"). No magic!

Concerning your database operations, it seems they are distributed
over a lot of different modules, and that might also cause problems,
whatever programming language we use. In typical database
applications, you need to keep track of transactions properly.

For each opened connection, you can perform a number of transactions
after each other. A transaction starts with the first database
operation after a connect, commit or rollback. A cursor should only
live within a transaction. In other words, you should close all
cursors before you perform a commit or rollback.

I find it very difficult to manage transactions properly if the
commits are spread out in the code. Usually I want one module to
contain some kind of transaction management logic, where I determine
the transaction boundries. This logic will hand out cursor object
to various pieces of code, and determine when to close the cursors
and commit the transaction.

I haven't really written multithreaded applications, so I don't
have any experiences in the problems that might cause. I know that
it's a fairly common pattern to have all database transactions in
one thread though, and to use Queue.Queue instances to pass data
to and from the thread that handles DB.

Anyway, you can only have one transaction going on at a time for
a connection, so if you share connections between threads (or use
a separate DB thread and queues) a rollback or commit in one thread
will affect the other threads as well...

Each DB-API 2.0 compliant library should be able to declare how it
can be used in a threaded application. See the DB-API 2.0 spec: Look for "threadsafety".

More information about the Python-list mailing list