From andychambers2002 at yahoo.co.uk Thu Dec 1 03:22:52 2005 From: andychambers2002 at yahoo.co.uk (Andrew Chambers) Date: Thu, 1 Dec 2005 02:22:52 +0000 Subject: [DB-SIG] DB-API unittests Message-ID: <20051201022252.GA20340@yapgi> Hi All, I've been doing some work on some unittests for the Python DB-API. So far, I've created tests for... - Attempt to connect using invalid dbname should raise Error - Attempt to divide by zero should raise DataError - Attempt to violate foreign/primery key should raise IntegrityError - Attempt to create identical tables should raise ProgrammingError - Connection object should have minimum set of methods - Cursor object should have minimum set of methods - Simple select query should result in expected description/rowcount - Type_codes in description attr should evaluate to corresponding Singletons. It only works for postgresql drivers since it connects to template1 which should be present on most postgresql databases It raises a few points (these may have been discussed here before but googling the archives didn't reveal them. - Evidently there is ambiguity about the way that dates should be entered. pgdb - curs.execute("select '%s'" % Date(2005, 01, 01)) psycopg - curs.execute("select %s" % Date(2005, 01, 01)) Don't know about the others - I don't have them installed - Exceptions. My interpretation of the API seems to deviate from that of psycopg and pgdb. Does anyone have any views on which is correct? I wrote the tests without having any existing implementation in mind just going from the spec. Are the tests at fault? I'd also be interested in what else you think should be tested. I've started to write some performance tests that simply query the pg_type table for a fixed time period (30 secs). These need more work though. I'll post if anyone asks. Finally, I couldn't think of a way to run these tests across a number of modules without duplicating the code (although to be honest I haven't given it much thought yet. If anyone has any ideas I'd be interested to hear those too. #!/usr/bin/env python import unittest import time from import * class ConnectionTestCase(unittest.TestCase): def testConnection(self): """Connection Object returned should have minimum operations""" conn = connect(database="template1") self.assert_(set(('close', 'commit', 'cursor', 'rollback')).issubset(set(dir(conn)))) def testConnectFail(self): """Should raise error on connecting with bad parameters""" self.assertRaises(DatabaseError, connect, database="fun'y name") class CursorTestCase(unittest.TestCase): def setUp(self): self.conn = connect(database="template1") self.curs = self.conn.cursor() def tearDown(self): del self.curs del self.conn def testHasAttrs(self): """Cursor object should have listed attributes\n """ self.curs.execute("select 1, 2;") required = ('arraysize', 'callproc', 'close', 'description', 'execute', 'executemany', 'fetchall', 'fetchmany', 'fetchone', 'rowcount', 'setinputsizes', 'setoutputsize') actual = (dir(self.curs)) diff = set(required).difference(set(actual)) self.assert_(len(diff) == 0, "Absentees: %s" % diff) def testSelect(self): """Data returned should be as expected For the query below, rowcount = 4 description = [(n, int4, 4, None, None, None, None), (t, 'text', -1, None, None, None, None)] """ sql = """select 1 as n, 'text' as t union select 2, 'two' union select 3, 'three' """ required_rows = 3 required_desc = [('n', 'int4', 4, None, None, None, None), ('t', 'text', -1, None, None, None, None)] self.curs.execute(sql) self.assert_(self.curs.rowcount == required_rows, "Actual rows: %s" % self.curs.rowcount) self.assert_(self.curs.description == required_desc, "Actual description: %s" % self.curs.description) def testSelectParams(self): """Query using parameters in simple pyformat """ sql = """select 1 as %s, '%s' as t union select %i, 'two' union select 3, 'three' """ self.curs.execute(sql, ('one', 'text', 2)) required_rows = 3 self.assert_(self.curs.rowcount == required_rows, "Actual rows: %s" % self.curs.rowcount) def testSelectTypes(self): """Description types should evaluate equal to module constants""" sql = """select '%(string)s', %(number)i, %(datetime)s""" self.curs.execute(sql, {'string' : "String", 'number' : 42, 'datetime' : Date(2005, 01, 01)}) desc = self.curs.description self.assertEqual(desc[0][1], STRING, str(desc)) self.assertEqual(desc[1][1], NUMBER) self.assertEqual(desc[2][1], DATETIME) def testInsertOK(self): """Successfully insert valid data into test table """ create = """create table pygres_ins_test ( order_no int PRIMARY KEY, supp_addr text, order_date date ); """ insert = """insert into pygres_ins_test values (%i, '%s', %s)""" destroy = "drop table pygres_ins_test;" self.curs.execute(create) self.curs.execute(insert, (1, "Any'town", Date(2005, 11, 11))) self.curs.execute(destroy) def testProgrammingError(self): """Attempt to create identical tables should raise ProgrammingError """ create = """create table dup ( a integer ); """ self.curs.execute(create) self.assertRaises(ProgrammingError, self.curs.execute, create) def testIntegrityErrorPK(self): """Attempt to violate primary key should raise IntegrityError """ create = """create table order ( order_id integer PRIMARY KEY, order_desc text ); """ insert = "insert into order values (%i, '%s')" self.curs.execute(create) self.curs.execute(insert, (1, 'First Order')) self.assertRaises(IntegrityError, self.curs.execute, insert, (1, 'Second Order')) def testIntegrityErrorFK(self): """Attempt to violate foreign key should raise IntegrityError""" create_1 = """ create table order ( order_id integer PRIMARY KEY, order_desc text );""" create_2 = """ create table order_item ( order_id integer REFERENCES order(order_id), product_id integer, quantity numeric, PRIMARY KEY(order_id, product_id) ); """ insert_ord = "insert into order values (%i, '%s')" insert_ord_line = "insert into order_item values (%i, %i, %i)" self.curs.execute(create_1) self.curs.execute(insert_ord, (1, "First Order")) self.conn.commit() self.curs.execute(create_2) self.conn.commit() self.assertRaises(IntegrityError, self.curs.execute, insert_ord_line, (2, 2, 2)) self.conn.commit() self.curs.execute("drop table order cascade;") self.curs.execute("drop table order_item;") self.conn.commit() def testDataError(self): """Attempt to divide by zero should raise DataError """ sql = "select (10 / 0)" self.assertRaises(DataError, self.curs.execute, sql) class PerformanceTestCase(unittest.TestCase): def __connect(self): self.conn = connect(database="template1") self.curs = self.conn.cursor() def __close(self): del self.curs del self.conn def testRetrievals(self): self.__connect() rows = 0 start = time.time() while True: self.curs.execute("select * from pg_type") while True: row = self.curs.fetchone() rows += 1 if time.time() - start > 30: print rows self.__close() return def connectionSuite(): """Returns the Connection test suite""" return unittest.makeSuite(ConnectionTestCase) def cursorSuite(): return unittest.makeSuite(CursorTestCase) if __name__ == '__main__': unittest.main() From fog at initd.org Thu Dec 1 09:49:43 2005 From: fog at initd.org (Federico Di Gregorio) Date: Thu, 01 Dec 2005 09:49:43 +0100 Subject: [DB-SIG] DB-API unittests In-Reply-To: <20051201022252.GA20340@yapgi> References: <20051201022252.GA20340@yapgi> Message-ID: <1133426983.4337.3.camel@localhost> Il giorno gio, 01/12/2005 alle 02.22 +0000, Andrew Chambers ha scritto: > - Evidently there is ambiguity about the way that dates should be entered. > pgdb - curs.execute("select '%s'" % Date(2005, 01, 01)) > psycopg - curs.execute("select %s" % Date(2005, 01, 01)) > Don't know about the others - I don't have them installed Ambiguity? IMHO, having to quote in the query string is obviously the wrong one. federico -- Federico Di Gregorio http://people.initd.org/fog Debian GNU/Linux Developer fog at debian.org INIT.D Developer fog at initd.org Don't dream it. Be it. -- Dr. Frank'n'further -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Questa parte del messaggio =?ISO-8859-1?Q?=E8?= firmata Url : http://mail.python.org/pipermail/db-sig/attachments/20051201/27cb6464/attachment.pgp From fumanchu at amor.org Thu Dec 1 10:07:56 2005 From: fumanchu at amor.org (Robert Brewer) Date: Thu, 1 Dec 2005 01:07:56 -0800 Subject: [DB-SIG] DB-API unittests References: <20051201022252.GA20340@yapgi> Message-ID: Andrew Chambers wrote: > I've been doing some work on some unittests for > the Python DB-API. > ... > Finally, I couldn't think of a way to run these > tests across a number of modules without duplicating > the code (although to be honest I haven't given > it much thought yet. If anyone has any ideas > I'd be interested to hear those too. See the zoo_fixture module (particularly the "run" function) and any of the test_store* modules at http://projects.amor.org/dejavu/browser/trunk/test for an example of how to run different databases against a common unittest suite, one at a time. The test.py module in that folder will run them all in order (any that are installed on the system, that is). Robert Brewer System Architect Amor Ministries fumanchu at amor.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/db-sig/attachments/20051201/de4dbd29/attachment.htm From mal at egenix.com Thu Dec 1 11:07:07 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 01 Dec 2005 11:07:07 +0100 Subject: [DB-SIG] DB-API unittests In-Reply-To: <20051201022252.GA20340@yapgi> References: <20051201022252.GA20340@yapgi> Message-ID: <438ECB4B.5070304@egenix.com> Andrew Chambers wrote: > Hi All, > > I've been doing some work on some unittests for the Python DB-API. I wonder whether you are aware of the existing test suite for DB-API 2.0 by Stuart Bishop: http://stuartbishop.net/Software/DBAPI20TestSuite/ In particular, it tests psycopg. > So far, I've created tests for... > > - Attempt to connect using invalid dbname should raise Error > - Attempt to divide by zero should raise DataError > - Attempt to violate foreign/primery key should raise IntegrityError > - Attempt to create identical tables should raise ProgrammingError > - Connection object should have minimum set of methods > - Cursor object should have minimum set of methods > - Simple select query should result in expected description/rowcount > - Type_codes in description attr should evaluate to corresponding > Singletons. > > It only works for postgresql drivers since it connects to template1 > which should be present on most postgresql databases > > It raises a few points (these may have been discussed here before but > googling the archives didn't reveal them. > > - Evidently there is ambiguity about the way that dates should be entered. > pgdb - curs.execute("select '%s'" % Date(2005, 01, 01)) > psycopg - curs.execute("select %s" % Date(2005, 01, 01)) > Don't know about the others - I don't have them installed psycopg is doing the right thing here - whatever Date() returns should be directly usable with the %s parameter marker. > - Exceptions. My interpretation of the API seems to deviate from that > of psycopg and pgdb. Does anyone have any views on which is correct? > I wrote the tests without having any existing implementation in mind > just going from the spec. Are the tests at fault? Hard to tell... what are your views ? > I'd also be interested in what else you think should be tested. I've > started to write some performance tests that simply query the pg_type > table for a fixed time period (30 secs). These need more work though. > I'll post if anyone asks. > > Finally, I couldn't think of a way to run these tests across a number of > modules without duplicating the code (although to be honest I haven't > given it much thought yet. If anyone has any ideas I'd be interested to > hear those too. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 01 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From andychambers2002 at yahoo.co.uk Thu Dec 1 14:32:10 2005 From: andychambers2002 at yahoo.co.uk (Andy Chambers) Date: Thu, 1 Dec 2005 13:32:10 +0000 (GMT) Subject: [DB-SIG] DB-API unittests In-Reply-To: <1133426983.4337.3.camel@localhost> Message-ID: <20051201133211.74251.qmail@web26910.mail.ukl.yahoo.com> --- Federico Di Gregorio wrote: > Il giorno gio, 01/12/2005 alle 02.22 +0000, Andrew Chambers ha scritto: > > - Evidently there is ambiguity about the way that dates should be entered. > > pgdb - curs.execute("select '%s'" % Date(2005, 01, 01)) > > psycopg - curs.execute("select %s" % Date(2005, 01, 01)) > > Don't know about the others - I don't have them installed > > Ambiguity? IMHO, having to quote in the query string is obviously the > wrong one. Sorry Federico. Forgot to reply to the list That's what I thought too. Although doesn't pgdb have the largest userbase out of existing APIs? Regards, Andy ___________________________________________________________ Yahoo! Model Search 2005 - Find the next catwalk superstars - http://uk.news.yahoo.com/hot/model-search/ From carsten at uniqsys.com Thu Dec 1 14:36:27 2005 From: carsten at uniqsys.com (Carsten Haese) Date: Thu, 01 Dec 2005 08:36:27 -0500 Subject: [DB-SIG] DB-API unittests In-Reply-To: <438ECB4B.5070304@egenix.com> References: <20051201022252.GA20340@yapgi> <438ECB4B.5070304@egenix.com> Message-ID: <1133444187.13621.30.camel@dot.uniqsys.com> On Thu, 2005-12-01 at 05:07, M.-A. Lemburg wrote: > Andrew Chambers wrote: > > - Evidently there is ambiguity about the way that dates should be entered. > > pgdb - curs.execute("select '%s'" % Date(2005, 01, 01)) > > psycopg - curs.execute("select %s" % Date(2005, 01, 01)) > > Don't know about the others - I don't have them installed > > psycopg is doing the right thing here - whatever Date() returns > should be directly usable with the %s parameter marker. Since nobody else seems to have noticed this, I'll point this out: The OP is not using %s as a parameter marker, since he is not passing a parameter tuple. He is using the %-operator to build a query as if DB-API didn't have parametrized queries. The correct way of course is curs.execute("select %s", (Date(2005, 01, 01),) ) which should work on any DB-API implementation that uses this particular marker style. IMHO this parameter style should be deprecated because it makes it way too easy to make -- and overlook -- this kind of mistake. -Carsten Haese From andychambers2002 at yahoo.co.uk Thu Dec 1 14:40:59 2005 From: andychambers2002 at yahoo.co.uk (Andy Chambers) Date: Thu, 1 Dec 2005 13:40:59 +0000 (GMT) Subject: [DB-SIG] DB-API unittests In-Reply-To: <438ECB4B.5070304@egenix.com> Message-ID: <20051201134100.33568.qmail@web26902.mail.ukl.yahoo.com> --- "M.-A. Lemburg" wrote: > I wonder whether you are aware of the existing test suite > for DB-API 2.0 by Stuart Bishop: > > http://stuartbishop.net/Software/DBAPI20TestSuite/ Mmm. No I wasn't. Thanks for the link. > > - Exceptions. My interpretation of the API seems to deviate from that > > of psycopg and pgdb. Does anyone have any views on which is correct? > > I wrote the tests without having any existing implementation in mind > > just going from the spec. Are the tests at fault? > > Hard to tell... what are your views ? The unittests in my first post represent as much of my views as I've had time to think about so far. I don't think I can make it any clearer than those (see in particular test.*Error(self)) Regards, Andy ___________________________________________________________ Yahoo! Model Search 2005 - Find the next catwalk superstars - http://uk.news.yahoo.com/hot/model-search/ From fog at initd.org Thu Dec 1 15:03:17 2005 From: fog at initd.org (Federico Di Gregorio) Date: Thu, 01 Dec 2005 15:03:17 +0100 Subject: [DB-SIG] DB-API unittests In-Reply-To: <20051201133211.74251.qmail@web26910.mail.ukl.yahoo.com> References: <20051201133211.74251.qmail@web26910.mail.ukl.yahoo.com> Message-ID: <1133445797.4450.20.camel@localhost> Il giorno gio, 01/12/2005 alle 13.32 +0000, Andy Chambers ha scritto: > --- Federico Di Gregorio wrote: > > > Il giorno gio, 01/12/2005 alle 02.22 +0000, Andrew Chambers ha scritto: > > > - Evidently there is ambiguity about the way that dates should be entered. > > > pgdb - curs.execute("select '%s'" % Date(2005, 01, 01)) > > > psycopg - curs.execute("select %s" % Date(2005, 01, 01)) > > > Don't know about the others - I don't have them installed > > > > Ambiguity? IMHO, having to quote in the query string is obviously the > > wrong one. > > Sorry Federico. Forgot to reply to the list > > That's what I thought too. Although doesn't pgdb have the > largest userbase out of existing APIs? I don't know, but judging from the drivers suggested by most Python applications out there (fast search on freshmeat) that's not the case. psycopg and pypgsql seem to be the most used. -- Federico Di Gregorio http://people.initd.org/fog Debian GNU/Linux Developer fog at debian.org INIT.D Developer fog at initd.org Gli esseri umani, a volte, sono destinati, per il solo fatto di esistere, a fare del male a qualcuno. -- Haruki Murakami -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Questa parte del messaggio =?ISO-8859-1?Q?=E8?= firmata Url : http://mail.python.org/pipermail/db-sig/attachments/20051201/fcd9c8ed/attachment.pgp From marcos at burke.ath.cx Sat Dec 3 18:44:21 2005 From: marcos at burke.ath.cx (Marcos =?ISO-8859-1?Q?S=E1nchez?= Provencio) Date: Sat, 03 Dec 2005 18:44:21 +0100 Subject: [DB-SIG] DB-API unittests In-Reply-To: <20051201133211.74251.qmail@web26910.mail.ukl.yahoo.com> References: <20051201133211.74251.qmail@web26910.mail.ukl.yahoo.com> Message-ID: <1133631861.7473.1.camel@localhost.localdomain> Just some information; pgdb is the only pg driver for python2.4 on debian stable. El jue, 01-12-2005 a las 13:32 +0000, Andy Chambers escribi?: > --- Federico Di Gregorio wrote: > > > Il giorno gio, 01/12/2005 alle 02.22 +0000, Andrew Chambers ha scritto: > > > - Evidently there is ambiguity about the way that dates should be entered. > > > pgdb - curs.execute("select '%s'" % Date(2005, 01, 01)) > > > psycopg - curs.execute("select %s" % Date(2005, 01, 01)) > > > Don't know about the others - I don't have them installed > > > > Ambiguity? IMHO, having to quote in the query string is obviously the > > wrong one. > > Sorry Federico. Forgot to reply to the list > > That's what I thought too. Although doesn't pgdb have the > largest userbase out of existing APIs? > > Regards, > Andy > > > > > ___________________________________________________________ > Yahoo! Model Search 2005 - Find the next catwalk superstars - http://uk.news.yahoo.com/hot/model-search/ > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > http://mail.python.org/mailman/listinfo/db-sig From anthony.tuininga at gmail.com Fri Dec 16 22:41:08 2005 From: anthony.tuininga at gmail.com (Anthony Tuininga) Date: Fri, 16 Dec 2005 14:41:08 -0700 Subject: [DB-SIG] cx_Oracle 4.1.1 Message-ID: <703ae56b0512161341x65d40fb6wcdb44bcd932012b2@mail.gmail.com> What is cx_Oracle? cx_Oracle is a Python extension module that allows access to Oracle and conforms to the Python database API 2.0 specifications with a few exceptions. Where do I get it? http://starship.python.net/crew/atuining What's new? 1) Add support for dropping a connection from a session pool. 2) Add support for write only attributes "module", "action" and "clientinfo" which work only in Oracle 10g as requested by Egor Starostin. 3) Add support for pickling database errors. 4) Use the previously created bind variable as a template if available when creating a new variable of a larger size. Thanks to Ted Skolnick for the initial patch. 5) Fixed tests to work properly in the Python 2.4 environment where dates and timestamps are different Python types. Thanks to Henning von Bargen for pointing this out. 6) Added additional directories to search for include files and libraries in order to better support the Oracle 10g instant client. 7) Set the internal fetch number to 0 in order to satisfy very picky source analysis tools as requested by Amaury Fogeot d'Arc. 8) Improve the documentation for building and installing the module from source as some people are unaware of the standard methods for building Python modules using distutils. 9) Added note in the documentation indicating that the arraysize attribute can drastically affect performance of queries since this seems to be a common misunderstanding of first time users of cx_Oracle. 10) Add a comment indicating that on HP-UX Itanium with Oracle 10g the library ttsh10 must alos be linked against. Thanks to Bernard Delmee for the information. Anthony Tuininga From ianb at colorstudy.com Sun Dec 18 20:33:05 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 18 Dec 2005 13:33:05 -0600 Subject: [DB-SIG] [Web-SIG] WSGI thread affinity/interleaving In-Reply-To: References: <5.1.1.6.0.20051217171840.01e1ab80@mail.telecommunity.com> Message-ID: <43A5B971.1010408@colorstudy.com> James Y Knight wrote: > I'm worried about database access. Most DBAPI adapters have > threadsafety level 2: "Threads may share the module and > connections.". So with those, at least, it should be fine to move a > connection between threads, since "share OK" implies "move OK". > However, no documentation I've found has said anything separately > about whether it's safe to _move_ a cursor between threads. It seems > likely to me that it would not be safe, at least in some database > adapters. And if it's not safe, that means a WSGI result iterator > cannot use any DBAPI cursor functionality which seems a drag. > > Does anybody have practical experience with the safety of moving a > DBAPI cursor between threads? I haven't done that, but SQLite (2?) notably doesn't allow you to move a connection between threads. I'm not actually sure what problems it causes if you do move them -- it may simply be an overzealous warning. CCing DB-SIG -- people there might know more details. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From gh at ghaering.de Sun Dec 18 23:18:23 2005 From: gh at ghaering.de (=?ISO-8859-1?Q?Gerhard_H=E4ring?=) Date: Sun, 18 Dec 2005 23:18:23 +0100 Subject: [DB-SIG] [Web-SIG] WSGI thread affinity/interleaving In-Reply-To: <43A5B971.1010408@colorstudy.com> References: <5.1.1.6.0.20051217171840.01e1ab80@mail.telecommunity.com> <43A5B971.1010408@colorstudy.com> Message-ID: <43A5E02F.4090206@ghaering.de> Ian Bicking wrote: > James Y Knight wrote: >> I'm worried about database access. Most DBAPI adapters have >> threadsafety level 2: "Threads may share the module and >> connections.". So with those, at least, it should be fine to move a >> connection between threads, since "share OK" implies "move OK". >> However, no documentation I've found has said anything separately >> about whether it's safe to _move_ a cursor between threads. It seems >> likely to me that it would not be safe, at least in some database >> adapters. And if it's not safe, that means a WSGI result iterator >> cannot use any DBAPI cursor functionality which seems a drag. >> >> Does anybody have practical experience with the safety of moving a >> DBAPI cursor between threads? > > I haven't done that, but SQLite (2?) notably doesn't allow you to move a > connection between threads. I'm not actually sure what problems it > causes if you do move them -- it may simply be an overzealous warning. It's the same for SQLite 3. The problem is, as far as I understand, that POSIX file locks don't work reliably when they're accessed from multiple threads. That's why the SQLite *docs* always said that you cannot share a SQLite database handle between threads. And pysqlite as well as apsw both fire exceptions if you try to do so. In recent SQLite 3.x versions, SQLite itself would detect this and return an error on *nix too FWIW. pysqlite does have an option to turn the check off, for people who want to shoot themselves in the foot. Fortunately for them, they nowadays get an error-message from SQLite on non-Windows systems anyway ;-) -- Gerhard From mal at egenix.com Mon Dec 19 00:57:11 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 19 Dec 2005 00:57:11 +0100 Subject: [DB-SIG] [Web-SIG] WSGI thread affinity/interleaving In-Reply-To: <43A5B971.1010408@colorstudy.com> References: <5.1.1.6.0.20051217171840.01e1ab80@mail.telecommunity.com> <43A5B971.1010408@colorstudy.com> Message-ID: <43A5F757.2030906@egenix.com> Ian Bicking wrote: > James Y Knight wrote: >> I'm worried about database access. Most DBAPI adapters have >> threadsafety level 2: "Threads may share the module and >> connections.". So with those, at least, it should be fine to move a >> connection between threads, since "share OK" implies "move OK". What exactly do you mean with "move" ? Sharing a connection refers to multiple threads creating cursors on this connection. >> However, no documentation I've found has said anything separately >> about whether it's safe to _move_ a cursor between threads. It seems >> likely to me that it would not be safe, at least in some database >> adapters. Thread level 3 adapters would allow for sharing a cursor meaning that you can call cursor.execute() from different threads. Given that you usually already have to be careful with sharing connections, sharing cursors is rather unlikely to work in a general setting. > And if it's not safe, that means a WSGI result iterator >> cannot use any DBAPI cursor functionality which seems a drag. >> >> Does anybody have practical experience with the safety of moving a >> DBAPI cursor between threads? > > I haven't done that, but SQLite (2?) notably doesn't allow you to move a > connection between threads. I'm not actually sure what problems it > causes if you do move them -- it may simply be an overzealous warning. > > CCing DB-SIG -- people there might know more details. Sharing cursors is possible with some database drivers and can be used to e.g. pool cursors with prepared commands. mxODBC does support this if the ODBC driver is thread-safe (which it should be if it adheres to the ODBC standard). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 19 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From p.f.moore at gmail.com Sun Dec 18 22:33:12 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 18 Dec 2005 21:33:12 +0000 Subject: [DB-SIG] [Web-SIG] WSGI thread affinity/interleaving In-Reply-To: <43A5B971.1010408@colorstudy.com> References: <5.1.1.6.0.20051217171840.01e1ab80@mail.telecommunity.com> <43A5B971.1010408@colorstudy.com> Message-ID: <79990c6b0512181333h7445b21ch4153f127b74ca556@mail.gmail.com> On 12/18/05, Ian Bicking wrote: > James Y Knight wrote: > > Does anybody have practical experience with the safety of moving a > > DBAPI cursor between threads? > > I haven't done that, but SQLite (2?) notably doesn't allow you to move a > connection between threads. I'm not actually sure what problems it > causes if you do move them -- it may simply be an overzealous warning. > > CCing DB-SIG -- people there might know more details. I can confirm that cx_Oracle does not like cursors being shared between threads. I even recall crashes (but can't verify this - once I checked and found I shouldn't be doing this, I stopped - the problem was intermittent, as is the nature of thread bugs :-(). Paul. From foom at fuhm.net Mon Dec 19 01:34:56 2005 From: foom at fuhm.net (James Y Knight) Date: Sun, 18 Dec 2005 19:34:56 -0500 Subject: [DB-SIG] [Web-SIG] WSGI thread affinity/interleaving In-Reply-To: <43A5F757.2030906@egenix.com> References: <5.1.1.6.0.20051217171840.01e1ab80@mail.telecommunity.com> <43A5B971.1010408@colorstudy.com> <43A5F757.2030906@egenix.com> Message-ID: <6B850331-4947-4824-84A3-2C04BC32BEA8@fuhm.net> On Dec 18, 2005, at 6:57 PM, M.-A. Lemburg wrote: > Ian Bicking wrote: > >> James Y Knight wrote: >> >>> I'm worried about database access. Most DBAPI adapters have >>> threadsafety level 2: "Threads may share the module and >>> connections.". So with those, at least, it should be fine to move a >>> connection between threads, since "share OK" implies "move OK". >>> > > What exactly do you mean with "move" ? Sharing a > connection refers to multiple threads creating cursors > on this connection. I'm asking about moving a cursor, that is, accessing it sequentially first from one thread, then later from another thread. This is potentially asking less than sharing, that is, accessing it simultaneously from two threads. For example, a simple class without any locking, that only modifies itself, would generally be movable between threads, but not sharable. Adding a mutex would make it both. James From mal at egenix.com Mon Dec 19 15:17:49 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 19 Dec 2005 15:17:49 +0100 Subject: [DB-SIG] [Web-SIG] WSGI thread affinity/interleaving In-Reply-To: <6B850331-4947-4824-84A3-2C04BC32BEA8@fuhm.net> References: <5.1.1.6.0.20051217171840.01e1ab80@mail.telecommunity.com> <43A5B971.1010408@colorstudy.com> <43A5F757.2030906@egenix.com> <6B850331-4947-4824-84A3-2C04BC32BEA8@fuhm.net> Message-ID: <43A6C10D.5000503@egenix.com> James Y Knight wrote: > On Dec 18, 2005, at 6:57 PM, M.-A. Lemburg wrote: > >> Ian Bicking wrote: >> >>> James Y Knight wrote: >>> >>>> I'm worried about database access. Most DBAPI adapters have >>>> threadsafety level 2: "Threads may share the module and >>>> connections.". So with those, at least, it should be fine to move a >>>> connection between threads, since "share OK" implies "move OK". >>>> >> What exactly do you mean with "move" ? Sharing a >> connection refers to multiple threads creating cursors >> on this connection. > > I'm asking about moving a cursor, that is, accessing it sequentially > first from one thread, then later from another thread. This is > potentially asking less than sharing, that is, accessing it > simultaneously from two threads. > > For example, a simple class without any locking, that only modifies > itself, would generally be movable between threads, but not sharable. > Adding a mutex would make it both. Ok. In that sense, I think "moving" is not really possible with database connections or cursors: these always rely on external resources and these may be relying on having the same thread context around when being called. Why would you want to "move" cursors or connections around ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 19 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From anthony.tuininga at gmail.com Mon Dec 19 18:28:16 2005 From: anthony.tuininga at gmail.com (Anthony Tuininga) Date: Mon, 19 Dec 2005 10:28:16 -0700 Subject: [DB-SIG] cx_Oracle 4.1.2 Message-ID: <703ae56b0512190928v529141a7pe2577a3cbf0688ae@mail.gmail.com> What is cx_Oracle? cx_Oracle is a Python extension module that allows access to Oracle and conforms to the Python database API 2.0 specifications with a few exceptions. Where do I get it? http://starship.python.net/crew/atuining What's new? 1) Restore support of Oracle 9i features when using the Oracle 10g client. Anthony Tuininga From goodger at python.org Wed Dec 28 16:12:19 2005 From: goodger at python.org (David Goodger) Date: Wed, 28 Dec 2005 10:12:19 -0500 Subject: [DB-SIG] PyCon TX 2006: Early-bird registration ends Dec. 31! Message-ID: <43B2AB53.1000409@python.org> Early bird registration for PyCon TX 2006 ends on December 31st, so there are only a few days LEFT. To register, please visit: http://us.pycon.org/TX2006/Registration You can still register after Dec. 31st, but the cost will go up by US$65 (US$25 for students). This year PyCon will feature a day of tutorials before the three days of regular presentations. Course outlines for all the tutorials have been posted; see http://wiki.python.org/moin/PyCon2006/Tutorials All of the PyCon tutorials are still open for new registrations, but space is limited, and we suspect they'll all be filled up by the time early-bird registration closes. Don't forget to book your hotel room, too. PyCon TX 2006 is being held at a Dallas/Addison hotel, and we have negotiated a special low rate: http://us.pycon.org/Addison/Hotels We hope to see you in Texas! -- David Goodger (on behalf of A.M. Kuchling, Chair, PyCon 2006) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 249 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/db-sig/attachments/20051228/5fef451d/signature.pgp From guido at python.org Sat Dec 31 06:10:02 2005 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Dec 2005 21:10:02 -0800 Subject: [DB-SIG] [Web-SIG] WSGI thread affinity/interleaving In-Reply-To: <43A6C10D.5000503@egenix.com> References: <5.1.1.6.0.20051217171840.01e1ab80@mail.telecommunity.com> <43A5B971.1010408@colorstudy.com> <43A5F757.2030906@egenix.com> <6B850331-4947-4824-84A3-2C04BC32BEA8@fuhm.net> <43A6C10D.5000503@egenix.com> Message-ID: On 12/19/05, M.-A. Lemburg wrote: > Ok. In that sense, I think "moving" is not really possible > with database connections or cursors: these always rely on > external resources and these may be relying on having the > same thread context around when being called. > > Why would you want to "move" cursors or connections around ? A typical connection (or cursor) caching implementation used from a multi-threaded program might easily do this: a resource is created in one thread, used for a while, then given back to the cache; when another thread requests a resource, it gets one from the cache that might have been used previously in a different thread. Keeping a cache per thread is a bit cumbersome and reduces the efficacy of the cache (if a thread goes away all the resources cached on its behalf must be closed instead of being made available to other threads). I'm not sure I understand what resources a typical DB client library might have that are associated with a thread instead of with a connection or cursor -- IOW I don't understand why you think moving resources between threads would be a problem, as long as only one thread "owns" them at any time. IOW if I maintain my own locking, why would I still be limited in sharing connections/cursors between threads? What am I missing? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mal at egenix.com Sat Dec 31 13:13:13 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 31 Dec 2005 13:13:13 +0100 Subject: [DB-SIG] [Web-SIG] WSGI thread affinity/interleaving In-Reply-To: References: <5.1.1.6.0.20051217171840.01e1ab80@mail.telecommunity.com> <43A5B971.1010408@colorstudy.com> <43A5F757.2030906@egenix.com> <6B850331-4947-4824-84A3-2C04BC32BEA8@fuhm.net> <43A6C10D.5000503@egenix.com> Message-ID: <43B675D9.4090708@egenix.com> Guido van Rossum wrote: > On 12/19/05, M.-A. Lemburg wrote: >> Ok. In that sense, I think "moving" is not really possible >> with database connections or cursors: these always rely on >> external resources and these may be relying on having the >> same thread context around when being called. >> >> Why would you want to "move" cursors or connections around ? > > A typical connection (or cursor) caching implementation used from a > multi-threaded program might easily do this: a resource is created in > one thread, used for a while, then given back to the cache; when > another thread requests a resource, it gets one from the cache that > might have been used previously in a different thread. Keeping a cache > per thread is a bit cumbersome and reduces the efficacy of the cache > (if a thread goes away all the resources cached on its behalf must be > closed instead of being made available to other threads). > > I'm not sure I understand what resources a typical DB client library > might have that are associated with a thread instead of with a > connection or cursor -- IOW I don't understand why you think moving > resources between threads would be a problem, as long as only one > thread "owns" them at any time. IOW if I maintain my own locking, why > would I still be limited in sharing connections/cursors between > threads? What am I missing? All this would be easily possible if the Python cursor object had full control over the external resources in use. However, most Python database cursor objects rely on external libraries and therefore do not have control over where state is stored. If you use a resource from a different thread than the one where it was created, this can cause situations where part of the state is missing, or worse, a different state is used. Many database libraries do their own caching at various levels (network connection, logical connections, cursors). Not all of them are fully thread-safe and its hard to find out. To be on the safe side, you should only use connections from the thread they were created with. It's still worthwhile to cache the connections (and even cursors on that connection): * connecting to a database can take anything from micro-seconds to several seconds; * preparing statements for execution on a cursor also takes time (and in most cases costs a network roundtrip), so caching already prepared cursors also makes sense for commonly used statements. The latter is especially useful with bound parameters since the database will usually only have to prepare the statement once and can then take any number of parameter sets to execute the statement with. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 31 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::