From cito at online.de Thu Nov 1 15:01:40 2012 From: cito at online.de (Christoph Zwerschke) Date: Thu, 01 Nov 2012 15:01:40 +0100 Subject: [DB-SIG] Use of context managers with DB API 2 Message-ID: <509280C4.10806@online.de> We're currently preparing a new release of PyGreSQL and want to make use of context managers. My obvious idea was to let connections and cursors act as context managers that just close themselves, and add an extra context manager in form of a "transaction" property on the connection object that can be used to wrap transactions. However, I then noticed that PySqlite, cx_Oracle, mx_odbc and pyodbc use connection as context managers differently, they do not close the connection on exit, but execute a rollback or commit instead. Though I felt a separate context manager for wrapping transactions would have been a better solution, I now think I should better follow the above examples and wrap transactions in the context manager of the connection, it looks like it already has become a quasi standard. Any opinions on that or reasons why it has been implemented this way? -- Christoph From mike_mp at zzzcomputing.com Thu Nov 1 15:21:47 2012 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Thu, 1 Nov 2012 10:21:47 -0400 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <509280C4.10806@online.de> References: <509280C4.10806@online.de> Message-ID: <6C5D15EC-1B87-4743-B500-002031CEE2E1@zzzcomputing.com> On Nov 1, 2012, at 10:01 AM, Christoph Zwerschke wrote: > We're currently preparing a new release of PyGreSQL and want to make use of context managers. My obvious idea was to let connections and cursors act as context managers that just close themselves, and add an extra context manager in form of a "transaction" property on the connection object that can be used to wrap transactions. > > However, I then noticed that PySqlite, cx_Oracle, mx_odbc and pyodbc use connection as context managers differently, they do not close the connection on exit, but execute a rollback or commit instead. > > Though I felt a separate context manager for wrapping transactions would have been a better solution, I now think I should better follow the above examples and wrap transactions in the context manager of the connection, it looks like it already has become a quasi standard. Any opinions on that or reasons why it has been implemented this way? IMO most real-world applications don't open and close raw database connections multiple times within their stream of execution, as the app will certainly need to talk to the database again, and the re-engagement of a new connection each time, including the messaging that occurs, is unnecessarily wasteful. The connection is usually stored in some global way or in a connection pool. The one exception to this would be an application that builds upon a transparent pooling service like PGBouncer. Whereas the transactional state on the connection is really the "resource" that is transitory - the snapshots and locks held for that transaction - that need to be cleanly closed surrounding many distinct operations as the application proceeds. From mal at egenix.com Thu Nov 1 16:43:06 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 01 Nov 2012 16:43:06 +0100 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <509280C4.10806@online.de> References: <509280C4.10806@online.de> Message-ID: <5092988A.1010709@egenix.com> Christoph Zwerschke wrote: > We're currently preparing a new release of PyGreSQL and want to make use of context managers. My > obvious idea was to let connections and cursors act as context managers that just close themselves, > and add an extra context manager in form of a "transaction" property on the connection object that > can be used to wrap transactions. > > However, I then noticed that PySqlite, cx_Oracle, mx_odbc and pyodbc use connection as context > managers differently, they do not close the connection on exit, but execute a rollback or commit > instead. > > Though I felt a separate context manager for wrapping transactions would have been a better > solution, I now think I should better follow the above examples and wrap transactions in the context > manager of the connection, it looks like it already has become a quasi standard. Any opinions on > that or reasons why it has been implemented this way? While I'm not much of a fan of using connections as context managers (error handling gets tricky), my impression is that most database modules implement the rollback/commit approach. Cursors are, if implemented, always closed automatically when used as context managers. Again, error handling can get tricky because of this, but there is definity demand for using both connections and cursors as context managers. If there's agreement, we could add this kind of use as context managers to the DB-API 2.0 as standard extension. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From Chris.Clark at actian.com Thu Nov 1 16:57:54 2012 From: Chris.Clark at actian.com (Chris Clark) Date: Thu, 01 Nov 2012 08:57:54 -0700 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <5092988A.1010709@egenix.com> References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> Message-ID: <50929C02.5040703@actian.com> On Thursday 2012-11-01 08:57 (-0700), M.-A. Lemburg wrote: > Christoph Zwerschke wrote: >> We're currently preparing a new release of PyGreSQL and want to make use of context managers. My >> obvious idea was to let connections and cursors act as context managers that just close themselves, >> and add an extra context manager in form of a "transaction" property on the connection object that >> can be used to wrap transactions. >> >> However, I then noticed that PySqlite, cx_Oracle, mx_odbc and pyodbc use connection as context >> managers differently, they do not close the connection on exit, but execute a rollback or commit >> instead. >> >> Though I felt a separate context manager for wrapping transactions would have been a better >> solution, I now think I should better follow the above examples and wrap transactions in the context >> manager of the connection, it looks like it already has become a quasi standard. Any opinions on >> that or reasons why it has been implemented this way? > While I'm not much of a fan of using connections as context managers > (error handling gets tricky), my impression is that most database modules > implement the rollback/commit approach. > > Cursors are, if implemented, always closed automatically when used as > context managers. Again, error handling can get tricky because of this, > but there is definity demand for using both connections and cursors > as context managers. > > If there's agreement, we could add this kind of use as context managers > to the DB-API 2.0 as standard extension. If we do this, we should probably provide pure python reference context manager implementations in the spec. Chris From vernondcole at gmail.com Fri Nov 2 01:15:41 2012 From: vernondcole at gmail.com (Vernon Cole) Date: Thu, 1 Nov 2012 18:15:41 -0600 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <50929C02.5040703@actian.com> References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> <50929C02.5040703@actian.com> Message-ID: I've been scratching my head over this question, too. Someone (Christoph?) please craft for us a little sample program, as small as possible, showing how this ought to look in practice. It ought to be obvious (see PEP 20 aphorism 13) but I am in need of a Dutch person (see PEP 20 aphorism 14) to point it out to me. I will happily implement a sample of the right-looking solution into a fork of adodbapi (which is in pure Python and can talk to a LOT of different databases) to kick it around for testing. -- Vernon Cole -------------- next part -------------- An HTML attachment was scrubbed... URL: From cito at online.de Fri Nov 2 10:53:54 2012 From: cito at online.de (Christoph Zwerschke) Date: Fri, 02 Nov 2012 10:53:54 +0100 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> <50929C02.5040703@actian.com> Message-ID: <50939832.9070006@online.de> Am 02.11.2012 01:15, schrieb Vernon Cole: > I've been scratching my head over this question, too. Someone > (Christoph?) please craft for us a little sample program, as small as > possible, showing how this ought to look in practice. The implementation would be like this: class ConnectionWithContextManager(Connection): def __enter__(self): return self def __exit__(self, et, ev, tb): if et is None and ev is None and tb is None: self.commit() else: self.rollback() Cursor objects should also have an __enter__ method that returns self, but their __exit__ method should just execute self.close(). You can also call contextlib.closing on a normal cursor to get that. The usage would be like this: con = dbapi2.connect(...) with con: # 1st transaction with con.cursor() as cur: cur.execute("insert into debit(amount) values (-100)") cur.execute("insert into credit(amount) values (100)") with con: # 2nd transaction with con.cursor() as cur: cur.execute("insert into debit(amount) values (-200)") cur.execute("insert into credit(amount) values (200)") con.close() SQLite hasn't implemented cursors as context managers, but it has added execute() as a shortcut method on the connection, so here you can do: con = sqlite3.connect(...) with con: # 1st transaction con.execute("insert into debit(amount) values (-100)") con.execute("insert into credit(amount) values (100)") with con: # 2nd transaction con.execute("insert into debit(amount) values (-200)") con.execute("insert into credit(amount) values (200)") con.close() See also http://docs.python.org/2/library/sqlite3.html#using-the-connection-as-a-context-manager What I find a bit ugly about this approach is that now the context managers of connections and cursors behave differently, and that code written using these context managers is not self-explanatory. If connections and cursors would just have closing context managers, and connections had an extra "transaction" member that is a context manager for handling the transaction, the code would be more readable and explicit which is better than implicit: with dbapi2.connect(...) as con: with con.transaction: with con.cursor() as cur: cur.execute("insert into debit(amount) values (-100)") cur.execute("insert into credit(amount) values (100)") with con.transaction: with con.cursor() as cur: cur.execute("insert into debit(amount) values (-200)") cur.execute("insert into credit(amount) values (200)") Or, when using the shortcut methods: with dbapi2.connect(...) as con: with con.transaction: con.execute("insert into debit(amount) values (-100)") con.execute("insert into credit(amount) values (100)") with con.transaction: con.execute("insert into debit(amount) values (-200)") con.execute("insert into credit(amount) values (200)") -- Christoph From vernondcole at gmail.com Fri Nov 2 15:28:47 2012 From: vernondcole at gmail.com (Vernon Cole) Date: Fri, 2 Nov 2012 08:28:47 -0600 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <50939832.9070006@online.de> References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> <50929C02.5040703@actian.com> <50939832.9070006@online.de> Message-ID: Christoph, I think you may be Dutch -- your design proposal feels right to me. (+1) from Vernon On Fri, Nov 2, 2012 at 3:53 AM, Christoph Zwerschke wrote: > > What I find a bit ugly about this approach is that now the context > managers of connections and cursors behave differently, and that code > written using these context managers is not self-explanatory. If > connections and cursors would just have closing context managers, and > connections had an extra "transaction" member that is a context manager for > handling the transaction, the code would be more readable and explicit > which is better than implicit: > > with dbapi2.connect(...) as con: > with con.transaction: > with con.cursor() as cur: > cur.execute("insert into debit(amount) values (-100)") > cur.execute("insert into credit(amount) values (100)") > with con.transaction: > with con.cursor() as cur: > cur.execute("insert into debit(amount) values (-200)") > cur.execute("insert into credit(amount) values (200)") > > Or, when using the shortcut methods: > > with dbapi2.connect(...) as con: > with con.transaction: > con.execute("insert into debit(amount) values (-100)") > con.execute("insert into credit(amount) values (100)") > with con.transaction: > con.execute("insert into debit(amount) values (-200)") > con.execute("insert into credit(amount) values (200)") > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Sat Nov 3 20:56:52 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 03 Nov 2012 20:56:52 +0100 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <50939832.9070006@online.de> References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> <50929C02.5040703@actian.com> <50939832.9070006@online.de> Message-ID: <50957704.4040509@egenix.com> On 02.11.2012 10:53, Christoph Zwerschke wrote: > ... > What I find a bit ugly about this approach is that now the context managers of connections and > cursors behave differently, and that code written using these context managers is not > self-explanatory. If connections and cursors would just have closing context managers, and > connections had an extra "transaction" member that is a context manager for handling the > transaction, the code would be more readable and explicit which is better than implicit: > > with dbapi2.connect(...) as con: > with con.transaction: > with con.cursor() as cur: > cur.execute("insert into debit(amount) values (-100)") > cur.execute("insert into credit(amount) values (100)") > with con.transaction: > with con.cursor() as cur: > cur.execute("insert into debit(amount) values (-200)") > cur.execute("insert into credit(amount) values (200)") > The described use of connections as context managers is already implemented in several database modules and so I'm not sure whether it's worth adding an extra attribute to connections would really help. Also note that people would likely start to think that they'd somehow gain access to a separate transaction object by using the extra attribute, which is not the case, since the connection logically encapsulates the transaction. Another nit I have with connections being used as context managers is that the code reads as if a transaction is started at the beginning of the block and ended when leaving it, but in reality, this is not the case. That said, using connections as context managers in the described way is popular and probably already a standard practice, so I guess it's better to standardize on it and document it properly, rather than leaving it open for interpretation - even if just to settle on one interpretation. > Or, when using the shortcut methods: > > with dbapi2.connect(...) as con: > with con.transaction: > con.execute("insert into debit(amount) values (-100)") > con.execute("insert into credit(amount) values (100)") > with con.transaction: > con.execute("insert into debit(amount) values (-200)") > con.execute("insert into credit(amount) values (200)") The "shortcut" methods you describe were explicitly removed when moving from DB-API 1.0 to 2.0, so it's not a good idea to reintroduce them :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 03 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From cito at online.de Sat Nov 3 23:50:34 2012 From: cito at online.de (Christoph Zwerschke) Date: Sat, 03 Nov 2012 23:50:34 +0100 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <50957704.4040509@egenix.com> References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> <50929C02.5040703@actian.com> <50939832.9070006@online.de> <50957704.4040509@egenix.com> Message-ID: <50959FBA.6000506@online.de> Am 03.11.2012 20:56, schrieb M.-A. Lemburg: > That said, using connections as context managers in the described > way is popular and probably already a standard practice, so I > guess it's better to standardize on it and document it properly, > rather than leaving it open for interpretation - even if just to > settle on one interpretation. Yep. The fact that PySqlite is now part of the standard library has already created a de facto standard, and most other database modules seem to follow it. >> Or, when using the shortcut methods: >> >> with dbapi2.connect(...) as con: >> with con.transaction: >> con.execute("insert into debit(amount) values (-100)") >> con.execute("insert into credit(amount) values (100)") >> with con.transaction: >> con.execute("insert into debit(amount) values (-200)") >> con.execute("insert into credit(amount) values (200)") > > The "shortcut" methods you describe were explicitly removed when > moving from DB-API 1.0 to 2.0, so it's not a good idea to reintroduce > them :-) But PySqlite has already reintrocuced them. They are now also part of the standard library: http://docs.python.org/2/library/sqlite3.html#using-shortcut-methods Is there anything bad about these shortcut methods? -- Christoph From mal at egenix.com Sun Nov 4 00:07:20 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 04 Nov 2012 00:07:20 +0100 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <50959FBA.6000506@online.de> References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> <50929C02.5040703@actian.com> <50939832.9070006@online.de> <50957704.4040509@egenix.com> <50959FBA.6000506@online.de> Message-ID: <5095A3A8.6080702@egenix.com> On 03.11.2012 23:50, Christoph Zwerschke wrote: > Am 03.11.2012 20:56, schrieb M.-A. Lemburg: >> That said, using connections as context managers in the described >> way is popular and probably already a standard practice, so I >> guess it's better to standardize on it and document it properly, >> rather than leaving it open for interpretation - even if just to >> settle on one interpretation. > > Yep. The fact that PySqlite is now part of the standard library has already created a de facto > standard, and most other database modules seem to follow it. > >>> Or, when using the shortcut methods: >>> >>> with dbapi2.connect(...) as con: >>> with con.transaction: >>> con.execute("insert into debit(amount) values (-100)") >>> con.execute("insert into credit(amount) values (100)") >>> with con.transaction: >>> con.execute("insert into debit(amount) values (-200)") >>> con.execute("insert into credit(amount) values (200)") >> >> The "shortcut" methods you describe were explicitly removed when >> moving from DB-API 1.0 to 2.0, so it's not a good idea to reintroduce >> them :-) > > But PySqlite has already reintrocuced them. They are now also part of the standard library: > > http://docs.python.org/2/library/sqlite3.html#using-shortcut-methods > > Is there anything bad about these shortcut methods? I don't remember the details (the discussion is somewhere in the archives). From today's perspective, here are some reasons for the removal: * DB-API 1.0 had all cursors methods/attributes duplicated on the connections to allow writing database modules for backends which don't support cursors. Today, the cursor concept is widely accepted, so non-support is a non-issue. * Having those methods on connection objects blurs the distinction between connections and cursors, making them harder to explain to newcomers. * The short-cuts don't make the underlying cursors available for error handling. * Creating new cursors for each .execute() is inefficient, since setting up and tearing down cursors can be time consuming (e.g. if cursors are mirrored on the server side). * Optimizations such as reusing prepared statements is not (easily) possible using such short-cuts, unless the database module works with cursor pools. * The shortcut methods only work for non result-set generating SQL statements, since there's no way to fetch the result sets without access to the cursors. ... and probably a few more :-) Overall, it boils down to "explicit is better than implicit". -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 03 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Sun Nov 4 01:11:22 2012 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Sat, 3 Nov 2012 20:11:22 -0400 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <5095A3A8.6080702@egenix.com> References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> <50929C02.5040703@actian.com> <50939832.9070006@online.de> <50957704.4040509@egenix.com> <50959FBA.6000506@online.de> <5095A3A8.6080702@egenix.com> Message-ID: <10C0E596-D712-45CD-A7CC-FE36F5585C1E@zzzcomputing.com> On Nov 3, 2012, at 7:07 PM, M.-A. Lemburg wrote: > > Overall, it boils down to "explicit is better than implicit". not to mention "There should be one-- and preferably only one --obvious way to do it." From peter at eisentraut.org Fri Nov 2 19:11:36 2012 From: peter at eisentraut.org (Peter Eisentraut) Date: Fri, 02 Nov 2012 14:11:36 -0400 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <50939832.9070006@online.de> References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> <50929C02.5040703@actian.com> <50939832.9070006@online.de> Message-ID: <50940CD8.2050706@eisentraut.org> On 11/2/12 5:53 AM, Christoph Zwerschke wrote: > What I find a bit ugly about this approach is that now the context > managers of connections and cursors behave differently, and that code > written using these context managers is not self-explanatory. If > connections and cursors would just have closing context managers, and > connections had an extra "transaction" member that is a context manager > for handling the transaction, the code would be more readable and > explicit which is better than implicit: > > with dbapi2.connect(...) as con: > with con.transaction: > with con.cursor() as cur: > cur.execute("insert into debit(amount) values (-100)") > cur.execute("insert into credit(amount) values (100)") > with con.transaction: > with con.cursor() as cur: > cur.execute("insert into debit(amount) values (-200)") > cur.execute("insert into credit(amount) values (200)") I agree that that makes a lot more sense. From cito at online.de Sun Nov 4 12:31:43 2012 From: cito at online.de (Christoph Zwerschke) Date: Sun, 04 Nov 2012 12:31:43 +0100 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <5095A3A8.6080702@egenix.com> References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> <50929C02.5040703@actian.com> <50939832.9070006@online.de> <50957704.4040509@egenix.com> <50959FBA.6000506@online.de> <5095A3A8.6080702@egenix.com> Message-ID: <5096521F.6010204@online.de> Am 04.11.2012 00:07, schrieb M.-A. Lemburg: > * The short-cuts don't make the underlying cursors available for > error handling. > > * The shortcut methods only work for non result-set generating > SQL statements, since there's no way to fetch the result sets > without access to the cursors. But the cursor could be returned as the result value of con.execute() as it's implemented in SQLite. > * Creating new cursors for each .execute() is inefficient, since > setting up and tearing down cursors can be time consuming > (e.g. if cursors are mirrored on the server side). > > * Optimizations such as reusing prepared statements is not > (easily) possible using such short-cuts, unless the database > module works with cursor pools. That does not hold for SQLite, since it does not have real cursors anyway. But it's an argument for not following the example of SQLite in "real database" modules. -- Christoph From cito at online.de Sun Nov 4 12:37:32 2012 From: cito at online.de (Christoph Zwerschke) Date: Sun, 04 Nov 2012 12:37:32 +0100 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <10C0E596-D712-45CD-A7CC-FE36F5585C1E@zzzcomputing.com> References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> <50929C02.5040703@actian.com> <50939832.9070006@online.de> <50957704.4040509@egenix.com> <50959FBA.6000506@online.de> <5095A3A8.6080702@egenix.com> <10C0E596-D712-45CD-A7CC-FE36F5585C1E@zzzcomputing.com> Message-ID: <5096537C.90200@online.de> Am 04.11.2012 01:11, schrieb Michael Bayer: > not to mention "There should be one-- and preferably > only one --obvious way to do it." Which can easily be overtrumped by "simple is better than complex", "readability counts", "flat is better than nested", and "practicality beats purity" ;-) -- Christoph From mike_mp at zzzcomputing.com Sun Nov 4 19:32:34 2012 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Sun, 4 Nov 2012 13:32:34 -0500 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <5096537C.90200@online.de> References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> <50929C02.5040703@actian.com> <50939832.9070006@online.de> <50957704.4040509@egenix.com> <50959FBA.6000506@online.de> <5095A3A8.6080702@egenix.com> <10C0E596-D712-45CD-A7CC-FE36F5585C1E@zzzcomputing.com> <5096537C.90200@online.de> Message-ID: <84B3C290-E15F-47B4-9B94-3F1ED65401B6@zzzcomputing.com> On Nov 4, 2012, at 6:37 AM, Christoph Zwerschke wrote: > Am 04.11.2012 01:11, schrieb Michael Bayer: > > not to mention "There should be one-- and preferably > > only one --obvious way to do it." > > Which can easily be overtrumped by "simple is better than complex", "readability counts", "flat is better than nested", and "practicality beats purity" ;-) IMHO the DBAPI is a low level API that should do just one thing, allow rudimental functionality to a relational database with as much consistency as possible. When some DBAPIs decide to put an .execute() method on the Connection, others don't, and it then has hidden performance implications on some backends and not others, ambiguity and confusion is introduced into the API. The usage of cursor() for databases that don't actually have the concept of a cursor, like SQLite, is serving a greater purpose in its consistency versus the potential disservice by providing for an extra step. There is a space for database APIs that are designed to follow the underlying behavior of the database, rather than a blanket specification - such an example is the APSW api for SQLite (http://code.google.com/p/apsw/), which explicitly disregards the DBAPI in favor of offering a variety of behavioral and performance enhancements specific to SQLite, which would not otherwise fit into the DBAPI. The majority of database applications are actually using higher level abstraction systems on top of the DBAPI, such as the Django ORM, SQLAlchemy, Storm ORM, and others. It's not the place for the DBAPI itself to present a rich set of programming choices, as these can, and should, be easily implemented as wrappers or abstraction systems around a compliant driver. Basically I think the task of providing database programming patterns should be shipped separately from a DBAPI itself, so that DBAPI authors only need implement a minimal and clear set of behaviors that accomplish the task of basic database communication and nothing more. Real world applications should not be using the DBAPI directly without some abstraction system, either home grown or separately shipped, in between. From cito at online.de Sun Nov 4 23:24:53 2012 From: cito at online.de (Christoph Zwerschke) Date: Sun, 04 Nov 2012 23:24:53 +0100 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <84B3C290-E15F-47B4-9B94-3F1ED65401B6@zzzcomputing.com> References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> <50929C02.5040703@actian.com> <50939832.9070006@online.de> <50957704.4040509@egenix.com> <50959FBA.6000506@online.de> <5095A3A8.6080702@egenix.com> <10C0E596-D712-45CD-A7CC-FE36F5585C1E@zzzcomputing.com> <5096537C.90200@online.de> <84B3C290-E15F-47B4-9B94-3F1ED65401B6@zzzcomputing.com> Message-ID: <5096EB35.9000706@online.de> Am 04.11.2012 19:32, schrieb Michael Bayer: > IMHO the DBAPI is a low level API that should do just one thing, > allow rudimental functionality to a relational database with as much > consistency as possible. Essentially agreed, but I think it's excusable for SQLite as a light-weight database to make an exception here so that it can be used easily and directly from the standard lib without the need to install any additional higher level wrapper. To wrap everything up, as a result of our discussion, I will do the following in the next version of PyGreSQL: * make connections context managers for transactions (may not be the optimal solution, but it's already established) * make cursors self-closing context managers (evident) * not implement shortcut functions (see above) Is everybody happy with that? -- Christoph From mal at egenix.com Mon Nov 5 10:28:28 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 05 Nov 2012 10:28:28 +0100 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <5096EB35.9000706@online.de> References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> <50929C02.5040703@actian.com> <50939832.9070006@online.de> <50957704.4040509@egenix.com> <50959FBA.6000506@online.de> <5095A3A8.6080702@egenix.com> <10C0E596-D712-45CD-A7CC-FE36F5585C1E@zzzcomputing.com> <5096537C.90200@online.de> <84B3C290-E15F-47B4-9B94-3F1ED65401B6@zzzcomputing.com> <5096EB35.9000706@online.de> Message-ID: <509786BC.5060904@egenix.com> On 04.11.2012 23:24, Christoph Zwerschke wrote: > To wrap everything up, as a result of our discussion, I will do the following in the next version of > PyGreSQL: > > * make connections context managers for transactions > (may not be the optimal solution, but it's already established) > * make cursors self-closing context managers (evident) > * not implement shortcut functions (see above) > > Is everybody happy with that? Sounds good :-) Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 05 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From daniele.varrazzo at gmail.com Mon Nov 5 10:46:19 2012 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Mon, 5 Nov 2012 09:46:19 +0000 Subject: [DB-SIG] reST DBAPI [was: Use of context managers with DB API 2] Message-ID: On Mon, Nov 5, 2012 at 9:28 AM, M.-A. Lemburg wrote: > On 04.11.2012 23:24, Christoph Zwerschke wrote: >> Is everybody happy with that? > > Sounds good :-) While at changing the DBAPI: would it be ok to reformat it in reST? Not only for aesthetic reasons, but also to provide anchors to refer to: in psycopg docs there are several cross references to the api and they can only be to the whole file, not to the point being referenced. I can do it if you want. As soon as the API is updated I'll implement the context managers for the next Psycopg release. -- Daniele From mal at egenix.com Mon Nov 5 11:16:15 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 05 Nov 2012 11:16:15 +0100 Subject: [DB-SIG] reST DBAPI [was: Use of context managers with DB API 2] In-Reply-To: References: Message-ID: <509791EF.7070709@egenix.com> On 05.11.2012 10:46, Daniele Varrazzo wrote: > On Mon, Nov 5, 2012 at 9:28 AM, M.-A. Lemburg wrote: >> On 04.11.2012 23:24, Christoph Zwerschke wrote: > >>> Is everybody happy with that? >> >> Sounds good :-) > > While at changing the DBAPI: would it be ok to reformat it in reST? > Not only for aesthetic reasons, but also to provide anchors to refer > to: in psycopg docs there are several cross references to the api and > they can only be to the whole file, not to the point being referenced. > I can do it if you want. If you'd like to do the conversion, we can go ahead with this. I'd suggest you first do the conversion and then we add the new text snippet for context managers. Thanks for volunteering. > As soon as the API is updated I'll implement the context managers for > the next Psycopg release. Nice :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 05 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From daniele.varrazzo at gmail.com Mon Nov 5 18:29:05 2012 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Mon, 5 Nov 2012 17:29:05 +0000 Subject: [DB-SIG] reST DBAPI [was: Use of context managers with DB API 2] In-Reply-To: <509791EF.7070709@egenix.com> References: <509791EF.7070709@egenix.com> Message-ID: On Mon, Nov 5, 2012 at 10:16 AM, M.-A. Lemburg wrote: > On 05.11.2012 10:46, Daniele Varrazzo wrote: >> On Mon, Nov 5, 2012 at 9:28 AM, M.-A. Lemburg wrote: >>> On 04.11.2012 23:24, Christoph Zwerschke wrote: >> >>>> Is everybody happy with that? >>> >>> Sounds good :-) >> >> While at changing the DBAPI: would it be ok to reformat it in reST? >> Not only for aesthetic reasons, but also to provide anchors to refer >> to: in psycopg docs there are several cross references to the api and >> they can only be to the whole file, not to the point being referenced. >> I can do it if you want. > > If you'd like to do the conversion, we can go ahead with this. I'd > suggest you first do the conversion and then we add the new text > snippet for context managers. Here is a mockup of the pep rendered with stylesheets in place: . Please review it: if it's fine we can submit it for inclusion in the peps repos. Current version of the pep in reST is at . -- Daniele From fog at dndg.it Mon Nov 5 10:34:57 2012 From: fog at dndg.it (Federico Di Gregorio) Date: Mon, 05 Nov 2012 10:34:57 +0100 Subject: [DB-SIG] Use of context managers with DB API 2 In-Reply-To: <509786BC.5060904@egenix.com> References: <509280C4.10806@online.de> <5092988A.1010709@egenix.com> <50929C02.5040703@actian.com> <50939832.9070006@online.de> <50957704.4040509@egenix.com> <50959FBA.6000506@online.de> <5095A3A8.6080702@egenix.com> <10C0E596-D712-45CD-A7CC-FE36F5585C1E@zzzcomputing.com> <5096537C.90200@online.de> <84B3C290-E15F-47B4-9B94-3F1ED65401B6@zzzcomputing.com> <5096EB35.9000706@online.de> <509786BC.5060904@egenix.com> Message-ID: <50978841.7020903@dndg.it> On 05/11/2012 10:28, M.-A. Lemburg wrote: > On 04.11.2012 23:24, Christoph Zwerschke wrote: >> > To wrap everything up, as a result of our discussion, I will do the following in the next version of >> > PyGreSQL: >> > >> > * make connections context managers for transactions >> > (may not be the optimal solution, but it's already established) >> > * make cursors self-closing context managers (evident) >> > * not implement shortcut functions (see above) >> > >> > Is everybody happy with that? > Sounds good :-) Absolutely. :) federico -- Federico Di Gregorio federico.digregorio at dndg.it Studio Associato Di Nunzio e Di Gregorio http://dndg.it We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. -- D.E.Knuth From mal at egenix.com Tue Nov 6 15:49:17 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 06 Nov 2012 15:49:17 +0100 Subject: [DB-SIG] reST DBAPI [was: Use of context managers with DB API 2] In-Reply-To: References: <509791EF.7070709@egenix.com> Message-ID: <5099236D.9030302@egenix.com> On 05.11.2012 18:29, Daniele Varrazzo wrote: > On Mon, Nov 5, 2012 at 10:16 AM, M.-A. Lemburg wrote: >> On 05.11.2012 10:46, Daniele Varrazzo wrote: >>> On Mon, Nov 5, 2012 at 9:28 AM, M.-A. Lemburg wrote: >>>> On 04.11.2012 23:24, Christoph Zwerschke wrote: >>> >>>>> Is everybody happy with that? >>>> >>>> Sounds good :-) >>> >>> While at changing the DBAPI: would it be ok to reformat it in reST? >>> Not only for aesthetic reasons, but also to provide anchors to refer >>> to: in psycopg docs there are several cross references to the api and >>> they can only be to the whole file, not to the point being referenced. >>> I can do it if you want. >> >> If you'd like to do the conversion, we can go ahead with this. I'd >> suggest you first do the conversion and then we add the new text >> snippet for context managers. > > Here is a mockup of the pep rendered with stylesheets in place: > . Please review it: if it's fine we > can submit it for inclusion in the peps repos. Current version of the > pep in reST is at . Thanks, Daniele. I'll have a look later today. Which version of the file is more current ? The one on your site or the one on the ticket ? http://bugs.python.org/issue16420 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 06 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From daniele.varrazzo at gmail.com Tue Nov 6 16:38:32 2012 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Tue, 6 Nov 2012 15:38:32 +0000 Subject: [DB-SIG] reST DBAPI [was: Use of context managers with DB API 2] In-Reply-To: <5099236D.9030302@egenix.com> References: <509791EF.7070709@egenix.com> <5099236D.9030302@egenix.com> Message-ID: On Tue, Nov 6, 2012 at 2:49 PM, M.-A. Lemburg wrote: > On 05.11.2012 18:29, Daniele Varrazzo wrote: >> On Mon, Nov 5, 2012 at 10:16 AM, M.-A. Lemburg wrote: >>> On 05.11.2012 10:46, Daniele Varrazzo wrote: >>>> On Mon, Nov 5, 2012 at 9:28 AM, M.-A. Lemburg wrote: >>>>> On 04.11.2012 23:24, Christoph Zwerschke wrote: >>>> >>>>>> Is everybody happy with that? >>>>> >>>>> Sounds good :-) >>>> >>>> While at changing the DBAPI: would it be ok to reformat it in reST? >>>> Not only for aesthetic reasons, but also to provide anchors to refer >>>> to: in psycopg docs there are several cross references to the api and >>>> they can only be to the whole file, not to the point being referenced. >>>> I can do it if you want. >>> >>> If you'd like to do the conversion, we can go ahead with this. I'd >>> suggest you first do the conversion and then we add the new text >>> snippet for context managers. >> >> Here is a mockup of the pep rendered with stylesheets in place: >> . Please review it: if it's fine we >> can submit it for inclusion in the peps repos. Current version of the >> pep in reST is at . > > Thanks, Daniele. I'll have a look later today. Which version of > the file is more current ? The one on your site or the one on > the ticket ? > > http://bugs.python.org/issue16420 They should be equal, haven't changed anything after submitting it to the tracker. I'll keep the tracker informed if I change anything else. I've seen you have just dropped the note [9] (http://hg.python.org/peps/rev/888c87639210): I'll leave you to do the same on the reST version. -- Daniele From mal at egenix.com Tue Nov 6 16:42:04 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 06 Nov 2012 16:42:04 +0100 Subject: [DB-SIG] reST DBAPI [was: Use of context managers with DB API 2] In-Reply-To: References: <509791EF.7070709@egenix.com> <5099236D.9030302@egenix.com> Message-ID: <50992FCC.6050202@egenix.com> On 06.11.2012 16:38, Daniele Varrazzo wrote: > On Tue, Nov 6, 2012 at 2:49 PM, M.-A. Lemburg wrote: >> On 05.11.2012 18:29, Daniele Varrazzo wrote: >>> On Mon, Nov 5, 2012 at 10:16 AM, M.-A. Lemburg wrote: >>>> On 05.11.2012 10:46, Daniele Varrazzo wrote: >>>>> On Mon, Nov 5, 2012 at 9:28 AM, M.-A. Lemburg wrote: >>>>>> On 04.11.2012 23:24, Christoph Zwerschke wrote: >>>>> >>>>>>> Is everybody happy with that? >>>>>> >>>>>> Sounds good :-) >>>>> >>>>> While at changing the DBAPI: would it be ok to reformat it in reST? >>>>> Not only for aesthetic reasons, but also to provide anchors to refer >>>>> to: in psycopg docs there are several cross references to the api and >>>>> they can only be to the whole file, not to the point being referenced. >>>>> I can do it if you want. >>>> >>>> If you'd like to do the conversion, we can go ahead with this. I'd >>>> suggest you first do the conversion and then we add the new text >>>> snippet for context managers. >>> >>> Here is a mockup of the pep rendered with stylesheets in place: >>> . Please review it: if it's fine we >>> can submit it for inclusion in the peps repos. Current version of the >>> pep in reST is at . >> >> Thanks, Daniele. I'll have a look later today. Which version of >> the file is more current ? The one on your site or the one on >> the ticket ? >> >> http://bugs.python.org/issue16420 > > They should be equal, haven't changed anything after submitting it to > the tracker. I'll keep the tracker informed if I change anything else. > > I've seen you have just dropped the note [9] > (http://hg.python.org/peps/rev/888c87639210): I'll leave you to do the > same on the reST version. Yes, for some reason this was not committed last time around (I think I committed it to the website SVN rather than the HG repo for PEPs). I'll add it to the ReST version as well. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 06 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Wed Nov 7 09:53:57 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 07 Nov 2012 09:53:57 +0100 Subject: [DB-SIG] ANN: DB-API 2.0 PEP now in ReST format In-Reply-To: References: <509791EF.7070709@egenix.com> <5099236D.9030302@egenix.com> Message-ID: <509A21A5.3080105@egenix.com> Thanks to Daniele Varrazzo, we now have the DB-API 2.0 converted to ReST format, which allows linking to the various parts :-) http://www.python.org/dev/peps/pep-0249/ -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 07 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From peter_e at gmx.net Mon Nov 26 21:31:26 2012 From: peter_e at gmx.net (Peter Eisentraut) Date: Mon, 26 Nov 2012 15:31:26 -0500 Subject: [DB-SIG] mocking DB API Message-ID: <50B3D19E.9030106@gmx.net> Is there some kind of ready-to-use way to mock an entire DB-API driver, for use in a unit test suite? The popular "mock" module can only mock classes and methods, but not modules, as would be required here. As a target to aim for, I'd be interested to see after a test a list or count of the execute calls that were made. From mal at egenix.com Tue Nov 27 21:15:05 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 27 Nov 2012 21:15:05 +0100 Subject: [DB-SIG] mocking DB API In-Reply-To: <50B3D19E.9030106@gmx.net> References: <50B3D19E.9030106@gmx.net> Message-ID: <50B51F49.1020508@egenix.com> On 26.11.2012 21:31, Peter Eisentraut wrote: > Is there some kind of ready-to-use way to mock an entire DB-API driver, > for use in a unit test suite? The popular "mock" module can only mock > classes and methods, but not modules, as would be required here. > > As a target to aim for, I'd be interested to see after a test a list or > count of the execute calls that were made. I'm not aware of any existing mock module for DB-API drivers, but given that it's easy to write a module that defines mock Cursor and Connection classes, it should be relatively simple to just hack on up with a little __get/setattr__ magic. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 27 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/