From mike_mp at zzzcomputing.com Tue Nov 1 09:03:20 2022 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Tue, 01 Nov 2022 09:03:20 -0400 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> Message-ID: <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> On Sun, Oct 30, 2022, at 5:38 PM, Erlend Egeberg Aasland wrote: > > > On 28 Oct 2022, at 18:14, Marc-Andre Lemburg wrote: > > > > On 28.10.2022 16:51, Mike Bayer wrote: > >> On Fri, Oct 28, 2022, at 10:20 AM, Marc-Andre Lemburg wrote: > >>> On 28.10.2022 15:58, Marc-Andre Lemburg wrote: > >>> > If there is a pending transaction, though, there are three approaches > >>> > we could take: > >>> > > >>> > 1. The database module raises an exception, to force an explicit > >>> > .commit() or .rollback() by the programmer. > >>> > > >>> > 2. The module automatically commits the pending transaction, > >>> > since that's what autocommit is all about. > >>> > >>> Just checked: Option 2 is what ODBC mandates... > >>> https://learn.microsoft.com/en-us/sql/odbc/reference/syntax/sqlsetconnectattr-function > >>> (search for SQL_ATTR_AUTOCOMMIT) > >> That's a C API, which has different programming conventions than what Python has, and it's also referring towards a function that's being called, so while that's a datapoint to note, I dont think by itself it really indicates how a Python API should organize itself one way or the other. > > > > I just wanted to note that the semantics of what to do when switching > > from False to True are already defined in one of the major database > > APIs standards, so it's good practice to follow such a standard. > > +1 I checked JDBC and it also follows this convention: https://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#setAutoCommit(boolean) *however*, in both cases of ODBC and JDBC, these are function calls. If DBAPI had a connection method define called `set_autocommit()`, the implicit COMMIT would be fine. In Python, running an attribute set operation should not incur IO at the point at which the attribute is being assigned. So if we are following existing standards from different programming languages, the .autocommit attribute should be changed to be a method `set_autocommit()`. > > My preference would be 2. or 3. > > > Overall, I believe that important settings such as autocommit > > should only be set in the connection constructor, since the > > semantics change dramatically between autocommit and manual > > commit. > > > > In such a world, we'd only have an autocommit keyword argument > > in the Connection constructor and a read-only attribute on the > > object to query this after creation. > > I would be ok with that. this is the only variant of the proposal that would be a breaking change for SQLAlchemy, since it removes existing functionality that most DBAPIs have now and change a use case that is now possible to be one that is impossible. The reality would be that some would comply, and others (most others) would not, because it is extremely difficult (and mostly unnecessary) to remove functionality from an API. It's also inconsistent with the previous notion of following other existing database standards as both ODBC and JDBC feature means of affecting the autocommit behavior on an already opened connection. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Nov 1 09:37:29 2022 From: mal at egenix.com (Marc-Andre Lemburg) Date: Tue, 1 Nov 2022 14:37:29 +0100 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> Message-ID: <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> I've thought some more about this over the last couple of days. Since we're adding a standard extension to the DB-API 2.0 based on existing use in the wild, we should really stick to what database module authors have implemented and not propose new semantics or add anything which is not yet widely accepted to the 2.0 version. Instead, I believe we should document what many modules already implement, which is the read/write .autocommit attribute. As for semantics, we should probably go with option 3 and only mention that many modules will actually already implement option 2. Regarding the problem with having attribute write access result in possible I/O, I think we should at the same time deprecate the write nature of the .autocommit attribute and announce that it'll be replaced with a new method, e.g. .setautocommit() (the DB-API has traditionally not added underscores to the names, except for the optional TPC API group), for DB-API 3.0. Related to this, I'd also suggest adding a new keyword parameter to the connection constructor (autocommit), which defaults to False and can be used to create autocommit connections right from the start. For the next version of the DB-API we should then also consider async methods and functions and try to make sure that all important API parts can be used in an async manner, in particular, making sure that API parts which can result in I/O are always defined in form of methods or functions (I believe that most are already, except for the .autocommit attribute which many modules implement). More generally speaking, I think Python is missing async support for properties. Perhaps this will be added at some point, so that we can write e.g. await connection.autocommit = True OTOH, I'm not really a fan of complex properties and even less so, ones which can result in I/O. I'll put together a new proposal for the new standard extension, addressing the above. On 01.11.2022 14:03, Mike Bayer wrote: > > > On Sun, Oct 30, 2022, at 5:38 PM, Erlend Egeberg Aasland wrote: >> >> > On 28 Oct 2022, at 18:14, Marc-Andre Lemburg wrote: >> > >> > On 28.10.2022 16:51, Mike Bayer wrote: >> >> On Fri, Oct 28, 2022, at 10:20 AM, Marc-Andre Lemburg wrote: >> >>> On 28.10.2022 15:58, Marc-Andre Lemburg wrote: >> >>> > If there is a pending transaction, though, there are three >> approaches >> >>> > we could take: >> >>> > >> >>> > 1. The database module raises an exception, to force an explicit >> >>> >???? .commit() or .rollback() by the programmer. >> >>> > >> >>> > 2. The module automatically commits the pending transaction, >> >>> >???? since that's what autocommit is all about. >> >>> >> >>> Just checked: Option 2 is what ODBC mandates... >> >>> >> https://learn.microsoft.com/en-us/sql/odbc/reference/syntax/sqlsetconnectattr-function >> >> >>> (search for SQL_ATTR_AUTOCOMMIT) >> >> That's a C API, which has different programming conventions than >> what Python has, and it's also referring towards a function that's >> being called, so while that's a datapoint to note, I dont think by >> itself it really indicates how a Python API should organize itself >> one way or the other. >> > >> > I just wanted to note that the semantics of what to do when switching >> > from False to True are already defined in one of the major database >> > APIs standards, so it's good practice to follow such a standard. >> >> +1 > > I checked JDBC and it also follows this convention: > https://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#setAutoCommit(boolean) > > > *however*, in both cases of ODBC and JDBC, these are function calls.?? > If DBAPI had a connection method define called `set_autocommit()`, the > implicit COMMIT would be fine.?? In Python, running an attribute set > operation should not incur IO at the point at which the attribute is > being assigned.? So if we are following existing standards from > different programming languages, the .autocommit attribute should be > changed to be a method `set_autocommit()`. > > >> >> My preference would be 2. or 3. >> >> > Overall, I believe that important settings such as autocommit >> > should only be set in the connection constructor, since the >> > semantics change dramatically between autocommit and manual >> > commit. >> > >> > In such a world, we'd only have an autocommit keyword argument >> > in the Connection constructor and a read-only attribute on the >> > object to query this after creation. >> >> I would be ok with that. > > this is the only variant of the proposal that would be a breaking > change for SQLAlchemy, since it removes existing functionality that > most DBAPIs have now and change a use case that is now possible to be > one that is impossible.?? The reality would be that some would comply, > and others (most others) would not, because it is extremely difficult > (and mostly unnecessary) to remove functionality from an API.??? It's > also inconsistent with the previous notion of following other existing > database standards as both ODBC and JDBC feature means of affecting > the autocommit behavior on an already opened connection. > > > _______________________________________________ > DB-SIG maillist -DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 01 2022) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike_mp at zzzcomputing.com Tue Nov 1 09:44:48 2022 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Tue, 01 Nov 2022 09:44:48 -0400 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> Message-ID: On Tue, Nov 1, 2022, at 9:37 AM, Marc-Andre Lemburg wrote: > I've thought some more about this over the last couple of days. > > Since we're adding a standard extension to the DB-API 2.0 based on existing use in the wild, we should really stick to what database module authors have implemented and not propose new semantics or add anything which is not yet widely accepted to the 2.0 version. > > Instead, I believe we should document what many modules already implement, which is the read/write .autocommit attribute. As for semantics, we should probably go with option 3 and only mention that many modules will actually already implement option 2. > > Regarding the problem with having attribute write access result in possible I/O, I think we should at the same time deprecate the write nature of the .autocommit attribute and announce that it'll be replaced with a new method, e.g. .setautocommit() (the DB-API has traditionally not added underscores to the names, except for the optional TPC API group), for DB-API 3.0. > > Related to this, I'd also suggest adding a new keyword parameter to the connection constructor (autocommit), which defaults to False and can be used to create autocommit connections right from the start. > > For the next version of the DB-API we should then also consider async methods and functions and try to make sure that all important API parts can be used in an async manner, in particular, making sure that API parts which can result in I/O are always defined in form of methods or functions (I believe that most are already, except for the .autocommit attribute which many modules implement). > > More generally speaking, I think Python is missing async support for properties. Perhaps this will be added at some point, so that we can write e.g. > > await connection.autocommit = True > > OTOH, I'm not really a fan of complex properties and even less so, ones which can result in I/O. > > I'll put together a new proposal for the new standard extension, addressing the above. > I am good with all of this. I would be pretty surprised if the asyncio folks wanted to add await to properties like that as you mentioned people aren't generally in favor of attribute sets having large side effects. > > > On 01.11.2022 14:03, Mike Bayer wrote: >> >> >> On Sun, Oct 30, 2022, at 5:38 PM, Erlend Egeberg Aasland wrote: >>> >>> > On 28 Oct 2022, at 18:14, Marc-Andre Lemburg wrote: >>> > >>> > On 28.10.2022 16:51, Mike Bayer wrote: >>> >> On Fri, Oct 28, 2022, at 10:20 AM, Marc-Andre Lemburg wrote: >>> >>> On 28.10.2022 15:58, Marc-Andre Lemburg wrote: >>> >>> > If there is a pending transaction, though, there are three approaches >>> >>> > we could take: >>> >>> > >>> >>> > 1. The database module raises an exception, to force an explicit >>> >>> > .commit() or .rollback() by the programmer. >>> >>> > >>> >>> > 2. The module automatically commits the pending transaction, >>> >>> > since that's what autocommit is all about. >>> >>> >>> >>> Just checked: Option 2 is what ODBC mandates... >>> >>> https://learn.microsoft.com/en-us/sql/odbc/reference/syntax/sqlsetconnectattr-function >>> >>> (search for SQL_ATTR_AUTOCOMMIT) >>> >> That's a C API, which has different programming conventions than what Python has, and it's also referring towards a function that's being called, so while that's a datapoint to note, I dont think by itself it really indicates how a Python API should organize itself one way or the other. >>> > >>> > I just wanted to note that the semantics of what to do when switching >>> > from False to True are already defined in one of the major database >>> > APIs standards, so it's good practice to follow such a standard. >>> >>> +1 >> >> I checked JDBC and it also follows this convention: https://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#setAutoCommit(boolean) >> >> *however*, in both cases of ODBC and JDBC, these are function calls. If DBAPI had a connection method define called `set_autocommit()`, the implicit COMMIT would be fine. In Python, running an attribute set operation should not incur IO at the point at which the attribute is being assigned. So if we are following existing standards from different programming languages, the .autocommit attribute should be changed to be a method `set_autocommit()`. >> >> >>> >>> My preference would be 2. or 3. >>> >>> > Overall, I believe that important settings such as autocommit >>> > should only be set in the connection constructor, since the >>> > semantics change dramatically between autocommit and manual >>> > commit. >>> > >>> > In such a world, we'd only have an autocommit keyword argument >>> > in the Connection constructor and a read-only attribute on the >>> > object to query this after creation. >>> >>> I would be ok with that. >> >> this is the only variant of the proposal that would be a breaking change for SQLAlchemy, since it removes existing functionality that most DBAPIs have now and change a use case that is now possible to be one that is impossible. The reality would be that some would comply, and others (most others) would not, because it is extremely difficult (and mostly unnecessary) to remove functionality from an API. It's also inconsistent with the previous notion of following other existing database standards as both ODBC and JDBC feature means of affecting the autocommit behavior on an already opened connection. >> >> >> _______________________________________________ >> DB-SIG maillist - DB-SIG at python.org >> https://mail.python.org/mailman/listinfo/db-sig >> > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Nov 01 2022) > >>> Python Projects, Coaching and Support ... https://www.egenix.com/ > >>> Python Product Development ... https://consulting.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > https://www.egenix.com/company/contact/ > https://www.malemburg.com/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlenda at gmail.com Tue Nov 1 09:46:18 2022 From: erlenda at gmail.com (Erlend Egeberg Aasland) Date: Tue, 1 Nov 2022 14:46:18 +0100 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> Message-ID: On Tue, 1 Nov 2022 at 14:37, Marc-Andre Lemburg wrote: > [?] Instead, I believe we should document what many modules already > implement, which is the read/write .autocommit attribute. As for semantics, > we should probably go with option 3 and only mention that many modules will > actually already implement option 2. > +1 Regarding the problem with having attribute write access result in possible > I/O, I think we should at the same time deprecate the write nature of the > .autocommit attribute and announce that it'll be replaced with a newmethod, > e.g. .setautocommit() (the DB-API has traditionally not added underscores > to the names, except for the optional TPC API group), for DB-API 3.0. > For the upcoming autocommit attribute of the stdlib sqlite3 module, I wonder if we should stick with these conventions right from the start. That is, autocommit can either be set from the connection constructor (using the new autocommit property), or using a setautocommit() method. The autocommit attribute will be read-only. Related to this, I'd also suggest adding a new keyword parameter to the > connection constructor (autocommit), which defaults to False and can be > used to create autocommit connections right from the start. > +1 Erlend -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Thu Nov 3 15:51:57 2022 From: mal at egenix.com (Marc-Andre Lemburg) Date: Thu, 3 Nov 2022 20:51:57 +0100 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> Message-ID: <5138e418-8b64-f4a9-577b-6749ed68e82f@egenix.com> On 03.11.2022 19:25, Tony Locke wrote: > Hello all, with pg8000 if autocommit is turned on while a transaction > is in progress, the autocommit behaviour only takes effect after the > current transaction has been explicitly committed or rolled back. Of > the options given, I'd go for option 1, raising an exception. I don't > like the idea of SQL statements being executed implicitly, which is > why I'd be against option 2. That's a fair point, but please remember that no statements are executed. The transaction control is completely with the driver library and negotiated with the server (some use implicit statements for this, but that's just a technical detail). In fact, users should /not/ use transaction statements on connections managed by drivers. That's why we have .rollback() and .commit() as separate APIs in the DB-API. I think we've already settled on option 3, with the added note that many database modules implement option 2. pg8000 would not be one of those, but that's not a problem. Writing to the attribute will be deprecated at the same time as documenting it, so that we can prepare for DB-API 3.0 using a method instead. This would then also address your concern. > > A radical suggestion for DBAPI-3.0 would be to remove autocommit > completely. I think this would make the dbapi interface easier for > people to understand, because there would be no implicit SQL commands, > which is what I think confuses people. Of course the downside is less > portability, but I think people accept that if they change their > database a lot of things will be different, and autocommit is just one > of them. A higher level library such as SQLAlchemy could still emit > the correct SQL via the dialect so that users of SQLAlchemy wouldn't > need to know about autocommit. That's not going to work out :-) Autocommit is essential for some database operations, e.g. to avoid locking. Many databases also don't permit running certain DDL statements inside transactions. BTW: Django's default is to run in autocommit mode... not a particularly good choice, but that's how it is: https://docs.djangoproject.com/en/4.1/topics/db/transactions/#autocommit Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 03 2022) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike_mp at zzzcomputing.com Thu Nov 3 16:07:34 2022 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Thu, 03 Nov 2022 16:07:34 -0400 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <5138e418-8b64-f4a9-577b-6749ed68e82f@egenix.com> References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> <5138e418-8b64-f4a9-577b-6749ed68e82f@egenix.com> Message-ID: On Thu, Nov 3, 2022, at 3:51 PM, Marc-Andre Lemburg wrote: > On 03.11.2022 19:25, Tony Locke wrote: >> Hello all, with pg8000 if autocommit is turned on while a transaction is in progress, the autocommit behaviour only takes effect after the current transaction has been explicitly committed or rolled back. Of the options given, I'd go for option 1, raising an exception. I don't like the idea of SQL statements being executed implicitly, which is why I'd be against option 2. > That's a fair point, but please remember that no statements are executed. The transaction control is completely with the driver library and negotiated with the server (some use implicit statements for this, but that's just a technical detail). In fact, users should *not* use transaction statements on connections managed by drivers. That's why we have .rollback() and .commit() as separate APIs in the DB-API. > > I think we've already settled on option 3, with the added note that many database modules implement option 2. pg8000 would not be one of those, but that's not a problem. > > Writing to the attribute will be deprecated at the same time as documenting it, so that we can prepare for DB-API 3.0 using a method instead. This would then also address your concern. > >> >> A radical suggestion for DBAPI-3.0 would be to remove autocommit completely. I think this would make the dbapi interface easier for people to understand, because there would be no implicit SQL commands, which is what I think confuses people. Of course the downside is less portability, but I think people accept that if they change their database a lot of things will be different, and autocommit is just one of them. A higher level library such as SQLAlchemy could still emit the correct SQL via the dialect so that users of SQLAlchemy wouldn't need to know about autocommit. > That's not going to work out :-) Autocommit is essential for some database operations, e.g. to avoid locking. Many databases also don't permit running certain DDL statements inside transactions. > I dont think I got Tony's email directly, but re: autocommit, the fact that basically all DBAPIs now implement autocommit is very advantageous to SQLAlchemy and in version 2.0 we have finally removed all semblances of SQLAlchemy's original homegrown "autocommit" feature, which we now refer towards as "library level autocommit" in contrast to "driver level autocommit". What's important about "autocommit" is that in most cases, it doesnt equate to the driver emitting "COMMIT" after every SQL statement, it instead allows the driver to forego emitting BEGIN in the first place, and the database itself runs in its own "autocommit" mode, which is typically higher performing. With DBAPI including the behavior, this first off allows us to simplify our library, as users who want to work in "autocommit" fashion can do so by just setting up the driver to work in this way, without us having to provide / test entirely different ways of working. But that's only the beginning of the advantages. The bigger advantage is that support for many other styles of database connectivity is supported, including for some PGBouncer configurations which don't tolerate transaction boundaries well, connecting to read-only replica databases where the overhead of DBAPI's implicit "BEGIN" followed by our necessary ".rollback()" for a pooled connection can be removed (this is a big MySQL / MariaDB use case), and of course we can support the various DDL scenarios (mostly on PostgreSQL) that require autocommit. When SQLAlchemy used "library level autocommit", we had no way to affect that the DBAPI was still emitting "BEGIN" and that our pooled connections still had to use ROLLBACK. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Clark at actian.com Thu Nov 3 18:20:15 2022 From: Chris.Clark at actian.com (Chris Clark) Date: Thu, 3 Nov 2022 22:20:15 +0000 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> Message-ID: Optional extension to pep-249, DB-API 2.0 works for me, where #3 is: 3. We leave these semantics open and up to the database module to determine. For DB-API 3.0 I would prefer we go with option #1 when the setautocommit() method is called. I wouldn?t be completely devastated if I was option #2 but I believe #1 is a better option. Chris From: DB-SIG On Behalf Of Marc-Andre Lemburg Sent: Tuesday, November 1, 2022 6:37 AM To: Mike Bayer ; Vishal Gupta via DB-SIG Subject: Re: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) I've thought some more about this over the last couple of days. Since we're adding a standard extension to the DB-API 2.0 based on existing use in the wild, we should really stick to what database module authors have implemented and not propose new semantics or add anything which is not yet widely accepted to the 2.0 version. Instead, I believe we should document what many modules already implement, which is the read/write .autocommit attribute. As for semantics, we should probably go with option 3 and only mention that many modules will actually already implement option 2. Regarding the problem with having attribute write access result in possible I/O, I think we should at the same time deprecate the write nature of the .autocommit attribute and announce that it'll be replaced with a new method, e.g. .setautocommit() (the DB-API has traditionally not added underscores to the names, except for the optional TPC API group), for DB-API 3.0. Related to this, I'd also suggest adding a new keyword parameter to the connection constructor (autocommit), which defaults to False and can be used to create autocommit connections right from the start. For the next version of the DB-API we should then also consider async methods and functions and try to make sure that all important API parts can be used in an async manner, in particular, making sure that API parts which can result in I/O are always defined in form of methods or functions (I believe that most are already, except for the .autocommit attribute which many modules implement). More generally speaking, I think Python is missing async support for properties. Perhaps this will be added at some point, so that we can write e.g. await connection.autocommit = True OTOH, I'm not really a fan of complex properties and even less so, ones which can result in I/O. I'll put together a new proposal for the new standard extension, addressing the above. On 01.11.2022 14:03, Mike Bayer wrote: On Sun, Oct 30, 2022, at 5:38 PM, Erlend Egeberg Aasland wrote: > On 28 Oct 2022, at 18:14, Marc-Andre Lemburg > wrote: > > On 28.10.2022 16:51, Mike Bayer wrote: >> On Fri, Oct 28, 2022, at 10:20 AM, Marc-Andre Lemburg wrote: >>> On 28.10.2022 15:58, Marc-Andre Lemburg wrote: >>> > If there is a pending transaction, though, there are three approaches >>> > we could take: >>> > >>> > 1. The database module raises an exception, to force an explicit >>> > .commit() or .rollback() by the programmer. >>> > >>> > 2. The module automatically commits the pending transaction, >>> > since that's what autocommit is all about. >>> >>> Just checked: Option 2 is what ODBC mandates... >>> https://learn.microsoft.com/en-us/sql/odbc/reference/syntax/sqlsetconnectattr-function >>> (search for SQL_ATTR_AUTOCOMMIT) >> That's a C API, which has different programming conventions than what Python has, and it's also referring towards a function that's being called, so while that's a datapoint to note, I dont think by itself it really indicates how a Python API should organize itself one way or the other. > > I just wanted to note that the semantics of what to do when switching > from False to True are already defined in one of the major database > APIs standards, so it's good practice to follow such a standard. +1 I checked JDBC and it also follows this convention: https://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#setAutoCommit(boolean) *however*, in both cases of ODBC and JDBC, these are function calls. If DBAPI had a connection method define called `set_autocommit()`, the implicit COMMIT would be fine. In Python, running an attribute set operation should not incur IO at the point at which the attribute is being assigned. So if we are following existing standards from different programming languages, the .autocommit attribute should be changed to be a method `set_autocommit()`. My preference would be 2. or 3. > Overall, I believe that important settings such as autocommit > should only be set in the connection constructor, since the > semantics change dramatically between autocommit and manual > commit. > > In such a world, we'd only have an autocommit keyword argument > in the Connection constructor and a read-only attribute on the > object to query this after creation. I would be ok with that. this is the only variant of the proposal that would be a breaking change for SQLAlchemy, since it removes existing functionality that most DBAPIs have now and change a use case that is now possible to be one that is impossible. The reality would be that some would comply, and others (most others) would not, because it is extremely difficult (and mostly unnecessary) to remove functionality from an API. It's also inconsistent with the previous notion of following other existing database standards as both ODBC and JDBC feature means of affecting the autocommit behavior on an already opened connection. _______________________________________________ DB-SIG maillist - DB-SIG at python.org https://mail.python.org/mailman/listinfo/db-sig -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 01 2022) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tlocke at tlocke.org.uk Thu Nov 3 14:25:15 2022 From: tlocke at tlocke.org.uk (Tony Locke) Date: Thu, 3 Nov 2022 18:25:15 +0000 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> Message-ID: Hello all, with pg8000 if autocommit is turned on while a transaction is in progress, the autocommit behaviour only takes effect after the current transaction has been explicitly committed or rolled back. Of the options given, I'd go for option 1, raising an exception. I don't like the idea of SQL statements being executed implicitly, which is why I'd be against option 2. A radical suggestion for DBAPI-3.0 would be to remove autocommit completely. I think this would make the dbapi interface easier for people to understand, because there would be no implicit SQL commands, which is what I think confuses people. Of course the downside is less portability, but I think people accept that if they change their database a lot of things will be different, and autocommit is just one of them. A higher level library such as SQLAlchemy could still emit the correct SQL via the dialect so that users of SQLAlchemy wouldn't need to know about autocommit. Regards, Tony. On Wed, 2 Nov 2022 at 06:08, Erlend Egeberg Aasland wrote: > > On Tue, 1 Nov 2022 at 14:37, Marc-Andre Lemburg wrote: > >> [?] Instead, I believe we should document what many modules already >> implement, which is the read/write .autocommit attribute. As for semantics, >> we should probably go with option 3 and only mention that many modules will >> actually already implement option 2. >> > +1 > > Regarding the problem with having attribute write access result in >> possible I/O, I think we should at the same time deprecate the write nature >> of the .autocommit attribute and announce that it'll be replaced with a >> newmethod, e.g. .setautocommit() (the DB-API has traditionally not added >> underscores to the names, except for the optional TPC API group), for >> DB-API 3.0. >> > For the upcoming autocommit attribute of the stdlib sqlite3 module, I > wonder if we should stick with these conventions right from the start. That > is, autocommit can either be set from the connection constructor (using the > new autocommit property), or using a setautocommit() method. The autocommit > attribute will be read-only. > > Related to this, I'd also suggest adding a new keyword parameter to the >> connection constructor (autocommit), which defaults to False and can be >> used to create autocommit connections right from the start. >> > +1 > > > Erlend > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dieter at handshake.de Sat Nov 5 12:44:11 2022 From: dieter at handshake.de (Dieter Maurer) Date: Sat, 5 Nov 2022 17:44:11 +0100 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> Message-ID: <25446.37595.665896.640710@ixdm.fritz.box> Tony Locke wrote at 2022-11-3 18:25 +0000: > ... >A radical suggestion for DBAPI-3.0 would be to remove autocommit >completely. -1 "autocommit" is useful for some applications and should be supported. From erlenda at gmail.com Mon Nov 7 09:42:44 2022 From: erlenda at gmail.com (Erlend Egeberg Aasland) Date: Mon, 7 Nov 2022 15:42:44 +0100 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <5138e418-8b64-f4a9-577b-6749ed68e82f@egenix.com> References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> <5138e418-8b64-f4a9-577b-6749ed68e82f@egenix.com> Message-ID: <9CD9C429-24A8-43C1-806A-A2739C68B10E@gmail.com> > On 3 Nov 2022, at 20:51, Marc-Andre Lemburg wrote: > > I think we've already settled on option 3, with the added note that many database modules implement option 2. [...] Fine with me. > Writing to the attribute will be deprecated at the same time as documenting it, so that we can prepare for DB-API 3.0 using a method instead. [...] For the upcoming sqlite3 autocommit attribute, I'll keep the PR as it is (that is: implement option 2). I'll deprecate writing to the autocommit attribute after the initial feature PR is merged. Erlend From mal at egenix.com Mon Nov 14 08:23:48 2022 From: mal at egenix.com (Marc-Andre Lemburg) Date: Mon, 14 Nov 2022 14:23:48 +0100 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <9CD9C429-24A8-43C1-806A-A2739C68B10E@gmail.com> References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> <5138e418-8b64-f4a9-577b-6749ed68e82f@egenix.com> <9CD9C429-24A8-43C1-806A-A2739C68B10E@gmail.com> Message-ID: Here's the updated wording for the addition, which should cover everything we have discussed: """ Connection.autocommit Attribute to query and set the autocommit mode of the connection. Returns True if the connection is operating in autocommit (non- transactional) mode. Returns False is the connection is operating in manual commit (transactional) mode. Setting the attribute to True or False adjusts the connection's mode accordingly. Changing the setting from True to False (disabling autocommit) will have the database leave autocommit mode and starts a new transaction. Changing from False to True (enabling autocommit) has database dependent semantics with respect to how pending transactions are handled [1]. Deprecation notice: Even though several database modules implement both the read and write nature of this attribute, setting the autocommit mode by writing to the attribute is deprecated, since this may result in I/O and related exceptions, making it difficult to implement in an async context [2]. [1] Many database modules implementing the autocommit attribute will automatically commit any pending transaction and then enter autocommit mode. [2] For DB-API 3.0, we are going to introduce a new method .setautocommit(value), which will allow setting the autocommit mode, and make .autocommit a read-only attribute. Additionally, we are considering to add a new standard keyword parameter autocommit to the Connection constructor. Modules authors are encouraged to add these changes in preparation for this change. """ Please let me know whether I forgot anything. I'll prepare the PR for PEP 249 later this week. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 14 2022) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ From mike_mp at zzzcomputing.com Mon Nov 14 09:14:19 2022 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Mon, 14 Nov 2022 09:14:19 -0500 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> <5138e418-8b64-f4a9-577b-6749ed68e82f@egenix.com> <9CD9C429-24A8-43C1-806A-A2739C68B10E@gmail.com> Message-ID: LGTM thanks! On Mon, Nov 14, 2022, at 8:23 AM, Marc-Andre Lemburg wrote: > Here's the updated wording for the addition, which should cover > everything we have discussed: > > """ > Connection.autocommit > > Attribute to query and set the autocommit mode of the connection. > > Returns True if the connection is operating in autocommit (non- > transactional) mode. Returns False is the connection is operating in > manual commit (transactional) mode. > > Setting the attribute to True or False adjusts the connection's mode > accordingly. > > Changing the setting from True to False (disabling autocommit) will > have the database leave autocommit mode and starts a new transaction. > Changing from False to True (enabling autocommit) has database > dependent semantics with respect to how pending transactions are > handled [1]. > > Deprecation notice: Even though several database modules implement > both the read and write nature of this attribute, setting the > autocommit mode by writing to the attribute is deprecated, since this > may result in I/O and related exceptions, making it difficult to > implement in an async context [2]. > > [1] Many database modules implementing the autocommit attribute will > automatically commit any pending transaction and then enter autocommit > mode. > > [2] For DB-API 3.0, we are going to introduce a new method > .setautocommit(value), which will allow setting the autocommit mode, and > make .autocommit a read-only attribute. Additionally, we are considering > to add a new standard keyword parameter autocommit to the Connection > constructor. Modules authors are encouraged to add these changes in > preparation for this change. > """ > > Please let me know whether I forgot anything. I'll prepare the PR > for PEP 249 later this week. > > Thanks, > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Nov 14 2022) > >>> Python Projects, Coaching and Support ... https://www.egenix.com/ > >>> Python Product Development ... https://consulting.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > https://www.egenix.com/company/contact/ > https://www.malemburg.com/ > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniele.varrazzo at gmail.com Mon Nov 14 10:23:46 2022 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Mon, 14 Nov 2022 15:23:46 +0000 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> <5138e418-8b64-f4a9-577b-6749ed68e82f@egenix.com> <9CD9C429-24A8-43C1-806A-A2739C68B10E@gmail.com> Message-ID: On Mon, 14 Nov 2022 at 13:23, Marc-Andre Lemburg wrote: > Please let me know whether I forgot anything. I'll prepare the PR > for PEP 249 later this week. Looks ok to me :) Cheers -- Daniele From anthony.tuininga at gmail.com Mon Nov 14 13:17:05 2022 From: anthony.tuininga at gmail.com (Anthony Tuininga) Date: Mon, 14 Nov 2022 11:17:05 -0700 Subject: [DB-SIG] python-oracledb 1.2.0 Message-ID: What is python-oracledb? python-oracledb is a Python extension module that enables access to Oracle Database for Python and conforms to the Python database API 2.0 specifications with a number of enhancements. This module is intended to eventually replace cx_Oracle. Where do I get it? https://pypi.org/project/oracledb/1.2.0/ The easiest method to install/upgrade python-oracledb is via pip as in python -m pip install oracledb --upgrade What's new? This release adds support for binding and fetching data of type oracledb.DB_TYPE_OBJECT (including SYS.XMLTYPE) in thin mode. It also adds official support for Python 3.11 and addresses a number of smaller enhancements and bug fixes. See the full release notes for all of the details: https://python-oracledb.readthedocs.io/en/latest/release_notes.html#oracledb-1-2-0-november-2022 Please provide any feedback via GitHub issues: https://github.com/oracle/ python-oracledb/issues or discussions: https://github.com/oracle/python- oracledb/discussions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Wed Nov 16 09:35:42 2022 From: mal at egenix.com (Marc-Andre Lemburg) Date: Wed, 16 Nov 2022 15:35:42 +0100 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> <5138e418-8b64-f4a9-577b-6749ed68e82f@egenix.com> <9CD9C429-24A8-43C1-806A-A2739C68B10E@gmail.com> Message-ID: <796705a0-a6a6-2012-dac1-c76aa46e58ba@egenix.com> FYI: I've created a PR with these changes... https://github.com/python/peps/pull/2887 On 14.11.2022 14:23, Marc-Andre Lemburg wrote: > Here's the updated wording for the addition, which should cover > everything we have discussed: > > """ > Connection.autocommit > > ? Attribute to query and set the autocommit mode of the connection. > > ? Returns True if the connection is operating in autocommit (non- > ? transactional) mode. Returns False is the connection is operating in > ? manual commit (transactional) mode. > > ? Setting the attribute to True or False adjusts the connection's mode > ? accordingly. > > ? Changing the setting from True to False (disabling autocommit) will > ? have the database leave autocommit mode and starts a new transaction. > ? Changing from False to True (enabling autocommit) has database > ? dependent semantics with respect to how pending transactions are > ? handled [1]. > > ? Deprecation notice: Even though several database modules implement > ? both the read and write nature of this attribute, setting the > ? autocommit mode by writing to the attribute is deprecated, since this > ? may result in I/O and related exceptions, making it difficult to > ? implement in an async context [2]. > > [1] Many database modules implementing the autocommit attribute will > automatically commit any pending transaction and then enter autocommit > mode. > > [2] For DB-API 3.0, we are going to introduce a new method > .setautocommit(value), which will allow setting the autocommit mode, and > make .autocommit a read-only attribute. Additionally, we are considering > to add a new standard keyword parameter autocommit to the Connection > constructor. Modules authors are encouraged to add these changes in > preparation for this change. > """ > > Please let me know whether I forgot anything. I'll prepare the PR > for PEP 249 later this week. > > Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 16 2022) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ From mal at egenix.com Wed Nov 16 13:05:52 2022 From: mal at egenix.com (Marc-Andre Lemburg) Date: Wed, 16 Nov 2022 19:05:52 +0100 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <796705a0-a6a6-2012-dac1-c76aa46e58ba@egenix.com> References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <125870e5-000c-427c-8e2d-e1b695724aa4@app.fastmail.com> <5ce10da0-e3e4-d2be-b601-22ded1358db0@egenix.com> <5138e418-8b64-f4a9-577b-6749ed68e82f@egenix.com> <9CD9C429-24A8-43C1-806A-A2739C68B10E@gmail.com> <796705a0-a6a6-2012-dac1-c76aa46e58ba@egenix.com> Message-ID: On 16.11.2022 15:35, Marc-Andre Lemburg wrote: > FYI: I've created a PR with these changes... > > https://github.com/python/peps/pull/2887 Erlend reviewed the PR and I have merged it now. It should be online on the PEP website in a bit. Thank you everyone for your input and help. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 16 2022) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/