From mike_mp at zzzcomputing.com Fri Apr 7 13:14:37 2023 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Fri, 07 Apr 2023 13:14:37 -0400 Subject: [DB-SIG] documenting whether or not the seq_of_parameters to executemany is expected to be run in order given Message-ID: hey list - as $subject says, if we run: cursor.executemany("insert into table (a, b) values (?, ?)", [(1, 1), (2, 2), (3, 3)]) it should be obvious that most people would *expect* that the three parameter sets given are INSERTed in the order that was given. This could be an issue if perhaps the rows being inserted contained values that refer to previous rows via foreign key. Or if attempting to run an UPDATE, where we would like the order of rows UPDATEd to be deterministic, so that we can avoid deadlocks with other processes that may be UPDATEing some subset of those same rows in a different transaction. it's obviously also an issue for developers expecting server-generated values to follow some sequence, however if you bring that use case up you will get a flock of lecturers scolding you for this suggestion, so let's ignore that use case (that is not my use case). However pep-249 doesn't indicate this behavior one way or the other, that is, whether we should not expect this, or we should expect this, or that it's up to the DBAPI to tell us what to expect. I bring this up because a common optimization for executemany of an INSERT is to rewrite the statement like this: "INSERT INTO table (a, b) VALUES (1, 1), (2, 2), (3, 3)" For example see what Pymysql does, using the regex at https://github.com/PyMySQL/PyMySQL/blob/885841f3fee416c222a75d83a81f74d3dcd71b51/pymysql/cursors.py#L5 to rewrite the statement here : https://github.com/PyMySQL/PyMySQL/blob/885841f3fee416c222a75d83a81f74d3dcd71b51/pymysql/cursors.py#L162 So it's also the case that most databases given the INSERT statement above will run the VALUES entries in that order, after all, why *wouldnt* they. But it turns out a similar statement run on MS SQL Server, using explicit table-valued entries in order, in some cases will actually insert the rows in some other order (optimizing for it seems like indexing of values in some way that relate to some foreign key constraint). My actual problem from there gets into that I'm also trying to use RETURNING , but that's not the scope of the question here. The scope here is, should pep-249 add some verbiage: "the order in which parameters are processed by executemany() should not be assumed to be in the order the parameters were given". which IMO would be crazy. but if that's the reality, maybe it should be stated. I'd of course *prefer* if it were stated that executemany() should process the given params in the order given. But I'm not too optimistic about that :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Clark at actian.com Fri Apr 7 13:46:40 2023 From: Chris.Clark at actian.com (Chris Clark) Date: Fri, 7 Apr 2023 17:46:40 +0000 Subject: [DB-SIG] documenting whether or not the seq_of_parameters to executemany is expected to be run in order given In-Reply-To: References: Message-ID: Good idea! A statement to the effect that it is DBMS (and potentially driver) dependent seems a good clarification. Chris From: DB-SIG On Behalf Of Mike Bayer Sent: Friday, April 7, 2023 10:15 AM To: db-sig at python.org Subject: [DB-SIG] documenting whether or not the seq_of_parameters to executemany is expected to be run in order given hey list - as $subject says, if we run: cursor.executemany("insert into table (a, b) values (?, ?)", [(1, 1), (2, 2), (3, 3)]) it should be obvious that most people would *expect* that the three parameter sets given are INSERTed in the order that was given. This could be an issue if perhaps the rows being inserted contained values that refer to previous rows via foreign key. Or if attempting to run an UPDATE, where we would like the order of rows UPDATEd to be deterministic, so that we can avoid deadlocks with other processes that may be UPDATEing some subset of those same rows in a different transaction. it's obviously also an issue for developers expecting server-generated values to follow some sequence, however if you bring that use case up you will get a flock of lecturers scolding you for this suggestion, so let's ignore that use case (that is not my use case). However pep-249 doesn't indicate this behavior one way or the other, that is, whether we should not expect this, or we should expect this, or that it's up to the DBAPI to tell us what to expect. I bring this up because a common optimization for executemany of an INSERT is to rewrite the statement like this: "INSERT INTO table (a, b) VALUES (1, 1), (2, 2), (3, 3)" For example see what Pymysql does, using the regex at https://github.com/PyMySQL/PyMySQL/blob/885841f3fee416c222a75d83a81f74d3dcd71b51/pymysql/cursors.py#L5 to rewrite the statement here : https://github.com/PyMySQL/PyMySQL/blob/885841f3fee416c222a75d83a81f74d3dcd71b51/pymysql/cursors.py#L162 So it's also the case that most databases given the INSERT statement above will run the VALUES entries in that order, after all, why *wouldnt* they. But it turns out a similar statement run on MS SQL Server, using explicit table-valued entries in order, in some cases will actually insert the rows in some other order (optimizing for it seems like indexing of values in some way that relate to some foreign key constraint). My actual problem from there gets into that I'm also trying to use RETURNING , but that's not the scope of the question here. The scope here is, should pep-249 add some verbiage: "the order in which parameters are processed by executemany() should not be assumed to be in the order the parameters were given". which IMO would be crazy. but if that's the reality, maybe it should be stated. I'd of course *prefer* if it were stated that executemany() should process the given params in the order given. But I'm not too optimistic about that :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike_mp at zzzcomputing.com Fri Apr 7 14:09:18 2023 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Fri, 07 Apr 2023 14:09:18 -0400 Subject: [DB-SIG] documenting whether or not the seq_of_parameters to executemany is expected to be run in order given In-Reply-To: References: Message-ID: <602223f4-e153-41e9-b9dc-84ae5d35a81b@app.fastmail.com> OK that's two "sure, let's absolve ourselves of this problem" responses :) any opinion on executemany() being less useful if this requirement is not established, and/or encouraging DBAPI authors to at least *document* this themselves and maybe *prefer* maintaining ordering ? On Fri, Apr 7, 2023, at 1:54 PM, Erlend Egeberg Aasland wrote: > On Fri, 7 Apr 2023 at 19:15, Mike Bayer wrote: >> >> [?] The scope here is, should pep-249 add some verbiage: "the order in which parameters are processed by executemany() should not be assumed to be in the order the parameters were given". [?] > > Sounds good to me. > > Erlend >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlenda at gmail.com Fri Apr 7 13:54:49 2023 From: erlenda at gmail.com (Erlend Egeberg Aasland) Date: Fri, 7 Apr 2023 19:54:49 +0200 Subject: [DB-SIG] documenting whether or not the seq_of_parameters to executemany is expected to be run in order given In-Reply-To: References: Message-ID: On Fri, 7 Apr 2023 at 19:15, Mike Bayer wrote: > > [?] The scope here is, should pep-249 add some verbiage: "the order in > which parameters are processed by executemany() should not be assumed to be > in the order the parameters were given". [?] > Sounds good to me. Erlend > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tlocke at tlocke.org.uk Sun Apr 9 05:19:38 2023 From: tlocke at tlocke.org.uk (Tony Locke) Date: Sun, 9 Apr 2023 10:19:38 +0100 Subject: [DB-SIG] documenting whether or not the seq_of_parameters to executemany is expected to be run in order given In-Reply-To: <602223f4-e153-41e9-b9dc-84ae5d35a81b@app.fastmail.com> References: <602223f4-e153-41e9-b9dc-84ae5d35a81b@app.fastmail.com> Message-ID: My initial thought is that the driver must pass the executemany parameters on to the server without changing the order of the parameters. Maybe that should be explicitly stated in the spec. Once the server has the parameters then I think the behaviour becomes DBMS dependent. I guess my philosophy is that the humble driver should just faithfully pass things back and forth while interfering as little as possible. On Fri, 7 Apr 2023 at 19:09, Mike Bayer wrote: > > OK that's two "sure, let's absolve ourselves of this problem" responses :) > > any opinion on executemany() being less useful if this requirement is not established, and/or encouraging DBAPI authors to at least *document* this themselves and maybe *prefer* maintaining ordering ? > > > > On Fri, Apr 7, 2023, at 1:54 PM, Erlend Egeberg Aasland wrote: > > On Fri, 7 Apr 2023 at 19:15, Mike Bayer wrote: > > > [?] The scope here is, should pep-249 add some verbiage: "the order in which parameters are processed by executemany() should not be assumed to be in the order the parameters were given". [?] > > > Sounds good to me. > > Erlend > > > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig From mike_mp at zzzcomputing.com Sun Apr 9 09:13:52 2023 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Sun, 09 Apr 2023 09:13:52 -0400 Subject: [DB-SIG] documenting whether or not the seq_of_parameters to executemany is expected to be run in order given In-Reply-To: References: <602223f4-e153-41e9-b9dc-84ae5d35a81b@app.fastmail.com> Message-ID: It's my understanding that a lot of executemany() implementations will actually run the given statement once per each element in the sequence of parameters. such as, it creates a prepared statement handle for the statement, then runs each parameter set. a driver that works this way can document that executemany() is guaranteed to invoke in that order, because it's a driver-determined behavior, not the database. a driver that packages all the parameter sets into a single concatenated statement of course has a different story, they can't guarantee this order. but in both cases it can be documented and in the former case it can be documented as the order being guaranteed also. On Sun, Apr 9, 2023, at 5:19 AM, Tony Locke wrote: > My initial thought is that the driver must pass the executemany > parameters on to the server without changing the order of the > parameters. Maybe that should be explicitly stated in the spec. Once > the server has the parameters then I think the behaviour becomes DBMS > dependent. I guess my philosophy is that the humble driver should just > faithfully pass things back and forth while interfering as little as > possible. > > On Fri, 7 Apr 2023 at 19:09, Mike Bayer wrote: > > > > OK that's two "sure, let's absolve ourselves of this problem" responses :) > > > > any opinion on executemany() being less useful if this requirement is not established, and/or encouraging DBAPI authors to at least *document* this themselves and maybe *prefer* maintaining ordering ? > > > > > > > > On Fri, Apr 7, 2023, at 1:54 PM, Erlend Egeberg Aasland wrote: > > > > On Fri, 7 Apr 2023 at 19:15, Mike Bayer wrote: > > > > > > [?] The scope here is, should pep-249 add some verbiage: "the order in which parameters are processed by executemany() should not be assumed to be in the order the parameters were given". [?] > > > > > > Sounds good to me. > > > > Erlend > > > > > > > > _______________________________________________ > > DB-SIG maillist - DB-SIG at python.org > > https://mail.python.org/mailman/listinfo/db-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Apr 10 11:31:16 2023 From: mal at egenix.com (Marc-Andre Lemburg) Date: Mon, 10 Apr 2023 17:31:16 +0200 Subject: [DB-SIG] documenting whether or not the seq_of_parameters to executemany is expected to be run in order given In-Reply-To: References: <602223f4-e153-41e9-b9dc-84ae5d35a81b@app.fastmail.com> Message-ID: On 09.04.2023 15:13, Mike Bayer wrote: > It's my understanding that a lot of executemany() implementations will > actually run the given statement once per each element in the sequence > of parameters.? such as, it creates a prepared statement handle for > the statement, then runs each parameter set.? a driver that works this > way can document that executemany() is guaranteed to invoke in that > order, because it's a driver-determined behavior, not the database. > > a driver that packages all the parameter sets into a single > concatenated statement of course has a different story, they can't > guarantee this order. > > but in both cases it can be documented and in the former case it can > be documented as the order being guaranteed also. The DB-API wording is vague in this respect, mostly because the Python driver cannot guarantee whether the statements actually get executed in the same order as the parameters are passed to the method. Some databases optimize such bulk executes to gain performance (e.g. reorder INSERTs to make them match partitions) or seemingly randomize executes (e.g. when running against a cluster to spread load). I agree that we should probably clearly state this in PEP 249 using your text: """ The order in which parameters are processed by .executemany() should not be assumed to be in the same order as the parameters were given, since the underlying database drivers or backends may optimize the way the operations are processed. """ Related to this: databases often also don't necessarily guarantee that INSERT order is stable, i.e. a SELECT will return the rows in the same order they were added or in primary key order. Many do, but esp. cluster ones typically don't. When writing code trying to work with different backends, this can easily create problems, if you're not aware. I was once bitten by this a long time ago, when running tests against a Terraform data warehouse. > > On Sun, Apr 9, 2023, at 5:19 AM, Tony Locke wrote: >> My initial thought is that the driver must pass the executemany >> parameters on to the server without changing the order of the >> parameters. Maybe that should be explicitly stated in the spec. Once >> the server has the parameters then I think the behaviour becomes DBMS >> dependent. I guess my philosophy is that the humble driver should just >> faithfully pass things back and forth while interfering as little as >> possible. >> >> On Fri, 7 Apr 2023 at 19:09, Mike Bayer wrote: >> > >> > OK that's two "sure, let's absolve ourselves of this problem" >> responses :) >> > >> > any opinion on executemany() being less useful if this requirement >> is not established, and/or encouraging DBAPI authors to at least >> *document* this themselves and maybe *prefer* maintaining ordering ? >> > >> > >> > >> > On Fri, Apr 7, 2023, at 1:54 PM, Erlend Egeberg Aasland wrote: >> > >> > On Fri, 7 Apr 2023 at 19:15, Mike Bayer >> wrote: >> > >> > >> > [?] The scope here is, should pep-249 add some verbiage: "the order >> in which parameters are processed by executemany() should not be >> assumed to be in the order the parameters were given". [?] >> > >> > >> > Sounds good to me. >> > >> > Erlend >> > >> > >> > >> > _______________________________________________ >> > DB-SIG maillist? - DB-SIG at python.org >> > https://mail.python.org/mailman/listinfo/db-sig >> > > > _______________________________________________ > DB-SIG maillist -DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Apr 10 2023) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthony.tuininga at gmail.com Mon Apr 24 16:53:49 2023 From: anthony.tuininga at gmail.com (Anthony Tuininga) Date: Mon, 24 Apr 2023 14:53:49 -0600 Subject: [DB-SIG] python-oracledb 1.3.1 Message-ID: What is python-oracledb? python-oracledb is a Python extension module that enables access to Oracle Database for Python and conforms to the Python database API 2.0 specifications with a number of enhancements. This module is intended to eventually replace cx_Oracle. Where do I get it? https://pypi.org/project/oracledb/1.3.1/ The easiest method to install/upgrade python-oracledb is via pip as in python -m pip install oracledb --upgrade What's new? This release addresses a number of reported issues. See the full release notes for all of the details: https://python-oracledb.readthedocs.io/en/latest/release_notes.html#oracledb-1-3-1-april-2023 Please provide any feedback via GitHub issues: https://github.com/oracle/ python-oracledb/issues or discussions: https://github.com/oracle/python- oracledb/discussions -------------- next part -------------- An HTML attachment was scrubbed... URL: