From eliswilson at hushmail.com Wed May 1 00:07:45 2013 From: eliswilson at hushmail.com (eliswilson at hushmail.com) Date: Tue, 30 Apr 2013 18:07:45 -0400 Subject: [DB-SIG] Biggest Fake Conference in Computer Science Message-ID: <20130430220746.1DE5B14DBDE@smtp.hushmail.com> Biggest Fake Conference in Computer Science We are researchers from different parts of the world and conducted a study on the world?s biggest bogus computer science conference WORLDCOMP http://sites.google.com/site/worlddump1 organized by Prof. Hamid Arabnia from University of Georgia, USA. We submitted a fake paper to WORLDCOMP 2011 and again (the same paper with a modified title) to WORLDCOMP 2012. This paper had numerous fundamental mistakes. Sample statements from that paper include: (1). Binary logic is fuzzy logic and vice versa (2). Pascal developed fuzzy logic (3). Object oriented languages do not exhibit any polymorphism or inheritance (4). TCP and IP are synonyms and are part of OSI model (5). Distributed systems deal with only one computer (6). Laptop is an example for a super computer (7). Operating system is an example for computer hardware Also, our paper did not express any conceptual meaning. However, it was accepted both the times without any modifications (and without any reviews) and we were invited to submit the final paper and a payment of $500+ fee to present the paper. We decided to use the fee for better purposes than making Prof. Hamid Arabnia richer. After that, we received few reminders from WORLDCOMP to pay the fee but we never responded. This fake paper is different from the two fake papers already published (see https://sites.google.com/site/worlddump4 for details) in WORLDCOMP. We MUST say that you should look at the above website if you have any thoughts of participating in WORLDCOMP. DBLP and other indexing agencies have stopped indexing WORLDCOMP?s proceedings since 2011 due to its fakeness. See http://www.informatik.uni-trier.de/~ley/db/conf/icai/index.html for of one of the conferences of WORLDCOMP and notice that there is no listing after 2010. See Section 2 of http://sites.google.com/site/dumpconf for comments from well-known researchers about WORLDCOMP. The status of your WORLDCOMP papers can be changed from scientific to other (i.e., junk or non-technical) at any time. Better not to have a paper than having it in WORLDCOMP and spoil the resume and peace of mind forever! Our study revealed that WORLDCOMP is money making business, using University of Georgia mask, for Prof. Hamid Arabnia. He is throwing out a small chunk of that money (around 20 dollars per paper published in WORLDCOMP?s proceedings) to his puppet (Mr. Ashu Solo or A.M.G. Solo) who publicizes WORLDCOMP and also defends it at various forums, using fake/anonymous names. The puppet uses fake names and defames other conferences to divert traffic to WORLDCOMP. He also makes anonymous phone calls and threatens the critiques of WORLDCOMP (See Item 7 of Section 5 of above website). That is, the puppet does all his best to get a maximum number of papers published at WORLDCOMP to get more money into his (and Prof. Hamid Arabnia?s) pockets. Prof. Hamid Arabnia makes a lot of tricks. For example, he appeared in a newspaper to fool the public, claiming him a victim of cyber-attack (see Item 8 in Section 5 of above website). Monte Carlo Resort (the venue of WORLDCOMP for more than 10 years, until 2012) has refused to provide the venue for WORLDCOMP?13 because of the fears of their image being tarnished due to WORLDCOMP?s fraudulent activities. That is why WORLDCOMP?13 is taking place at a different resort. WORLDCOMP will not be held after 2013. The draft paper submission deadline is over but still there are no committee members, no reviewers, and there is no conference Chairman. The only contact details available on WORLDCOMP?s website is just an email address! We ask Prof. Hamid Arabnia to publish all reviews for all the papers (after blocking identifiable details) since 2000 conference. Reveal the names and affiliations of all the reviewers (for each year) and how many papers each reviewer had reviewed on average. We also ask him to look at the Open Challenge (Section 6) at https://sites.google.com/site/moneycomp1 and respond if he has any professional values. Sorry for posting to multiple lists. Spreading the word is the only way to stop this bogus conference. Please forward this message to other mailing lists and people. We are shocked with Prof. Hamid Arabnia and his puppet?s activities at http://worldcomp-fake-bogus.blogspot.com Search Google using the keyword worldcomp fake for additional links. From mal at egenix.com Thu May 2 14:16:50 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 02 May 2013 14:16:50 +0200 Subject: [DB-SIG] checking column types from cursor object in a database-independent way? In-Reply-To: <0DF2B047-0ACF-4286-A00D-7D8BF42D824D@zzzcomputing.com> References: <337E5B06-72F0-47A0-ACCA-BB5AB4256798@zzzcomputing.com> <2AD29189-745E-4127-91BD-85775D980E73@zzzcomputing.com> <09CFDDF1-B874-447E-A4CA-1B07F1179CF0@zzzcomputing.com> <51763871.1040705@egenix.com> <0DF2B047-0ACF-4286-A00D-7D8BF42D824D@zzzcomputing.com> Message-ID: <51825932.1050208@egenix.com> On 23.04.2013 16:41, Michael Bayer wrote: > > On Apr 23, 2013, at 3:29 AM, M.-A. Lemburg wrote: > >> >> >> A method doing the lookup via the sys.modules dictionary >> could resolve those issues: >> >> database = connection.database() >> try: >> cursor = connection.cursor() >> cursor.execute(...) >> except database.DataError: >> ... >> >> Thoughts ? > > > we probably want to call it "module" or "namespace" or "api" or something like that. The method itself can be a Python @property which would cause it to behave like an attribute in any case, or if we want to keep it as a method call to make it clear that code is being executed, that's fine as well. But the spec would need to say in either case, "don't assign the module itself to the connection". I like the idea with a property. I'm not sure about those other name candidates, though: "connection.database" make the code very readable, IMO. Is there a possible name clash with existing attributes on database modules that we would need to take into account ? Another readable candidate would be "connection.dbapi". -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 02 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-04-30: Released eGenix PyRun 1.2.0 ... http://egenix.com/go44 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From vernondcole at gmail.com Thu May 2 14:33:43 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Thu, 2 May 2013 06:33:43 -0600 Subject: [DB-SIG] checking column types from cursor object in a database-independent way? In-Reply-To: <51825932.1050208@egenix.com> References: <337E5B06-72F0-47A0-ACCA-BB5AB4256798@zzzcomputing.com> <2AD29189-745E-4127-91BD-85775D980E73@zzzcomputing.com> <09CFDDF1-B874-447E-A4CA-1B07F1179CF0@zzzcomputing.com> <51763871.1040705@egenix.com> <0DF2B047-0ACF-4286-A00D-7D8BF42D824D@zzzcomputing.com> <51825932.1050208@egenix.com> Message-ID: I think that "database" as the property identifier carries too much baggage -- there are too many meanings of, and too much use of the term, therefore it is not immediately obvious what the property might be used for. It reads well, but does not have an "ahha!" factor. connection.dbapi does. I think it would be a good choice. +1 -- Vernon Cole On Thu, May 2, 2013 at 6:16 AM, M.-A. Lemburg wrote: > On 23.04.2013 16:41, Michael Bayer wrote: > > > > On Apr 23, 2013, at 3:29 AM, M.-A. Lemburg wrote: > > > >> > >> > >> A method doing the lookup via the sys.modules dictionary > >> could resolve those issues: > >> > >> database = connection.database() > >> try: > >> cursor = connection.cursor() > >> cursor.execute(...) > >> except database.DataError: > >> ... > >> > >> Thoughts ? > > > > > > we probably want to call it "module" or "namespace" or "api" or > something like that. The method itself can be a Python @property which > would cause it to behave like an attribute in any case, or if we want to > keep it as a method call to make it clear that code is being executed, > that's fine as well. But the spec would need to say in either case, > "don't assign the module itself to the connection". > > I like the idea with a property. I'm not sure about those other name > candidates, though: "connection.database" make the code very > readable, IMO. > > Is there a possible name clash with existing attributes on database > modules that we would need to take into account ? > > Another readable candidate would be "connection.dbapi". > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, May 02 2013) > >>> Python Projects, Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2013-04-30: Released eGenix PyRun 1.2.0 ... http://egenix.com/go44 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > http://mail.python.org/mailman/listinfo/db-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Thu May 2 14:36:47 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 02 May 2013 14:36:47 +0200 Subject: [DB-SIG] checking column types from cursor object in a database-independent way? In-Reply-To: References: <337E5B06-72F0-47A0-ACCA-BB5AB4256798@zzzcomputing.com> <2AD29189-745E-4127-91BD-85775D980E73@zzzcomputing.com> <09CFDDF1-B874-447E-A4CA-1B07F1179CF0@zzzcomputing.com> <51763871.1040705@egenix.com> Message-ID: <51825DDF.6050002@egenix.com> On 23.04.2013 17:00, Daniel Lenski wrote: > On Tue, Apr 23, 2013 at 12:29 AM, M.-A. Lemburg wrote: >> >> I don't think we should clutter up the connection objects with >> module scope attributes that hardly ever get used. >> >> The exceptions are used a lot, so it makes sense to have them >> easily available via the connection object - even though I'm not >> sure whether that particular DB-API extension was such a >> good idea w/r to API design (it's one of those practicality beats >> purity things). > > Again, I understand the desire not to cruft up the cursor or > connection namespaces, but I am sort of surprised that you consider > the type objects to be little-used. It seems to me that they are a > necessity for any abstract DB-independent layer. Database data type handling is very complex and in practice you need to deal with a whole lot of issues that the high level DB-API type objects don't capture, e.g. supported data ranges, encoding support, precision, etc. Today's databases also often support additional types for e.g. geo location, network addresses, XML, ranges, etc. for which there's no corresponding DB-API type. See http://www.postgresql.org/docs/9.2/static/datatype.html for a typical example of what databases provide today. Such logic is better left to higher-level database abstractions, which then support a handful of backends. I'd be +0 on deprecating the type objects for DB-API 3.0. While they don't introduce a lot of overhead for database module authors, they aren't all that helpful to users and can lead to confusion. >> Adding access to the database module via the connection object >> would allow to resolve all this. The problem >> with doing so is that you typically don't want the module >> object to be referenced directly by hundreds of objects in your >> application and you can also run into problems when reloading >> modules or (depending on how this is implemented) circular >> references. >> >> A method doing the lookup via the sys.modules dictionary >> could resolve those issues: >> >> database = connection.database() >> try: >> cursor = connection.cursor() >> cursor.execute(...) >> except database.DataError: >> ... >> >> Thoughts ? > > This seems like a reasonable solution to me, that would solve both > this problem as well as similar ones. > > Alternatively, a cursor.coltypes attribute (in term containing > .STRING, .DATETIME, etc.) would solve the present problem. > > Dan > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > http://mail.python.org/mailman/listinfo/db-sig > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 02 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-04-30: Released eGenix PyRun 1.2.0 ... http://egenix.com/go44 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Thu May 2 15:53:23 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Thu, 2 May 2013 09:53:23 -0400 Subject: [DB-SIG] checking column types from cursor object in a database-independent way? In-Reply-To: <51825932.1050208@egenix.com> References: <337E5B06-72F0-47A0-ACCA-BB5AB4256798@zzzcomputing.com> <2AD29189-745E-4127-91BD-85775D980E73@zzzcomputing.com> <09CFDDF1-B874-447E-A4CA-1B07F1179CF0@zzzcomputing.com> <51763871.1040705@egenix.com> <0DF2B047-0ACF-4286-A00D-7D8BF42D824D@zzzcomputing.com> <51825932.1050208@egenix.com> Message-ID: On May 2, 2013, at 8:16 AM, M.-A. Lemburg wrote: > > I like the idea with a property. I'm not sure about those other name > candidates, though: "connection.database" make the code very > readable, IMO. > > Is there a possible name clash with existing attributes on database > modules that we would need to take into account ? > > Another readable candidate would be "connection.dbapi". +1 for connection.dbapi > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, May 02 2013) >>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2013-04-30: Released eGenix PyRun 1.2.0 ... http://egenix.com/go44 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ From mal at egenix.com Thu May 2 16:54:44 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 02 May 2013 16:54:44 +0200 Subject: [DB-SIG] checking column types from cursor object in a database-independent way? In-Reply-To: References: <337E5B06-72F0-47A0-ACCA-BB5AB4256798@zzzcomputing.com> <2AD29189-745E-4127-91BD-85775D980E73@zzzcomputing.com> <09CFDDF1-B874-447E-A4CA-1B07F1179CF0@zzzcomputing.com> <51763871.1040705@egenix.com> <0DF2B047-0ACF-4286-A00D-7D8BF42D824D@zzzcomputing.com> <51825932.1050208@egenix.com> Message-ID: <51827E34.1060202@egenix.com> On 02.05.2013 15:53, Michael Bayer wrote: >> Another readable candidate would be "connection.dbapi". > > +1 for connection.dbapi Sounds like we're reaching consensus :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 02 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-04-30: Released eGenix PyRun 1.2.0 ... http://egenix.com/go44 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From vernondcole at gmail.com Thu May 2 19:12:02 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Thu, 2 May 2013 11:12:02 -0600 Subject: [DB-SIG] checking column types from cursor object in a database-independent way? In-Reply-To: <51827E34.1060202@egenix.com> References: <337E5B06-72F0-47A0-ACCA-BB5AB4256798@zzzcomputing.com> <2AD29189-745E-4127-91BD-85775D980E73@zzzcomputing.com> <09CFDDF1-B874-447E-A4CA-1B07F1179CF0@zzzcomputing.com> <51763871.1040705@egenix.com> <0DF2B047-0ACF-4286-A00D-7D8BF42D824D@zzzcomputing.com> <51825932.1050208@egenix.com> <51827E34.1060202@egenix.com> Message-ID: So -- here is a prototype... > >>> import adodbapi > > >>> c = adodbapi.connect("provider=Microsoft.ACE.OLEDB.12.0;data >> source=../examples/test.mdb") > > >>> c > > > > >>> c.dbapi > > > 'C:\Python27\lib\site-packages\adodbapi\apibase. > > pyc'> > > >>> c.dbapi.NUMBER > > > > My implementation is: > import apibase as api class Connection(object): # include connection attributes as class attributes required by api > definition. dbapi = property(lambda cls: api) Warning = api.Warning ... (etc.) > Is that what we want to see? -- Vernon Cole On Thu, May 2, 2013 at 8:54 AM, M.-A. Lemburg wrote: > On 02.05.2013 15:53, Michael Bayer wrote: > >> Another readable candidate would be "connection.dbapi". > > > > +1 for connection.dbapi > > Sounds like we're reaching consensus :-) > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, May 02 2013) > >>> Python Projects, Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2013-04-30: Released eGenix PyRun 1.2.0 ... http://egenix.com/go44 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike_mp at zzzcomputing.com Thu May 2 19:30:35 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Thu, 2 May 2013 13:30:35 -0400 Subject: [DB-SIG] checking column types from cursor object in a database-independent way? In-Reply-To: References: <337E5B06-72F0-47A0-ACCA-BB5AB4256798@zzzcomputing.com> <2AD29189-745E-4127-91BD-85775D980E73@zzzcomputing.com> <09CFDDF1-B874-447E-A4CA-1B07F1179CF0@zzzcomputing.com> <51763871.1040705@egenix.com> <0DF2B047-0ACF-4286-A00D-7D8BF42D824D@zzzcomputing.com> <51825932.1050208@egenix.com> <51827E34.1060202@egenix.com> Message-ID: On May 2, 2013, at 1:12 PM, Vernon D. Cole wrote: > So -- here is a prototype... > > >>> import adodbapi > >>> c = adodbapi.connect("provider=Microsoft.ACE.OLEDB.12.0;data source=../examples/test.mdb") > >>> c > > >>> c.dbapi > pyc'> > >>> c.dbapi.NUMBER > > > My implementation is: > > import apibase as api > class Connection(object): > # include connection attributes as class attributes required by api definition. > dbapi = property(lambda cls: api) > Warning = api.Warning > ... (etc.) > > Is that what we want to see? Python @property will pick up on the __doc__ of the method (which is also expected to be an instance method), so that can be specified traditionally: class Connection(object): @property def dbapi(self): "Return a reference to the DBAPI module for this Connection." return api > -- > Vernon Cole > > > > On Thu, May 2, 2013 at 8:54 AM, M.-A. Lemburg wrote: > On 02.05.2013 15:53, Michael Bayer wrote: > >> Another readable candidate would be "connection.dbapi". > > > > +1 for connection.dbapi > > Sounds like we're reaching consensus :-) > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, May 02 2013) > >>> Python Projects, Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2013-04-30: Released eGenix PyRun 1.2.0 ... http://egenix.com/go44 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vernondcole at gmail.com Thu May 2 19:55:35 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Thu, 2 May 2013 11:55:35 -0600 Subject: [DB-SIG] checking column types from cursor object in a database-independent way? In-Reply-To: References: <337E5B06-72F0-47A0-ACCA-BB5AB4256798@zzzcomputing.com> <2AD29189-745E-4127-91BD-85775D980E73@zzzcomputing.com> <09CFDDF1-B874-447E-A4CA-1B07F1179CF0@zzzcomputing.com> <51763871.1040705@egenix.com> <0DF2B047-0ACF-4286-A00D-7D8BF42D824D@zzzcomputing.com> <51825932.1050208@egenix.com> <51827E34.1060202@egenix.com> Message-ID: I forgot: pywin32 just dropped support for Python 2.3 so I can use decorators now. Michael, you have just contributed four lines of code to the next version of adodbapi. -- Vernon On Thu, May 2, 2013 at 11:30 AM, Michael Bayer wrote: > > On May 2, 2013, at 1:12 PM, Vernon D. Cole wrote: > > So -- here is a prototype... > > >> >>> import adodbapi >> >> >>> c = adodbapi.connect("provider=Microsoft.ACE.OLEDB.12.0;data >>> source=../examples/test.mdb") >> >> >>> c >> >> >> >> >>> c.dbapi >> >> >> 'C:\Python27\lib\site-packages\adodbapi\apibase. >> >> pyc'> >> >> >>> c.dbapi.NUMBER >> >> >> >> > My implementation is: > > >> import apibase as api > > class Connection(object): > > # include connection attributes as class attributes required by api >> definition. > > dbapi = property(lambda cls: api) > > Warning = api.Warning > > ... (etc.) >> > > Is that what we want to see? > > > Python @property will pick up on the __doc__ of the method (which is also > expected to be an instance method), so that can be specified traditionally: > > class Connection(object): > @property > def dbapi(self): > "Return a reference to the DBAPI module for this Connection." > return api > > > > > -- > Vernon Cole > > > > On Thu, May 2, 2013 at 8:54 AM, M.-A. Lemburg wrote: > >> On 02.05.2013 15:53, Michael Bayer wrote: >> >> Another readable candidate would be "connection.dbapi". >> > >> > +1 for connection.dbapi >> >> Sounds like we're reaching consensus :-) >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Source (#1, May 02 2013) >> >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >> >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >> ________________________________________________________________________ >> 2013-04-30: Released eGenix PyRun 1.2.0 ... http://egenix.com/go44 >> >> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri May 3 09:55:57 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 03 May 2013 09:55:57 +0200 Subject: [DB-SIG] checking column types from cursor object in a database-independent way? In-Reply-To: References: <337E5B06-72F0-47A0-ACCA-BB5AB4256798@zzzcomputing.com> <2AD29189-745E-4127-91BD-85775D980E73@zzzcomputing.com> <09CFDDF1-B874-447E-A4CA-1B07F1179CF0@zzzcomputing.com> <51763871.1040705@egenix.com> <0DF2B047-0ACF-4286-A00D-7D8BF42D824D@zzzcomputing.com> <51825932.1050208@egenix.com> <51827E34.1060202@egenix.com> Message-ID: <51836D8D.4050104@egenix.com> On 02.05.2013 19:55, Vernon D. Cole wrote: > I forgot: pywin32 just dropped support for Python 2.3 so I can use > decorators now. > > Michael, you have just contributed four lines of code to the next version > of adodbapi. I think we can use an example like the one Michael gave for the DB-API v2 extension. BTW: I'm not yet sure about the implications of using "api" directly in the property definition w/r to reload(dbapi_module). It may be safer to use sys.modules['dbapi_module'], but that's an implementation detail. > -- > Vernon > > > > On Thu, May 2, 2013 at 11:30 AM, Michael Bayer wrote: > >> >> On May 2, 2013, at 1:12 PM, Vernon D. Cole wrote: >> >> So -- here is a prototype... >> >> >>>>>> import adodbapi >>> >>>>>> c = adodbapi.connect("provider=Microsoft.ACE.OLEDB.12.0;data >>>> source=../examples/test.mdb") >>> >>>>>> c >>> >>> >>> >>>>>> c.dbapi >>> >>> >>> 'C:\Python27\lib\site-packages\adodbapi\apibase. >>> >>> pyc'> >>> >>>>>> c.dbapi.NUMBER >>> >>> >>> >>> >> My implementation is: >> >> >>> import apibase as api >> >> class Connection(object): >> >> # include connection attributes as class attributes required by api >>> definition. >> >> dbapi = property(lambda cls: api) >> >> Warning = api.Warning >> >> ... (etc.) >>> >> >> Is that what we want to see? >> >> >> Python @property will pick up on the __doc__ of the method (which is also >> expected to be an instance method), so that can be specified traditionally: >> >> class Connection(object): >> @property >> def dbapi(self): >> "Return a reference to the DBAPI module for this Connection." >> return api >> >> >> >> >> -- >> Vernon Cole >> >> >> >> On Thu, May 2, 2013 at 8:54 AM, M.-A. Lemburg wrote: >> >>> On 02.05.2013 15:53, Michael Bayer wrote: >>>>> Another readable candidate would be "connection.dbapi". >>>> >>>> +1 for connection.dbapi >>> >>> Sounds like we're reaching consensus :-) >>> >>> -- >>> Marc-Andre Lemburg >>> eGenix.com >>> >>> Professional Python Services directly from the Source (#1, May 02 2013) >>>>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >>> ________________________________________________________________________ >>> 2013-04-30: Released eGenix PyRun 1.2.0 ... http://egenix.com/go44 >>> >>> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >>> >>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >>> Registered at Amtsgericht Duesseldorf: HRB 46611 >>> http://www.egenix.com/company/contact/ >>> >> >> >> > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 03 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-04-30: Released eGenix PyRun 1.2.0 ... http://egenix.com/go44 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From info at egenix.com Mon May 6 10:01:00 2013 From: info at egenix.com (eGenix Team: M.-A. Lemburg) Date: Mon, 06 May 2013 10:01:00 +0200 Subject: [DB-SIG] ANN: eGenix mxODBC 3.2.3 - Python ODBC Database Interface Message-ID: <5187633C.1040802@egenix.com> ________________________________________________________________________ ANNOUNCING eGenix.com mxODBC Python ODBC Database Interface Version 3.2.3 mxODBC is our commercially supported Python extension providing ODBC database connectivity to Python applications on Windows, Mac OS X, Unix and BSD platforms This announcement is also available on our web-site for online reading: http://www.egenix.com/company/news/eGenix-mxODBC-3.2.3-GA.html ________________________________________________________________________ INTRODUCTION mxODBC provides an easy-to-use, high-performance, reliable and robust Python interface to ODBC compatible databases such as MS SQL Server, MS Access, Oracle Database, IBM DB2 and Informix , Sybase ASE and Sybase Anywhere, MySQL, PostgreSQL, SAP MaxDB and many more: http://www.egenix.com/products/python/mxODBC/ The "eGenix mxODBC - Python ODBC Database Interface" product is a commercial extension to our open-source eGenix mx Base Distribution: http://www.egenix.com/products/python/mxBase/ ________________________________________________________________________ NEWS The 3.2.3 release of our mxODBC is the latest patch level release of our popular Python ODBC Interface. In this release, we've included the following the following enhancements and fixes: Compatibility Enhancements -------------------------- * Added a work-around for a regression in Python 2.7.4 that results in a segfault when exiting Python after loading mxODBC. The regression will be fixed in Python 2.7.5, but we don't want to expose our users to segfaults, so added a work-around. See http://bugs.python.org/issue17703 for the bug ticket. * Please also see the mxODBC 3.2.2 announcement for an important new feature which allows to dramatically increase the fetch performance when working with MS SQL Server and IBM DB2 databases: http://www.egenix.com/company/news/eGenix-mxODBC-3.2.2-GA.html Fixes ----- * Fixed a bugglet which caused connection.__members__ to no longer list connection attributes. For the full set of changes please check the mxODBC change log: http://www.egenix.com/products/python/mxODBC/changelog.html ________________________________________________________________________ FEATURES mxODBC 3.2 was released on 2012-08-28. Please see the full announcement for highlights of the 3.2 release: http://www.egenix.com/company/news/eGenix-mxODBC-3.2.0-GA.html For the full set of features mxODBC has to offer, please see: http://www.egenix.com/products/python/mxODBC/#Features ________________________________________________________________________ EDITIONS mxODBC is available in these three editions: * The low-cost Standard Edition which provides data connectivity to a single database type, e.g. just MS SQL Server. * The Professional Edition, which gives full access to all mxODBC features. * The Product Development Edition, which allows including mxODBC in applications you develop. Compared to mxODBC 3.0, we have simplified our license terms to clarify the situation on multi-core and virtual machines. In most cases, you no longer need to purchase more than one license per processor or virtual machine, scaling down the overall license costs significantly compared to earlier mxODBC releases. For a complete overview of the new editions, please see the product page. http://www.egenix.com/products/python/mxODBC/#mxODBCEditions ________________________________________________________________________ DOWNLOADS The download archives and instructions for installing the package can be found at: http://www.egenix.com/products/python/mxODBC/ In order to use the eGenix mxODBC package you will first need to install the eGenix mx Base package: http://www.egenix.com/products/python/mxBase/ ________________________________________________________________________ UPGRADING Users are encouraged to upgrade to this latest mxODBC release to benefit from the new features and updated ODBC driver support. We have taken special care, not to introduce backwards incompatible changes, making the upgrade experience as smooth as possible. Customers who have purchased mxODBC 3.2 license can continue to use their licenses with this patch level release. Customers who have purchased mxODBC 2.x, 3.0 or 3.1 licenses, can benefit from upgrade discounts. We will give out 20% discount coupons going from mxODBC 2.x to 3.2 and 50% coupons for upgrades from mxODBC 3.x to 3.2. After upgrade, use of the original license from which you upgraded is no longer permitted. Please contact the eGenix.com Sales Team at sales at egenix.com with your existing license serials for details for an upgrade discount coupon. If you want to try the new release before purchace, you can request 30-day evaluation licenses by visiting our web-site http://www.egenix.com/products/python/mxODBC/#Evaluation or by writing to sales at egenix.com, stating your name (or the name of the company) and the number of eval licenses that you need. _______________________________________________________________________ SUPPORT Commercial support for this product is available from eGenix.com. Please see http://www.egenix.com/services/support/ for details about our support offerings. _______________________________________________________________________ INFORMATION About Python (http://www.python.org/): Python is an object-oriented Open Source programming language which runs on all modern platforms. By integrating ease-of-use, clarity in coding, enterprise application connectivity and rapid application design, Python establishes an ideal programming platform for today's IT challenges. About eGenix (http://www.egenix.com/): eGenix is a software project, consulting and product company focusing on expert services and professional quality products for companies, Python users and developers. Enjoy, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 06 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-04-30: Released eGenix PyRun 1.2.0 ... http://egenix.com/go44 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From vernondcole at gmail.com Wed May 8 11:48:52 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Wed, 8 May 2013 10:48:52 +0100 Subject: [DB-SIG] should precision and sign of decimal(0) be significant? Message-ID: I am working on code (django-mssql) where I found the following test: ## expected = ( ## Decimal('0.00'), ## Decimal('0.0'), ## Decimal('-0.00')) ## ## cur = con.cursor() ## cur.paramstyle = 'format' # a nonstandard extension -- VDC ## cur.execute("SELECT %s as A, %s as B, %s as C", expected) ## ## result = cur.fetchall() ## self.assertEqual(result[0], expected) I added the paramstyle alteration, to match the original test's assumption -- that is not what I am asking about. My question is: is this test reasonable on it's face? I personally think that all values of zero are equal, and the user should have no expectation that neqative zero, or an accurately precise zero, should be returned. My present code returns three copies of Decimal(0). But someone must disagree, or this test would not have been written. What is the feeling of the group? -- Vernon Cole -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Wed May 8 12:59:08 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 08 May 2013 12:59:08 +0200 Subject: [DB-SIG] should precision and sign of decimal(0) be significant? In-Reply-To: References: Message-ID: <518A2FFC.7020406@egenix.com> On 08.05.2013 11:48, Vernon D. Cole wrote: > I am working on code (django-mssql) where I found the following test: > > ## expected = ( > ## Decimal('0.00'), > ## Decimal('0.0'), > ## Decimal('-0.00')) > ## > ## cur = con.cursor() > > ## cur.paramstyle = 'format' # a nonstandard extension -- VDC > > ## cur.execute("SELECT %s as A, %s as B, %s as C", expected) > ## > ## result = cur.fetchall() > ## self.assertEqual(result[0], expected) > > I added the paramstyle alteration, to match the original test's assumption > -- that is not what I am asking about. My question is: is this test > reasonable on it's face? > > I personally think that all values of zero are equal, and the user should > have no expectation that neqative zero, or an accurately precise zero, > should be returned. My present code returns three copies of Decimal(0). > > But someone must disagree, or this test would not have been written. > > What is the feeling of the group? It's possible that the DB module passes decimals as strings to the database and the database could treat them differently in the resp. string form (e.g. raise an error in case the precision doesn't match). The test will pass either way, since all values compare equal. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46 2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From Chris.Clark at actian.com Wed May 8 18:22:07 2013 From: Chris.Clark at actian.com (Chris Clark) Date: Wed, 08 May 2013 09:22:07 -0700 Subject: [DB-SIG] should precision and sign of decimal(0) be significant? In-Reply-To: <518A2FFC.7020406@egenix.com> References: <518A2FFC.7020406@egenix.com> Message-ID: <518A7BAF.3040104@actian.com> On Wed, 08 May 2013 12:59:08 +0200, M.-A. Lemburg wrote: > On 08.05.2013 11:48, Vernon D. Cole wrote: >> I am working on code (django-mssql) where I found the following test: >> >> ## expected = ( >> ## Decimal('0.00'), >> ## Decimal('0.0'), >> ## Decimal('-0.00')) >> ## >> ## cur = con.cursor() >> >> ## cur.paramstyle = 'format' # a nonstandard extension -- VDC >> >> ## cur.execute("SELECT %s as A, %s as B, %s as C", expected) >> ## >> ## result = cur.fetchall() >> ## self.assertEqual(result[0], expected) >> >> I added the paramstyle alteration, to match the original test's assumption >> -- that is not what I am asking about. My question is: is this test >> reasonable on it's face? >> >> I personally think that all values of zero are equal, and the user should >> have no expectation that neqative zero, or an accurately precise zero, >> should be returned. My present code returns three copies of Decimal(0). >> >> But someone must disagree, or this test would not have been written. >> >> What is the feeling of the group? > It's possible that the DB module passes decimals as > strings to the database and the database could treat them > differently in the resp. string form (e.g. raise an error > in case the precision doesn't match). > > The test will pass either way, since all values compare > equal. To reiterate MA's reply. It is DBMS/driver dependent on how these values are treated (and sent/retrieved to/from the DBMS). Most servers will thunk it down to a plain zero but you can't predict what each vendor does. In python those are _different_ but as per MA's comment they compare the same: Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from decimal import Decimal >>> Decimal('0.00') == Decimal('0.0') == Decimal('-0.00') True >>> Decimal('0.00') Decimal('0.00') >>> Decimal('0.0') Decimal('0.0') >>> Decimal('-0.00') Decimal('-0.00') I personally have a bunch of similar tests to make sure the "different" values are handled correctly. Chris From vernondcole at gmail.com Fri May 17 12:09:40 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Fri, 17 May 2013 04:09:40 -0600 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) Message-ID: I short time ago I commented to this forum that the 'format' paramstyle was important to keep, because of widespread use, citing django as an example. I have taken the question to the django developers group and received a response that they don't really like the 'format' paramstyle and would be happy to have a reason to change. I therefor rescind my previous objection. -- Vernon Cole -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri May 17 12:42:48 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 17 May 2013 12:42:48 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: Message-ID: <519609A8.6080407@egenix.com> On 17.05.2013 12:09, Vernon D. Cole wrote: > I short time ago I commented to this forum that the 'format' paramstyle was > important to keep, because of widespread use, citing django as an example. > > I have taken the question to the django developers group and received a > response that they don't really like the 'format' paramstyle and would be > happy to have a reason to change. > > I therefor rescind my previous objection. psychopg2 uses the 'format' paramstyle and while I agree that it has issues, I think the existing code base using it is large enough that we cannot easily remove that paramstyle :-( I'd love to hear what Daniele thinks about this. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46 2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Fri May 17 15:59:11 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Fri, 17 May 2013 09:59:11 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <519609A8.6080407@egenix.com> References: <519609A8.6080407@egenix.com> Message-ID: <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> On May 17, 2013, at 6:42 AM, M.-A. Lemburg wrote: > On 17.05.2013 12:09, Vernon D. Cole wrote: >> I short time ago I commented to this forum that the 'format' paramstyle was >> important to keep, because of widespread use, citing django as an example. >> >> I have taken the question to the django developers group and received a >> response that they don't really like the 'format' paramstyle and would be >> happy to have a reason to change. >> >> I therefor rescind my previous objection. > > psychopg2 uses the 'format' paramstyle and while I agree that > it has issues, I think the existing code base using > it is large enough that we cannot easily remove that > paramstyle :-( > > I'd love to hear what Daniele thinks about this. there's no reason a DBAPI can't keep a particular paramstyle, we just want to make it so that *all* DBAPIs definitely support "named" and "qmark". The rest are just optional. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, May 17 2013) >>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46 > 2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > http://mail.python.org/mailman/listinfo/db-sig From mal at egenix.com Fri May 17 16:43:07 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 17 May 2013 16:43:07 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> Message-ID: <519641FB.3080707@egenix.com> On 17.05.2013 15:59, Michael Bayer wrote: > > On May 17, 2013, at 6:42 AM, M.-A. Lemburg wrote: > >> On 17.05.2013 12:09, Vernon D. Cole wrote: >>> I short time ago I commented to this forum that the 'format' paramstyle was >>> important to keep, because of widespread use, citing django as an example. >>> >>> I have taken the question to the django developers group and received a >>> response that they don't really like the 'format' paramstyle and would be >>> happy to have a reason to change. >>> >>> I therefor rescind my previous objection. >> >> psychopg2 uses the 'format' paramstyle and while I agree that >> it has issues, I think the existing code base using >> it is large enough that we cannot easily remove that >> paramstyle :-( >> >> I'd love to hear what Daniele thinks about this. > > there's no reason a DBAPI can't keep a particular paramstyle, we just want to make it so that *all* DBAPIs definitely support "named" and "qmark". The rest are just optional. The original plan was to simplify the DB-API by reducing the number of possible paramstyles, but you're probably right: it's better to just require support for a few and leave other paramstyles as option to the module authors, if they choose to support more than just the two basic ones. That way people writing new code can focus on just those two styles, while existing applications can continue to work with the optional ones. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46 2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From vernondcole at gmail.com Fri May 17 17:01:29 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Fri, 17 May 2013 16:01:29 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> Message-ID: Good point, the others would be optional. How should the switching be done? My present development head has four methods: 1) adodbapi.apibase.paramstyle = 'named' I think this one should be deprecated. Stepping on a module quasi-constant is never a good idea. 2) adodbapi.connect(connection_string, paramstyle='named') I just added this one, so a django adapter can hand me a bucketfull of keyword arguments. I think I like this the best in terms of ease-of-use. 3) conn = adodbapi.connect(connection_string) conn.paramstyle = 'named' This is the most obvious, and provides lots of power. The programmer could use 'qmark' to load big INSERT statements, then switch to 'named' for detailed SELECT or UPDATE. The default value is determined by the module author. 4) cursor.paramstyle = 'named' I have this implemented, but the more I think about it, the more it seems unnecessary. What other options should be considered? -- Vernon On Fri, May 17, 2013 at 2:59 PM, Michael Bayer wrote: > > On May 17, 2013, at 6:42 AM, M.-A. Lemburg wrote: > > > On 17.05.2013 12:09, Vernon D. Cole wrote: > >> I short time ago I commented to this forum that the 'format' paramstyle > was > >> important to keep, because of widespread use, citing django as an > example. > >> > >> I have taken the question to the django developers group and received a > >> response that they don't really like the 'format' paramstyle and would > be > >> happy to have a reason to change. > >> > >> I therefor rescind my previous objection. > > > > psychopg2 uses the 'format' paramstyle and while I agree that > > it has issues, I think the existing code base using > > it is large enough that we cannot easily remove that > > paramstyle :-( > > > > I'd love to hear what Daniele thinks about this. > > there's no reason a DBAPI can't keep a particular paramstyle, we just want > to make it so that *all* DBAPIs definitely support "named" and "qmark". > The rest are just optional. > > > > > > > > -- > > Marc-Andre Lemburg > > eGenix.com > > > > Professional Python Services directly from the Source (#1, May 17 2013) > >>>> Python Projects, Consulting and Support ... http://www.egenix.com/ > >>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > > ________________________________________________________________________ > > 2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46 > > 2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45 > > > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > > Registered at Amtsgericht Duesseldorf: HRB 46611 > > http://www.egenix.com/company/contact/ > > _______________________________________________ > > DB-SIG maillist - DB-SIG at python.org > > http://mail.python.org/mailman/listinfo/db-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cito at online.de Fri May 17 17:10:42 2013 From: cito at online.de (Christoph Zwerschke) Date: Fri, 17 May 2013 17:10:42 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> Message-ID: <51964872.30603@online.de> Am 17.05.2013 17:01, schrieb Vernon D. Cole: > What other options should be considered? Another option would be to get rid of the parameter completely, and silently accept both styles, whatever is used in the sql command passed to the execute method. -- Christoph From mike_mp at zzzcomputing.com Fri May 17 17:10:50 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Fri, 17 May 2013 11:10:50 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> Message-ID: On May 17, 2013, at 11:01 AM, Vernon D. Cole wrote: > Good point, the others would be optional. > > How should the switching be done? My present development head has four methods: > > 1) adodbapi.apibase.paramstyle = 'named' > I think this one should be deprecated. Stepping on a module quasi-constant is never a good idea. > > 2) adodbapi.connect(connection_string, paramstyle='named') > I just added this one, so a django adapter can hand me a bucketfull of keyword arguments. I think I like this the best in terms of ease-of-use. > > 3) conn = adodbapi.connect(connection_string) > conn.paramstyle = 'named' > This is the most obvious, and provides lots of power. The programmer could use 'qmark' to load big INSERT statements, then switch to 'named' for detailed SELECT or UPDATE. The default value is determined by the module author. > > 4) cursor.paramstyle = 'named' > I have this implemented, but the more I think about it, the more it seems unnecessary. > > What other options should be considered? My vote is at the module level: from psycopg2 import dbapi2 vs. from psycopg2 import dbapi3 and if you are using dbapi3, it only supports named and qmark. If you're in dbapi2, then you can have "paramstyle" keywords and all that. I'm hoping for dbapi3 to be more consistent and less switchy. From mike_mp at zzzcomputing.com Fri May 17 17:15:36 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Fri, 17 May 2013 11:15:36 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <51964872.30603@online.de> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> Message-ID: <0CB6BDFE-8E93-4C21-86DF-041A2E3BB625@zzzcomputing.com> On May 17, 2013, at 11:10 AM, Christoph Zwerschke wrote: > Am 17.05.2013 17:01, schrieb Vernon D. Cole: > > What other options should be considered? > > Another option would be to get rid of the parameter completely, and silently accept both styles, whatever is used in the sql command passed to the execute method. that's how dbapi3 should work for sure. for dbapi2 compatibility, some kind of switching would be needed. From daniele.varrazzo at gmail.com Fri May 17 17:33:59 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Fri, 17 May 2013 16:33:59 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <51964872.30603@online.de> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> Message-ID: On Fri, May 17, 2013 at 4:10 PM, Christoph Zwerschke wrote: > > Another option would be to get rid of the parameter completely, and silently > accept both styles, whatever is used in the sql command passed to the > execute method. This is impossible: cur.execute("""Select 'Guess how many params this query has?? ? ? %s';""", args) Please, to everybody involved in this discussion: stop thinking that giving more choices makes people's life easier. No: it makes it more complicated, whatever the person in question is a database driver author, a single database user or a multi-library database author. -- Daniele From vernondcole at gmail.com Fri May 17 17:36:08 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Fri, 17 May 2013 16:36:08 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <0CB6BDFE-8E93-4C21-86DF-041A2E3BB625@zzzcomputing.com> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <0CB6BDFE-8E93-4C21-86DF-041A2E3BB625@zzzcomputing.com> Message-ID: If there are only the two styles, you can tell which is expected by looking at the parameters. If you have a sequence, you use 'qmark', if a mapping, you use 'named'. Follow the example of string.format() which will accept either keywords or positional args. Attempting to determine the style by scanning the SQL is expensive and not a sure thing. On Fri, May 17, 2013 at 4:15 PM, Michael Bayer wrote: > > On May 17, 2013, at 11:10 AM, Christoph Zwerschke wrote: > > > Am 17.05.2013 17:01, schrieb Vernon D. Cole: > > > What other options should be considered? > > > > Another option would be to get rid of the parameter completely, and > silently accept both styles, whatever is used in the sql command passed to > the execute method. > > that's how dbapi3 should work for sure. for dbapi2 compatibility, some > kind of switching would be needed. > > > > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > http://mail.python.org/mailman/listinfo/db-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri May 17 17:36:30 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 17 May 2013 17:36:30 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <51964872.30603@online.de> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> Message-ID: <51964E7E.9010507@egenix.com> On 17.05.2013 17:10, Christoph Zwerschke wrote: > Am 17.05.2013 17:01, schrieb Vernon D. Cole: >> What other options should be considered? > > Another option would be to get rid of the parameter completely, and silently accept both styles, > whatever is used in the sql command passed to the execute method. This would violate explicit is better than implicit, and it also creates problems with raising proper errors, e.g. passing a dictionary of parameters to a SQL command which uses a mix of qmark and named style parameter markers. We do need to keep this explicit. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46 2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From vernondcole at gmail.com Fri May 17 17:44:32 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Fri, 17 May 2013 16:44:32 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <51964E7E.9010507@egenix.com> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> Message-ID: On Fri, May 17, 2013 at 4:36 PM, M.-A. Lemburg wrote: > e.g. passing a > dictionary of parameters to a SQL command which uses a mix > of qmark and named style parameter markers. > That is a plain, simple syntax error, and will be reported as such by the SQL engine, along with a million other syntax errors that I have no way of detecting in Python. The error reporting is the same in all cases: "SQL does not like your stuff." -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri May 17 17:45:04 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 17 May 2013 17:45:04 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> Message-ID: <51965080.5040605@egenix.com> On 17.05.2013 17:01, Vernon D. Cole wrote: > Good point, the others would be optional. > > How should the switching be done? My present development head has four > methods: > > 1) adodbapi.apibase.paramstyle = 'named' > I think this one should be deprecated. Stepping on a module quasi-constant > is never a good idea. Not a good idea, agreed. It's very difficult for a database module to detect changes like these to e.g. detect wrong parameter values. Example: module.paramstyle = 'oracle' would likely not generate an exception, but a subsequent query then would and you'd have to search your application code for the assignment line, provided the database module tells you what to look for :-) > 2) adodbapi.connect(connection_string, paramstyle='named') > I just added this one, so a django adapter can hand me a bucketfull of > keyword arguments. I think I like this the best in terms of ease-of-use. Interesting idea. > 3) conn = adodbapi.connect(connection_string) > conn.paramstyle = 'named' > This is the most obvious, and provides lots of power. The programmer could > use 'qmark' to load big INSERT statements, then switch to 'named' for > detailed SELECT or UPDATE. The default value is determined by the module > author. > > 4) cursor.paramstyle = 'named' > I have this implemented, but the more I think about it, the more it seems > unnecessary. Those two are the most natural in terms of how the rest of the DB-API works, so +1 on those. I think we already agreed on this approach being added as optional extension to DB-API 2.0, but could be wrong. > What other options should be considered? I think those are sufficient to configure your application to whatever param style you need/like best. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46 2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From daniele.varrazzo at gmail.com Fri May 17 17:50:40 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Fri, 17 May 2013 16:50:40 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <519609A8.6080407@egenix.com> References: <519609A8.6080407@egenix.com> Message-ID: Sorry, originally sent this message only to M.-A.L. On Fri, May 17, 2013 at 11:42 AM, M.-A. Lemburg wrote: > psychopg2 uses the 'format' paramstyle and while I agree that > it has issues, I think the existing code base using > it is large enough that we cannot easily remove that > paramstyle :-( > > I'd love to hear what Daniele thinks about this. Of course switching psycopg to named/qmark placeholders means immediately destroying the psycopg compatibility with 100% of the applications written using it. I cannot imagine earning any love from our user base for this choice. Ditto for the mysql driver and I guess others. Reading back the upstream thread I also see proposal to leave support for all the current placeholders but mandate drivers to implement qmark and named too. I think this is only going to complicate the implementation of the drivers and of the programs using it. Even making paramstyle a connection or cursor property there will always be the case of a connection passed to library functions written for the previous param style. Every function taking a connection as input should check what is the paramstyle used and dynamically generate a query, instead of using a constant string. Or get into a pattern of storing the paramstyle, switch to a known one, run the query and go back to the original one. Also note that the paramstyle indication leaves unspecified how to disambiguate the placeholder "magic" character. How do I hardcode a "?" in a query for a qmark driver? "\?" or "??". The specs don't say it, and afaik drivers may have solved this problem in different ways, which doesn't make interoperability magically happen. Likewise how do you escape the colon in the named format? For extra fun, postgres has a widely used "::" cast operator, applying which would become :value::::integer. Or :value\:\:integer (to be written in a raw string, otherwise it better be :value\\:\\:integer). At least the format/pyformat style borrow a well defined escape syntax from the python spec. Also, what characters is the placeholder name made of? What is the placeholder name in :fun() or in :array[3]? And in ":vari?t?"? What in :value3? And is :3value a numeric or a named placeholder? In my opinion everything but format/pyformat just too underspecified to be really sound. Also note that the original reason different paramstyles exist is that different client libraries use different placeholders (qmark for sqlite, named for oracle...) and using them the Python module can avoid conversions, just passing-through the query. What if the dbapi decides to define a placeholder syntax that happens to be different from the underlying driver? A subtle conversion would now be required. In this line of thought please note that the parameter required by the postgres libraries is $1, $2, ... not supported by the dbapi. I'm not advocating for their introduction: I just want to point out that qmark exists because it was handy for sqlite and named exists because it was handy for oracle: making them mandatory for drivers that natively support other placeholder formats means taking an implementation detail of a random database and make a mandatory conversion for all the others. While at it, I also want to point out that the entire idea of the "paramstyle" module attribute is not sufficient to describe the behaviour of the drivers that implement more than one paramstyle, such as the proposed qmark/named or the widespread format/pyformat. My (provocative) proposal for the dbapi3 is to drop the paramstyle attribute altogether and suggest the driver users to read the manual to know how to pass parameters to the queries. If that's too provocative I think the Python dbapi should mandate the %s and %(name)s formats because they are the only ones to have well defined syntax and escaping rule and are well known to every python developer. All the other formats are implementation details of the underlying drivers, so it would be pointless to standardize on them. At which point the paramstyle again becomes redundant and can be dropped. Also note that it would be easy for a driver to convert %s and %(name)s into ? or :1 or :name applying a regexp; not so much the contrary (because of the above ambiguities). The absolutely worse decision that could be made defining an API is to mandate one format but allow something else too: this would definitely not help interoperability and would create confusion even in environments where a single database and a single driver for it is used. If a driver wanted to expose a pass-through cursor using whatever format the underlying driver takes it should be so outside the DBAPI. As an example, psycopg would use a different cursor subclass to allow $1 parameters, but this is strictly a consequence of psycopg architecture and may easily be different for other drivers. -- Daniele From mal at egenix.com Fri May 17 17:57:31 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 17 May 2013 17:57:31 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> Message-ID: <5196536B.5070408@egenix.com> On 17.05.2013 17:44, Vernon D. Cole wrote: > On Fri, May 17, 2013 at 4:36 PM, M.-A. Lemburg wrote: > >> e.g. passing a >> dictionary of parameters to a SQL command which uses a mix >> of qmark and named style parameter markers. >> > > That is a plain, simple syntax error, and will be reported as such by the > SQL engine, along with a million other syntax errors that I have no way of > detecting in Python. The error reporting is the same in all cases: "SQL > does not like your stuff." Not necessarily, since the database module will likely convert the SQL to one of the two param styles, so you end up with a SQL string that contains both the converted param markers and the ones you had in the original SQL string. Detecting the used paramstyle from the SQL statement is really difficult, so modules would more likely used the parameter object type to determine the paramstyle, but this also has it's problems: For qmark, you'd look for a sequence, for named, you'd look for a mapping. There are situations (e.g. in ORMs) where the parameter object actually supports both interface types. Which brings us to "In the face of ambiguity, refuse the temptation to guess." :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46 2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Fri May 17 18:07:18 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Fri, 17 May 2013 12:07:18 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <51964E7E.9010507@egenix.com> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> Message-ID: <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> On May 17, 2013, at 11:36 AM, M.-A. Lemburg wrote: > On 17.05.2013 17:10, Christoph Zwerschke wrote: >> Am 17.05.2013 17:01, schrieb Vernon D. Cole: >>> What other options should be considered? >> >> Another option would be to get rid of the parameter completely, and silently accept both styles, >> whatever is used in the sql command passed to the execute method. > > This would violate explicit is better than implicit, and it also > creates problems with raising proper errors, e.g. passing a > dictionary of parameters to a SQL command which uses a mix > of qmark and named style parameter markers. > > We do need to keep this explicit. IMHO it is explicit, whether you say: cursor.execute(stmt, { < dictionary>} ) vs. cursor.execute(stmt, [ ] ) the type of parameters passed indicates the style of params to search for. It's explicit via type inference. if a dictionary is passed, "?" symbols are left alone. If a list is passed, ":param" symbols are left alone. If neither is passed, then neither ? nor :param are searched for. From daniele.varrazzo at gmail.com Fri May 17 18:15:06 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Fri, 17 May 2013 17:15:06 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> Message-ID: On Fri, May 17, 2013 at 5:07 PM, Michael Bayer wrote: > > On May 17, 2013, at 11:36 AM, M.-A. Lemburg wrote: > >> On 17.05.2013 17:10, Christoph Zwerschke wrote: >>> Am 17.05.2013 17:01, schrieb Vernon D. Cole: >>>> What other options should be considered? >>> >>> Another option would be to get rid of the parameter completely, and silently accept both styles, >>> whatever is used in the sql command passed to the execute method. >> >> This would violate explicit is better than implicit, and it also >> creates problems with raising proper errors, e.g. passing a >> dictionary of parameters to a SQL command which uses a mix >> of qmark and named style parameter markers. >> >> We do need to keep this explicit. > > > IMHO it is explicit, whether you say: > > cursor.execute(stmt, { < dictionary>} ) > > vs. > > cursor.execute(stmt, [ ] ) > > the type of parameters passed indicates the style of params to search for. It's explicit via type inference. if a dictionary is passed, "?" symbols are left alone. If a list is passed, ":param" symbols are left alone. If neither is passed, then neither ? nor :param are searched for. This only resolves the ambiguity between positional or mapping arguments. It doesn't help a driver to disambiguate between two different mapping styles, such as named or pyformat. -- Daniele From cito at online.de Fri May 17 18:17:58 2013 From: cito at online.de (Christoph Zwerschke) Date: Fri, 17 May 2013 18:17:58 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> Message-ID: <51965836.5030908@online.de> Am 17.05.2013 17:33, schrieb Daniele Varrazzo:> On Fri, May 17, 2013 at 4:10 PM, Christoph Zwerschke wrote: >> >> Another option would be to get rid of the parameter completely, and silently >> accept both styles, whatever is used in the sql command passed to the >> execute method. > > This is impossible: > > cur.execute("""Select 'Guess how many params this query has?? ? ? %s';""", args) Hm, I forgot DBAPI does not care about SQL; it replaces parameters even inside SQL strings. So then, you're right, it can be ambiguous. By the way, this is really unclear from the DBAPI 2 documentation: The example in the dbapi 2 docs is "WHERE name=?" and "WHERE name=%s" which seems to indicate that the value is automatically put in quotes, particularly in view of footnote 5 which says "The client should not be required to "escape" the value so that it can be used ? the value should be equal to the actual database value." In this example this means, the value would be a string without surrounding quotes. The example clause should then be "WHERE name='?'" and "WHERE name='%s'". Maybe this should be changed in DBAPI 3? This would allow the driver to use prepared statements under the hood. -- Chris From daniele.varrazzo at gmail.com Fri May 17 18:21:28 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Fri, 17 May 2013 17:21:28 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <51965836.5030908@online.de> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51965836.5030908@online.de> Message-ID: On Fri, May 17, 2013 at 5:17 PM, Christoph Zwerschke wrote: > > Hm, I forgot DBAPI does not care about SQL; it replaces parameters even > inside SQL strings. So then, you're right, it can be ambiguous. > > By the way, this is really unclear from the DBAPI 2 documentation: > > The example in the dbapi 2 docs is "WHERE name=?" and "WHERE name=%s" which > seems to indicate that the value is automatically put in quotes, > particularly in view of footnote 5 which says "The client should not be > required to "escape" the value so that it can be used ? the value should be > equal to the actual database value." In this example this means, the value > would be a string without surrounding quotes. The example clause should then > be "WHERE name='?'" and "WHERE name='%s'". Quite the contrary: the quotes should never appear in the query and these examples are correct but... > Maybe this should be changed in DBAPI 3? This would allow the driver to use > prepared statements under the hood. ... I would limit this thread to the issue of the paramstyle and leave everything wished for dbapi3 to a different thread. -- Daniele From mike_mp at zzzcomputing.com Fri May 17 18:22:58 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Fri, 17 May 2013 12:22:58 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> Message-ID: <1DAC1699-088F-4F78-A441-75E12DD68E30@zzzcomputing.com> On May 17, 2013, at 11:50 AM, Daniele Varrazzo wrote: > Sorry, originally sent this message only to M.-A.L. > > On Fri, May 17, 2013 at 11:42 AM, M.-A. Lemburg wrote: >> psychopg2 uses the 'format' paramstyle and while I agree that >> it has issues, I think the existing code base using >> it is large enough that we cannot easily remove that >> paramstyle :-( >> >> I'd love to hear what Daniele thinks about this. > > Of course switching psycopg to named/qmark placeholders means > immediately destroying the psycopg compatibility with 100% of the > applications written using it. I cannot imagine earning any love from > our user base for this choice. Ditto for the mysql driver and I guess > others. we're talking here about DBAPI3. psycopg2 would offer this as a new name within the module namespace. The existing DBAPI2 system would remain unaffected. > > Reading back the upstream thread I also see proposal to leave support > for all the current placeholders but mandate drivers to implement > qmark and named too. I think this is only going to complicate the > implementation of the drivers and of the programs using it. Drivers written against the DBAPI3 spec should only support qmark and named. Existing DBAPI2 drivers can remain supporting whatever they do now. > Even > making paramstyle a connection or cursor property there will always be > the case of a connection passed to library functions written for the > previous param style. > Every function taking a connection as input > should check what is the paramstyle used and dynamically generate a > query, instead of using a constant string. Or get into a pattern of > storing the paramstyle, switch to a known one, run the query and go > back to the original one. I agree, making paramstyle a settable attribute is not helping anyone. > Also note that the paramstyle indication leaves unspecified how to > disambiguate the placeholder "magic" character. How do I hardcode a > "?" in a query for a qmark driver? "\?" or "??". The specs don't say > it, and afaik drivers may have solved this problem in different ways, > which doesn't make interoperability magically happen. this would be great, I already have to struggle quite a bit with percent signs among drivers that accept "format" vs. those that don't vs. those that sometimes do. > Likewise how do > you escape the colon in the named format? For extra fun, postgres has > a widely used "::" cast operator, applying which would become > :value::::integer. Or :value\:\:integer (to be written in a raw > string, otherwise it better be :value\\:\\:integer). We should probably look to see what JDBC does as it supports both ? and :named. > At least the > format/pyformat style borrow a well defined escape syntax from the > python spec. The major issue with format/pyformat is that too many DBAPIs take the easy way out and just run their string through Python's "%" operator. This causes DBAPIs that use the format styles to behave differently from those that don't. In particular, a problem I have with MySQL-python and *maybe* recent versions of psycopg2 (would have to check again) is that the behavior changes whether or not I pass parameters to cursor.execute(). The same statement that includes a "%" sign as part of the string must be passed without escaping when no list/dictionary is passed as the second argument, but when a list/dictionary is passed, the parsing changes and now the % sign has to be escaped normally. This is an undocumented behavior that's directly a side effect of the DBAPI saying, "oh there's parameters, let's call "stmt % params", vs. not. I'd prefer every DBAPI use extremely similar parsing routines, so that escaping behavior and all of that is completely consistent. > Also, what characters is the placeholder name made of? > What is the placeholder name in :fun() or in :array[3]? And in > ":vari?t?"? What in :value3? And is :3value a numeric or a named > placeholder? In my opinion everything but format/pyformat just too > underspecified to be really sound. see JDBC > > Also note that the original reason different paramstyles exist is that > different client libraries use different placeholders (qmark for > sqlite, named for oracle...) and using them the Python module can > avoid conversions, just passing-through the query. What if the dbapi > decides to define a placeholder syntax that happens to be different > from the underlying driver? A subtle conversion would now be required. having six parameter styles is extremely inconvenient, and there is no DBAPI I'm aware of that doesn't end up doing some kind of conversion in any case. > In this line of thought please note that the parameter required by the > postgres libraries is $1, $2, ... not supported by the dbapi. I'm not > advocating for their introduction: I just want to point out that qmark > exists because it was handy for sqlite and named exists because it was > handy for oracle: making them mandatory for drivers that natively > support other placeholder formats means taking an implementation > detail of a random database and make a mandatory conversion for all > the others. qmark and colon are widely accepted standard formats I've seen in use since the 90's > > While at it, I also want to point out that the entire idea of the > "paramstyle" module attribute is not sufficient to describe the > behaviour of the drivers that implement more than one paramstyle, such > as the proposed qmark/named or the widespread format/pyformat. a DBAPI3 that supports only qmark and named, based on the type of collection passed to execute(), no longer needs a ".paramstyle" attribute at all. > > My (provocative) proposal for the dbapi3 is to drop the paramstyle > attribute altogether and suggest the driver users to read the manual > to know how to pass parameters to the queries. Agree on drop the paramstyle, disagree on "read the manual" - the enormous issue with DBAPI2 is how all the drivers make up their own conventions and behaviors for what should be entirely consistent systems. I shouldn't have to read any DBAPI2 manual to be able to use a particular DBAPI in plain "vanilla" mode - the spec should be all that's needed, unless I would like to use specific features offered by that DBAPI. > > The absolutely worse decision that could be made defining an API is to > mandate one format but allow something else too: this would definitely > not help interoperability and would create confusion even in > environments where a single database and a single driver for it is > used. agree, I'd prefer fewer "choices" in DBAPI, and it's all the "optional" things allowed now that turns it into the wild west. I think DBAPI3 should be presented as a new module within DBAPIs, using the format "from import dbapi3", presenting a "connect()" function in dbapi3, and only dbapi3 behaviors present in that namespace. It would be absolutely great if I could install any of 20 different DBAPIs, and for each one call upon "from import dbapi3; dbapi3.connect()" in exactly the same way. From mike_mp at zzzcomputing.com Fri May 17 18:23:45 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Fri, 17 May 2013 12:23:45 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> Message-ID: On May 17, 2013, at 12:15 PM, Daniele Varrazzo wrote: > On Fri, May 17, 2013 at 5:07 PM, Michael Bayer wrote: >> >> >> >> IMHO it is explicit, whether you say: >> >> cursor.execute(stmt, { < dictionary>} ) >> >> vs. >> >> cursor.execute(stmt, [ ] ) >> >> the type of parameters passed indicates the style of params to search for. It's explicit via type inference. if a dictionary is passed, "?" symbols are left alone. If a list is passed, ":param" symbols are left alone. If neither is passed, then neither ? nor :param are searched for. > > This only resolves the ambiguity between positional or mapping > arguments. It doesn't help a driver to disambiguate between two > different mapping styles, such as named or pyformat. the proposal includes that only qmark and named are available. There would be no pyformat, format, numeric. From cito at online.de Fri May 17 18:33:17 2013 From: cito at online.de (Christoph Zwerschke) Date: Fri, 17 May 2013 18:33:17 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51965836.5030908@online.de> Message-ID: <51965BCD.4040300@online.de> Am 17.05.2013 18:21, schrieb Daniele Varrazzo: > Quite the contrary: the quotes should never appear in the query and > these examples are correct but... Just tested this with psycopg2. If you use the clause "WHERE name=%s" and pass a value without quotes, you get a ProgrammingError from the database. So is psycopg2 not properly implementing DBAPI 2? -- Christoph From daniele.varrazzo at gmail.com Fri May 17 18:36:35 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Fri, 17 May 2013 17:36:35 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> Message-ID: On Fri, May 17, 2013 at 5:23 PM, Michael Bayer wrote: > > the proposal includes that only qmark and named are available. There would be no pyformat, format, numeric. What you said before doesn't seem the same: On Fri, May 17, 2013 at 2:59 PM, Michael Bayer wrote: > there's no reason a DBAPI can't keep a particular paramstyle, we just want to make it so that *all* DBAPIs definitely support "named" and "qmark". The rest are just optional. You seem suggesting psycopg MUST offer qmark and MAY (at its own discretion) offer pyformat. No: leaving this choice open is insanely bad. If dbapi wants to take a position it should mandate one format with a carefully specified syntax and reject any other formats. If a driver wanted to implement support for a different placeholder syntax (e.g. for query pass-through) it must be do such outside the dbapi realm and tools like SQLAlchemy can be implemented using the DBAPI part only, without resorting to driver-specific dialects. -- Daniele From mike_mp at zzzcomputing.com Fri May 17 18:39:54 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Fri, 17 May 2013 12:39:54 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> Message-ID: <86EAA918-3892-47F3-8E61-008FA93BB430@zzzcomputing.com> On May 17, 2013, at 11:50 AM, Daniele Varrazzo wrote: > > If that's too provocative I think the Python dbapi should mandate the > %s and %(name)s formats because they are the only ones to have well > defined syntax and escaping rule and are well known to every python > developer. I forgot also, not to mention that Python itself is moving away from "%" as a formatting operator, in favor of the newer .format() method which has yet *another* syntax: see pep 3101: http://www.python.org/dev/peps/pep-3101/ http://docs.python.org/3/library/string.html#formatspec I love the format and pyformat styles a lot for Python programming but I feel like they cause a lot of confusion when they are additionally interpreted by a DBAPI. It runs you into various "double-escaping" types of situations. From mike_mp at zzzcomputing.com Fri May 17 18:41:04 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Fri, 17 May 2013 12:41:04 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <51965BCD.4040300@online.de> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51965836.5030908@online.de> <51965BCD.4040300@online.de> Message-ID: <1F0C3C44-7DC2-4897-AF7E-8D05E96143A9@zzzcomputing.com> On May 17, 2013, at 12:33 PM, Christoph Zwerschke wrote: > Am 17.05.2013 18:21, schrieb Daniele Varrazzo: > > Quite the contrary: the quotes should never appear in the query and > > these examples are correct but... > > Just tested this with psycopg2. If you use the clause "WHERE name=%s" and pass a value without quotes, you get a ProgrammingError from the database. So is psycopg2 not properly implementing DBAPI 2? right, did you pass any parameters to execute()? I believe if no parameters are present, it decides not to do the "%" operation. Other DBAPIs that don't use format/pyformat don't have this behavior. From daniele.varrazzo at gmail.com Fri May 17 18:45:10 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Fri, 17 May 2013 17:45:10 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <86EAA918-3892-47F3-8E61-008FA93BB430@zzzcomputing.com> References: <519609A8.6080407@egenix.com> <86EAA918-3892-47F3-8E61-008FA93BB430@zzzcomputing.com> Message-ID: On Fri, May 17, 2013 at 5:39 PM, Michael Bayer wrote: > > On May 17, 2013, at 11:50 AM, Daniele Varrazzo wrote: > >> >> If that's too provocative I think the Python dbapi should mandate the >> %s and %(name)s formats because they are the only ones to have well >> defined syntax and escaping rule and are well known to every python >> developer. > > I forgot also, not to mention that Python itself is moving away from "%" as a formatting operator, in favor of the newer .format() method which has yet *another* syntax: see pep 3101: > > http://www.python.org/dev/peps/pep-3101/ > > http://docs.python.org/3/library/string.html#formatspec > > I love the format and pyformat styles a lot for Python programming but I feel like they cause a lot of confusion when they are additionally interpreted by a DBAPI. It runs you into various "double-escaping" types of situations. I purposely avoided calling format() in cause as a suggestion to move there. The one supported by format() is to allow richness in the way the values are merged to the format string, whereas for the dbapi the driver knows better than the user. For example the driver only needs support for %s, not for %d or for %0.2f. -- Daniele From mike_mp at zzzcomputing.com Fri May 17 18:46:20 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Fri, 17 May 2013 12:46:20 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> Message-ID: <95BCD002-12C5-47E1-A9D8-F086A36BF36C@zzzcomputing.com> On May 17, 2013, at 12:36 PM, Daniele Varrazzo wrote: > What you said before doesn't seem the same: > > On Fri, May 17, 2013 at 2:59 PM, Michael Bayer wrote: > >> there's no reason a DBAPI can't keep a particular paramstyle, we just want to make it so that *all* DBAPIs definitely support "named" and "qmark". The rest are just optional. > > You seem suggesting psycopg MUST offer qmark and MAY (at its own > discretion) offer pyformat. No: leaving this choice open is insanely > bad. If dbapi wants to take a position it should mandate one format > with a carefully specified syntax and reject any other formats. > > If a driver wanted to implement support for a different placeholder > syntax (e.g. for query pass-through) it must be do such outside the > dbapi realm and tools like SQLAlchemy can be implemented using the > DBAPI part only, without resorting to driver-specific dialects. yes, you are correct. So please modify my statement to read: "there's no reason a DBAPI can't keep their DBAPI2 implementation available, whatever it does now, and also provide a DBAPI3-compliant adapter via "from import dbapi3". that *only* provides qmark and named via a well-known usage pattern." So I have modified my position of an hour ago to include that DBAPI3 drivers should be entirely in their own module namespace. SQLAlchemy can of course dance around whatever the DBAPI decides to do here, as far as SQLA I'm just trying to push the spec away from ambiguous edge cases such as having to conditionally escape "%" characters based not just on driver but on whether parameters are present. Our issue is more about the new and interesting surprises every single new DBAPI brings us, so to that extent a more consistent and minimal DBAPI spec would hopefully help. From vernondcole at gmail.com Fri May 17 18:46:42 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Fri, 17 May 2013 17:46:42 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <5196536B.5070408@egenix.com> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> <5196536B.5070408@egenix.com> Message-ID: One of the things I like about Python is that I do not have to declare my data types in advance, as long as I use them correctly. I _like_ the idea of not having to pre-set my parameter style. (Having a mix of styles in one SQL statement is just wrong and need not be considered.) Having said that, if a module wants to support more than two -- which may often be the case -- there _must_ be a way to do the pre-set. So what if pre-sets were optional, and automatic selection based on parameter characteristics were the default? I could easily add that to my existing code by auto-switching when I see connection.paramstyle == NotImplemented. If anyone (like Danielle or myself) wants to continue supporting odd-ball paramstyles (i.e. anything other than the approved pair) then use an appropriate pre-set. If connection.paramstyle == None then hand it straight to the engine without looking. If one of the (previously defined) standard values, then reformat as needed, if needed. I am not trying to complicate a version 3 api -- but I also do not want to support two code bases -- so I want my version 3 module to be able to do anything my version 2 module does, including handle (newly deprecated or version specific) parameter styles. But I want that old behaviour to be optional. Think of a "from __past__ import paramstyle" feature. ;-) -- Vernon Cole On Fri, May 17, 2013 at 4:57 PM, M.-A. Lemburg wrote: > On 17.05.2013 17:44, Vernon D. Cole wrote: > > On Fri, May 17, 2013 at 4:36 PM, M.-A. Lemburg wrote: > > > >> e.g. passing a > >> dictionary of parameters to a SQL command which uses a mix > >> of qmark and named style parameter markers. > >> > > > > That is a plain, simple syntax error, and will be reported as such by the > > SQL engine, along with a million other syntax errors that I have no way > of > > detecting in Python. The error reporting is the same in all cases: "SQL > > does not like your stuff." > > Not necessarily, since the database module will likely convert > the SQL to one of the two param styles, so you end up with > a SQL string that contains both the converted param markers > and the ones you had in the original SQL string. > > Detecting the used paramstyle from the SQL statement is really > difficult, so modules would more likely used the parameter > object type to determine the paramstyle, but this also has it's > problems: > > For qmark, you'd look for a sequence, for named, you'd look > for a mapping. There are situations (e.g. in ORMs) where the > parameter object actually supports both interface types. > > Which brings us to "In the face of ambiguity, refuse the temptation > to guess." :-) > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, May 17 2013) > >>> Python Projects, Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46 > 2013-05-06 : Released mxODBC 3.2.3 ... > http://egenix.com/go45 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniele.varrazzo at gmail.com Fri May 17 18:41:58 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Fri, 17 May 2013 17:41:58 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <51965BCD.4040300@online.de> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51965836.5030908@online.de> <51965BCD.4040300@online.de> Message-ID: On Fri, May 17, 2013 at 5:33 PM, Christoph Zwerschke wrote: > Am 17.05.2013 18:21, schrieb Daniele Varrazzo: > >> Quite the contrary: the quotes should never appear in the query and >> these examples are correct but... > > Just tested this with psycopg2. If you use the clause "WHERE name=%s" and > pass a value without quotes, you get a ProgrammingError from the database. > So is psycopg2 not properly implementing DBAPI 2? You have tested it wrong, and haven't bothered reading the first few lines of the first page of the documentation of the driver library. -- Daniele From mal at egenix.com Fri May 17 18:57:07 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 17 May 2013 18:57:07 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> Message-ID: <51966163.5090507@egenix.com> On 17.05.2013 17:50, Daniele Varrazzo wrote: > Sorry, originally sent this message only to M.-A.L. > > On Fri, May 17, 2013 at 11:42 AM, M.-A. Lemburg wrote: >> psychopg2 uses the 'format' paramstyle and while I agree that >> it has issues, I think the existing code base using >> it is large enough that we cannot easily remove that >> paramstyle :-( >> >> I'd love to hear what Daniele thinks about this. > > Of course switching psycopg to named/qmark placeholders means > immediately destroying the psycopg compatibility with 100% of the > applications written using it. I cannot imagine earning any love from > our user base for this choice. Ditto for the mysql driver and I guess > others. Thought so :-) So I think we all agree that the existing paramstyles cannot simply be dropped. This leaves the option of making things easier for new users and existing users who write new code, by allowing them to choose which paramstyle they prefer instead of having the database modules mandate the style. > Reading back the upstream thread I also see proposal to leave support > for all the current placeholders but mandate drivers to implement > qmark and named too. I think this is only going to complicate the > implementation of the drivers and of the programs using it. Why is that ? > Even > making paramstyle a connection or cursor property there will always be > the case of a connection passed to library functions written for the > previous param style. Every function taking a connection as input > should check what is the paramstyle used and dynamically generate a > query, instead of using a constant string. Or get into a pattern of > storing the paramstyle, switch to a known one, run the query and go > back to the original one. The .paramstyle on the connection is only used as default value for the cursor.paramstyle. Functions that take a connection as argument and don't know which .paramstyle is being used by the application would then do: cursor = connection.cursor() cursor.paramstyle = 'named' cursor.execute(...) This also make the intention very clear in the function for someone who's reading the code. > Also note that the paramstyle indication leaves unspecified how to > disambiguate the placeholder "magic" character. How do I hardcode a > "?" in a query for a qmark driver? "\?" or "??". The specs don't say > it, and afaik drivers may have solved this problem in different ways, > which doesn't make interoperability magically happen. You normally don't have to escape '?' in SQL statements. It is allowed inside SQL string literals, but I've not seen it used outside those literals for anything other than a binding parameter marker. > Likewise how do > you escape the colon in the named format? The format usually is defined as ":[a-z0-9]+", with the additional requirement that the marker is not preceded by a colon, e.g. "::integer" is not a valid marker. > For extra fun, postgres has > a widely used "::" cast operator, applying which would become > :value::::integer. Or :value\:\:integer (to be written in a raw > string, otherwise it better be :value\\:\\:integer). At least the > format/pyformat style borrow a well defined escape syntax from the > python spec. Also, what characters is the placeholder name made of? > What is the placeholder name in :fun() or in :array[3]? And in > ":vari?t?"? What in :value3? And is :3value a numeric or a named > placeholder? In my opinion everything but format/pyformat just too > underspecified to be really sound. True, we would have to clarify the definitions of those parameter markers a bit. They are currently defined by the database driver or backend and only serve an informational purpose. > Also note that the original reason different paramstyles exist is that > different client libraries use different placeholders (qmark for > sqlite, named for oracle...) and using them the Python module can > avoid conversions, just passing-through the query. Aside: qmark is the ODBC standard and also used by MS SQL Server, Sybase and DB2 as native paramstyle. named is the native style used by Oracle. You're right, that the original motivation was to avoid having to parse/convert the SQL statements in the database module. However, given that it's not all that hard to write such conversion routines, I think we should reconsider the case and make things easier for the user. > What if the dbapi > decides to define a placeholder syntax that happens to be different > from the underlying driver? A subtle conversion would now be required. > In this line of thought please note that the parameter required by the > postgres libraries is $1, $2, ... not supported by the dbapi. I'm not > advocating for their introduction: I just want to point out that qmark > exists because it was handy for sqlite and named exists because it was > handy for oracle: making them mandatory for drivers that natively > support other placeholder formats means taking an implementation > detail of a random database and make a mandatory conversion for all > the others. We'd only make the support for 'named' and 'qmark' mandatory. Database modules would still be free to continue to also offer 'format' or 'pyformat' or even other paramstyles such as the $n one. > While at it, I also want to point out that the entire idea of the > "paramstyle" module attribute is not sufficient to describe the > behaviour of the drivers that implement more than one paramstyle, such > as the proposed qmark/named or the widespread format/pyformat. The module scope variable provides the default for all connections created with the module, so the sequence goes like this: module.paramstyle -> connection.paramstyle -> cursor.paramstyle > My (provocative) proposal for the dbapi3 is to drop the paramstyle > attribute altogether and suggest the driver users to read the manual > to know how to pass parameters to the queries. > > If that's too provocative I think the Python dbapi should mandate the > %s and %(name)s formats because they are the only ones to have well > defined syntax and escaping rule and are well known to every python > developer. All the other formats are implementation details of the > underlying drivers, so it would be pointless to standardize on them. > At which point the paramstyle again becomes redundant and can be > dropped. Also note that it would be easy for a driver to convert %s > and %(name)s into ? or :1 or :name applying a regexp; not so much the > contrary (because of the above ambiguities). Well, I guess you're biased on this one :-) We've already had the discussion and the poll on the paramstyle markers and consensus was to focus on qmark and named. > The absolutely worse decision that could be made defining an API is to > mandate one format but allow something else too: this would definitely > not help interoperability and would create confusion even in > environments where a single database and a single driver for it is > used. If a driver wanted to expose a pass-through cursor using > whatever format the underlying driver takes it should be so outside > the DBAPI. As an example, psycopg would use a different cursor > subclass to allow $1 parameters, but this is strictly a consequence of > psycopg architecture and may easily be different for other drivers. I don't understand this one. How would we make things more difficult for users if we mandate two of the already available paramstyles ? I think it would greatly simplify the use of the DB-API, since applications could then be written to use a single paramstyle, even when using multiple database modules. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46 2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Fri May 17 18:57:21 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Fri, 17 May 2013 12:57:21 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> <5196536B.5070408@egenix.com> Message-ID: On May 17, 2013, at 12:46 PM, "Vernon D. Cole" wrote: > One of the things I like about Python is that I do not have to declare my data types in advance, as long as I use them correctly. I _like_ the idea of not having to pre-set my parameter style. (Having a mix of styles in one SQL statement is just wrong and need not be considered.) > > Having said that, if a module wants to support more than two -- which may often be the case -- there _must_ be a way to do the pre-set. > > So what if pre-sets were optional, and automatic selection based on parameter characteristics were the default? > > I could easily add that to my existing code by auto-switching when I see connection.paramstyle == NotImplemented. If anyone (like Danielle or myself) wants to continue supporting odd-ball paramstyles (i.e. anything other than the approved pair) then use an appropriate pre-set. If connection.paramstyle == None then hand it straight to the engine without looking. If one of the (previously defined) standard values, then reformat as needed, if needed. > > I am not trying to complicate a version 3 api -- but I also do not want to support two code bases -- so I want my version 3 module to be able to do anything my version 2 module does, including handle (newly deprecated or version specific) parameter styles. But I want that old behaviour to be optional. Think of a "from __past__ import paramstyle" feature. ;-) you'd never need to support two entirely separate codebases. There's many ways to have an API present two different facades, calling down to a common set of functions. We're only talking about facades here. From mal at egenix.com Fri May 17 19:00:14 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 17 May 2013 19:00:14 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> Message-ID: <5196621E.2050005@egenix.com> On 17.05.2013 18:07, Michael Bayer wrote: > > On May 17, 2013, at 11:36 AM, M.-A. Lemburg wrote: > >> On 17.05.2013 17:10, Christoph Zwerschke wrote: >>> Am 17.05.2013 17:01, schrieb Vernon D. Cole: >>>> What other options should be considered? >>> >>> Another option would be to get rid of the parameter completely, and silently accept both styles, >>> whatever is used in the sql command passed to the execute method. >> >> This would violate explicit is better than implicit, and it also >> creates problems with raising proper errors, e.g. passing a >> dictionary of parameters to a SQL command which uses a mix >> of qmark and named style parameter markers. >> >> We do need to keep this explicit. > > > IMHO it is explicit, whether you say: > > cursor.execute(stmt, { < dictionary>} ) > > vs. > > cursor.execute(stmt, [ ] ) > > the type of parameters passed indicates the style of params to search for. It's explicit via type inference. if a dictionary is passed, "?" symbols are left alone. If a list is passed, ":param" symbols are left alone. If neither is passed, then neither ? nor :param are searched for. Hmm, then how do you tell whether this code is correct or not ? :-) cursor.execute('select * from mytable where id=?', parameters) Type inference also doesn't help universally, since objects can easily implement both the sequence and mapping ABC. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46 2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From cito at online.de Fri May 17 19:06:26 2013 From: cito at online.de (Christoph Zwerschke) Date: Fri, 17 May 2013 19:06:26 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51965836.5030908@online.de> <51965BCD.4040300@online.de> Message-ID: <51966392.9060905@online.de> Am 17.05.2013 18:41, schrieb Daniele Varrazzo: > You have tested it wrong, and haven't bothered reading the first few > lines of the first page of the documentation of the driver library. Yes, sorry for creating confusion today, I tested it wrong and psycopg2 *does* add the quotes. But then I wonder why you gave this example: Select 'Guess how many params this query has?? ? ? %s' Because if params must not be quoted this makes little sense. How do you expect the command above to be executed by the driver? Should the parameters now appear without quotes because they are already in a literal string? At least psycopg2 quotes them anyway, so you would have double quotes and a syntax error. So I think it's safe to assume or require that parameters appear only outside of strings and then it's not ambiguous any more, because ? and : cannot appear in SQL outside quotes. -- Chris From mal at egenix.com Fri May 17 19:30:47 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 17 May 2013 19:30:47 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <5196621E.2050005@egenix.com> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> <5196621E.2050005@egenix.com> Message-ID: <51966947.1040708@egenix.com> Ok, I think we've heard all sides. We have two extreme positions: 1. only allow qmark and named, let the driver figure out which one is used 2. only allow format/pyformat and the original idea of having an adjustable paramstyle, with the only requirement that modules must provide the two agreed styles qmark and named. I don't think the extreme positions are going to go anywhere. Both options break too much code and/or alienate database coders which are used to the database's native parameter style. As for how to adjust the paramstyle, I think Vernon's summary is a good one. SQLAlchemy and other applications could then do (*): import dbapi_module # We want named paramstyle on all connections: dbapi_module.paramstyle = 'named' and reduce the overhead of having to escape percent signs. The automatic paramstyle detection that has been suggested appears fragile to me, but perhaps it does have some merit. We could have an 'detect' paramstyle which tells the module to apply auto-detection of the used style (in whatever way it chooses). Still, I'm not really convinced this is a workable approach to solid database programming :-) Let's think about these things some more over the weekend - hopefully you all can enjoy some sunshine; it's awfully cold and rainy in Germany. (*) I know this contradicts what I wrote to Vernon. After thinking about it some more, I think the module level parameter should not go away and may actually serve a good purpose for defining the overall default of the module. The DB-API would have to make it clear, though, that adjusting the module level parameter will only have an effect on newly created connections. Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46 2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From vernondcole at gmail.com Fri May 17 19:30:34 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Fri, 17 May 2013 18:30:34 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <5196621E.2050005@egenix.com> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> <5196621E.2050005@egenix.com> Message-ID: On Fri, May 17, 2013 at 6:00 PM, M.-A. Lemburg wrote: > > IMHO it is explicit, whether you say: > > > > cursor.execute(stmt, { < dictionary>} ) > > > > vs. > > > > cursor.execute(stmt, [ ] ) > > > > the type of parameters passed indicates the style of params to search > for. It's explicit via type inference. if a dictionary is passed, "?" > symbols are left alone. If a list is passed, ":param" symbols are left > alone. If neither is passed, then neither ? nor :param are searched for. > > Hmm, then how do you tell whether this code is correct or not ? :-) > > cursor.execute('select * from mytable where id=?', parameters) > > Type inference also doesn't help universally, since objects can > easily implement both the sequence and mapping ABC. I don't need to "tell whether the code is correct". I am not writing a compiler -- just acting on what my user tells me to do. This happens at run time, so I will know at that moment whether the parameters passed to me have keys or not. If there are keys, I will run my "named-to-qmark" converter. If not, I will hand off the command as is (since my target engine expects qmark). There is no guess work here. Try for a key (and use 'named'). No key? Use the sequence (and use qmark). No sequence? Send the command as is. If the SQL engine complains about the result, tell the user. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike_mp at zzzcomputing.com Fri May 17 19:44:42 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Fri, 17 May 2013 13:44:42 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <51966947.1040708@egenix.com> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> <5196621E.2050005@egenix.com> <51966947.1040708@egenix.com> Message-ID: On May 17, 2013, at 1:30 PM, M.-A. Lemburg wrote: > As for how to adjust the paramstyle, I think Vernon's summary > is a good one. SQLAlchemy and other applications could then > do (*): > > import dbapi_module > # We want named paramstyle on all connections: > dbapi_module.paramstyle = 'named' > > and reduce the overhead of having to escape percent signs. just to confirm - that'll work on connections and cursors too, right? I wouldn't want to do it at the module level. From mal at egenix.com Fri May 17 19:51:23 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 17 May 2013 19:51:23 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> <5196621E.2050005@egenix.com> <51966947.1040708@egenix.com> Message-ID: <51966E1B.1040201@egenix.com> On 17.05.2013 19:44, Michael Bayer wrote: > > On May 17, 2013, at 1:30 PM, M.-A. Lemburg wrote: > >> As for how to adjust the paramstyle, I think Vernon's summary >> is a good one. SQLAlchemy and other applications could then >> do (*): >> >> import dbapi_module >> # We want named paramstyle on all connections: >> dbapi_module.paramstyle = 'named' >> >> and reduce the overhead of having to escape percent signs. > > just to confirm - that'll work on connections and cursors too, right? I wouldn't want to do it at the module level. Yes: """ The module scope variable provides the default for all connections created with the module, so the sequence goes like this: module.paramstyle -> connection.paramstyle -> cursor.paramstyle """ -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46 2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From vernondcole at gmail.com Fri May 17 19:52:19 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Fri, 17 May 2013 18:52:19 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <51966947.1040708@egenix.com> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> <5196621E.2050005@egenix.com> <51966947.1040708@egenix.com> Message-ID: I am worried about what the module-level attribute does to thread-safety. If this feature is kept, it should be a class attribute, not a module attribute. We should be doing connections using new style classes. conn = dbapi_module.Connection() conn.paramstyle = 'oddball' conn.connect(your, parameters, here) or dbapi_module.Connection.paramstyle = 'oddball' conn = dbapi_module.Connection().connect(parameters) -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniele.varrazzo at gmail.com Fri May 17 19:55:00 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Fri, 17 May 2013 18:55:00 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <51966947.1040708@egenix.com> References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> <5196621E.2050005@egenix.com> <51966947.1040708@egenix.com> Message-ID: On Fri, May 17, 2013 at 6:30 PM, M.-A. Lemburg wrote: > > We have two extreme positions: > > 1. only allow qmark and named, let the driver figure out which one > is used > > 2. only allow format/pyformat Just FYI, my position is not to only allow format/pyformat: as you said that would be too partisan :) My position is that it's too late to make such a choice. Now I'm leaving home but I should be able to better clarify my point across the weekend. -- Daniele From mal at egenix.com Fri May 17 20:12:49 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 17 May 2013 20:12:49 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> <51964E7E.9010507@egenix.com> <478B4CF1-396D-4F48-94AC-E2714205BA8E@zzzcomputing.com> <5196621E.2050005@egenix.com> <51966947.1040708@egenix.com> Message-ID: <51967321.3060604@egenix.com> On 17.05.2013 19:52, Vernon D. Cole wrote: > I am worried about what the module-level attribute does to thread-safety. We should clearly document the fact that the module level attribute serves as default value for new connections - it should really only be set right after importing the module or before opening any connections. > If this feature is kept, it should be a class attribute, not a module > attribute. We should be doing connections using new style classes. > > conn = dbapi_module.Connection() > conn.paramstyle = 'oddball' > conn.connect(your, parameters, here) > > or > > dbapi_module.Connection.paramstyle = 'oddball' > conn = dbapi_module.Connection().connect(parameters) Hmm, setting attributes on connections or cursors isn't thread-safe either, so I'm not sure what the above would solve. It also introduces problems with the notion of connection objects always having an open transaction and be a significant change in the API. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46 2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From daniele.varrazzo at gmail.com Sat May 18 02:07:53 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Sat, 18 May 2013 01:07:53 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <1DAC1699-088F-4F78-A441-75E12DD68E30@zzzcomputing.com> References: <519609A8.6080407@egenix.com> <1DAC1699-088F-4F78-A441-75E12DD68E30@zzzcomputing.com> Message-ID: On Fri, May 17, 2013 at 5:22 PM, Michael Bayer wrote: > > The major issue with format/pyformat is that too many DBAPIs take the easy way out and just run their string through Python's "%" operator. This causes DBAPIs that use the format styles to behave differently from those that don't. In particular, a problem I have with MySQL-python and *maybe* recent versions of psycopg2 (would have to check again) is that the behavior changes whether or not I pass parameters to cursor.execute(). The same statement that includes a "%" sign as part of the string must be passed without escaping when no list/dictionary is passed as the second argument, but when a list/dictionary is passed, the parsing changes and now the % sign has to be escaped normally. This is an undocumented behavior that's directly a side effect of the DBAPI saying, "oh there's parameters, let's call "stmt % params", vs. not. On this specific point you are right, and just yesterday I had to apologise for the inconsistency with a colleague of mine. This is something I would probably fix in an eventual "psycopg3" that would break compatibility with the past, but note that it would only affect functions that use the % operator and have no parameter: a minimal number (can't remember on top of my head if the % is used for other syntactical elements in postgres: not that I recall). OTOH if you want consistent behaviour with the %s and %%s just pass an empty tuple to queries taking no argument. You are a framework, you can do that (I would do that in psycopg by specifying the empty tuple as default argument for execute's params, or or something semantically equivalent). -- Daniele From daniele.varrazzo at gmail.com Sat May 18 03:29:22 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Sat, 18 May 2013 02:29:22 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <51966163.5090507@egenix.com> References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> Message-ID: On Fri, May 17, 2013 at 5:57 PM, M.-A. Lemburg wrote: > >> Reading back the upstream thread I also see proposal to leave support >> for all the current placeholders but mandate drivers to implement >> qmark and named too. I think this is only going to complicate the >> implementation of the drivers and of the programs using it. > > Why is that ? I'll try to explain that again. Let's say I work in a postgres-only shop: currently there is no interoperability problems, and people writes python-postgres programs and py-pg libraries. Let's say psycopg3 (implementing dbapi3) implements a method to switch paramstyle, whatever it is. Alice is writing a script program, reading the database, doing some business-related math and generating a report. Of course the business model is already implemented in the well tested and uniformly used "shoplib" library, written by Bob: this library takes a psycopg2 connection and runs whatever complicated query they need to implement such model. In dbapi2 this is a piece of cake. In dbapi3 Alice is free to have her opinions: she is hip and dbapi3 is all the rage, so she will create a psycopg3 pyformat connection and pass that to the shoplib: of course the program will crash and burn immediately. Alice may point out that Bob is not providing complete support to a valid psycopg connection. There are two possible outcomes: 1. most likely Alice will be laughed at and the shop will assume a policy of not using different paramstyle than pyformat, hence making impossible to use libraries or frameworks (sqlalchemy, django) that would of course benefit of the uniformly used named format; 2. Alice is good enough into talking everybody that dbapi3 is the future: now Bob in order to offer the shoplib service to psycopg connection in either mode will have to jump through hoops: he will either store the paramstyle of the connection received in a temporary variable, switch it to pyformat and finally restore it (with all the concurrency problems this would involve, as the connection may be shared across threads); or he will have to dynamically generate queries that would have otherwise been expressed as string constants. This is the reason why I firmly believe that giving excessive freedom to the users when designing a standard only means troubles not only for who implements the service (the dbapi driver in our case) that has to implement several equivalent options, but also for the users of such driver, because writing a function library accepting connections that may demand different parameter style is much more complicated than one where you would just have done with a string, and offers new and unexpected modes of failure. Doing that only to second stylistic matters would be just irresponsible. If we are designing a standard we have to make decisions: every decision we don't make and leave to somebody else only increases the complexity, doesn't definitely add democracy and peace in the world. And open choices don't increment complexity in additive way: they do it rather in multiplicative way. My thoughts about the specific issue in this thread (not my wishlist for dbapi3, with which I'll be glad to bore you to death in another thread) is that 1. demanding a single paramstyle for the driver will only make millions of people angry 2. making the paramstyle switchable will result in programs more complicated, libraries *way* more complicated and database drivers *way, way* more complicated, inevitably causing bugs on either side, making millions of people angry 3. leaving the things as they are w.r.t. the paramstyle (and fix the several shortcomings that dbapi 1 and 2 have shown in the about 10 years they have been around) would make a dbapi3 more gradually acceptable and not a watershed event. And would leave a couple of people angry, which have chosen as mission in their lives to write compatibility layers on top of dbapi modules :) Sorry Michael, you were a good developer but I'd rather sacrifice you than million of people :) Out of joke, Michael, I know that wrestling the idiosyncrasies of each and every dbapi module must feel like herding cats, but I don't believe the different paramstyles are the biggest of the issues you have to solve. I bet bigger problems are in the different syntax offered by the database servers, different data type supported and different ways to represent them, different ways to perform introspectiAnd after on and who knows what else (well, you do :). With postgres that wants "select ... from a join b on a.id = b.a_id" and informix that wants "select ... from a, outer b where a.id = b.a_id" I just don't think a common placeholder is going to be a decisive help for anybody who wants to deal with database abstraction layers. Oh, yes, and there was 4. "have the driver automatically detect the paramstyle". Please, don't joke: this idea is so brittle it's not even funny to demand it as part of a standard. > I think it would greatly simplify the use of the DB-API, since > applications could then be written to use a single paramstyle, > even when using multiple database modules. Come on, you are a database expert: do you really believe that because two commands can be written with the same paramstyle it would be possible to send the same query to different databases? With so different sql syntaxes, so different data models, so different runtime models? I don't think interoperability across database is a problem that can be solved at dbapi level: the aim of the dbapi (and the benefit it has provided in its not short life) is to provide uniformity in the python code to access a database; I don't think it being a database abstraction layer has ever been its scope. -- Daniele From vernondcole at gmail.com Sat May 18 10:29:20 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Sat, 18 May 2013 09:29:20 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> Message-ID: Dear Daniele: I spent some time this morning considering your long and well presented appeal. At my age, I have to walk, rather than run, for exercise, and it helps give me a lot of time to think. I have experience in a small, monolithic shop like the one you hypothesize in your example. I say "small", but that is a relative term -- it was the corporate data center for Georgia Pacific, at that time (around 1980) number 52 on the Fortune 500 list. G.P. had, I think wisely, decided _not_ to upgrade to the latest fad, interactive mainframes, and kept their 1968 era IBM 360/65 well maintained, and doing what it does very well: high speed batch processing. Where they needed to be interactive, they decided to go with another solution: Digital Equipment minicomputers. (That's why they hired me.) They saved millions of dollars by sticking with that venerable model 65 and its OS-360 operating system. But the day finally came when they had to roll an IBM 370 into the shop and go interactive, running CICS on OS/MVS. It was, as you suggested, a _very_ painful period for that shop. Painful, but inevitable. I hope that I may have made that transition easier by providing some tools to smooth things out. CICS is a resource hogging badly performing dog. Our little $100,000 PDP-11/70 would run circles around the $3,000,000 370. I could tell stories... Similarly, MS SQL Server is plainly inferior to PostgreSQL. Thursday, I ran my first django test suite on SQL Server. It took _six_ _hours_. (On my Postgres server it took 42 minutes.) Why would anyone pay actual money for such a monster?? But thousands of people do, and I must interact with them. So, here is the thing. I want you to take the time to make your database driver "*way, way* more complicated" by supporting the SQL format I must use, which is 'qmark'. (I think that is fair, since I have already taken the time to make _my_ database driver support the format you prefer. It added 65 lines to my Python code to support both new formats.) Then, someday in the future, some fortunate person will be able to say: "let's get rid of this pig SQL Server and run our stuff on Postgres" and it will be within the financial realm of possibility for them to do that, all because you invested that time. Meanwhile, the people in your neat little monolithic shop will add one line of code the their programs, telling your database driver to go ahead and do things the old (efficient) way. The fact that they will also be enabled to sneak up on inevitable pain rather than being blind-sided by it is a plus. At least, that's what I think. -- Vernon Cole -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniele.varrazzo at gmail.com Sun May 19 04:22:01 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Sun, 19 May 2013 03:22:01 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> Message-ID: On Sat, May 18, 2013 at 9:29 AM, Vernon D. Cole wrote: > So, here is the thing. I want you to take the time to make your database > driver "*way, way* more complicated" by supporting the SQL format I must > use, which is 'qmark'. Aaahhh... is it that what you want? If what you wanted is a psycopg talking qmark, why on the earth haven't you asked for that? Why instead are you asking the DBAPI to rule that every db driver must implement qmark? There: just released qmarkpg, allowing using qmark and named parameters with psycopg: - Source: https://github.com/dvarrazzo/qmarkpg - PyPI: https://pypi.python.org/pypi/qmarkpg/ This is not a mock. The module passes the DBAPI test suite (2PC included). Nobody in my about 10 years of frequentation of the psycopg mailing list has ever asked psycopg to implement qmark/named. I repeat it: this is a need nobody has ever expressed. And believe me we have been asked for a lot of things. This module is the proof that qmark or not qmark syntax is a non-issue: converting across types is a trivial operation. Writing a wrapper module is a solution infinitely superior to any module/connection/cursor writable attribute dancing; perfectly thread safe and 100% backward compatible. My impression from this long thread is that we are in front of bikeshedding in its purest form: I haven't read a rationale yet to justify the rewrite of *every query* written in Python against MySQL, PostgreSQL and god knows what other database server. The justification has been just a tad more than "we like the ?". The feature specification is "let's copy Java". Well, I've looked into that too: JDBC specifies *only* qmark, not named. Named is only supported by *a different* object (a wrapper, I understand): JDBC does not define a grammar where both qmark and named can be used. And unsurprisingly so: the grammar is ambiguous: cur.execute("select $$a ? and a :foo$$", args) This is a valid postgres query. If args is an object implementing Mapping and Sequence ABCs there's no way to disambiguate that. Now go on and tell me that we could double the ? even when the :foo are used: we are making up that syntax now, because JDBC hasn't event tried. Oh, and in the placeholder with unary "?" operator (valid in postgres) "???": is the operator prefix or suffix? Any driver implementing both positional and named arguments will face the ambiguities of a grammar the proponents have not specified: a problem that format/pyformat don't have. My bottom line is that it is provable that a spec mandating qmark/named is broken, and psycopg will never implement it: not only for its technical inferiority but above all because there is no real request for it, and people who believe they need that can obtain the same result in trivial ways without bothering the rest of the world. -- Daniele From mike_mp at zzzcomputing.com Sun May 19 05:54:15 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Sat, 18 May 2013 23:54:15 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> Message-ID: <581A1088-95A5-41EB-B62B-288BA5AA2111@zzzcomputing.com> On May 18, 2013, at 10:22 PM, Daniele Varrazzo wrote: > > My impression from this long thread is that we are in front of > bikeshedding in its purest form: I haven't read a rationale yet to > justify the rewrite of *every query* written in Python against MySQL, > PostgreSQL and god knows what other database server. The justification > has been just a tad more than "we like the ?". The feature > specification is "let's copy Java". as well as ODBC. > > Well, I've looked into that too: JDBC specifies *only* qmark, not > named. > Named is only supported by *a different* object (a wrapper, I > understand): JDBC does not define a grammar where both qmark and named > can be used. believe it or not I'd prefer to *just* have qmark rather than continue to support six different paramstyles, two of which have an awkward overlap with Python string formatting (read: "format" and "pyformat" present the *most* problems of all the formats, due to their easy overlap with Python's built in formatting). > And unsurprisingly so: the grammar is ambiguous: > > cur.execute("select $$a ? and a :foo$$", args) > > This is a valid postgres query. If args is an object implementing > Mapping and Sequence ABCs there's no way to disambiguate that. Now go > on and tell me that we could double the ? even when the :foo are used: > we are making up that syntax now, because JDBC hasn't event tried. Oh, > and in the placeholder with unary "?" operator (valid in postgres) > "???": is the operator prefix or suffix? I'd suggest we spend a little time seeing how these two universally used formats handle this. The ? and :format styles have been in use for decades; I'm sure we can find adequate precedents for how escaping should be done. We've already failed to specify things unambiguously with percent signs, I can show you a good handful of SQLAlchemy tests that we now mark as "failing" only for MySQL-python and psycopg2, no other DBAPIs - since at some point both DBAPIs changed how when they decide to handle % signs and when they don't, and it was just too much of an edge case for me to bother reacting again (tests that specifically involve putting a % sign in a table or column name - who the heck would do that? beats me, but we had users reporting it). I also have had some very annoying bugs where SQL statements being placed into error messages found their way into log lines, which then caused logging to fail because Python's logging also implicitly expects %s/%(name)s symbols in the strings, the strings had the wrong number of symbols and the logging of errors failed. I've also had confused users who were trying to format string with %d, %f, etc. expecting that the database driver would interpret the types correctly. It's specifically this overlap with a very common formatting system in Python that causes issues. The strings we send to our databases are specifically *not* being sent to a Python interpreter, and this is the deeper reason why I feel that %s/%(name)s is less appropriate than a database-dedicated format. > > My bottom line is that it is provable that a spec mandating > qmark/named is broken, and psycopg will never implement it: not only > for its technical inferiority but above all because there is no real > request for it, There's a reason you've never had these requests - people that actually write software using direct DBAPI access in my experience have no idea that other DBAPIs do things differently (and typically, they're newcomers to Python and often application-level programming overall anyway). Anyone who starts writing a new application in a *second* DBAPI, is shocked to see how this new DBAPI is nothing like the first DBAPI they used so much, realizes immediately how impossible it is to write cross-DBAPI code, starts using a system like SQLAlchemy, and you never hear from them. I hear from them, especially those poor users that have "%" signs in their DDL definitions and trip over the "provable" inconsistencies present in the current % sign implementations. A bigger issue here, that's interesting to consider, is that I think you agree that there should only be two paramstyles, one name-based and one positional - how come the needs of the cx_oracle, pyodbc, or SQLite DBAPIs, who don't consistently use format/pyformat and who also would be greatly concerned about backwards compatibility and the inconvenience of change, are less valid than those of psycopg2? It seems like moving from 6 to 2 paramstyles means lots of DBAPIs need to change no matter what. Surely psycopg2, who is by far the most on the ball with changes and staying up to date could do the best job of making this transition smooth ..... just a thought. From phd at phdru.name Sat May 18 12:08:50 2013 From: phd at phdru.name (Oleg Broytman) Date: Sat, 18 May 2013 14:08:50 +0400 Subject: [DB-SIG] SQLObject 1.4.0 Message-ID: <20130518100850.GA10938@iskra.aviel.ru> Hello! I'm pleased to announce version 1.4.0, the first stable release of branch 1.4 of SQLObject. What's new in SQLObject ======================= Features & Interface -------------------- * Support for PostgreSQL 8.1 is dropped. The minimal supported version of PostgreSQL is 8.2 now. * Optimization in PostgresConnection: use INSERT...RETURNING id to get the autoincremented id in one query instead of two (INSERT + SELECT id). * Changed the way to get if the table has identity in MS SQL. * NCHAR/NVARCHAR and N''-quoted strings for MS SQL. Contributors for this release are Ken Lalonde and Andrew Ziem. For a more complete list, please see the news: http://sqlobject.org/News.html What is SQLObject ================= SQLObject is an object-relational mapper. Your database tables are described as classes, and rows are instances of those classes. SQLObject is meant to be easy to use and quick to get started with. SQLObject supports a number of backends: MySQL, PostgreSQL, SQLite, Firebird, Sybase, MSSQL and MaxDB (also known as SAPDB). Where is SQLObject ================== Site: http://sqlobject.org Development: http://sqlobject.org/devel/ Mailing list: https://lists.sourceforge.net/mailman/listinfo/sqlobject-discuss Archives: http://news.gmane.org/gmane.comp.python.sqlobject Download: https://pypi.python.org/pypi/SQLObject/1.4.0 News and changes: http://sqlobject.org/News.html Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From vernondcole at gmail.com Sun May 19 18:10:17 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Sun, 19 May 2013 10:10:17 -0600 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> Message-ID: Dear Daniele: I am a bit confused. Two posts ago you said that supporting other parmamstyle values would be "*way, way* more complicated". Now you say: " qmark or not qmark syntax is a non-issue: converting across types is a trivial operation." Clearly you are against the "two paramstyles are mandatory, others are optional" rule. But I cannot quite make out whether you are against implementing it because it is too hard, or because it is too easy. My own experience, as a developer who has already done this, is that the job is somewhere in the middle -- somewhat complex. It took me a day or two to write the code and the tests. Not a job to give a beginner, but not a real head breaker either. You also state that the modification is not needed because it has never been requested by one of *your* users. <> ---- Dear Daniele: I am a software engineer who is developing a system to aid in the eradication of polio from the earth. Our company uses django, and PostgreSQL databases on Ubuntu servers and workstations, using psycopg2. We now have a need to interface our data with SQL Server databases and analysis tools used by other agencies. In order to do so, I am updating a db-api driver I maintained when I worked for a Microsoft-only shop, and have begun working with the core developers of django to better inter-operate with SQL Server. A discussion on the subject of 'format' paramstyle with my mentors on the django-developers group revealed that they do _not_ favor the style, and in fact it causes problems by its interaction with the Python '%' operator. They would welcome the opportunity to simply their lives by gradually moving to 'qmark' and 'named' styles, which do not suffer from '%' pollution. Please use your time and influence to make this possible for our premier database system. Will this method of communication be sufficient, or should I open a trouble ticket addressing the issue? Your sincere user, Vernon Cole software engineer eHealthAfrica.org ------ <<\shirt>> Now, you have had a user request. Please don't just take your ball and go home. That will not stop the ball game, but it will rain on it. Work with us. -- Vernon Cole On Sat, May 18, 2013 at 8:22 PM, Daniele Varrazzo < daniele.varrazzo at gmail.com> wrote: > On Sat, May 18, 2013 at 9:29 AM, Vernon D. Cole > wrote: > > So, here is the thing. I want you to take the time to make your database > > driver "*way, way* more complicated" by supporting the SQL format I must > > use, which is 'qmark'. > > Aaahhh... is it that what you want? If what you wanted is a psycopg > talking qmark, why on the earth haven't you asked for that? Why > instead are you asking the DBAPI to rule that every db driver must > implement qmark? > > There: just released qmarkpg, allowing using qmark and named > parameters with psycopg: > > - Source: https://github.com/dvarrazzo/qmarkpg > - PyPI: https://pypi.python.org/pypi/qmarkpg/ > > This is not a mock. The module passes the DBAPI test suite (2PC included). > > Nobody in my about 10 years of frequentation of the psycopg mailing > list has ever asked psycopg to implement qmark/named. I repeat it: > this is a need nobody has ever expressed. And believe me we have been > asked for a lot of things. > > This module is the proof that qmark or not qmark syntax is a > non-issue: converting across types is a trivial operation. Writing a > wrapper module is a solution infinitely superior to any > module/connection/cursor writable attribute dancing; perfectly thread > safe and 100% backward compatible. > > My impression from this long thread is that we are in front of > bikeshedding in its purest form: I haven't read a rationale yet to > justify the rewrite of *every query* written in Python against MySQL, > PostgreSQL and god knows what other database server. The justification > has been just a tad more than "we like the ?". The feature > specification is "let's copy Java". > > Well, I've looked into that too: JDBC specifies *only* qmark, not > named. Named is only supported by *a different* object (a wrapper, I > understand): JDBC does not define a grammar where both qmark and named > can be used. And unsurprisingly so: the grammar is ambiguous: > > cur.execute("select $$a ? and a :foo$$", args) > > This is a valid postgres query. If args is an object implementing > Mapping and Sequence ABCs there's no way to disambiguate that. Now go > on and tell me that we could double the ? even when the :foo are used: > we are making up that syntax now, because JDBC hasn't event tried. Oh, > and in the placeholder with unary "?" operator (valid in postgres) > "???": is the operator prefix or suffix? > > Any driver implementing both positional and named arguments will face > the ambiguities of a grammar the proponents have not specified: a > problem that format/pyformat don't have. > > My bottom line is that it is provable that a spec mandating > qmark/named is broken, and psycopg will never implement it: not only > for its technical inferiority but above all because there is no real > request for it, and people who believe they need that can obtain the > same result in trivial ways without bothering the rest of the world. > > -- Daniele > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniele.varrazzo at gmail.com Sun May 19 21:01:28 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Sun, 19 May 2013 20:01:28 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> Message-ID: On May 19, 2013 5:10 PM, "Vernon D. Cole" wrote: > > Dear Daniele: > > I am a bit confused. Two posts ago you said that supporting other parmamstyle values would be "*way, way* more complicated". Now you say: " qmark or not qmark syntax is a non-issue: converting across types is a trivial operation." It is uselessly painful maintaining a driver allowing paramstyle switching and very hard developing a program making use of it. What has been very easy is wrapping a dbapi module with another dbapi module that support *a single* paramstyle (actually two: one positional and one mapped). qmarkpg does no attempt to support pyformat. > Clearly you are against the "two paramstyles are mandatory, others are optional" rule. But I cannot quite make out whether you are against implementing it because it is too hard, or because it is too easy. Because it's too hard inside a single module and laughably easy outside if it, keeping previous module users happy. If I'm failing making this point understood I must be a total lamer at explainations. Or I am being thoroughly trolled. > My own experience, as a developer who has already done this, is that the job is somewhere in the middle -- somewhat complex. It took me a day or two to write the code and the tests. Not a job to give a beginner, but not a real head breaker either. > > You also state that the modification is not needed because it has never been requested by one of *your* users. > > <> > ---- > Dear Daniele: > I am a software engineer who is developing a system to aid in the eradication of polio from the earth. Our company uses django, and PostgreSQL databases on Ubuntu servers and workstations, using psycopg2. We now have a need to interface our data with SQL Server databases and analysis tools used by other agencies. In order to do so, I am updating a db-api driver I maintained when I worked for a Microsoft-only shop, and have begun working with the core developers of django to better inter-operate with SQL Server. > A discussion on the subject of 'format' paramstyle with my mentors on the django-developers group revealed that they do _not_ favor the style, and in fact it causes problems by its interaction with the Python '%' operator. They would welcome the opportunity to simply their lives by gradually moving to 'qmark' and 'named' styles, which do not suffer from '%' pollution. > Please use your time and influence to make this possible for our premier database system. Will this method of communication be sufficient, or should I open a trouble ticket addressing the issue? > Your sincere user, > Vernon Cole > software engineer > eHealthAfrica.org > ------ > <<\shirt>> > Now, you have had a user request. Dear Vernon, I know the %s placeholders may be a little confusing: they are used because psycopg predates the parameter passing feature of the postgres client library. This doesn't save implementation troubles to psycopg: internally it doesn't use the python % operator (because py3 doesn't offer it for bytes - also note that psycopg2 predates the dbapi on py3 and Unicode queries support, for which the dbsig hasn't had any discussion). On the other hand recent research has shown that even if %s may have chosen by accident (or may not: fog is a clever guy) it offers the only way to support both positional and mapped parameters support, whereas other adapters offer only one of the style (more precisely: other adapters just offer whatever the underlying client library offer: ? for sqlite, :name for Oracle etc. Had psycopg done the same it would have been $n). Even if Psycopg authors wanted to review the placeholder syntax it would result in serious backward incompatibility (meaning 100% incompatibility!) with already written code: you can bet your favourite grandma we would be forked in no time! We have also been requested to drop support for parameters passed as dicts and only implement ?, at which I believe there would be a Kickstarter project to hire a couple of hitmen and send them where I live (and I'd give them 15 quids). Incidentally I've started a project to support qmark and named placeholder: please check the qmarkpg project on pypi and github. It is a psycopg wrapper and it inherits all its features. The project is recent but not less mature than what psycopg would be if we changed placeholders internally. Please test it: patches and bug reports (both on github) are welcome. > Please don't just take your ball and go home. That will not stop the ball game, but it will rain on it. Work with us. I have expressed my views about how the idea of switchable placeholders is ill thought and why qmark and named don't play well together. If dbapi decided to make qmark mandatory qmarkpg would be our dbapi3 driver while psycopg would stay dbapi2. -- Daniele -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at personnelware.com Sun May 19 21:45:35 2013 From: carl at personnelware.com (Carl Karsten) Date: Sun, 19 May 2013 14:45:35 -0500 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> Message-ID: All, I have been following this parmamstyle issue. I'll admit, I don't see an easy answer so don't get too defensive if it seems like I am attacking anyone's position. I really don't have strong feelings for or against anything. But I do have some ideas that might help direct progress out of this circle things seem to be stuck in. I don't think it is reasonable to expect python's dbapi to be as db agnostic as we would like it to be. I think anytime multi db support is coded, this pattern will emerge: for each server (MySql, PostgreSQL, MsSql...) there will be code specific to that server. If only 2 servers are coded for, and a 3rd is desired, it will require additional code. (I bet this pattern has a name. Any design pattern experts here?) Python's dbapi exists to make it easy to write that additional code. It does not exist to eliminate the need to write additional code. To help support or dispute my opinions and I think this whole thread, I think it would help to define the dbapi users. I see 4 or so groups 1. single app/db/instance. I have had plenty of jobs where I worked on an app that had only 1 install, so I controlled everything: I got to pick the whole stack: OS, db server, application language, framework, db module. 2. multi db/platform framework developer. I have a bit of experience here too: http://www.f1tech.com/products/visual-foxexpress and http://dabodev.com. I have hacked on those code bases at the core developer's houses. 3. framework user. Like when I use Django to develop a website for a client. I only use the ORM's interface and don't use .raw('select ....) 4. single app (like #1) but mulit db. App gets installed in various environments and needs to support more than one db engine. Doesn't use an existing ORM or abstraction because they need to use features of the SQL engine that are too hard to work with using the abstractions. I have no experience here. I am wondering if anyone in this thread does, and if this group actually has any members. I think most of the disagreement in this thread is trying to address the needs of this group. Here is my personal case study of what happens when there is nothing like dbapi. The code I am going to reference is all here https://github.com/CarlFK/veyepar/tree/master/dj/scripts but I would not recommend diving in. It is scary and the details are not really relevant to this discussion. My video processing app interfaces with 4 video hosting services (youtube, vimeo, blip, archive). Each service is basically the same: they provide a place to store a video and some metadata, plus have an html interface to a player. They also have unique features: blip and archive host the original file and provide a public url anyone can download that file from. blip lets me upload an image to use for a thumb nail. youtube generates one on it's own, archive doesn't have thumbs. All of them support tags, but youtube does not allow a slash char in the tag. Each has an API. They all use http over tcp/ip, So I could use the requests module to interface with them, but that would be hard. Lucky all 4 have python modules that use who knows what modules to do the http. But there is no consistency how these modules are called. They expose the features of each service, but each one has a unique way of doing the same thing, so I wrote 4 wrappers that give me a constant interface: Each wrapper has: class Uploader(object): def upload(self): and then the following lines that calls into the provided api: youtube_uploader.py yt_service.InsertVideoEntry(video_entry, pf) archive_uploader.py key.set_contents_from_file(pf) vimeo_uploader.py vup.upload(self.files[0]['pathname']) blip_uploader.py (OK, my blip code is a mess. This was the first one I coded for, and I had no idea what the separation was going to look like. class Blip(object): def PostMultipart self.PostMultipart(self.BLIP_UPLOAD_URL, fields, files) So now my app uses the wrappers and regardless of what service I am calling, I do: post_arc.py: ret = uploader.upload() post_vimeo.py: ret = uploader.upload() post_yt.py: youtube_success = uploader.upload() post_blip.py: response_obj = blip_cli.Upload(video_id, blip_user, blip_pw, files, meta, thumb) Like i said, the blip code is a mess, but I think my mess supports what I see the value of dbapi is. All 4 foo_uploader's are very similar. In fact to make a new one I copy the last one I worked on and adjust it to make it work. Yes, I expect I could have created an abstract class.. that may happen someday. Same goes post_foo.py. I have a dream of consolidating all 4 post_foo's into one that takes a parameter and uses the corresponding foo_uploader module. I am not sure if that will ever happen, or if it should. Here is how this relates to dbapi: foo_upload.py does what dbapi modules do: make it easier to write each new post_foo. If a new video service called bar comes along that I need to post to, I fully expect to first create the bar_upload wrapper, and then post_bar.py. If a new SQL server came along, I would expect there would be a client lib written in C. A small group of people (likely just one person) to write a new dbapi module for it, and then all the developers who wanted to add support for that server would write code. All the developers would thank the dbapi module writer for providing a pythonic interface to the server's client lib.. I hope you were able to follow all of this, and get something out of it. I think this group should examine how things are currently being used, and give some concrete examples of how they think things could be different. On Sun, May 19, 2013 at 11:10 AM, Vernon D. Cole wrote: > Dear Daniele: > > I am a bit confused. Two posts ago you said that supporting other > parmamstyle values would be "*way, way* more complicated". Now you say: " > qmark or not qmark syntax is a non-issue: converting across types is a > trivial operation." > > Clearly you are against the "two paramstyles are mandatory, others are > optional" rule. But I cannot quite make out whether you are against > implementing it because it is too hard, or because it is too easy. > > My own experience, as a developer who has already done this, is that the job > is somewhere in the middle -- somewhat complex. It took me a day or two to > write the code and the tests. Not a job to give a beginner, but not a real > head breaker either. > > You also state that the modification is not needed because it has never been > requested by one of *your* users. > > < and put on my Linux-django-developer shirt>> > ---- > Dear Daniele: > I am a software engineer who is developing a system to aid in the > eradication of polio from the earth. Our company uses django, and PostgreSQL > databases on Ubuntu servers and workstations, using psycopg2. We now have a > need to interface our data with SQL Server databases and analysis tools used > by other agencies. In order to do so, I am updating a db-api driver I > maintained when I worked for a Microsoft-only shop, and have begun working > with the core developers of django to better inter-operate with SQL Server. > A discussion on the subject of 'format' paramstyle with my mentors on the > django-developers group revealed that they do _not_ favor the style, and in > fact it causes problems by its interaction with the Python '%' operator. > They would welcome the opportunity to simply their lives by gradually moving > to 'qmark' and 'named' styles, which do not suffer from '%' pollution. > Please use your time and influence to make this possible for our premier > database system. Will this method of communication be sufficient, or should > I open a trouble ticket addressing the issue? > Your sincere user, > Vernon Cole > software engineer > eHealthAfrica.org > ------ > <<\shirt>> > Now, you have had a user request. > > Please don't just take your ball and go home. That will not stop the ball > game, but it will rain on it. Work with us. > -- > Vernon Cole > > > On Sat, May 18, 2013 at 8:22 PM, Daniele Varrazzo > wrote: >> >> On Sat, May 18, 2013 at 9:29 AM, Vernon D. Cole >> wrote: >> > So, here is the thing. I want you to take the time to make your >> > database >> > driver "*way, way* more complicated" by supporting the SQL format I must >> > use, which is 'qmark'. >> >> Aaahhh... is it that what you want? If what you wanted is a psycopg >> talking qmark, why on the earth haven't you asked for that? Why >> instead are you asking the DBAPI to rule that every db driver must >> implement qmark? >> >> There: just released qmarkpg, allowing using qmark and named >> parameters with psycopg: >> >> - Source: https://github.com/dvarrazzo/qmarkpg >> - PyPI: https://pypi.python.org/pypi/qmarkpg/ >> >> This is not a mock. The module passes the DBAPI test suite (2PC included). >> >> Nobody in my about 10 years of frequentation of the psycopg mailing >> list has ever asked psycopg to implement qmark/named. I repeat it: >> this is a need nobody has ever expressed. And believe me we have been >> asked for a lot of things. >> >> This module is the proof that qmark or not qmark syntax is a >> non-issue: converting across types is a trivial operation. Writing a >> wrapper module is a solution infinitely superior to any >> module/connection/cursor writable attribute dancing; perfectly thread >> safe and 100% backward compatible. >> >> My impression from this long thread is that we are in front of >> bikeshedding in its purest form: I haven't read a rationale yet to >> justify the rewrite of *every query* written in Python against MySQL, >> PostgreSQL and god knows what other database server. The justification >> has been just a tad more than "we like the ?". The feature >> specification is "let's copy Java". >> >> Well, I've looked into that too: JDBC specifies *only* qmark, not >> named. Named is only supported by *a different* object (a wrapper, I >> understand): JDBC does not define a grammar where both qmark and named >> can be used. And unsurprisingly so: the grammar is ambiguous: >> >> cur.execute("select $$a ? and a :foo$$", args) >> >> This is a valid postgres query. If args is an object implementing >> Mapping and Sequence ABCs there's no way to disambiguate that. Now go >> on and tell me that we could double the ? even when the :foo are used: >> we are making up that syntax now, because JDBC hasn't event tried. Oh, >> and in the placeholder with unary "?" operator (valid in postgres) >> "???": is the operator prefix or suffix? >> >> Any driver implementing both positional and named arguments will face >> the ambiguities of a grammar the proponents have not specified: a >> problem that format/pyformat don't have. >> >> My bottom line is that it is provable that a spec mandating >> qmark/named is broken, and psycopg will never implement it: not only >> for its technical inferiority but above all because there is no real >> request for it, and people who believe they need that can obtain the >> same result in trivial ways without bothering the rest of the world. >> >> -- Daniele > > > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > http://mail.python.org/mailman/listinfo/db-sig > -- Carl K From ktc.jkennedy at gmail.com Sun May 19 22:23:30 2013 From: ktc.jkennedy at gmail.com (John Kennedy) Date: Sun, 19 May 2013 16:23:30 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) Message-ID: Dearest DBAPI authors, I have lurked with great interest on this board for quite some time, as the topics are generally near and dear to my heart. I believe I can say that I have used nearly all of your drivers over the last ten years. I have purchased Marc's products at least a dozen times on behalf of my customers, and I will generally favor a python DBAPI solution to the data transportation issue du jeur if one is available. I'm not sure that my opinion is even worthy of consideration in this discussion, but I offer it nonetheless. My company specializes in the development of interfaces and "glue" applications. These are typically in the form of either a web or desktop application, or a background service that serves to automate, merge, and facilitate the presentation of various disparate internal systems into a more or less seamless facade. Python is our tool of choice because it allows us to construct our systems using tried-and-true modular design patterns, quickly, and without unnecessary licensing or technical restrictions. Our projects range in size from several hours of development to 1000 hours or so, with the typical project being best expressed in terms of days or a few weeks. As such, over the years our internal subversion repository has several thousand one-off tools developed with DBAPI in all different environments, primarily for data munging and conversion, and perhaps 40 or 50 "systems" that are deployed throughout our base of customers. I have developers ranging from young and inexperienced to old and wise on my team. My present concerns revolve around adopting internal standards that maximize the value of any code we write. Naturally, maximizing the opportunity for reuse of both techniques and entire code modules is important. This may not be the case throughout your user base, but in my case I am faced with the fact that human time is expensive, and machine time is cheap. So all of my effort is in the ongoing net reduction of human time, thereby increasing the value to my customers. It is in this context that I perceive 100% of the value of even having such a thing as DBAPI. With all due respect to the many years of effort that many of you have spent to assist me and people like me, I must confess that in my case the DBAPI itself is only as valuable as its ability to allow me to supplant the underlying database with minimal impact on my code. Perhaps I am in a fortunate niche, because I am frequently able to get by with a very minimal subset of standard SQL, and minimal use of database specific features. This is by choice, so as to maximize my ability to handle the things that are frequently outside of my control such as wholesale change in one or more of the domain specific systems that I need to integrate with. All of you, as developers of DBAPI compliant modules, have undertaken to supply me with some level of consistency across what can only be described as an exceedingly non-standard set of underlying databases. The harsh facts revolve around what you cannot offer me: - any sort of consistency of data types, particularly numeric types - very little in terms of help, despite your efforts, with metadata - any ability to unify the various forms of fully qualified table references - any ability to get consistent locking behavior - any ability to get consistent query cache behavior - any ability to have reusable stored procedures - any ability to NOT have SQL and its inherent impedance mismatches in my code - a fixed point data type with speed comparable to floats But if not the hope for some sort of database independence, the only thing it seems DBAPI gives me to date is that I will have a module, a connection object, a cursor object with some execute methods, and a description of fields in my cursor for which I can only really count on the field name being consistently available. Through no fault of any of yours, I cannot even find a reasonable way to write routines that do datetime manipulation in a reusable fashion, because I don't know ahead of time whether I will be using an mxDateTime (which I consider to be unfortunately superior as well as non-standard) or a python datetime, or a java.util.date or a java.sql.date, or a CLR System.DateTime. >From my humble perspective Daniele made the most salient point of the discussion in reference to optional behavior making things more difficult for everyone rather than simpler. I believe that in reference to "standard behavior" such as proposed behavior for DBAPI this is true. The purpose of a standard should be to standardize. As such, I fully support the mandate that DBAPI have SOME SORT of standard parameter substitution behavior that I can rely upon consistently -- because selfishly I know this will inherently increase the value of any code for which I am able to develop to the standard. The unfortunate reality is that any application developer who competently deals with real life data where it lives can easily code around any differences in your drivers. I have a humorous set of wrappers, proxies, and custom modules for this purpose. So humbly, I suggest perhaps categorizing users along the classification of whether or not their intent is relative database independence. For those who are using your driver merely because it is the de-facto tool one must use to get access to the database, sadly I believe that DBAPI itself is just another layer of throwaway middleware -- and more likely to stop me, as an app developer, from utilizing the special feature I need, than to help me. But from the perspective of a user who could benefit a great deal from database independence, the DBAPI is my only hope. As to WHICH paramstyle is best, my opinion is nearly worthless, but I will give it anyway. Again, humbly, I prefer qmark. My reasoning for this is that I find myself doing a large amount of string manipulation in my code, and python is quite powerful in this respect. It would be trivial (albeit spectacularly time consuming) for me to adjust all of my code to NEVER need to pass parameters to a DBAPI method call. However, a large subset of my code set involves translations from another language or environment, and these are most frequently ODBC. The real reason I use parameters is in the hope that the SQL engine will prepare my statement and reduce my round trip time. But in reality in most environments there are either better alternatives (i.e. inserts and updates with a list of parameters instead of a loop with single set of parameters), or the gain is trivial (i.e. a query cache is used anyway, or I am front-ending with memcache because ALL DB access is too slow). The qmark, being positional, allows me to write the most abstract form of statements, and at least makes it clear in my code which "layer" is performing the substitution. Again I wish to express my gratitude for all of you in this sometimes thankless task! Your Fan, J. Kennedy -------------- next part -------------- An HTML attachment was scrubbed... URL: From ktc.jkennedy at gmail.com Mon May 20 00:40:01 2013 From: ktc.jkennedy at gmail.com (John Kennedy) Date: Sun, 19 May 2013 18:40:01 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) Message-ID: I have often thought that the human race would unite, were we only to be attacked by Martians. To serve that end, I will sacrifice myself and any credibility I might have otherwise retained to present such an attack here. The one thing that the DBAPI group could do to make my wildest DBAPI dreams come true in the next version would be to give me a single method on the cursor object that guaranteed to function uniformly across all databases, albeit at the expense of any "native" performance and reliability advantages in the face of corner cases that I might otherwise have had. In my dream world, I would call this "cur.sexecute". The "sexecute" method would give me a number of huge advantages in terms of code reuse, because, without tracing through all of the code that might have instantiated the cursor object now in front of me, I would KNOW that this object had a "sexecute" method, and I would know (almost like a contract) that it would behave as follows, consistently across my entire codebase: - 1st param - my SQL - parameterized with "?" - 2nd param - my parameters -- either guaranteed to require a particular form of sequence nesting, or guaranteed to not care. - return type - list of tuples I am not intelligent enough to determine the side effects in such a large user base as you folks have. For those of you who have developed drivers that are subclassable, there could be users who created their own "sexecute" methods with profoundly different intentions than I have :-) However, I can state the following positives with certainty: - This would only require ME to change my existing code if I prefer to maximize its re-usability in the face of database uncertainty. - My teammates would remember the method name, even when they are forced to edit in an environment that doesn't support code completion. - As mentioned, tracing through inheritance and runtime hierarchies to determine if "sexecute" would work would be unnecessary. - You have either implemented this already, or can delegate to another method in your implementation. - Everyone would think you are hip for choosing such an awesome method name. Humbly Yours, J. Kennedy -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniele.varrazzo at gmail.com Mon May 20 11:35:09 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Mon, 20 May 2013 10:35:09 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> Message-ID: On Sun, May 19, 2013 at 8:01 PM, Daniele Varrazzo wrote: > On May 19, 2013 5:10 PM, "Vernon D. Cole" wrote: >> Please don't just take your ball and go home. That will not stop the ball >> game, but it will rain on it. Work with us. > > I have expressed my views about how the idea of switchable placeholders is > ill thought and why qmark and named don't play well together. If dbapi > decided to make qmark mandatory qmarkpg would be our dbapi3 driver while > psycopg would stay dbapi2. Vernon, because I know my communication skills are not the best and I probably cut my sentences too short: don't see my position as "I'm taking the ball home and not playing with you anymore". You have discussed two solutions in this thread and on these I have my opinions: - a single placeholders style would be ok but who has proposed it should be specify it in a complete and unambiguous way, not first decide to go that way and then hope in a miracle - a switchable grammar is bad, and who has proposed it should be aware that makes thread safety at level 2 (sharing the connection) impossible. - if a non-backward compatible road is decided, an update path to survive the period of coexistence of the two standards as has been with py2.6 and 2to3 in the long road for py3: none of that has been proposed. What I ask is, everybody, please, show some responsibility. Think about the consequence of your decisions. The dbapi is used by *everybody* who uses python and any database. Michael: your only observations are "the feature X (or its abolition) would make SQLAlchemy n lines of code smaller". Well, guess what, proposing to sacrifice useful features because of use case is absolute selfishness. My main responsibility is for psycopg2 users, but I'm not against the dbapi evolution, and I'm not against working (as I've always done, for free) writing code to meet the expectation of a solid and standard-compliant driver. My choice of responsibility is to meet the request of a dbapi3 to use a non-backward-compatible placeholder style by developing a dbapi3 compliant wrapper module, leaving Psycopg dbapi2 compliant, and to talk you out of switchable placeholders. Mine is the only suggestion of an upgrade path that has been proposed in this thread. So, please, be responsible: the dbsig has too little active users, I believe really too little for the amount of responsibility it brings. You have the duty to be more responsible than this, throwing balls of mud at the wall and see if they stick will only cause problems at massive scale and most likely will leave dbapi3 discredited and ignored. -- Daniele From vernondcole at gmail.com Mon May 20 13:51:31 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Mon, 20 May 2013 05:51:31 -0600 Subject: [DB-SIG] Test suite update for format conversion accuracy. Message-ID: Stuart: I have just uploaded a branch for your consideration for https://code.launchpad.net/dbapi-compliance. This version adds a second field to the second test table (barflies) and loads that field with the literal 'thi%s :may cau%(e)s? troub:1e'. The value of this literal is a trap for unwary format conversion routines which fail to notice that they are re-formatting a literal value. This version also includes patches for both outstanding bug/patch requests, and a correction to one of them which is not quite complete. The version number is '1.4.1' -- Vernon Cole -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike_mp at zzzcomputing.com Mon May 20 15:20:29 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Mon, 20 May 2013 09:20:29 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> Message-ID: On May 20, 2013, at 5:35 AM, Daniele Varrazzo wrote: > > What I ask is, everybody, please, show some responsibility. Think > about the consequence of your decisions. The dbapi is used by > *everybody* who uses python and any database. Michael: your only > observations are "the feature X (or its abolition) would make > SQLAlchemy n lines of code smaller". Well, guess what, proposing to > sacrifice useful features because of use case is absolute selfishness. Sorry, I don't normally say things like this but to accuse me of "selfishness" is really ridiculous. SQLAlchemy is hardly at all affected whether or not DBAPI has two or eighteen paramstyles, as I've said, we can deal with anything, we already deal with dozens of inconsistencies at many levels and the paramstyle issue is one of the most trivial, it is very easy to code around - which is why it's all the more absurd that DBAPI authors themselves would express vast reservation about a change on their end, as it is similarly not a big deal no matter what the paramstyle is. My agenda here is that, since we're talking about improving the DBAPI, to suggest how I really think it can be improved, and seeing as this one little part of DBAPI is actually being discussed to be improved, I've made my suggestion which is, to remove the unnecessary number of paramstyles and standardize on just two. If it turns out that all the DBAPI authors simply cannot agree as to which two styles to choose and we stay with six, then that is DBAPI's unfortunate loss, not SQLAlchemy's. > > My main responsibility is for psycopg2 users, but I'm not against the > dbapi evolution, and I'm not against working (as I've always done, for > free) writing code to meet the expectation of a solid and > standard-compliant driver. My choice of responsibility is to meet the > request of a dbapi3 to use a non-backward-compatible placeholder style > by developing a dbapi3 compliant wrapper module, leaving Psycopg > dbapi2 compliant, and to talk you out of switchable placeholders. Mine > is the only suggestion of an upgrade path that has been proposed in > this thread. Nobody is debating that at all, psycopg2 can take whatever upgrade path it chooses. I was under the impression we were just talking about a spec here. From daniele.varrazzo at gmail.com Mon May 20 16:12:16 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Mon, 20 May 2013 15:12:16 +0100 Subject: [DB-SIG] Test suite update for format conversion accuracy. In-Reply-To: References: Message-ID: On Mon, May 20, 2013 at 12:51 PM, Vernon D. Cole wrote: > Stuart: > > I have just uploaded a branch for your consideration for > https://code.launchpad.net/dbapi-compliance. This version adds a second > field to the second test table (barflies) and loads that field with the > literal 'thi%s :may cau%(e)s? troub:1e'. The value of this literal is a > trap for unwary format conversion routines which fail to notice that they > are re-formatting a literal value. You are requiring the sql string to be parsed, literal string identified and placeholders inside them not treated. This is not a DBAPI2 requirement. Without this requirement a driver can be implemented just passing the string to the client library, an extremely reasonable way to implement a client library wrapper. If you add it every driver will need to know the parsing rule of the server (and of its version). This is not a task as trivial as counting the quotes or applying a regexp: take a look at the Postgres literals grammar: . Of course PG developers are free to complicate that as much as they want: a client driver is expected to be agnostic and just forward the query. I don't think any currently implemented driver behaves as you wish. Here is stdlib's sqlite3: In [1]: import sqlite3 In [2]: cnn = sqlite3.connect(':memory:') In [3]: cur = cnn.cursor() In [4]: cur.execute("create table test (a text, b text)") Out[4]: In [5]: cur.execute("insert into test values (?, ?)", ('hi', 'mum')) Out[5]: In [6]: cur.execute("insert into test values (?, 'wat?')", ('hi')) --------------------------------------------------------------------------- ProgrammingError Traceback (most recent call last) /home/piro/src/qmarkpg/ in () ----> 1 cur.execute("insert into test values (?, 'wat?')", ('hi')) ProgrammingError: Incorrect number of bindings supplied. The current statement uses 1, and there are 2 supplied. (for extra fun, I don't think sqlite3 even contemplates the possibility to use ? into a literal and neither ?? nor \? appear to work, see ) -- Daniele From carl at personnelware.com Mon May 20 17:20:58 2013 From: carl at personnelware.com (Carl Karsten) Date: Mon, 20 May 2013 10:20:58 -0500 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> Message-ID: Does anyone on in this discussion actually want named parameters for their own use, or does it seem like something someone might want, so lets try to make that happen just in case? I can see the benefit (easier to read code) but given the amount of problem it would seem to cause, it doesn't seem worth it. But I have lost track of the pros-n-cons of the various options. I think it is time to start a wiki with pointers to example code. -- Carl K From carl at personnelware.com Mon May 20 17:49:23 2013 From: carl at personnelware.com (Carl Karsten) Date: Mon, 20 May 2013 10:49:23 -0500 Subject: [DB-SIG] Test suite update for format conversion accuracy. In-Reply-To: References: Message-ID: On Mon, May 20, 2013 at 9:12 AM, Daniele Varrazzo wrote: > > In [6]: cur.execute("insert into test values (?, 'wat?')", ('hi')) > --------------------------------------------------------------------------- > ProgrammingError Traceback (most recent call last) > /home/piro/src/qmarkpg/ in () > ----> 1 cur.execute("insert into test values (?, 'wat?')", ('hi')) > > ProgrammingError: Incorrect number of bindings supplied. The current > statement uses 1, and there are 2 supplied. Shouldn't ('hi') be ('hi',) ? >>> cur.execute("insert into test values (?, 'wat?')", ('hi',)) -- Carl K From daniele.varrazzo at gmail.com Mon May 20 18:16:52 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Mon, 20 May 2013 17:16:52 +0100 Subject: [DB-SIG] Test suite update for format conversion accuracy. In-Reply-To: References: Message-ID: <1369066612.29169.16.camel@risotto.smithersbet.com> On Mon, 2013-05-20 at 10:49 -0500, Carl Karsten wrote: > On Mon, May 20, 2013 at 9:12 AM, Daniele Varrazzo > wrote: > > > > In [6]: cur.execute("insert into test values (?, 'wat?')", ('hi')) > > --------------------------------------------------------------------------- > > ProgrammingError Traceback (most recent call last) > > /home/piro/src/qmarkpg/ in () > > ----> 1 cur.execute("insert into test values (?, 'wat?')", ('hi')) > > > > ProgrammingError: Incorrect number of bindings supplied. The current > > statement uses 1, and there are 2 supplied. > > Shouldn't ('hi') be ('hi',) ? > > >>> cur.execute("insert into test values (?, 'wat?')", ('hi',)) > Oh, right, sqlite (the library, not the python module) can deal with a question mark in a string ok. Wrong example then. Grepping for '?' in the _sqlite directory in Python source suggests that literals ? are not looked for anywhere; this implies that the module doesn't process them before passing the string to the client library (that in the sqlite specific case is *the* database, in client-server libraries this is not the case but I don't have any handy to test). If any module wanted to support a placeholder different than the native one would find the need to parse the query string, which I don't think is a reasonable thing to ask to a client library. -- Daniele From carl at personnelware.com Mon May 20 18:26:28 2013 From: carl at personnelware.com (Carl Karsten) Date: Mon, 20 May 2013 11:26:28 -0500 Subject: [DB-SIG] Test suite update for format conversion accuracy. In-Reply-To: <1369066612.29169.16.camel@risotto.smithersbet.com> References: <1369066612.29169.16.camel@risotto.smithersbet.com> Message-ID: On Mon, May 20, 2013 at 11:16 AM, Daniele Varrazzo wrote: > If any module wanted to support a placeholder different than the native > one would find the need to parse the query string, which I don't think > is a reasonable thing to ask to a client library. +1 I have an idea: two layers. 1. thin python interface to the underling binary lib that actaully talks to the server. 2. a bunch of helper code that everyone is going to write anyway. -- Carl K From mike_mp at zzzcomputing.com Mon May 20 18:40:04 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Mon, 20 May 2013 12:40:04 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> Message-ID: <1EAB1FE2-DB79-4444-9149-568560286C1A@zzzcomputing.com> On May 20, 2013, at 11:20 AM, Carl Karsten wrote: > Does anyone on in this discussion actually want named parameters for > their own use, or does it seem like something someone might want, so > lets try to make that happen just in case? > > I can see the benefit (easier to read code) but given the amount of > problem it would seem to cause, it doesn't seem worth it. > > But I have lost track of the pros-n-cons of the various options. I > think it is time to start a wiki with pointers to example code. We should re-locate the original thread where I think a lot more DBAPI authors weighed in, and we had decided on qmark and named after much deliberation. I maintain that a paramstyle that specifically does not overlap with Python's own built-in formats is more appropriate, for the reasons I've outlined (works poorly with other systems that already deal with Python formatting, leads DBAPI authors to be lazy, rely too heavily on Python built-in behavior rather than considering bound parameters as a special case, leads to inconsistent implementation schemes, confuses end users since it isn't "true" Python formatting, is not in line with commonly accepted standard conventions in the greater database adapter community). From vernondcole at gmail.com Mon May 20 18:46:36 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Mon, 20 May 2013 10:46:36 -0600 Subject: [DB-SIG] Test suite update for format conversion accuracy. In-Reply-To: References: Message-ID: On Mon, May 20, 2013 at 8:12 AM, Daniele Varrazzo < daniele.varrazzo at gmail.com> wrote: > On Mon, May 20, 2013 at 12:51 PM, Vernon D. Cole > wrote: > > Stuart: > > > > I have just uploaded a branch for your consideration for > > https://code.launchpad.net/dbapi-compliance. This version adds a second > > field to the second test table (barflies) and loads that field with the > > literal 'thi%s :may cau%(e)s? troub:1e'. The value of this literal is a > > trap for unwary format conversion routines which fail to notice that they > > are re-formatting a literal value. > > You are requiring the sql string to be parsed, literal string > identified and placeholders inside them not treated. This is not a > DBAPI2 requirement. > Indeed, the DB API2 does not address parameter reformatting at all. We are just starting to play with that. I am merely making sure that a reformat operation does not step on the contents of an ANSI standard literal. And, yes, when I am reformatting a string I do have to parse it. How else would I find my format markers? > > Without this requirement a driver can be implemented just passing the > string to the client library, an extremely reasonable way to implement > a client library wrapper. If you add it every driver will need to know > the parsing rule of the server (and of its version). This is not a > task as trivial as counting the quotes As a matter of fact, counting quotes handles this problem quite nicely. Actually, in my code I don't really count them. I do a .split("'") [that is using a single quote] and then only parse the odd-numbered tokens. Then do a "'".join(). > or applying a regexp: take a > look at the Postgres literals grammar: > . > v v v v v v v v 4.1.2.1. String Constants A string constant in SQL is an arbitrary sequence of characters bounded by single quotes ('), for example 'This is a string'. To include a single-quote character within a string constant, write two adjacent single quotes, e.g., 'Dianne''s horse'. Note that this is not the same as a double-quote character ("). ^ ^ ^ ^ ^ ^ That's what I am using for this test: an arbitrary sequence of characters bounded by single quotes. > Of course PG developers are free to complicate that as much as they > want: a client driver is expected to be agnostic and just forward the > query. > Yes, that is the point: forward the query, not mess it up. > > I don't think any currently implemented driver behaves as you wish. > I happen to _know_ that at least _one_ does. ( My code is GPL, feel free to copy. I copied from yours yesterday morning.) > Here is stdlib's sqlite3: > > In [1]: import sqlite3 > > In [2]: cnn = sqlite3.connect(':memory:') > > In [3]: cur = cnn.cursor() > > In [4]: cur.execute("create table test (a text, b text)") > Out[4]: > > In [5]: cur.execute("insert into test values (?, ?)", ('hi', 'mum')) > Out[5]: > > In [6]: cur.execute("insert into test values (?, 'wat?')", ('hi')) > --------------------------------------------------------------------------- > ProgrammingError Traceback (most recent call last) > /home/piro/src/qmarkpg/ in () > ----> 1 cur.execute("insert into test values (?, 'wat?')", ('hi')) > > ProgrammingError: Incorrect number of bindings supplied. The current > statement uses 1, and there are 2 supplied. > > That is, indeed, a programming error. "h" and "i" are two values. The line should have another comma: In [6]: cur.execute("insert into test values (?, 'wat?')", ('hi',)) I usually pass a single parameter in a list simply to avoid that pattern: In [6]: cur.execute("insert into test values (?, 'wat?')", ['hi']) (for extra fun, I don't think sqlite3 even contemplates the > possibility to use ? into a literal and neither ?? nor \? appear to > work, see ) > > Here is what the document you linked says: v v v v 4.1.2.1. String Constants A string constant in SQL is an arbitrary sequence of characters bounded by single quotes ('), for example 'This is a string'. To include a single-quote character within a string constant, write two adjacent single quotes, e.g., 'Dianne''s horse'. Note that this is not the same as a double-quote character ("). ^ ^ ^ ^ ^ Nothing in there about avoiding question marks. Now let's try it.... C:\Users\vernon\Projects\qmarkpg\tests>python Python 2.7.4 (default, Apr 6 2013, 19:55:15) [MSC v.1500 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import adodbapi >>> cnn = adodbapi.connect( ... '%(prov_drv)s;Server=%(host)s;Database=%(database)s;uid=%(user)s;pwd=%(password)s;', ... 'adotestuser', '12345678', "25.223.161.222", 'adotest', ... prov_drv='Provider=MSDASQL;Driver={PostgreSQL Unicode(x64)}') >>> cnn.dbms_name u'PostgreSQL' >>> cnn.dbms_version u'9.1.9' >>> cur = cnn.cursor() >>> cur.execute("create table test (a text, b text)") >>> cur.paramstyle 'qmark' >>> cur.execute("insert into test values (?, ?)", ('hi', 'mum')) >>> cur.execute("insert into test values (?, 'wat?')", ('hi',)) >>> cur.paramstyle = 'format' >>> cur.execute("insert into test values (%s, '%sure')", ('hello',)) >>> cnn.commit() >>> cur.execute('select * from test') >>> for row in cur: ... print row ... ('hi', 'mum') ('hi', 'wat?') ('hello', '%sure') >>> Seems to me that counting quotes works. -- Vernon -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike_mp at zzzcomputing.com Mon May 20 19:08:12 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Mon, 20 May 2013 13:08:12 -0400 Subject: [DB-SIG] Test suite update for format conversion accuracy. In-Reply-To: <1369066612.29169.16.camel@risotto.smithersbet.com> References: <1369066612.29169.16.camel@risotto.smithersbet.com> Message-ID: <7ED6FA78-E9EF-43A6-A9D4-34DFAC62C158@zzzcomputing.com> On May 20, 2013, at 12:16 PM, Daniele Varrazzo wrote: > > Oh, right, sqlite (the library, not the python module) can deal with a > question mark in a string ok. Wrong example then. > > Grepping for '?' in the _sqlite directory in Python source suggests that > literals ? are not looked for anywhere; this implies that the module > doesn't process them before passing the string to the client library > (that in the sqlite specific case is *the* database, in client-server > libraries this is not the case but I don't have any handy to test). > > If any module wanted to support a placeholder different than the native > one would find the need to parse the query string, which I don't think > is a reasonable thing to ask to a client library. I think part of the wisdom of "?" in any case is that the "?" symbol is not something that ever needs to be in a SQL statement that uses bound parameters. Whereas the % symbol certainly is - for one thing, its the modulus operator on many platforms, including Postgresql. The use case where a statement has a mix of directly embedded parameters and bound parameters isn't a real need. If you're using bound parameters, that's what you should be doing across the board; embedding literals directly into a SQL statement is something we only do for command-line one-offs - in a real application, it's one of the most commonly exploited security holes in modern software. The only need I can see where one would need to mix literals and bound parameters is when using drivers like that of Firebird and some variants of ODBC where you can't put bound parameters in certain places, like in the columns clause. For example, Firebird and certain flavors of ODBC won't let you do this: SELECT ? = ? because the driver wants to assign typing information to the incoming expressions, and it does this by looking at the context of the bound parameter. Quote-counting is an easy way to work around this, as the rules for quoting in SQL are very simple. If solid reference implementations for parsing the ? could be developed, all the DBAPIs could use them and then we'd have guaranteed consistent behavior on all platforms. From daniele.varrazzo at gmail.com Mon May 20 19:57:38 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Mon, 20 May 2013 18:57:38 +0100 Subject: [DB-SIG] Test suite update for format conversion accuracy. In-Reply-To: References: Message-ID: <1369072658.29169.71.camel@risotto.smithersbet.com> On Mon, 2013-05-20 at 10:46 -0600, Vernon D. Cole wrote: > On Mon, May 20, 2013 at 8:12 AM, Daniele Varrazzo < > daniele.varrazzo at gmail.com> wrote: > > > On Mon, May 20, 2013 at 12:51 PM, Vernon D. Cole > > wrote: > > > Stuart: > > > > > > I have just uploaded a branch for your consideration for > > > https://code.launchpad.net/dbapi-compliance. This version adds a second > > > field to the second test table (barflies) and loads that field with the > > > literal 'thi%s :may cau%(e)s? troub:1e'. The value of this literal is a > > > trap for unwary format conversion routines which fail to notice that they > > > are re-formatting a literal value. > > > > You are requiring the sql string to be parsed, literal string > > identified and placeholders inside them not treated. This is not a > > DBAPI2 requirement. > > > > Indeed, the DB API2 does not address parameter reformatting at all. We are > just starting to play with that. I am merely making sure that a reformat > operation does not step on the contents of an ANSI standard literal. And, > yes, when I am reformatting a string I do have to parse it. How else would > I find my format markers? This would restrict the DBAPI to only deal with ANSI standard literal, not what whatever the server is ready to accept, limiting the usability scope of the DBAPI3 (unless you are asking to amend the DBAPI2 introducing the restriction back, as it seem you want fixing the dbapi2 acceptance test suite without renaming to dbapi30.py) > > Without this requirement a driver can be implemented just passing the > > string to the client library, an extremely reasonable way to implement > > a client library wrapper. If you add it every driver will need to know > > the parsing rule of the server (and of its version). This is not a > > task as trivial as counting the quotes > > > As a matter of fact, counting quotes handles this problem quite nicely. > Actually, in my code I don't really count them. I do a .split("'") [that > is using a single quote] and then only parse the odd-numbered tokens. Then > do a "'".join(). Probably that's enough for ADO (is that a client library? I've worked with ODBC/ADO/DAO/Jet too much time ago). It's not for Postgres in my experience. > > or applying a regexp: take a > > look at the Postgres literals grammar: > > . > > > v v v v v v v v > 4.1.2.1. String Constants > > A string constant in SQL is an arbitrary sequence of characters bounded by > single quotes ('), for example 'This is a string'. To include a > single-quote character within a string constant, write two adjacent single > quotes, e.g., 'Dianne''s horse'. Note that this is not the same as a > double-quote character ("). > ^ ^ ^ ^ ^ ^ > > That's what I am using for this test: an arbitrary sequence of characters > bounded by single quotes. Look at the scroll bar: the page doesn't stop there. You are skipping beasts as: U&'d\0061t\+000061' U&'d!0061t!+000061' UESCAPE '!' $$Dianne's horse$$ $SomeTag$Dianne's horse$SomeTag$ the latter can also be nested: $function$ BEGIN RETURN ($1 ~ $q$[\t\r\n\v\\]$q$); END; $function$ Of course a postgres table could also be called ' (single quote): it's a matter of applying the syntax for identifiers quoting, so I think you can write insert into "'" values (?, '?') without the server complaining. But that should be translated into insert into "'" values ($1, '?') while I think your algorithm would generate insert into "'" values (?, '$1') And there are the comments too: -- comment'd Which can also be /* multiline /* and nest'd */ */ > > Of course PG developers are free to complicate that as much as they > > want: a client driver is expected to be agnostic and just forward the > > query. > > > > Yes, that is the point: forward the query, not mess it up. In this case I would be only bound to support $n placeholder, with no possibility whatsoever to be DBAPI compliant. If instead I don't have to make the distinction of the placeholder being inside or outside the string, placeholders replacement is an affordable operation (e.g. ? can be replaced with $1 and ?? can be replaced with ?. And ??? can be replaced alternatively by $1? or by ?$1, according to the current lunar phase). > > I don't think any currently implemented driver behaves as you wish. > > > > I happen to _know_ that at least _one_ does. ( My code is GPL, feel free > to copy. I copied from yours yesterday morning.) Thank you for the offer, but really I wouldn't like to put myself in the position of writing a client program to parse postgres grammar. > ProgrammingError: Incorrect number of bindings supplied. The current > > statement uses 1, and there are 2 supplied. > > > > That is, indeed, a programming error. "h" and "i" are two values. The line > should have another comma: > In [6]: cur.execute("insert into test values (?, 'wat?')", ('hi',)) > I usually pass a single parameter in a list simply to avoid that pattern: > In [6]: cur.execute("insert into test values (?, 'wat?')", ['hi']) > > (for extra fun, I don't think sqlite3 even contemplates the > > possibility to use ? into a literal and neither ?? nor \? appear to > > work, see ) > > > > Here is what the document you linked says: > v v v v > 4.1.2.1. String Constants > [...] > Nothing in there about avoiding question marks. That's the wrong link, it's postgres grammar, not sqlite (about which I've already ack'd it was the wrong example, so no need to go there and check). sqlite is not extensible so a ? can either be a placeholder or be in a string literal, otherwise it's a parse error. Furthermore, sqlite parser is in the client library, not in the server. Question marks are not special for postgres. Actually they are valid operators: . In your model it should be the driver's responsibility to convert select * from foo where a ?? ? and what = '?' select * from "'" where a ?? ? and what = '?' into select * from foo where a ? $1 and what = '?' select * from "'" where a ? $1 and what = '?' > Now let's try it.... > Seems to me that counting quotes works. Sorry, I don't have windows to test it. Does it work with a table called "a'b" too? Does cur.execute("insert into table values (? || $$?$$)", ('hello',)) do what expected? Does it deal with comments in the query string ok? -- Daniele From Chris.Clark at actian.com Mon May 20 21:31:25 2013 From: Chris.Clark at actian.com (Chris Clark) Date: Mon, 20 May 2013 12:31:25 -0700 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <1EAB1FE2-DB79-4444-9149-568560286C1A@zzzcomputing.com> References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> <1EAB1FE2-DB79-4444-9149-568560286C1A@zzzcomputing.com> Message-ID: <519A7A0D.5090407@actian.com> On Mon, 20 May 2013 12:40:04 -0400, Michael Bayer wrote: > On May 20, 2013, at 11:20 AM, Carl Karsten wrote: > >> Does anyone on in this discussion actually want named parameters for >> their own use, or does it seem like something someone might want, so >> lets try to make that happen just in case? >> >> I can see the benefit (easier to read code) but given the amount of >> problem it would seem to cause, it doesn't seem worth it. >> >> But I have lost track of the pros-n-cons of the various options. I >> think it is time to start a wiki with pointers to example code. > We should re-locate the original thread where I think a lot more DBAPI authors weighed in, and we had decided on qmark and named after much deliberation. > > I maintain that a paramstyle that specifically does not overlap with Python's own built-in formats is more appropriate, for the reasons I've outlined (works poorly with other systems that already deal with Python formatting, leads DBAPI authors to be lazy, rely too heavily on Python built-in behavior rather than considering bound parameters as a special case, leads to inconsistent implementation schemes, confuses end users since it isn't "true" Python formatting, is not in line with commonly accepted standard conventions in the greater database adapter community). Apologies for the long mail, I've trimmed it as much as I could in the time I had. We had a voting thread, for example http://mail.python.org/pipermail/db-sig/2007-June/005084.html (although there where earlier threads in 2006 April). I got the impression supporting (at least) 2 parms styles in a single was a comprise. From my perspective I'd prefer a single style, qmark. I think Daniele and are in agreement that supporting two different styles in a driver is not ideal. I suspect we disagree on what the single style should be :-) I'm ok with the other styles being optional, this helps with backwards compat for old applications (I'm ignoring how to enable that backwards compat). Gerhard had a suggestion on how to deal with transitioning over to a single style http://mail.python.org/pipermail/db-sig/2006-April/004681.html the issue then was that we didn't have any sample code, it looks like Daniele has started on that https://github.com/dvarrazzo/qmarkpg For most drivers (for example, ODBC and JDBC) there is a single bind parameter indicator and it is '?', the driver implementer MUST implement it. The original Python dbi spec was (I think) a thin veneer over the native drivers that did not attempt to hide the DBMS parameter passing symbols. This appears to be the real sticking point, we have 2 use cases 1. "I want to write a python database application" * Some users want a single way that works with all backends. I'm conveniently ignoring SQL dialect issues here, for a simple "SELECT col1 from some_table where col2 = ?" SQL dialects are not an issue. The ODBC (etc.) spec ignore SQL dialect issues too. 2. "I want to write a python database application". * they really want Python access to the database API that doesn't get in the way. They are not concerned about other vendors. There is nothing wrong with this approach for those users but this isn't a good use case when support for other backends is needed. Chris From carl at personnelware.com Mon May 20 22:25:10 2013 From: carl at personnelware.com (Carl Karsten) Date: Mon, 20 May 2013 15:25:10 -0500 Subject: [DB-SIG] Test suite update for format conversion accuracy. In-Reply-To: References: Message-ID: On Mon, May 20, 2013 at 11:46 AM, Vernon D. Cole wrote: > when I am reformatting a string I do have to parse it. I think we need to see a reason to reformat a string. In fact I think we need to see a reason to do any of this. I have a feeling this group is not in agreement what the over all goal is, but are debating implementations as if they all would achieve the same goal. One goal I think we can all agree on: Python bindings to the client library. I do not want to ever write this code: Py_BEGIN_ALLOW_THREADS ; conn = mysql_init(&(self->connection)); if (connect_timeout) { unsigned int timeout = connect_timeout; I want someone else to, and I want there to be one version that is the blessed version that 99% of all python developers use. In general when I run into the need for python bindings, I just want a 1:1 interface between my python code and the binary compiled library I am working with. When I don't have that, I am pretty much dead in the water. The few times I have tried to use swig of ctypes or whatever, I failed. In my dream world, here is my vision of the perfect API (which may or may not be dbapi) import foo con = foo.connect( server, credentials... ) cur = con.execute( command, parameters ) command and parameters wold get passed to the server with as little transformation as possible. I think that means the command would be passed char for char (note: unicode encoding takes place here) and the parameters get converted from the python implementation to whatever the client lib needs. At this layer, I only want to see the features of the foo server. If the foo server doesn't have the concept of transactions, then I don't want to see any transaction related code. The foo module could have a set of tests that test what it implements. Once that is implemented, foo can be wrapped to make a dbapi3_level2 compliant module. And that could could be wrapped to create a dbapi3_level3 compliant module. I hesitate to call foo dbapi3_level1 because who knows what the client lib's api look like. If we break the modules up into these 3 levels, I think it will be easier to define where features will live. For instance, I see level 3 being the place to transform tokens from a uniform "everything uses ?" to whatever the server is expecting. 3 levels will also make it way easier to fix problems, either existing bugs or future server features that bubble up into the client code and cause problems. (I am making up this future problem, I am not sure it applies to something as static as SQL servers.) Much of my motivation for the above comes from wanting to change a few lines of code in the the MySql module: query = query % tuple(( get_codec(a, self.encoders)(db, a) for a in args )) self._query(query) https://sourceforge.net/p/mysql-python/svn/HEAD/tree/trunk/MySQLdb/MySQLdb/cursors.py#l188 I can't fix that, because there is not python binding to the "pass parameters to server" thing. I think if the 2 levels were split apart I would have an easier time getting some ctypes hacker to fix the level 1, and then I wold have a chance of fixing the python code. I think what would come of this is ORM developers would use the level 2 module, and app developers who don't want to use an ORM would use the level 3 module. And everyone in this thread would be happy, because every feature described could be implemented, and features that are mutually exclusive would not step on each other. -- Carl K From daniele.varrazzo at gmail.com Tue May 21 00:44:02 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Mon, 20 May 2013 23:44:02 +0100 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <519A7A0D.5090407@actian.com> References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> <1EAB1FE2-DB79-4444-9149-568560286C1A@zzzcomputing.com> <519A7A0D.5090407@actian.com> Message-ID: On Mon, May 20, 2013 at 8:31 PM, Chris Clark wrote: Hi Chris, thanks for your summary, it has explained me a couple of things. > We had a voting thread, for example > http://mail.python.org/pipermail/db-sig/2007-June/005084.html (although > there where earlier threads in 2006 April). That's why I haven't found traces of the poll in my mail and just trusted it: I wasn't subscribed yet to the dbsig at the time, and I thought you were referring to some much more recent poll. > For most drivers (for example, ODBC and JDBC) there is a single bind > parameter indicator and it is '?', the driver implementer MUST implement it. > The original Python dbi spec was (I think) a thin veneer over the native > drivers that did not attempt to hide the DBMS parameter passing symbols. > This appears to be the real sticking point, we have 2 use cases > > 1. "I want to write a python database application" > * > > Some users want a single way that works with all backends. I'm > conveniently ignoring SQL dialect issues here, for a simple > "SELECT col1 from some_table where col2 = ?" SQL dialects are > not an issue. The ODBC (etc.) spec ignore SQL dialect issues too. > > 2. "I want to write a python database > application". > * they really want Python access to the database API that doesn't > get in the way. They are not concerned about other vendors. > There is nothing wrong with this approach for those users but > this isn't a good use case when support for other backends is > needed. This is a very interesting split, and shows pretty much where all of our disagreement comes from. As a driver developer I have the 2.-style users in mind and want to use my driver directly and allow to use whatever feature the database I support exposes. Michael thinking about SQLAlchemy, Vernon thinking about Django focus on an 1.-style usage, where they write a program using the driver and the final user only uses the driver indirectly. From the latter point of view I agree uniformity is more valuable than flexibility (e.g. to permit the same driver to support both sequence and mapping arguments). With uniformity in mind, I think the requirement in the title of this thread should be further restricted rejecting the :named style too, leaving indeed *only* qmark as supported paramstyle: a driver that would attempt to be flexible and implement both would find an ambiguous grammar to deal with, whereas leaving only the ? to juggle is a much easier problem. I will soon update qmarkpg to only support its eponymous paramstyle, dropping named. It is now a trivial task to write a generalized version of qmarkpg, taking a conforming dbapi2 driver and converting it into a driver supporting qmark and only qmark, getting rid of any ambiguities. I would call it qmarker. Ironically, looking to see if the name was already taken in PyPI I've found: https://pypi.python.org/pypi/sqlparams/ """ sqlparams is a utility module for simplifying the use of SQL parameters in queries. Some Python DB API 2.0 compliant modules only support the ordinal qmark or format style parameters (e.g., pyodbc only supports qmark). This utility module provides a helper class, SQLParams, that is used to support named parameter styles such as named, numeric and pyformat, and have them safely converted to the desired ordinal style. """ He must be a frustrated 2.-style user :) So I believe there is really no need for the spec to restrict placeholder styles: you may come out with two flavours of the specs: kinda dbapi3-human and dbapi3-orm, the latter demanding qmark only; but because you can mechanically convert any placeholder to qmark in a wrapper module I see no reason for the dbapi to rule on that and cut out the realistic use cases of direct users of a dbapi module (at the same time forcing an unworkable upgrade path). SQLAlchemy, Django, etc. may just use psycopg2 as: import qmarker import psycopg2 as _psycopg2 psycopg2 = qmarker.qmark_module(_psycopg2) # your magic here -- Daniele From vernondcole at gmail.com Tue May 21 00:44:53 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Mon, 20 May 2013 16:44:53 -0600 Subject: [DB-SIG] Test suite update for format conversion accuracy. In-Reply-To: <1369072658.29169.71.camel@risotto.smithersbet.com> References: <1369072658.29169.71.camel@risotto.smithersbet.com> Message-ID: Daniele, Daniele, Daniele: Calm down. It's only a __TEST__. It doesn't restrict _anything_. If you look in qmarkpg/tests/dbapi20.py, you will see code for Python 3 support, some of which is actually forbidden by PEP 249 -- because Python 3 did not exist when the PEP was written. We (*) had to guess what was the most reasonable thing to do in the spirit of the PEP, and made the changes we thought were best to move into the Python 3 world. (*) "We" meaning Stuart Bishop, Roger Upole, Mark Hammond, and myself, "kf7xm". Just because we made a test which works in Python 3, does that mean that everyone has to switch to Python 3 tomorrow and throw away all their old code? Of course not. MxODBC, for example, still does not work on Python 3. We got ready, that is all. Now we appear to be on the edge of a similar change cycle: SQL format converters are becoming more common. In the past few months you have written one, I have written one, there is a new one going into django's Oracle back end -- they are appearing all over the place. Taking a closer look is not a bad idea. I have noticed that some of these converters have not been careful about string literals -- and I have this funny, old fashioned idea that the value of a literal should not change anywhere between the source code and the data base. Thirty years ago, when I was writing Quality Assurance software for a living (please don't tell anyone that it was in COBOL) I learned that you do the simple, obvious test first, then later, if you have budget, you can mess with the corners and edges. Every test in the dbapi20 suite is of that simple-and-obvious variety. The existing parameter substitution test is, roughly: "Using the parameter style that you prefer, can I make one parameter substitution of one simple string variable, store it in the data base, and get it back out?" I am proposing to change that test to: "Using the parameter style that you prefer, can I make one parameter substitution of one simple string variable, store it in the data base, and get it back out, without altering the value of a nearby simple string literal?" I really don't think that is an unreasonable test -- of any database software. I am sorry that it was a program with your name attached that, I admit, I was looking at when I decided to get off my duff and write that extra test I have been thinking about for an age or two. I don't really understand Regular Expressions and was quite interested to find out the results, because I did not know what they would be. I do not write tests for products unless I believe in their possibilities. That's how open source improves. We look at each other's work and learn from it, and make suggestions -- even if that suggestion is only: "look at that, I think it may be a bug." Thank you, for "if isinstance(parameters, Mapping):" it is exactly the correct answer and I had spent hours trying to find it. You asked what ADO is. I am not sure what noun to use. Octopus comes close. I talk to it and it talks to a hundred or so various tabular data sources. The test suite runs against PostgreSQL, MySQL, SQL Server, and Jet. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike_mp at zzzcomputing.com Tue May 21 01:37:26 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Mon, 20 May 2013 19:37:26 -0400 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> <1EAB1FE2-DB79-4444-9149-568560286C1A@zzzcomputing.com> <519A7A0D.5090407@actian.com> Message-ID: <3531592C-1CD4-408D-AFE8-E56B43C04182@zzzcomputing.com> On May 20, 2013, at 6:44 PM, Daniele Varrazzo wrote: > > So I believe there is really no need for the spec to restrict > placeholder styles: you may come out with two flavours of the specs: > kinda dbapi3-human and dbapi3-orm, the latter demanding qmark only; > but because you can mechanically convert any placeholder to qmark in a > wrapper module I see no reason for the dbapi to rule on that and cut > out the realistic use cases of direct users of a dbapi module (at the > same time forcing an unworkable upgrade path). SQLAlchemy, Django, > etc. may just use psycopg2 as: > > import qmarker > import psycopg2 as _psycopg2 > psycopg2 = qmarker.qmark_module(_psycopg2) > # your magic here It is obvious that any number of wrappers can be made to make the paramstyle act like anything- the discussion here is only about a change for a new DBAPI3 specification. Again, SQLAlchemy has no need for such a wrapper as it already essentially *is* one, and if and when a DBAPI3 spec ever came out we will still be supporting DBAPI2 for decades seeing that we support many DBAPIs that haven't gotten updated in months/years, so again, nothing is changing in the slightest for SQLAlchemy. The discussion is strictly about attempting to improve a spec that for some inexplicable reason defines six different ways to do the same thing for no apparent reason - and if this one simple and glaring wart can't be improved upon, then I am very pessimistic about the reality of a DBAPI3 spec; we might as well just keep adding more optional bells, whistles and accessors to DBAPI2. There are of course upgrade paths if a real DBAPI3 with backwards incompatible changes came out, backwards incompatible upgrade paths are a basic fact of software development which we all must deal with every day. In this case, users of the new spec import from a new namespace, or add new arguments to how they connect, simple as that - no existing users need to be impacted in any way. Backwards incompatible changes are the only way you can remove cruft from an API. But if we're deciding that cruft removal is not an option, then the whole discussion here has no point. From vernondcole at gmail.com Tue May 21 01:37:52 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Mon, 20 May 2013 17:37:52 -0600 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> <1EAB1FE2-DB79-4444-9149-568560286C1A@zzzcomputing.com> <519A7A0D.5090407@actian.com> Message-ID: I just now realized that my quickly typed SUBJECT: line may have been the cause for misunderstanding here... It should have been "...limiting *mandatory* praramstyle to ['named','qmark']..." There has never been a suggestion that you cannot optionally support others .. just that you must support at least these two. Sorry. Vernon Cole -------------- next part -------------- An HTML attachment was scrubbed... URL: From dieter at handshake.de Tue May 21 07:43:28 2013 From: dieter at handshake.de (Dieter Maurer) Date: Tue, 21 May 2013 07:43:28 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> Message-ID: <20891.2432.794523.577001@localhost.localdomain> Carl Karsten wrote at 2013-5-20 10:20 -0500: >Does anyone on in this discussion actually want named parameters for >their own use, or does it seem like something someone might want, so >lets try to make that happen just in case? I you have commands with a large number of parameters, named parameters are more readable and less error prone. Otherwise, it is simple to get the parameter order wrong. >I can see the benefit (easier to read code) but given the amount of >problem it would seem to cause, it doesn't seem worth it. Well sometimes, SQL commands need a large number of parameters... -- Dieter From vernondcole at gmail.com Tue May 21 18:32:53 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Tue, 21 May 2013 17:32:53 +0100 Subject: [DB-SIG] Two sample implementations of ['named', 'qmark'] auto switch (a report) Message-ID: There have been a few comments that automatically detecting SQL parameter style based on the container that parameters are passed in might not work. I have discovered that sometimes trying to put something in code sheds a lot of light on my understanding, so I decided to try it. Indeed, my last comment to this group "just try to get the key" was completely unworkable. This is my report. I added another paramstyle, 'auto', to adodbapi.py The actual code addition was trivial: two lines altered... >>>accepted_paramstyles = ('qmark', 'named', 'format', 'auto') and >>>...elif self.paramstyle == 'named' or (self.paramstyle == 'auto' and isinstance(parameters, dict)): Just to experience working in an autoswitched world, I reset the default: >>>paramstyle='auto' # the default parameter style I added some appropriate tests to my unittest suite, and went on my way to other things. Several hours later, errors appeared in the tests for the django-mssql project I was working on. "Ahah", I thought: "I found a place where auto switch breaks something!" It was dbapi20.py (running as a sub-test) informing me about an unknown paramstyle. Later, I downloaded Daniele Varrazzo's qmarkpg package. Daniele uses pure autoswitching. His only reference to paramstyle is when it is defined. I quote: >>>paramstyle = 'named' # or qmark? whatever. The heart of his algorithm is as follows (I will cut out details).. > from collections import Mapping, Sequence > > ...clip... > def convert_params(query, args): > """ > Convert a qmark query into "format" or a named query into "pyformat". > > I'm not sure it is possible to disambiguate the two query styles, I'm not > even sure anybody has tried showing it is possible or the contrary. So we > try to infer from the args which type of arguments the query has. > """ > if args is None: > # no placeholder here > return query > > > elif isinstance(args, Mapping): > ...clip... > return RE_NAMED.sub(sub_mapping, query) > elif isinstance(args, Sequence): > > ...clip... > return RE_QMARK.sub(sub_sequence, query) > else: > raise TypeError('expected a sequence or mapping argument') > > A quick check of the psycopg2 documentation reveals that, indeed, it does not care whether you hand it 'format' or 'pyformat' queries: it autoswitches. It has done that for years. Daniele just hands off the autoswitch to it. (I admired his use of collections.Mapping, and immediately copied it. Made my code one line longer for the import.) My conclusion: Auto switching is easy, practical, and has years of precedent history. DB-api 3 should use it as the default action. - - - - - - - - - - - - - - - - - - - - - - - - - - References: https://sourceforge.net/projects/adodbapi/files/adodbapi/2.5.0/ (Works with most databases, but you have to have a Windows machine to act as a proxy server if you are running Linux. You will have to set .paramstyle = 'auto' -- it defaults to 'qmark') https://github.com/dvarrazzo/qmarkpg (Requires PostgreSQL and psycopg2. Works on Linux or Windows.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniele.varrazzo at gmail.com Tue May 21 19:24:16 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Tue, 21 May 2013 18:24:16 +0100 Subject: [DB-SIG] Two sample implementations of ['named', 'qmark'] auto switch (a report) In-Reply-To: References: Message-ID: <1369157056.9697.45.camel@risotto.smithersbet.com> On Tue, 2013-05-21 at 17:32 +0100, Vernon D. Cole wrote: > Daniele uses pure autoswitching. His only reference to paramstyle is when > it is defined. I quote: > > >>>paramstyle = 'named' # or qmark? whatever. > > The heart of his algorithm is as follows (I will cut out details).. > > > from collections import Mapping, Sequence > > > > ...clip... > > def convert_params(query, args): > > """ > > Convert a qmark query into "format" or a named query into "pyformat". > > > > I'm not sure it is possible to disambiguate the two query styles, I'm not > > even sure anybody has tried showing it is possible or the contrary. So we > > try to infer from the args which type of arguments the query has. > > """ > > if args is None: > > # no placeholder here > > return query > > > > > > elif isinstance(args, Mapping): > > ...clip... > > return RE_NAMED.sub(sub_mapping, query) > > elif isinstance(args, Sequence): > > > > ...clip... > > return RE_QMARK.sub(sub_sequence, query) > > else: > > raise TypeError('expected a sequence or mapping argument') > > This was the code from qmarkpg 0.1, which was supporting both qmark and named. Note a fundamental flaw: if args is a sequence it unescapes ?? into ?; if it is a mapping it unescapes :: into :. But if it is None it doesn't unescape anything, i.e. the kind of nuisance Michael was reporting about psycopg not doing %% -> % if no param is passed. Because of this shortcoming I've dropped the mapping support altogether from qmarkpg 0.2, because qmark is sufficient for the use in a framework looking for uniformity more than flexibility and because so I can unescape ?? -> ? in any situation - whether I got params or not. > > A quick check of the psycopg2 documentation reveals that, indeed, it does > not care whether you hand it 'format' or 'pyformat' queries: it > autoswitches. It has done that for years. Daniele just hands off the > autoswitch to it. Yes, but with a fundamental difference: psycopg switches according to the placeholder found in the query, it doesn't look at the type of the param; i.e. the algorithm used is something like: - find the first % in the query string - take the following char - is it a '%'? - add a single '%' to the query - is it an 's'? - we are dealing with a sequence - escape and add args[0] to the query - is it an '('? - we are dealing with a mapping - parse the name until the ')s' - escape and add args[name] to the query - everything else is an error and goes on but giving error if a mix of %s and %(name)s is found. It is a stripped-down version of the string % operator (which indeed can take no problem either a tuple or a mapping on the RHS). It works spectacularly fine but only because the placeholders are not ambiguous: looking at the char after each % in the query I can tell unambiguously if I am in format, pyformat or error state. I am not sure you can do the same with the qmark/named pair: this is the heart of my observation that format/pyformat are a superior combination w.r.t. parseability and it's not easy so write an adapter supporting qmark/named *together* (whereas supporting only one of them is straightforward). Really, it's not like I don't like to code :) > (I admired his use of collections.Mapping, and immediately copied it. Made > my code one line longer for the import.) > > My conclusion: Auto switching is easy, practical, and has years of > precedent history. DB-api 3 should use it as the default action. I'd love that, but please take your time to review the ambiguous cases and the consequences in usability of the specific grammar I'd chosen for qmarkpg 0.1, which implicitly demanded the query to: - have literal ? escaped as ??, : not escaped if args is a sequence - have literal : escaped as ::, ? not escaped if args is a mapping - don't escape neither ? nor : if no param is passed It doesn't feel very ergonomic. Have you got something better in mind? -- Daniele From wilk at flibuste.net Tue May 21 19:39:43 2013 From: wilk at flibuste.net (wilk) Date: Tue, 21 May 2013 17:39:43 +0000 (UTC) Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> Message-ID: On 19-05-2013, Carl Karsten wrote: > I don't think it is reasonable to expect python's dbapi to be as db > agnostic as we would like it to be. Of course, we will not resolv this issue, sql code will still be specific to database. But, as a developer using differents databases, even if i accept to write specific sql code, i like to use the same paramstyle. So i wrote my own paramstyle with $x and write a litle wrapper to database paramstyle (and i will gain 2 lines of code if i could know the paramstyle from the connection). My own because it's a lot of corner cases and i prefer to know exactly what i do... Don't forget that a lot of developers tired of paramstyle incompatibilities don't use any of them and use string concatenation instead ! What do you think about qmark mandatory and an sqlparamters function in stdlib ? Maybe an extension of string format, it could be fine to can do: i = "update mytable set x={v}".sqlformat cursor.execute(i(v=5)) -- William Dod? Informaticien Ind?pendant From mal at egenix.com Tue May 21 20:30:41 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 21 May 2013 20:30:41 +0200 Subject: [DB-SIG] paramstyle specification (was: Two sample implementations of ['named', 'qmark'] auto switch (a report)) In-Reply-To: <1369157056.9697.45.camel@risotto.smithersbet.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> Message-ID: <519BBD51.7050905@egenix.com> Wow, a lot of discussion to read :-) I'll start with some comments on the more important things. On 21.05.2013 19:24, Daniele Varrazzo wrote: > On Tue, 2013-05-21 at 17:32 +0100, Vernon D. Cole wrote: >> My conclusion: Auto switching is easy, practical, and has years of >> precedent history. DB-api 3 should use it as the default action. -1 on making this the default for the reasons I already stated earlier on. We can make it a possible paramstyle value, but not the default. Whether it works or not depends a lot on the supported paramstyles that the auto-detection can detect and even then the algorithm can run into problems when e.g. seeing a parameter object that implements both the mapping and sequence protocol, accidental use of (unresolved) Python formatting characters in a SQL statement, etc. Note that not all Python types are associated with an abstract base class, e.g. take a user defined class that implements the mapping or sequence protocol, but doesn't inherit from the ABCs. Also note that the DB-API 2.0 has so far mandated that .__getitem__() be used for lookups, so that both sequences and mappings can be used - even for positional paramstyles. I'm not sure many database modules implement this. We might want to require ABC inheritance to be used for parameter sequences/mappings in DB-API 3.0 to resolve this issue. > I'd love that, but please take your time to review the ambiguous cases > and the consequences in usability of the specific grammar I'd chosen for > qmarkpg 0.1, which implicitly demanded the query to: > > - have literal ? escaped as ??, : not escaped if args is a sequence > - have literal : escaped as ::, ? not escaped if args is a mapping > - don't escape neither ? nor : if no param is passed > > It doesn't feel very ergonomic. Have you got something better in mind? I agree that we need to think more carefully about the definition of the paramstyles if we want to make them standard in DB-API 3.0. However, escaping doesn't strike me as one of the important details, since I have yet to find an example where you'd actually need escaping :-) First of all, SQL string literals should not be subject to marker parsing. They must be skipped and care must be taken to make sure that the '-quoting is being respected ('' -> '). * qmark case: question marks don't appear outside SQL string literals, so you don't need to escape them - they will always refer to binding markers * named case: it should be enough to make sure that the colon of a named marker is not preceded by another colon (negative look-behind). If it is, the potential substring is not a named marker, e.g. "abc::integer < 3" is not a named marker, "abc < :integer" is a named marker (please correct me, if I'm missing an important use case) For other paramstyles, such as the format and pyformat ones, you probably need to apply the standard "%%" escaping, since "%" is sometimes used in SQL as operator (e.g. MySQL). However, I think we can leave those deliberately underspecified in DB-API 3.0 to both encourage use of one of the mandatory paramstyles, and to allow database module authors to continue exposing natively used paramstyles. Something we do need to address in a paramstyle spec for qmark and named is SQL comments. Perhaps easiest would be to disallow them in SQL statements passed to .execute*(). PS: mxODBC 3.2 implements both styles and does conversion from named to qmark. When choosing named style, we explicitly disallow question marks to appear in the SQL statement to detect errors early and give proper error messages to the user, rather than some obscure "too few parameters for statement". -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 21 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Tue May 21 20:55:05 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Tue, 21 May 2013 14:55:05 -0400 Subject: [DB-SIG] paramstyle specification (was: Two sample implementations of ['named', 'qmark'] auto switch (a report)) In-Reply-To: <519BBD51.7050905@egenix.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> Message-ID: <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> On May 21, 2013, at 2:30 PM, M.-A. Lemburg wrote: > > Something we do need to address in a paramstyle spec for > qmark and named is SQL comments. Perhaps easiest would be > to disallow them in SQL statements passed to .execute*(). um, meaning no SQL comments allowed in a call to cursor.execute() ? That would be a really big problem for folks who pass comments in their SQL as a means to help with database query log parsing. There are definitely folks who do that, and it's very useful. > > PS: mxODBC 3.2 implements both styles and does conversion from > named to qmark. When choosing named style, we explicitly disallow > question marks to appear in the SQL statement to detect errors > early and give proper error messages to the user, rather than > some obscure "too few parameters for statement". you used the word "underspecified" a moment ago which I found frightening, but then three paragraphs later it's just the word I need - I think KISS should be the rule here, the parser is only aware of one paramstyle at a time, and doesn't try to do any nannying of things that look like other paramstyles. From mal at egenix.com Tue May 21 21:07:01 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 21 May 2013 21:07:01 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> Message-ID: <519BC5D5.4000509@egenix.com> On 18.05.2013 03:29, Daniele Varrazzo wrote: > On Fri, May 17, 2013 at 5:57 PM, M.-A. Lemburg wrote: >> >>> Reading back the upstream thread I also see proposal to leave support >>> for all the current placeholders but mandate drivers to implement >>> qmark and named too. I think this is only going to complicate the >>> implementation of the drivers and of the programs using it. >> >> Why is that ? > > I'll try to explain that again. Let's say I work in a postgres-only > shop: currently there is no interoperability problems, and people > writes python-postgres programs and py-pg libraries. Let's say > psycopg3 (implementing dbapi3) implements a method to switch > paramstyle, whatever it is. > > Alice is writing a script program, reading the database, doing some > business-related math and generating a report. Of course the business > model is already implemented in the well tested and uniformly used > "shoplib" library, written by Bob: this library takes a psycopg2 > connection and runs whatever complicated query they need to implement > such model. > > In dbapi2 this is a piece of cake. In dbapi3 Alice is free to have her > opinions: she is hip and dbapi3 is all the rage, so she will create a > psycopg3 pyformat connection and pass that to the shoplib: of course > the program will crash and burn immediately. Alice may point out that > Bob is not providing complete support to a valid psycopg connection. > > There are two possible outcomes: 1. most likely Alice will be laughed > at and the shop will assume a policy of not using different paramstyle > than pyformat, hence making impossible to use libraries or frameworks > (sqlalchemy, django) that would of course benefit of the uniformly > used named format; Sticking with the DB-API 2.0 way of doing things is always an option, yes. I don't think we should change the paramstyle default of database modules to avoid excessive breakage. > 2. Alice is good enough into talking everybody that > dbapi3 is the future: now Bob in order to offer the shoplib service to > psycopg connection in either mode will have to jump through hoops: he > will either store the paramstyle of the connection received in a > temporary variable, switch it to pyformat and finally restore it (with > all the concurrency problems this would involve, as the connection may > be shared across threads); or he will have to dynamically generate > queries that would have otherwise been expressed as string constants. Bob's library would have to be updated to also handle connections that have a different .paramstyle setting, but the change is simple: he'd only have to make sure that when creating a cursor, he sets the cursor.paramstyle to whatever his library supports. There are no concurrency problems, as far as I can see. > This is the reason why I firmly believe that giving excessive freedom > to the users when designing a standard only means troubles not only > for who implements the service (the dbapi driver in our case) that has > to implement several equivalent options, but also for the users of > such driver, because writing a function library accepting connections > that may demand different parameter style is much more complicated > than one where you would just have done with a string, and offers new > and unexpected modes of failure. Doing that only to second stylistic > matters would be just irresponsible. Well, so far, we have given database module authors excessive freedom. DB-API 3.0 would move that freedom on to the users to make their lives easier :-) > If we are designing a standard we have to make decisions: every > decision we don't make and leave to somebody else only increases the > complexity, doesn't definitely add democracy and peace in the world. > And open choices don't increment complexity in additive way: they do > it rather in multiplicative way. > > My thoughts about the specific issue in this thread (not my wishlist > for dbapi3, with which I'll be glad to bore you to death in another > thread) is that > > 1. demanding a single paramstyle for the driver will only make > millions of people angry > > 2. making the paramstyle switchable will result in programs more > complicated, libraries *way* more complicated and database drivers > *way, way* more complicated, inevitably causing bugs on either side, > making millions of people angry > > 3. leaving the things as they are w.r.t. the paramstyle (and fix the > several shortcomings that dbapi 1 and 2 have shown in the about 10 > years they have been around) would make a dbapi3 more gradually > acceptable and not a watershed event. Fact is, we already have adjustable .paramstyles now in DB-API 2.0 modules. What we're trying to do here is standardize this a bit more, so that users can benefit from having multiple .paramstyles available. I don't think that you often run into a situation where you pass around database connection objects between libraries. If you do, those libraries will usually either be written for a single backend, or have knowledge of a few select databases and database modules. In the first case, they will adapt to whatever is standard for the used database module, in the second case, the library can be simplified to only have to deal with two paramstyles. Indeed, I believe that simplifying the paramstyle story will make libraries that work directly on connection objects easier to implement. If you look at JDBC, we're actually converging to an industry standard here :-) JDBC used to only support qmark style (since it built on the ODBC standard which mandates qmark style). In JDBC 3.0 they added named style as additional option. I can't say why, but from my experience with qmark style, named style does have its advantages when dealing with SQL statements that have lots of parameters, say e.g. INSERTs. So JDBC now has two paramstyles. The DB-API has had 5 paramstyles for many many years, and we're now trying to simplify this in a backwards compatible way to encourage use of only two paramstyles. > Oh, yes, and there was 4. "have the driver automatically detect the > paramstyle". Please, don't joke: this idea is so brittle it's not even > funny to demand it as part of a standard. Agreed :-) >> I think it would greatly simplify the use of the DB-API, since >> applications could then be written to use a single paramstyle, >> even when using multiple database modules. > > Come on, you are a database expert: do you really believe that because > two commands can be written with the same paramstyle it would be > possible to send the same query to different databases? With so > different sql syntaxes, so different data models, so different runtime > models? I don't think interoperability across database is a problem > that can be solved at dbapi level: the aim of the dbapi (and the > benefit it has provided in its not short life) is to provide > uniformity in the python code to access a database; I don't think it > being a database abstraction layer has ever been its scope. I was only referring to the parameter binding logic, not the SQL or data types involved. With adjustable paramstyles, people can greatly simplify their porting efforts when moving from one database module to another. E.g. take the reasoning why we added "named" paramstyle support to mxODBC in release 3.2: we had requests from customers who wanted to switch from a native Oracle driver to the Oracle Instant Client ODBC driver. Instead of having them rewrite thousands of lines of code to adapt from Oracle named to qmark paramstyle used in ODBC, we added automatic conversion and they are now able to run (mostly) the same code using the Oracle ODBC driver. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 21 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Tue May 21 21:24:19 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 21 May 2013 21:24:19 +0200 Subject: [DB-SIG] paramstyle specification In-Reply-To: <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> Message-ID: <519BC9E3.1010108@egenix.com> On 21.05.2013 20:55, Michael Bayer wrote: > > On May 21, 2013, at 2:30 PM, M.-A. Lemburg wrote: > >> >> Something we do need to address in a paramstyle spec for >> qmark and named is SQL comments. Perhaps easiest would be >> to disallow them in SQL statements passed to .execute*(). > > um, meaning no SQL comments allowed in a call to cursor.execute() ? That would be a really big problem for folks who pass comments in their SQL as a means to help with database query log parsing. There are definitely folks who do that, and it's very useful. Well, I know it's useful, but it also makes parsing SQL for parameter markers much harder :-) >> PS: mxODBC 3.2 implements both styles and does conversion from >> named to qmark. When choosing named style, we explicitly disallow >> question marks to appear in the SQL statement to detect errors >> early and give proper error messages to the user, rather than >> some obscure "too few parameters for statement". > > you used the word "underspecified" a moment ago which I found frightening, but then three paragraphs later it's just the word I need - I think KISS should be the rule here, the parser is only aware of one paramstyle at a time, and doesn't try to do any nannying of things that look like other paramstyles. I'm not sure I understand. With "underspecified" I meant that we leave the other paramstyles in the same unspecified state as they are now (or rather leave them specified by the database module implementations). For the two main paramstyles we're currently discussing, qmark and named, we will have to put more effort into specifying what they mean and how they work. I agree that adding magic auto-selection of SQL parsers is not a good idea. If you setup a cursor to expect qmark style, it should complain loudly when you try to pass in named style parameters or perhaps even named style SQL. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 21 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From Chris.Clark at actian.com Tue May 21 22:54:19 2013 From: Chris.Clark at actian.com (Chris Clark) Date: Tue, 21 May 2013 13:54:19 -0700 Subject: [DB-SIG] paramstyle specification In-Reply-To: <519BC9E3.1010108@egenix.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> <519BC9E3.1010108@egenix.com> Message-ID: <519BDEFB.3070308@actian.com> On Tue, 21 May 2013 21:24:19 +0200, M.-A. Lemburg wrote: > On 21.05.2013 20:55, Michael Bayer wrote: >> On May 21, 2013, at 2:30 PM, M.-A. Lemburg wrote: >> >>> Something we do need to address in a paramstyle spec for >>> qmark and named is SQL comments. Perhaps easiest would be >>> to disallow them in SQL statements passed to .execute*(). >> um, meaning no SQL comments allowed in a call to cursor.execute() ? That would be a really big problem for folks who pass comments in their SQL as a means to help with database query log parsing. There are definitely folks who do that, and it's very useful. > Well, I know it's useful, but it also makes parsing SQL for > parameter markers much harder :-) Oracle uses comments as optimizer hints, i.e. they are required for the SQL to run. If we have a parser, it needs to handle comments too. I.e. needs to know about strings (and skip them intelligently, including the 'It''s Monty Python' cases) as well as the various comments styles (which again should be skipped over), example comments styles are: 1. /* C style */ 2. -- Microsoft SQL Server style There may well be others. >>> PS: mxODBC 3.2 implements both styles and does conversion from >>> named to qmark. When choosing named style, we explicitly disallow >>> question marks to appear in the SQL statement to detect errors >>> early and give proper error messages to the user, rather than >>> some obscure "too few parameters for statement". >> you used the word "underspecified" a moment ago which I found frightening, but then three paragraphs later it's just the word I need - I think KISS should be the rule here, the parser is only aware of one paramstyle at a time, and doesn't try to do any nannying of things that look like other paramstyles. > I'm not sure I understand. > > With "underspecified" I meant that we leave the other paramstyles > in the same unspecified state as they are now (or rather leave them > specified by the database module implementations). > > For the two main paramstyles we're currently discussing, qmark > and named, we will have to put more effort into specifying > what they mean and how they work. > > I agree that adding magic auto-selection of SQL parsers is > not a good idea. If you setup a cursor to expect qmark style, > it should complain loudly when you try to pass in named style > parameters or perhaps even named style SQL. > I think I'm with Michael on this; I would prefer auto selection not be part of the spec, I also do not want auto detecting (and helper errors) either, I want to see the (possibly native DBMS) error when it finds "bad" SQL. If a driver authors wants to add detection logic (Marc-Andre clearly has customers who want and appreciate this) then the spec shouldn't disallow it either :-) I personally would like to see qmark the only required format (with others allowed), and have (as part of the spec) a reference implementation that could convert from qmark into X (under a permissive license) that driver authors could then use and include as part of their driver if they need it (the ideal would both a pure python and a C based implementation but I suspect python only would suffice). Chris From carl at personnelware.com Tue May 21 22:58:57 2013 From: carl at personnelware.com (Carl Karsten) Date: Tue, 21 May 2013 15:58:57 -0500 Subject: [DB-SIG] paramstyle specification In-Reply-To: <519BDEFB.3070308@actian.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> <519BC9E3.1010108@egenix.com> <519BDEFB.3070308@actian.com> Message-ID: On Tue, May 21, 2013 at 3:54 PM, Chris Clark wrote: > I personally would like to see qmark the only required format (with others > allowed), and have (as part of the spec) a reference implementation that > could convert from qmark into X (under a permissive license) that driver > authors could then use and include as part of their driver if they need it > (the ideal would both a pure python and a C based implementation but I > suspect python only would suffice). +1 -- Carl K From vernondcole at gmail.com Wed May 22 00:07:21 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Tue, 21 May 2013 16:07:21 -0600 Subject: [DB-SIG] paramstyle specification In-Reply-To: <519BC9E3.1010108@egenix.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> <519BC9E3.1010108@egenix.com> Message-ID: I think that we should be careful not to mandate any sophisticated grammar processing of the query string. The most notice we should take of comments would be to suggest that any reformatting operation might choose to stop at a double dash. /* C style */ comments and other non-ANSI-standard standard things should not be addressed. Consider what is the worst that could happen? Let's say a programmer sets my module into 'named' and passes me a query with an extra format pattern inside an extended syntax comment: SELECT a, b FROM table_c WHERE name = :nam /* :weird */ AND state = :st I search for a colon and find one. Valid names consist of alphanumeric and underscore, so I collect characters until I find the end. I replace the whole mess with a question mark, look for 'nam' in my parameter dictionary, and place the value in the parameter list I am building. Doing it again, I find 'weird' as the next token. Two things can happen: if 'weird' is in the parameter dictionary, I emit a question mark and add the value to the list. If not, then I raise an error. In the former case, the query gets passed to the SQL engine with three parameters for two (that it sees) question marks, and it will raise an error. No data is lost or altered. The programmer will learn not to do that again. This is a very different case from performing that same substitution in a string literal, where data _does_ get altered. Even then, I am not going to try to find every possible funny literal encoding scheme for every possible SQL dialect. I am only responsible for things between two single quotes. ANSI standard. The SQL compiler is the compiler. I am just running a little macro program in front. If running the macro creates syntax errors then the compiler lets them know, and they fix it. The simpler the rules for the macro the easier to debug the output. In particular, if there are no parameters, or if the paramstyle is in the native format for my engine, I will not look at the command at all. If a module author wants to do more elaborate syntax checking, fine. But it should not be included in the standard. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike_mp at zzzcomputing.com Wed May 22 00:52:39 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Tue, 21 May 2013 18:52:39 -0400 Subject: [DB-SIG] paramstyle specification In-Reply-To: <519BC9E3.1010108@egenix.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> <519BC9E3.1010108@egenix.com> Message-ID: On May 21, 2013, at 3:24 PM, M.-A. Lemburg wrote: > On 21.05.2013 20:55, Michael Bayer wrote: >> >> On May 21, 2013, at 2:30 PM, M.-A. Lemburg wrote: >> >>> >>> Something we do need to address in a paramstyle spec for >>> qmark and named is SQL comments. Perhaps easiest would be >>> to disallow them in SQL statements passed to .execute*(). >> >> um, meaning no SQL comments allowed in a call to cursor.execute() ? That would be a really big problem for folks who pass comments in their SQL as a means to help with database query log parsing. There are definitely folks who do that, and it's very useful. > > Well, I know it's useful, but it also makes parsing SQL for > parameter markers much harder :-) I'd definitely want DBAPIs to be willing to take that on, though, rather than disallowing comments. > >>> PS: mxODBC 3.2 implements both styles and does conversion from >>> named to qmark. When choosing named style, we explicitly disallow >>> question marks to appear in the SQL statement to detect errors >>> early and give proper error messages to the user, rather than >>> some obscure "too few parameters for statement". >> >> you used the word "underspecified" a moment ago which I found frightening, but then three paragraphs later it's just the word I need - I think KISS should be the rule here, the parser is only aware of one paramstyle at a time, and doesn't try to do any nannying of things that look like other paramstyles. > > I'm not sure I understand. > > With "underspecified" I meant that we leave the other paramstyles > in the same unspecified state as they are now (or rather leave them > specified by the database module implementations). Right, my vague idea here is that how to deal with ? or :names in comments would also be "underspecified". I.e. let's not even get into it. > For the two main paramstyles we're currently discussing, qmark > and named, we will have to put more effort into specifying > what they mean and how they work. > > I agree that adding magic auto-selection of SQL parsers is > not a good idea. If you setup a cursor to expect qmark style, > it should complain loudly when you try to pass in named style > parameters or perhaps even named style SQL. I'm guessing a bit what we mean by "auto-selection", here are the two scenarios I can see: 1. the cursor knows upfront what paramstyle is expected. This is achieved via the .paramstyle setting. When execute() is called, it parses the string appropriate to that paramstyle, and expects the parameter argument to be either a sequence or a mapping depending on if the style is positional or named. if the sequence/mapping is mismatched to the paramstyle, an error is raised. or 2. the cursor knows upfront that it will deal with a single "positional" or "named" parameter style. When execute() is called, it examines the parameter argument to see if it is a sequence or a mapping. Based on that, it determines to parse for the "positional" or "named" style. I'm guessing that #2 is the "magic auto-selection" here. I don't have a problem with it as I think some DBAPIs already do that, but I don't have a huge problem if we go with #1 either. But I would like to get it specified what happens when there are *no* parameters sent, as the DBAPIs behave inconsistently in this regard (some parse the string, others don't). As a total aside, I also have issues with some DBAPIs that accept *only* a list for the "sequence", others that accept *only* a tuple (this may be only when the parameter list is empty, however). So specifying the significance of "empty" parameter lists as well as how to test for "sequence" would be nice too, so that I can hit the DBAPI authors with bug reports backed up by the spec. From daniele.varrazzo at gmail.com Wed May 22 02:04:44 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Wed, 22 May 2013 01:04:44 +0100 Subject: [DB-SIG] paramstyle specification (was: Two sample implementations of ['named', 'qmark'] auto switch (a report)) In-Reply-To: <519BBD51.7050905@egenix.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> Message-ID: On Tue, May 21, 2013 at 7:30 PM, M.-A. Lemburg wrote: > However, escaping doesn't strike me as one of the important > details, since I have yet to find an example where you'd > actually need escaping :-) Uhm, I thought I'd shown a few of them. But they may have been scattered around in the long thread and spinoffs. So, my spanners in the machine are: > First of all, SQL string literals should not be subject to > marker parsing. They must be skipped and care must be taken > to make sure that the '-quoting is being respected ('' -> '). I've shown in a separate thread that postgres defines other string literals than the 'quoted' ones. For instance the multiline ones, using $$these$$ as delimiters, that may also be named and nested. There are of course comments too to take care of. And "quoted" identifiers. And on and on. Even the layman knowledge that quotes are escaped in sql by doubling them is shaken by the postgres parser: the string "a'b" can be expressed as E'a\'b'. In short, parsing postgres syntax requires a well built tokenizer on the client, an excessive requirement for a generic client-side library (especially one that is not supposed to "understand" the query being sent but whose only role is to pass it to the server). > * qmark case: question marks don't appear outside SQL string > literals, so you don't need to escape them - they will > always refer to binding markers In postgres ? is a valid operator, or part of valid operators. The following query reports there are 30 such operators predefined in postgres 9.1: =# select format('%s %s %s', oprleft::regtype, oprname, oprright::regtype) from pg_operator where oprname ~ E'\\?'; abstime tinterval path ?# path box ?# box ... There are hstore operators , geometric operators , and extension modules can -and do- add others . > * named case: it should be enough to make sure that the colon > of a named marker is not preceded by another colon > (negative look-behind). If it is, the potential substring > is not a named marker, e.g. "abc::integer < 3" is not a named > marker, "abc < :integer" is a named marker If this works, I can offer a life jacket proposing to always use the : as escape symbol, escaping ? as :? Don't know yet if it's valid: I'm not practical with negative look-behind parsing. Definitely doesn't look a familiar way to escape stuff. > However, I think we can leave those deliberately underspecified > in DB-API 3.0 to both encourage use of one of the mandatory > paramstyles, and to allow database module authors to continue > exposing natively used paramstyles. > > Something we do need to address in a paramstyle spec for > qmark and named is SQL comments. Perhaps easiest would be > to disallow them in SQL statements passed to .execute*(). It is definitely an "easy" route, but it leaves us with drivers that by spec may forbid the use of certain database features: I feel it an excessive price to pay in the name of consistency. Again, sorry if I'm repetitive, but note that many of the problems stem by a sort of implied feature request: that a driver can support both qmark and named placeholders. I see how a driver supporting both sequences and maps as parameters is handier than one only offering a single one, but I think that precisely the qmark/named pair is a dreadful combination to force into cohabitation, and this is the main force leading to unnatural requirements such as the impossibility to express some of the valid characters in the query. Even taking the hard case of postgres: if qmark was the only chosen style, single qmarks would become placeholders and double qmarks would become single (as operators, in literals, in comments, everywhere, without the need of a lexer). If instead only named was the chosen one people should write casts as a::::int, but the system would be provable correct. It's when both are allowed that escaping becomes an hairy thing. If the dbapi3 requirement becomes to mandate support for a single paramstyle things become manageable again. -- Daniele From dieter at handshake.de Wed May 22 08:44:15 2013 From: dieter at handshake.de (Dieter Maurer) Date: Wed, 22 May 2013 08:44:15 +0200 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: <519BC5D5.4000509@egenix.com> References: <519609A8.6080407@egenix.com> <51966163.5090507@egenix.com> <519BC5D5.4000509@egenix.com> Message-ID: <20892.26943.556812.307044@localhost.localdomain> M.-A. Lemburg wrote at 2013-5-21 21:07 +0200: > ... >I don't think that you often run into a situation where you >pass around database connection objects between libraries. If you drastically change things between version 2 and version 3 libraries, this may get more frequent: In a large system, you may have thousands of sql commands and it may be a heavy burden to adapt all of them from the version 2 to version 3 API. Thus, you may want to keep using a version 2 library for them. New (or updated) code, on the other hand, may want to use the new version 3 API. But, new and old code likely must work with the same database entities (tables) and therefore have a likely need to work in the same transaction. Transactions are now bound to a connection. Hence, one might want the same connection object for both version 2 and version 3 API. -- Dieter From vernondcole at gmail.com Wed May 22 10:58:45 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Wed, 22 May 2013 02:58:45 -0600 Subject: [DB-SIG] paramstyle specification (was: Two sample implementations of ['named', 'qmark'] auto switch (a report)) In-Reply-To: References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> Message-ID: The major difference -- the change of paradigm (to put it in elegant terms) -- between api 2 and our proposed api 3, is our use of 'paramstyle'. In api 2, we attempted to use it to tell our client programmer how to talk to us. That did not work out too well. In api 3, we propose to offer the service of letting our client programmer tell us how she wishes to talk, using some construct such as >>>cursr.paramstyle = 'named' This method of operation is so different that I wish we could use a different term for it, an actual English word. Except that I cannot think of one, and I refuse to type "parameter_replacement_variant", so paramstyle it remains, spell checker be damned. I am going to let Suzy Programmer use 'paramstyle' to say to me: "I would like to use named parameters at this moment." To which I am willing to reply: "Okay, encode them as a colon followed by the name. Please refrain from using that pattern inside of any obscuring structures, since that would foul up my ability to help you." I think that is a perfectly legitimate contract. But what about the fellow that worries Daniele -- the kind of dweeb who reads language manuals for entertainment and comes up with things like "Hey look!" "If I do _this_ I can make a table named 'single quote'." "Hold my beer while I show you." -- what about him? As much as we hate it, we all know that guy is out there, and perhaps, sometimes, maybe, might have a legitimate need. For that guy, we need a different contract, basically: "Get out of the way and let me do it myself." So I propose that we offer him that contract. Let's define (sorry) a new paramstyle, and call it 'native'. (I was first thinking of 'raw', but I like 'native' better.) That way, the expert (or ID 10 T) user _also_ has a way of expressing his wishes to us. He would have complete control and the ability to use any obscure features of his SQL dialect he wishes. He does not want or need parameter replacement. The list of mandated paramstyles would become three: ['named', 'qmark', 'native']. 'named' and 'qmark' may or may not get parameter replacement processing depending on the needs of the underlying engine. 'native' would always be passed to the engine unaltered. Pardon me if my proofreading failed. I broke my glasses last night. -- Vernon Cole -------------- next part -------------- An HTML attachment was scrubbed... URL: From niki.spahiev at gmail.com Wed May 22 15:54:25 2013 From: niki.spahiev at gmail.com (Niki Spahiev) Date: Wed, 22 May 2013 16:54:25 +0300 Subject: [DB-SIG] paramstyle specification In-Reply-To: References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> Message-ID: On 22.05.2013 11:58, Vernon D. Cole wrote: > The major difference -- the change of paradigm (to put it in elegant > terms) -- between api 2 and our proposed api 3, is our use of 'paramstyle'. > > In api 2, we attempted to use it to tell our client programmer how to > talk to us. That did not work out too well. > > In api 3, we propose to offer the service of letting our client > programmer tell us how she wishes to talk, using some construct such as > >>>cursr.paramstyle = 'named' > > This method of operation is so different that I wish we could use a > different term for it, an actual English word. Except that I cannot > think of one, and I refuse to type "parameter_replacement_variant", so > paramstyle it remains, spell checker be damned. Please help future googlers and use something distinctive - "param_variant"? Niki From mal at egenix.com Thu May 23 16:59:27 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 23 May 2013 16:59:27 +0200 Subject: [DB-SIG] paramstyle specification In-Reply-To: References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> <519BC9E3.1010108@egenix.com> Message-ID: <519E2ECF.2070104@egenix.com> On 22.05.2013 00:52, Michael Bayer wrote: > > On May 21, 2013, at 3:24 PM, M.-A. Lemburg wrote: > >> On 21.05.2013 20:55, Michael Bayer wrote: >>> >>> On May 21, 2013, at 2:30 PM, M.-A. Lemburg wrote: >>> >>>> >>>> Something we do need to address in a paramstyle spec for >>>> qmark and named is SQL comments. Perhaps easiest would be >>>> to disallow them in SQL statements passed to .execute*(). >>> >>> um, meaning no SQL comments allowed in a call to cursor.execute() ? That would be a really big problem for folks who pass comments in their SQL as a means to help with database query log parsing. There are definitely folks who do that, and it's very useful. >> >> Well, I know it's useful, but it also makes parsing SQL for >> parameter markers much harder :-) > > I'd definitely want DBAPIs to be willing to take that on, though, rather than disallowing comments. > > >> >>>> PS: mxODBC 3.2 implements both styles and does conversion from >>>> named to qmark. When choosing named style, we explicitly disallow >>>> question marks to appear in the SQL statement to detect errors >>>> early and give proper error messages to the user, rather than >>>> some obscure "too few parameters for statement". >>> >>> you used the word "underspecified" a moment ago which I found frightening, but then three paragraphs later it's just the word I need - I think KISS should be the rule here, the parser is only aware of one paramstyle at a time, and doesn't try to do any nannying of things that look like other paramstyles. >> >> I'm not sure I understand. >> >> With "underspecified" I meant that we leave the other paramstyles >> in the same unspecified state as they are now (or rather leave them >> specified by the database module implementations). > > Right, my vague idea here is that how to deal with ? or :names in comments would also be "underspecified". I.e. let's not even get into it. > >> For the two main paramstyles we're currently discussing, qmark >> and named, we will have to put more effort into specifying >> what they mean and how they work. >> >> I agree that adding magic auto-selection of SQL parsers is >> not a good idea. If you setup a cursor to expect qmark style, >> it should complain loudly when you try to pass in named style >> parameters or perhaps even named style SQL. > > I'm guessing a bit what we mean by "auto-selection", here are the two scenarios I can see: > > 1. the cursor knows upfront what paramstyle is expected. This is achieved via the .paramstyle setting. When execute() is called, it parses the string appropriate to that paramstyle, and expects the parameter argument to be either a sequence or a mapping depending on if the style is positional or named. if the sequence/mapping is mismatched to the paramstyle, an error is raised. This is the "explicit is better than implicit" version I'm in favor of for the DB-API standard. Database modules may, of course, opt to implement some for of auto-detection, but with .paramstyle set to a particular parameter binding style, the modules should not use such a mechanism. > 2. the cursor knows upfront that it will deal with a single "positional" or "named" parameter style. When execute() is called, it examines the parameter argument to see if it is a sequence or a mapping. Based on that, it determines to parse for the "positional" or "named" style. > > I'm guessing that #2 is the "magic auto-selection" here. I don't have a problem with it as I think some DBAPIs already do that, but I don't have a huge problem if we go with #1 either. > > But I would like to get it specified what happens when there are *no* parameters sent, as the DBAPIs behave inconsistently in this regard (some parse the string, others don't). Does it matter whether they parse the string or not ? If the SQL contains a parameter marker for which no parameter is provided, the database will complain, so you'd get an error either from the database backend or the database module. > As a total aside, I also have issues with some DBAPIs that accept *only* a list for the "sequence", others that accept *only* a tuple (this may be only when the parameter list is empty, however). So specifying the significance of "empty" parameter lists as well as how to test for "sequence" would be nice too, so that I can hit the DBAPI authors with bug reports backed up by the spec. The DB-API is already clear on this: any sequence should be accepted for .execute() and any sequence of sequences for .executemany(). Spelling out the special case of passing an empty sequence to .execute() and .executemany() would probably be wise. For .execute() this would only be valid for a statement that doesn't have parameter markers. For .executemany() this would be the same as a no-op (and only serve a purpose on the basis that it makes writing algorithms easier). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 23 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-07-01: EuroPython 2013, Florence, Italy ... 39 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Thu May 23 17:44:35 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 23 May 2013 17:44:35 +0200 Subject: [DB-SIG] Escaping parameter markers Message-ID: <519E3963.7000009@egenix.com> I've done some research on escaping parameter markers. Here's what I came up with: ODBC: Supports only qmark ('?'). The standard doesn't specify an escape sequence for ? in SQL statements. JDBC: Supports qmark ('?') and named (':param'). The standard doesn't specify an escape sequence for either of those in SQL statements. Oracle: Supports named (':param'). There's no mention of escaping the colon in their documentation. Interesting aside: The parameters can be bound based on name and based on position (if unambiguous), so I guess we'll have to spell out that with named we always mean binding by name. Since specifically PostgreSQL uses '?' as operator, there has been some discussion on how to resolve the issue, but I haven't found their resolution on the net: http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/2013-February/000053.html http://stackoverflow.com/questions/14779896/does-the-jdbc-spec-prevent-from-being-used-as-an-operator-outside-of-quotes In PostgreSQL, it's possible to avoid using '?' and '::' in SQL by simply using appropriate functions or CASTs instead, so you don't really need escaping. Since neither of those standards defines an escape mechanism for the parameter markers, I guess the DB-API should not either and instead leave this for the database modules to handle in whatever way is appropriate for the database backend. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 23 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-07-01: EuroPython 2013, Florence, Italy ... 39 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Thu May 23 17:47:12 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 23 May 2013 17:47:12 +0200 Subject: [DB-SIG] paramstyle specification In-Reply-To: References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> <519BC9E3.1010108@egenix.com> Message-ID: <519E3A00.6060607@egenix.com> On 22.05.2013 00:07, Vernon D. Cole wrote: > I think that we should be careful not to mandate any sophisticated grammar > processing of the query string. The most notice we should take of comments > would be to suggest that any reformatting operation might choose to stop at > a double dash. > /* C style */ comments and other non-ANSI-standard standard things should > not be addressed. Consider what is the worst that could happen? > > Let's say a programmer sets my module into 'named' and passes me a query > with an extra format pattern inside an extended syntax comment: > > SELECT a, b FROM table_c WHERE name = :nam /* :weird */ AND state = :st > > I search for a colon and find one. Valid names consist of alphanumeric and > underscore, so I collect characters until I find the end. I replace the > whole mess with a question mark, look for 'nam' in my parameter dictionary, > and place the value in the parameter list I am building. > Doing it again, I find 'weird' as the next token. Two things can happen: > if 'weird' is in the parameter dictionary, I emit a question mark and add > the value to the list. If not, then I raise an error. In the former case, > the query gets passed to the SQL engine with three parameters for two (that > it sees) question marks, and it will raise an error. > > No data is lost or altered. The programmer will learn not to do that > again. > > This is a very different case from performing that same substitution in > a string literal, where data _does_ get altered. Even then, I am not going > to try to find every possible funny literal encoding scheme for every > possible SQL dialect. I am only responsible for things between two single > quotes. ANSI standard. > > The SQL compiler is the compiler. I am just running a little macro > program in front. If running the macro creates syntax errors then the > compiler lets them know, and they fix it. The simpler the rules for the > macro the easier to debug the output. In particular, if there are no > parameters, or if the paramstyle is in the native format for my engine, I > will not look at the command at all. > > If a module author wants to do more elaborate syntax checking, fine. But > it should not be included in the standard. Agreed, but given other comments in this thread, I think we do need to at least allow standard SQL comments (--) to be used and correctly parsed by any parameter style converter. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 23 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-07-01: EuroPython 2013, Florence, Italy ... 39 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Thu May 23 18:24:41 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Thu, 23 May 2013 12:24:41 -0400 Subject: [DB-SIG] paramstyle specification In-Reply-To: <519E2ECF.2070104@egenix.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> <519BC9E3.1010108@egenix.com> <519E2ECF.2070104@egenix.com> Message-ID: On May 23, 2013, at 10:59 AM, M.-A. Lemburg wrote: >> 2. the cursor knows upfront that it will deal with a single "positional" or "named" parameter style. When execute() is called, it examines the parameter argument to see if it is a sequence or a mapping. Based on that, it determines to parse for the "positional" or "named" style. >> >> I'm guessing that #2 is the "magic auto-selection" here. I don't have a problem with it as I think some DBAPIs already do that, but I don't have a huge problem if we go with #1 either. >> >> But I would like to get it specified what happens when there are *no* parameters sent, as the DBAPIs behave inconsistently in this regard (some parse the string, others don't). > > Does it matter whether they parse the string or not ? If the SQL > contains a parameter marker for which no parameter is provided, > the database will complain, so you'd get an error either from the > database backend or the database module. OK well here's where there's a cultural mismatch between different DBAPI authors. Some DBAPIs need to perform substitutions in the string before they send it off to the server, like psycopg2 does. Others, like those that work with ODBC (I think) don't have to do as much, if at all. So I'd say no, the system doesn't "have to" parse the string if that's not already part of its behavior. >> As a total aside, I also have issues with some DBAPIs that accept *only* a list for the "sequence", others that accept *only* a tuple (this may be only when the parameter list is empty, however). So specifying the significance of "empty" parameter lists as well as how to test for "sequence" would be nice too, so that I can hit the DBAPI authors with bug reports backed up by the spec. > > The DB-API is already clear on this: any sequence should be > accepted for .execute() and any sequence of sequences for .executemany(). > > Spelling out the special case of passing an empty sequence to > .execute() and .executemany() would probably be wise. For .execute() > this would only be valid for a statement that doesn't have parameter > markers. For .executemany() this would be the same as a no-op (and > only serve a purpose on the basis that it makes writing algorithms > easier). The thing with the empty parameter lists becomes significant when we deal with DBAPIs that change their behavior based on the paramlist being present or not. This is why it gets really confusing; some DBAPIs don't like an empty parameter list, others will change their string parsing behavior (such as requiring % to be escaped or not) based on whether or not a collection is present. From Chris.Clark at actian.com Thu May 23 22:27:53 2013 From: Chris.Clark at actian.com (Chris Clark) Date: Thu, 23 May 2013 13:27:53 -0700 Subject: [DB-SIG] Escaping parameter markers In-Reply-To: <519E3963.7000009@egenix.com> References: <519E3963.7000009@egenix.com> Message-ID: <519E7BC9.2020203@actian.com> On Thu, 23 May 2013 17:44:35 +0200, M.-A. Lemburg wrote: > I've done some research on escaping parameter markers. Here's what > I came up with: > > ODBC: Supports only qmark ('?'). The standard doesn't specify an > escape sequence for ? in SQL statements. > > JDBC: Supports qmark ('?') and named (':param'). The standard > doesn't specify an escape sequence for either of those > in SQL statements. Being pedantic; JDBC requires qmark ('?'). The JDBC spec. only documents named (':param') for execute procedure ( e.g. http://docs.oracle.com/javase/7/docs/api/java/sql/CallableStatement.html#setString%28java.lang.String ). Any JDBC driver that implements named parms for queries, like SELECT, has added an extension, for example Oracle (or is an ORM feature). > Oracle: Supports named (':param'). There's no mention of escaping > the colon in their documentation. Interesting aside: The > parameters can be bound based on name and based on > position (if unambiguous), so I guess we'll have to spell > out that with named we always mean binding by name. It depends on the interface/api. If using ODBC it is qmark ('?'). If using something like ProC, then yes it is named only. > Since specifically PostgreSQL uses '?' as operator, there has > been some discussion on how to resolve the issue, but I haven't > found their resolution on the net: > > http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/2013-February/000053.html > http://stackoverflow.com/questions/14779896/does-the-jdbc-spec-prevent-from-being-used-as-an-operator-outside-of-quotes > > In PostgreSQL, it's possible to avoid using '?' and '::' > in SQL by simply using appropriate functions or CASTs instead, > so you don't really need escaping. > > Since neither of those standards defines an escape mechanism for > the parameter markers, I guess the DB-API should not either and > instead leave this for the database modules to handle in whatever > way is appropriate for the database backend. Agreed. Chris From Chris.Clark at actian.com Thu May 23 22:33:20 2013 From: Chris.Clark at actian.com (Chris Clark) Date: Thu, 23 May 2013 13:33:20 -0700 Subject: [DB-SIG] bind parameters In-Reply-To: <519E2ECF.2070104@egenix.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> <519BC9E3.1010108@egenix.com> <519E2ECF.2070104@egenix.com> Message-ID: <519E7D10.7010607@actian.com> On Thu, 23 May 2013 16:59:27 +0200, M.-A. Lemburg wrote: > On 22.05.2013 00:52, Michael Bayer wrote: >> As a total aside, I also have issues with some DBAPIs that accept *only* a list for the "sequence", others that accept *only* a tuple (this may be only when the parameter list is empty, however). So specifying the significance of "empty" parameter lists as well as how to test for "sequence" would be nice too, so that I can hit the DBAPI authors with bug reports backed up by the spec. > The DB-API is already clear on this: any sequence should be > accepted for .execute() and any sequence of sequences for .executemany(). > > Spelling out the special case of passing an empty sequence to > .execute() and .executemany() would probably be wise. For .execute() > this would only be valid for a statement that doesn't have parameter > markers. For .executemany() this would be the same as a no-op (and > only serve a purpose on the basis that it makes writing algorithms > easier). > I'd like to suggest that if the bind parameter is nonzero (i.e. __nonzero__ method call) then it is assumed there are no params. This way we don't have to get into is it an empty sequence, is it None, etc. Chris From Chris.Clark at actian.com Thu May 23 23:37:31 2013 From: Chris.Clark at actian.com (Chris Clark) Date: Thu, 23 May 2013 14:37:31 -0700 Subject: [DB-SIG] bind parameters In-Reply-To: <519E7D10.7010607@actian.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> <519BC9E3.1010108@egenix.com> <519E2ECF.2070104@egenix.com> <519E7D10.7010607@actian.com> Message-ID: <519E8C1B.6050303@actian.com> On Thu, 23 May 2013 13:33:20 -0700, Chris Clark wrote: > On Thu, 23 May 2013 16:59:27 +0200, M.-A. Lemburg wrote: >> On 22.05.2013 00:52, Michael Bayer wrote: >>> As a total aside, I also have issues with some DBAPIs that accept >>> *only* a list for the "sequence", others that accept *only* a tuple >>> (this may be only when the parameter list is empty, however). So >>> specifying the significance of "empty" parameter lists as well as >>> how to test for "sequence" would be nice too, so that I can hit the >>> DBAPI authors with bug reports backed up by the spec. >> The DB-API is already clear on this: any sequence should be >> accepted for .execute() and any sequence of sequences for >> .executemany(). >> >> Spelling out the special case of passing an empty sequence to >> .execute() and .executemany() would probably be wise. For .execute() >> this would only be valid for a statement that doesn't have parameter >> markers. For .executemany() this would be the same as a no-op (and >> only serve a purpose on the basis that it makes writing algorithms >> easier). >> > > I'd like to suggest that if the bind parameter is nonzero (i.e. > __nonzero__ method call) then it is assumed there are no params. This > way we don't have to get into is it an empty sequence, is it None, etc. Apologies, I meant the reverse :-( I'd like to suggest that if the bind parameter is nonzero (i.e. __nonzero__ method call returns True) then it is assumed there *are* params. This way we don't have to get into is it an empty sequence, is it None, etc. Chris From mal at egenix.com Fri May 24 10:08:13 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 24 May 2013 10:08:13 +0200 Subject: [DB-SIG] Escaping parameter markers In-Reply-To: <519E7BC9.2020203@actian.com> References: <519E3963.7000009@egenix.com> <519E7BC9.2020203@actian.com> Message-ID: <519F1FED.5020209@egenix.com> On 23.05.2013 22:27, Chris Clark wrote: > On Thu, 23 May 2013 17:44:35 +0200, M.-A. Lemburg wrote: >> I've done some research on escaping parameter markers. Here's what >> I came up with: >> >> ODBC: Supports only qmark ('?'). The standard doesn't specify an >> escape sequence for ? in SQL statements. >> >> JDBC: Supports qmark ('?') and named (':param'). The standard >> doesn't specify an escape sequence for either of those >> in SQL statements. > > Being pedantic; JDBC requires qmark ('?'). > > The JDBC spec. only documents named (':param') for execute procedure ( e.g. > http://docs.oracle.com/javase/7/docs/api/java/sql/CallableStatement.html#setString%28java.lang.String ). > Any JDBC driver that implements named parms for queries, like SELECT, has added an extension, for > example Oracle (or is an ORM feature). Thanks for the clarification. Indeed, the prepared statement does not expose methods for named binding: http://docs.oracle.com/javase/7/docs/api/java/sql/PreparedStatement.html >> Oracle: Supports named (':param'). There's no mention of escaping >> the colon in their documentation. Interesting aside: The >> parameters can be bound based on name and based on >> position (if unambiguous), so I guess we'll have to spell >> out that with named we always mean binding by name. > > It depends on the interface/api. If using ODBC it is qmark ('?'). Right. ODBC doesn't have a way of binding by name. It only supports binding by position - and it doesn't differentiate between prepared statements and callable ones. > If using something like ProC, then yes it is named only. > > >> Since specifically PostgreSQL uses '?' as operator, there has >> been some discussion on how to resolve the issue, but I haven't >> found their resolution on the net: >> >> http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/2013-February/000053.html >> http://stackoverflow.com/questions/14779896/does-the-jdbc-spec-prevent-from-being-used-as-an-operator-outside-of-quotes >> >> >> In PostgreSQL, it's possible to avoid using '?' and '::' >> in SQL by simply using appropriate functions or CASTs instead, >> so you don't really need escaping. >> >> Since neither of those standards defines an escape mechanism for >> the parameter markers, I guess the DB-API should not either and >> instead leave this for the database modules to handle in whatever >> way is appropriate for the database backend. > > Agreed. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 24 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-07-01: EuroPython 2013, Florence, Italy ... 38 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Fri May 24 10:23:41 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 24 May 2013 10:23:41 +0200 Subject: [DB-SIG] bind parameters and empty parameter lists In-Reply-To: <519E8C1B.6050303@actian.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> <519BC9E3.1010108@egenix.com> <519E2ECF.2070104@egenix.com> <519E7D10.7010607@actian.com> <519E8C1B.6050303@actian.com> Message-ID: <519F238D.8010302@egenix.com> On 23.05.2013 23:37, Chris Clark wrote: > On Thu, 23 May 2013 13:33:20 -0700, Chris Clark wrote: >> On Thu, 23 May 2013 16:59:27 +0200, M.-A. Lemburg wrote: >>> On 22.05.2013 00:52, Michael Bayer wrote: >>>> As a total aside, I also have issues with some DBAPIs that accept *only* a list for the >>>> "sequence", others that accept *only* a tuple (this may be only when the parameter list is >>>> empty, however). So specifying the significance of "empty" parameter lists as well as how to >>>> test for "sequence" would be nice too, so that I can hit the DBAPI authors with bug reports >>>> backed up by the spec. >>> The DB-API is already clear on this: any sequence should be >>> accepted for .execute() and any sequence of sequences for .executemany(). >>> >>> Spelling out the special case of passing an empty sequence to >>> .execute() and .executemany() would probably be wise. For .execute() >>> this would only be valid for a statement that doesn't have parameter >>> markers. For .executemany() this would be the same as a no-op (and >>> only serve a purpose on the basis that it makes writing algorithms >>> easier). >>> >> >> I'd like to suggest that if the bind parameter is nonzero (i.e. __nonzero__ method call) then it >> is assumed there are no params. This way we don't have to get into is it an empty sequence, is it >> None, etc. > > Apologies, I meant the reverse :-( > > I'd like to suggest that if the bind parameter is nonzero (i.e. __nonzero__ method call returns > True) then it is assumed there *are* params. This way we don't have to get into is it an empty > sequence, is it None, etc. There is no __nonzero__ in Python 3 anymore. The method was renamed to __bool__ :-) I'm also not sure whether we should allow any of these: .execute(sql, None) .executemany(sql, None) .executemany(sql, [None]) In order to support your suggestion, database module would have to call the __nonzero__/__bool__ slot on the parameters, which could be an expensive operation with no real value in case the parameters are not None, or even be impossible in the case of iterators. Using the slot, we'd also allow things like these: .execute(sql, 0) .executemany(sql, 0.0) .executemany(sql, [0]) which can hide programming errors. Using the __len__ slot would work, though, unless you have an iterator... Which leads to another idea for DB-API 3: I think we should allow iterators for the parameter list in .executemany(), and perhaps even for the parameter sequence in .execute(). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 24 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-07-01: EuroPython 2013, Florence, Italy ... 38 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From Chris.Clark at actian.com Fri May 24 17:54:13 2013 From: Chris.Clark at actian.com (Chris Clark) Date: Fri, 24 May 2013 08:54:13 -0700 Subject: [DB-SIG] bind parameters and empty parameter lists In-Reply-To: <519F238D.8010302@egenix.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> <519BC9E3.1010108@egenix.com> <519E2ECF.2070104@egenix.com> <519E7D10.7010607@actian.com> <519E8C1B.6050303@actian.com> <519F238D.8010302@egenix.com> Message-ID: <519F8D25.9080906@actian.com> On Fri, 24 May 2013 10:23:41 +0200, M.-A. Lemburg wrote: > On 23.05.2013 23:37, Chris Clark wrote: >> On Thu, 23 May 2013 13:33:20 -0700, Chris Clark wrote: >>> On Thu, 23 May 2013 16:59:27 +0200, M.-A. Lemburg wrote: >>>> On 22.05.2013 00:52, Michael Bayer wrote: >>>>> As a total aside, I also have issues with some DBAPIs that accept *only* a list for the >>>>> "sequence", others that accept *only* a tuple (this may be only when the parameter list is >>>>> empty, however). So specifying the significance of "empty" parameter lists as well as how to >>>>> test for "sequence" would be nice too, so that I can hit the DBAPI authors with bug reports >>>>> backed up by the spec. >>>> The DB-API is already clear on this: any sequence should be >>>> accepted for .execute() and any sequence of sequences for .executemany(). >>>> >>>> Spelling out the special case of passing an empty sequence to >>>> .execute() and .executemany() would probably be wise. For .execute() >>>> this would only be valid for a statement that doesn't have parameter >>>> markers. For .executemany() this would be the same as a no-op (and >>>> only serve a purpose on the basis that it makes writing algorithms >>>> easier). >>>> >>> I'd like to suggest that if the bind parameter is nonzero (i.e. __nonzero__ method call) then it >>> is assumed there are no params. This way we don't have to get into is it an empty sequence, is it >>> None, etc. >> Apologies, I meant the reverse :-( >> >> I'd like to suggest that if the bind parameter is nonzero (i.e. __nonzero__ method call returns >> True) then it is assumed there *are* params. This way we don't have to get into is it an empty >> sequence, is it None, etc. > There is no __nonzero__ in Python 3 anymore. The method was renamed > to __bool__ :-) > > I'm also not sure whether we should allow any of these: > > .execute(sql, None) > .executemany(sql, None) Personally I'd like to allow the above. If a driver has: def execute(self, sql_text, params=None): if params: # process params, more on this later # else assume no params this will be allowed but the param will be ignored > .executemany(sql, [None]) This is OK too, this drops in the the "we have params logic", and it is a single param that is NULL. > In order to support your suggestion, database module would have to > call the __nonzero__/__bool__ slot on the parameters, which could > be an expensive operation with no real value in case the parameters > are not None, Do you have an example of then this would be expensive? I do not understand the "real value in case the parameters are not None" comment, could you expand (possibly with a short example) on this please? > or even be impossible in the case of iterators. An iterator with values left is always going to be true, if empty it is false: >>> params = xrange(10) >>> if params: ... print True ... True >>> params = xrange(0) >>> if params: ... print True ... >>> this is true for generators too. > Using the slot, we'd also allow things like these: > > .execute(sql, 0) > .executemany(sql, 0.0) I'm OK with these being allowed through, they would NOT be treated as params. If there are param markers in the sql, a (DBMS) error should be raised with "missing bind params" if the driver doe snot then pass in params. > .executemany(sql, [0]) Again, this is valid, a single tuple with an integer value of zero. > which can hide programming errors. The dbms should raise an error for this, this shouldn't be hidden. > Using the __len__ slot would work, though, unless you have an > iterator... I'm not keen on that. For instance strings are sequences and if we have: .execute(sql, 'hello') it would be valid. I don't have a good answer for this one, I've been toying with the idea of a string check for param but that seems overly excessive. Coming back to param checks: def execute(self, sql_text, params=None): if params: for param in params: do stuff.... # else assume no params I've been toying with the idea of catching TypeErrors to detect cases like .execute(sql, 1) in the driver (and wrappers) that I maintain. E.g.: >>> for x in 1: ... print x ... Traceback (most recent call last): File "", line 1, in TypeError: 'int' object is not iterable Is catching a type error a good way to check if something isn't a sequence? What are other people doing? This doesn't handle users who accidentally pass in a single string (instead of a tuple with a single string). > Which leads to another idea for DB-API 3: I think we should allow > iterators for the parameter list in .executemany(), and perhaps > even for the parameter sequence in .execute(). They should be allowed and I think you are correct we should be explicit in the spec on this. I actually have code that relies on passing a generator into executemany (the easiest example is to pass a cursor in for instance or an ETL function that takes a cursor as a param) . Chris From daniele.varrazzo at gmail.com Fri May 24 18:53:39 2013 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Fri, 24 May 2013 17:53:39 +0100 Subject: [DB-SIG] bind parameters and empty parameter lists In-Reply-To: <519F8D25.9080906@actian.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> <519BC9E3.1010108@egenix.com> <519E2ECF.2070104@egenix.com> <519E7D10.7010607@actian.com> <519E8C1B.6050303@actian.com> <519F238D.8010302@egenix.com> <519F8D25.9080906@actian.com> Message-ID: <1369414419.1017.15.camel@risotto.smithersbet.com> On Fri, 2013-05-24 at 08:54 -0700, Chris Clark wrote: > An iterator with values left is always going to be true, if empty it is > false: > > >>> params = xrange(10) > >>> if params: > ... print True > ... > True > >>> params = xrange(0) > >>> if params: > ... print True > ... > >>> > > this is true for generators too. No, this is not true: it's xrange to be special-cased and has a length: In [1]: def emptyiter(): ...: return ...: yield 0 ...: In [4]: bool(emptyiter()) Out[4]: True > > Using the slot, we'd also allow things like these: > > > > .execute(sql, 0) > > .executemany(sql, 0.0) > > I'm OK with these being allowed through, they would NOT be treated as > params. If there are param markers in the sql, a (DBMS) error should be > raised with "missing bind params" if the driver doe snot then pass in > params. > > > .executemany(sql, [0]) > > Again, this is valid, a single tuple with an integer value of zero. For consistency between execute and executemany, I think ".executemany(sql, [0])" should behave as ".execute(sql, 0)", which is likely to raise an error. > > Which leads to another idea for DB-API 3: I think we should allow > > iterators for the parameter list in .executemany(), and perhaps > > even for the parameter sequence in .execute(). > > They should be allowed and I think you are correct we should be explicit > in the spec on this. > > I actually have code that relies on passing a generator into executemany > (the easiest example is to pass a cursor in for instance or an ETL > function that takes a cursor as a param) . I'm fine with iterators in executemany, not so much in execute. The reason is that execute (with positional placeholders) requires a well defined length of the parameters, because if the no. of placeholders is different from the no. of parameters it should give an error (the error may be returned by the driver, but as a matter of fact a query with 3 "?" and 2 arguments is an error). If execute received an iterable the first thing it should do to would be "args = list(args)" to get their number. Forcing by spec the args to be a sequence the driver can assume to find len() and __getitem__ working on 0..len-1, and that's about everything needed to implement parameters passing. Another question is about its usefulness: the args are regularly object of different types, you generally don't have an iterable that yields first an int, then a date, then a string. You generally expect iterators to yield homogeneous objects. An iterable as executemany argument, yielding sequences, is quite handy instead. -- Daniele From Chris.Clark at actian.com Fri May 24 19:00:48 2013 From: Chris.Clark at actian.com (Chris Clark) Date: Fri, 24 May 2013 10:00:48 -0700 Subject: [DB-SIG] bind parameters and empty parameter lists In-Reply-To: <1369414419.1017.15.camel@risotto.smithersbet.com> References: <1369157056.9697.45.camel@risotto.smithersbet.com> <519BBD51.7050905@egenix.com> <58A15700-D613-46EB-B936-BACBB24C535C@zzzcomputing.com> <519BC9E3.1010108@egenix.com> <519E2ECF.2070104@egenix.com> <519E7D10.7010607@actian.com> <519E8C1B.6050303@actian.com> <519F238D.8010302@egenix.com> <519F8D25.9080906@actian.com> <1369414419.1017.15.camel@risotto.smithersbet.com> Message-ID: <519F9CC0.9030207@actian.com> On Fri, 24 May 2013 17:53:39 +0100, Daniele Varrazzo wrote: > On Fri, 2013-05-24 at 08:54 -0700, Chris Clark wrote: >>> .executemany(sql, [0]) >> Again, this is valid, a single tuple with an integer value of zero. > For consistency between execute and executemany, I think > ".executemany(sql, [0])" should behave as ".execute(sql, 0)", which is > likely to raise an error. You're right I misread the executemany as execute, thanks for catching that! Chris From phd at phdru.name Sun May 26 19:37:46 2013 From: phd at phdru.name (Oleg Broytman) Date: Sun, 26 May 2013 21:37:46 +0400 Subject: [DB-SIG] SQLObject 1.3.3 and 1.4.1 Message-ID: <20130526173746.GA17101@iskra.aviel.ru> Hello! I'm pleased to announce bugfix releases 1.3.3 and 1.4.1. What's new in SQLObject ======================= * Fixed bugs in pickling and unpickling (remove/restore a weak proxy to self, fixed cache handling). * Added an example of using SQLObject with web.py to the links page. Contributors for this release are Andrew Trusty and Rhubarb Sin. For a more complete list, please see the news: http://sqlobject.org/News.html What is SQLObject ================= SQLObject is an object-relational mapper. Your database tables are described as classes, and rows are instances of those classes. SQLObject is meant to be easy to use and quick to get started with. SQLObject supports a number of backends: MySQL, PostgreSQL, SQLite, Firebird, Sybase, MSSQL and MaxDB (also known as SAPDB). Where is SQLObject ================== Site: http://sqlobject.org Development: http://sqlobject.org/devel/ Mailing list: https://lists.sourceforge.net/mailman/listinfo/sqlobject-discuss Archives: http://news.gmane.org/gmane.comp.python.sqlobject Download: https://pypi.python.org/pypi/SQLObject/1.3.3 https://pypi.python.org/pypi/SQLObject/1.4.1 News and changes: http://sqlobject.org/News.html Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN.