From msanchez at grupoburke.com Fri May 2 11:46:16 2003 From: msanchez at grupoburke.com (Marcos =?ISO-8859-1?Q?S=E1nchez?= Provencio) Date: Fri May 2 06:46:17 2003 Subject: [DB-SIG] ANN: adodbapi 2.0 In-Reply-To: References: Message-ID: <1051872365.757.22.camel@renata.macondo.pri> Does this module handle currency & decimal data without converting to inexact floating point numbers? I have browsed through the source and it did not seem to. Is it planned? El mi?, 30 de 04 de 2003 a las 18:35, Henrik Ekelund escribi?: > NAME: > adodbapi - A python DB API 2.0 interface to Microsoft ADO > > DESCRIPTION: > A 100% Python-DB API 2.0 compliant module that > makes it easy to use Microsoft ADO for connecting > with databases and other data sources. Includes > pyunit testcases. > > WHATS NEW: > 1. Improved performance through GetRows method. > 2. Flexible date conversions. > Supports eGenix mxExtensions, python time module and python 2.3 > datetime module. > 3. More exact mappings of numeric datatypes. > 4. User defined conversions of datatypes through "plug-ins". > 5. Improved testcases, not dependent on Northwind data. > 6. Works with Microsoft Access currency datatype. > 7. Improved DB-API 2.0 compliance. > 8. rowcount property works on statements Not returning records. > 9. "Optional" error handling extension, see DB-API 2.0 specification. > 10. Two maling lists have been created, one for users and one for > developers. > > PLATFORM: > Windows only > > HOMEPAGE: > > > LICENSE: > Lesser GPL (fine for most commercial vendors) > > > Henrik Ekelund > > > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig -- Marcos S?nchez Provencio www.burke.es From andy at nospam.com Mon May 5 02:02:39 2003 From: andy at nospam.com (Andy Sy) Date: Sun May 4 12:54:14 2003 Subject: [DB-SIG] Are a unified prepared statement and map-based result APIs a future goal? Message-ID: <009901c3125e$fe2f14b0$744d4eca@duron12> Upon my switch from kinterbasdb to MySQLdb, two warts of the Python DB-API immediately made themselves known. Different prepared statement syntax and non-standard ways of getting dictionary-indexed results throw a big monkey wrench into the portability promised by such an API. This is most unPythonic. Are there plans to fix this in future versions of the DB-API spec? ============================== email: andy @t netfxph d0t com From rmangaliag at slu.edu.ph Mon May 5 16:36:56 2003 From: rmangaliag at slu.edu.ph (ali) Date: Mon May 5 03:23:13 2003 Subject: [DB-SIG] a simple question on rdbms and berkeley db References: Message-ID: <000801c312d9$1d225ba0$e019a8c0@slu.edu.ph> a simple question on rdbms and berkeley db.... i needed a database for a simple message board that i am creating using python... then there was a question on databases??? i first considered firebird and was actually working well... and i came across berkely db... and i really got confused... my question is, what is the difference between rdbms and berkeley db or even gdb and others of the same genre??? another question, in my simple web-based message board?? what database is the most appropriate?? or am i wrong in assuming they are different... i'm new to databases, please be gentle with me... :) thanks... ali From henrik.ekelund at vpd.se Mon May 5 14:31:19 2003 From: henrik.ekelund at vpd.se (Henrik Ekelund) Date: Mon May 5 07:31:33 2003 Subject: [DB-SIG] ANN: adodbapi 2.0 Message-ID: Yes. It can convert it any way you want. By default "Currency" is mapped to a function cvtMoney that returns floats, and "decimal" is left untouched as a string. Do like this if you want to receive the raw data (a tuple) of a currency field: adodbapi.variantConversions[adCurrency]=adodbapi.identity You can also specify exactly how you want "decimal" and "numeric" handled by making your own converter: def myconverter(variant): return variantConvertedToWhatIWant(variant) adodbapi.variantConversions[adoExactNumericTypes]=myconverter There are also some testcases describing this in more detail. /Henrik > -----Original Message----- > From: Marcos S?nchez Provencio [mailto:msanchez@grupoburke.com] > Sent: den 2 maj 2003 12:46 > To: Henrik Ekelund > Cc: 'db-sig@python.org' > Subject: Re: [DB-SIG] ANN: adodbapi 2.0 > > > Does this module handle currency & decimal data without converting to > inexact floating point numbers? I have browsed through the > source and it > did not seem to. Is it planned? > > El mi?, 30 de 04 de 2003 a las 18:35, Henrik Ekelund escribi?: > > NAME: > > adodbapi - A python DB API 2.0 interface to Microsoft ADO > > > > DESCRIPTION: > > A 100% Python-DB API 2.0 compliant module that > > makes it easy to use Microsoft ADO for connecting > > with databases and other data sources. Includes > > pyunit testcases. > > > > WHATS NEW: > > 1. Improved performance through GetRows method. > > 2. Flexible date conversions. > > Supports eGenix mxExtensions, python time module and > python 2.3 > > datetime module. > > 3. More exact mappings of numeric datatypes. > > 4. User defined conversions of datatypes through "plug-ins". > > 5. Improved testcases, not dependent on Northwind data. > > 6. Works with Microsoft Access currency datatype. > > 7. Improved DB-API 2.0 compliance. > > 8. rowcount property works on statements Not returning records. > > 9. "Optional" error handling extension, see DB-API 2.0 > specification. > > 10. Two maling lists have been created, one for users > and one for > > developers. > > > > PLATFORM: > > Windows only > > > > HOMEPAGE: > > > > > > LICENSE: > > Lesser GPL (fine for most commercial vendors) > > > > > > Henrik Ekelund > > > > > > > > > _______________________________________________ > > DB-SIG maillist - DB-SIG@python.org > > http://mail.python.org/mailman/listinfo/db-sig > -- > Marcos S?nchez Provencio > www.burke.es > ********************************************************************** This email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com ********************************************************************** From msanchez at grupoburke.com Mon May 5 23:14:09 2003 From: msanchez at grupoburke.com (=?ISO-8859-1?Q?Marcos_S=E1nchez_Provencio?=) Date: Mon May 5 16:14:28 2003 Subject: [DB-SIG] a simple question on rdbms and berkeley db In-Reply-To: <000801c312d9$1d225ba0$e019a8c0@slu.edu.ph> References: <000801c312d9$1d225ba0$e019a8c0@slu.edu.ph> Message-ID: <3EB6C611.9080609@grupoburke.com> From http://www.skippingdot.net/2002/02/04 ---- No support for SQL at all. From the FAQ: /Berkeley DB does not support a schema layer or an SQL query interface, nor does it support ODBC or JDBC./ Digging a little deeper, check out What Berkeley DB Is Not in the online documentation. Excerpt: /Berkeley DB is not a relational system. Relational database systems are semantically rich and offer high-level database access. Compared to such systems, Berkeley DB is a high-performance, transactional library for record storage. It's possible to build a relational system on top of Berkeley DB. In fact, the popular MySQL relational system uses Berkeley DB for transaction-protected table management, and takes care of all the SQL parsing and execution. It uses Berkeley DB for the storage level, and provides the semantics and access tools./ ----- Basically, berkeley does one simple thing: It stores strings under a key. If you need more complicated functions (such as tables with columns, a client/server architecture, query language... you need some kind of SQL database. Also, the license is a bit restrictive, although this might not be a problem if your app is open source or private. ali wrote: >a simple question on rdbms and berkeley db.... > >i needed a database for a simple message board that i am creating using >python... then there was a question on databases??? i first considered >firebird and was actually working well... and i came across berkely db... >and i really got confused... > >my question is, what is the difference between rdbms and berkeley db or even >gdb and others of the same genre??? > >another question, in my simple web-based message board?? what database is >the most appropriate?? > >or am i wrong in assuming they are different... > >i'm new to databases, please be gentle with me... :) > >thanks... >ali > > >_______________________________________________ >DB-SIG maillist - DB-SIG@python.org >http://mail.python.org/mailman/listinfo/db-sig > > From chris at cogdon.org Mon May 5 14:29:18 2003 From: chris at cogdon.org (Chris Cogdon) Date: Mon May 5 16:29:22 2003 Subject: [DB-SIG] a simple question on rdbms and berkeley db In-Reply-To: <3EB6C611.9080609@grupoburke.com> Message-ID: <3EE6B23E-7F38-11D7-AF67-000393A6D442@cogdon.org> On Monday, May 5, 2003, at 13:14 US/Pacific, Marcos S?nchez Provencio wrote: > /Berkeley DB is not a relational system. Relational database systems > are semantically rich and offer high-level database access. Compared > to such systems, Berkeley DB is a high-performance, transactional > library for record storage. It's possible to build a relational system > on top of Berkeley DB. In fact, the popular MySQL relational system > uses Berkeley DB for transaction-protected table management, and takes > care of all the SQL parsing and execution. It uses Berkeley DB for the > storage level, and provides the semantics and access tools./ To avoid future confusion... Berkeley DB is just ONE of the storage types MySQL can use. Others include ISAM, MyISAM and InnoDB, which are probably increasing in order of 'power'. Ie, to enable certain MySQL features, you need to be using an appropriate backend storage type. MyISAM is mostly what people use nowadays. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From magnus at thinkware.se Mon May 5 23:45:23 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Mon May 5 16:44:43 2003 Subject: [DB-SIG] a simple question on rdbms and berkeley db In-Reply-To: Message-ID: <5.2.1.1.0.20030505215637.02684c10@www.thinkware.se> At Mon, 5 May 2003 15:36:56 +0800, ali wrote: >my question is, what is the difference between rdbms and berkeley db or even >gdb and others of the same genre??? The berkeley db, gdb etc are basically just persistent dictionaries. You use them as a normal dictionary, but you are restricted to using strings (I think) for both key and data. I contrast to a dictionary, their content is stored in files. In other words, each database file is like a plain table with one simple key that the rows are indexed on. So, single item updates, inserts and deletes are trivial dictionary operations, but aggregarions, queries or multi row updates that are more complex than "select * where primary_key = ?" have to be done by looping through all the elements in the python code, and modifying them as ordinary pyhon data. >another question, in my simple web-based message board?? what database is >the most appropriate?? Only you can say... If you have something working with an SQL database already, why switch? If you want a simpler database than Firebird and don't need simultaneous multiple users etc, maybe something that needs no separate server process, consider sqlite, which uses a fair subset of SQL, but will run embedded in you python process. -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The shortest path from thought to working program From rmangaliag at slu.edu.ph Thu May 8 15:08:36 2003 From: rmangaliag at slu.edu.ph (ali) Date: Thu May 8 01:54:31 2003 Subject: [DB-SIG] odbc vs mxodbc References: <5.2.1.1.0.20030505215637.02684c10@www.thinkware.se> Message-ID: <006501c31528$44e158c0$e019a8c0@slu.edu.ph> i've been using activepython for months now and i was quite pleased when i learned that an odbc api is already bundled with it... my question is: another odbc api is available, mxodbc, it is commercial and it sounds better (advertising wise-- no offense meant :) than odbc... how does odbc fare with mxodbc??? btw, i haven't tried mxodbc yet so i dont have any way of comparing??? thanks ali From mal at lemburg.com Thu May 8 09:58:19 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu May 8 02:59:10 2003 Subject: [DB-SIG] odbc vs mxodbc In-Reply-To: <006501c31528$44e158c0$e019a8c0@slu.edu.ph> References: <5.2.1.1.0.20030505215637.02684c10@www.thinkware.se> <006501c31528$44e158c0$e019a8c0@slu.edu.ph> Message-ID: <3EBA000B.1050601@lemburg.com> ali wrote: > i've been using activepython for months now and i was quite pleased when i > learned that an odbc api is already bundled with it... > > my question is: > > another odbc api is available, mxodbc, it is commercial and it sounds better > (advertising wise-- no offense meant :) than odbc... how does odbc fare with > mxodbc??? btw, i haven't tried mxodbc yet so i dont have any way of > comparing??? Why not simply give it a try and see for yourself ? mxODBC is DB-API 2.0, ODBC 2.x,3.x compatible, odbc still is DB-API 1.0, ODBC 1.x and doesn't give you access to all the goodies ODBC has to offer. Furthermore, odbc is unmaintained, not proven to work with todays ODBC drivers and you can't get support for it. Of course, that's my totally unbiased opinion ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, May 08 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 47 days left From rmangaliag at slu.edu.ph Thu May 8 17:19:07 2003 From: rmangaliag at slu.edu.ph (ali) Date: Thu May 8 04:05:12 2003 Subject: [DB-SIG] connecting to an access database... References: <5.2.1.1.0.20030505215637.02684c10@www.thinkware.se> <006501c31528$44e158c0$e019a8c0@slu.edu.ph> <3EBA000B.1050601@lemburg.com> Message-ID: <00a001c3153a$7fd5a460$e019a8c0@slu.edu.ph> is there an api that connects directly to an access database without using odbc??? much like kinterbasdb to firebird... (am i correct in my "parallelism"... anyway :) thanks again.... ali From mal at lemburg.com Thu May 8 11:13:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu May 8 04:13:55 2003 Subject: [DB-SIG] connecting to an access database... In-Reply-To: <00a001c3153a$7fd5a460$e019a8c0@slu.edu.ph> References: <5.2.1.1.0.20030505215637.02684c10@www.thinkware.se> <006501c31528$44e158c0$e019a8c0@slu.edu.ph> <3EBA000B.1050601@lemburg.com> <00a001c3153a$7fd5a460$e019a8c0@slu.edu.ph> Message-ID: <3EBA119B.70907@lemburg.com> ali wrote: > is there an api that connects directly to an access database without using > odbc??? ODBC *is* the native API of MS Access. There are other ways of accessing the Access files, but I would only you those for read-only purposes. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, May 08 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 47 days left From msanchez at grupoburke.com Thu May 8 11:15:22 2003 From: msanchez at grupoburke.com (Marcos =?ISO-8859-1?Q?S=E1nchez?= Provencio) Date: Thu May 8 06:15:23 2003 Subject: [DB-SIG] connecting to an access database... In-Reply-To: <00a001c3153a$7fd5a460$e019a8c0@slu.edu.ph> References: <5.2.1.1.0.20030505215637.02684c10@www.thinkware.se> <006501c31528$44e158c0$e019a8c0@slu.edu.ph> <3EBA000B.1050601@lemburg.com> <00a001c3153a$7fd5a460$e019a8c0@slu.edu.ph> Message-ID: <1052388882.15988.62.camel@cynar.proteus> You may try using COM+ADO or COM+DAO (or ActiveX or whatever the name is) to access Access. I believe DAO is also a native api for Jet databases. ?Is there any performance comparison? I suspect the best option is mxODBC if you need support or are planning to migrate from Jet to more serious db servers. The odbc module from win32all must be a dinosaur now, but it may work for you... El jue, 08 de 05 de 2003 a las 10:19, ali escribi?: > is there an api that connects directly to an access database without using > odbc??? > > much like kinterbasdb to firebird... > > (am i correct in my "parallelism"... anyway :) > > thanks again.... > > ali > > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig -- From mal at lemburg.com Thu May 8 13:57:11 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu May 8 06:58:01 2003 Subject: [DB-SIG] connecting to an access database... In-Reply-To: <1052388882.15988.62.camel@cynar.proteus> References: <5.2.1.1.0.20030505215637.02684c10@www.thinkware.se> <006501c31528$44e158c0$e019a8c0@slu.edu.ph> <3EBA000B.1050601@lemburg.com> <00a001c3153a$7fd5a460$e019a8c0@slu.edu.ph> <1052388882.15988.62.camel@cynar.proteus> Message-ID: <3EBA3807.2010101@lemburg.com> Marcos S?nchez Provencio wrote: > You may try using COM+ADO or COM+DAO (or ActiveX or whatever the name > is) to access Access. I believe DAO is also a native api for Jet > databases. I believe that's what MS wants you to think :-) If you scan the Jet Engine DLLs though, you'll find lots of ODBC APIs in there. ADO/DAO, .NET or whatever they've come up with recently simply adds a few layers on top of the ODBC API these DLLs expose. BTW, I'd suggest to have a look at MS Desktop Engine (MSDE) 2000; basically MS SQL Server 2000 without the GUI tools). That's certainly a powerful alternative to the Jet engine used in Access. > ?Is there any performance comparison? I suspect the best > option is mxODBC if you need support or are planning to migrate from Jet > to more serious db servers. > > The odbc module from win32all must be a dinosaur now, but it may work > for you... > > El jue, 08 de 05 de 2003 a las 10:19, ali escribi?: > >>is there an api that connects directly to an access database without using >>odbc??? >> >>much like kinterbasdb to firebird... >> >>(am i correct in my "parallelism"... anyway :) >> >>thanks again.... >> >>ali >> >> >>_______________________________________________ >>DB-SIG maillist - DB-SIG@python.org >>http://mail.python.org/mailman/listinfo/db-sig -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, May 08 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 47 days left From henrik.ekelund at vpd.se Thu May 8 13:58:08 2003 From: henrik.ekelund at vpd.se (Henrik Ekelund) Date: Thu May 8 06:58:26 2003 Subject: [DB-SIG] connecting to an access database... Message-ID: With adodbapi you can use ado and the Microsoft.Jet.OLEDB driver instead of odbc. http://adodbai.sourceforge.net /Henrik > -----Original Message----- > From: Marcos S?nchez Provencio [mailto:msanchez@grupoburke.com] > Sent: den 8 maj 2003 12:15 > To: ali > Cc: db-sig@python.org > Subject: Re: [DB-SIG] connecting to an access database... > > > You may try using COM+ADO or COM+DAO (or ActiveX or whatever the name > is) to access Access. I believe DAO is also a native api for Jet > databases. ?Is there any performance comparison? I suspect the best > option is mxODBC if you need support or are planning to > migrate from Jet > to more serious db servers. > > The odbc module from win32all must be a dinosaur now, but it may work > for you... > > El jue, 08 de 05 de 2003 a las 10:19, ali escribi?: > > is there an api that connects directly to an access > database without using > > odbc??? > > > > much like kinterbasdb to firebird... > > > > (am i correct in my "parallelism"... anyway :) > > > > thanks again.... > > > > ali > > > > > > _______________________________________________ > > DB-SIG maillist - DB-SIG@python.org > > http://mail.python.org/mailman/listinfo/db-sig > -- > > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig > ********************************************************************** This email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com ********************************************************************** From anthra.norell at tiscalinet.ch Fri May 9 00:07:47 2003 From: anthra.norell at tiscalinet.ch (Anthra Norell) Date: Thu May 8 17:10:24 2003 Subject: [DB-SIG] bsddb Message-ID: <002701c315a5$e14ccdc0$0201a8c0@mcuf7> Hi, I'm new with this and am having no luck with the bsddb module. It doesn't abide by the manual and shows a propensity to crash the python interactive window. (Python 2.2.2. Windows ME.) E.g. like this: id_db = bsddb.hashopen ('c:\\fre\\dok\\invest\\id.db', 'c') Producing this: Traceback (most recent call last): File "", line 1, in ? File "c:\fre\src\python\db_test.py-", line 35, in ? id_db = bsddb.hashopen (db_file_name, 'c') error: (22, 'Invalid argument') The same thing happens with btopen () instead of hashopen () Any idea anyone? Suggestions for downloading better alternatives. As I said, I'm new here. Frederic -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/db-sig/attachments/20030508/33337099/attachment.htm From andy47 at halfcooked.com Sat May 10 19:23:40 2003 From: andy47 at halfcooked.com (Andy Todd) Date: Sat May 10 13:25:42 2003 Subject: [DB-SIG] pyformat Parameter Style Message-ID: <3EBD359C.8030208@halfcooked.com> Is there a definitive specification of the pyformat parameter style anywhere? The usual sources come up dry. It is mentioned in the DB-API specification (http://www.python.org/peps/pep-0249.html) but not defined. It seems from observation that one should simply put '%()s' in the query string and then pass a mapping with a corresponding key when calling the execute method. But I'd just like to make absolutely sure. In particular, I'm curious if the 's' actually means anything (like string?). Should I be using different characters for different data types, and if so, which ones? Regards, Andy -- -------------------------------------------------------------------------------- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From chris at cogdon.org Sat May 10 11:32:00 2003 From: chris at cogdon.org (Chris Cogdon) Date: Sat May 10 13:32:05 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <3EBD359C.8030208@halfcooked.com> Message-ID: <4E691ADF-830D-11D7-BD84-000393A6D442@cogdon.org> On Saturday, May 10, 2003, at 10:23 US/Pacific, Andy Todd wrote: > Is there a definitive specification of the pyformat parameter style > anywhere? > > The usual sources come up dry. It is mentioned in the DB-API > specification (http://www.python.org/peps/pep-0249.html) but not > defined. > > It seems from observation that one should simply put '%( name>)s' in the query string and then pass a mapping with a > corresponding key when calling the execute method. But I'd just like > to make absolutely sure. > > In particular, I'm curious if the 's' actually means anything (like > string?). Should I be using different characters for different data > types, and if so, which ones? You just about nailed it. You should ALWAYS use 's', too, regardless of the data type. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From mal at lemburg.com Sat May 10 20:54:30 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat May 10 13:55:36 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <4E691ADF-830D-11D7-BD84-000393A6D442@cogdon.org> References: <4E691ADF-830D-11D7-BD84-000393A6D442@cogdon.org> Message-ID: <3EBD3CD6.8090001@lemburg.com> Chris Cogdon wrote: > > On Saturday, May 10, 2003, at 10:23 US/Pacific, Andy Todd wrote: > >> Is there a definitive specification of the pyformat parameter style >> anywhere? >> >> The usual sources come up dry. It is mentioned in the DB-API >> specification (http://www.python.org/peps/pep-0249.html) but not defined. >> >> It seems from observation that one should simply put '%(> name>)s' in the query string and then pass a mapping with a >> corresponding key when calling the execute method. But I'd just like >> to make absolutely sure. >> >> In particular, I'm curious if the 's' actually means anything (like >> string?). Should I be using different characters for different data >> types, and if so, which ones? > > You just about nailed it. You should ALWAYS use 's', too, regardless of > the data type. The DB API says "Python extended format codes" meaning that all valid Python formatting codes (including their parameters) may be used. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, May 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 45 days left From chris at cogdon.org Sat May 10 12:05:40 2003 From: chris at cogdon.org (Chris Cogdon) Date: Sat May 10 14:05:48 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <3EBD3CD6.8090001@lemburg.com> Message-ID: <02AF2FB7-8312-11D7-BD84-000393A6D442@cogdon.org> On Saturday, May 10, 2003, at 10:54 US/Pacific, M.-A. Lemburg wrote: > The DB API says "Python extended format codes" meaning that all > valid Python formatting codes (including their parameters) may > be used. That's very interesting. Looking at some of the code I have, using, for example %(key)d would not work, even if the parameter was an integer, because the API turns each and every single parameter into a string regardless of the type. This code snippet from pyPgSQL.PgSQL: if len(parms) == 1 and \ (type(parms[0]) in [DictType, ListType, TupleType] or \ isinstance(parms[0], PgResultSet)): parms = _quoteall(parms[0]) else: parms = tuple(map(_quote, parms)); self.res = self.conn.conn.query(_qstr % parms) Notice how the parameters are 'quoted' first (which will turn them ALL into strings), and THEN it's passed to the '%' operator. If you have any %d in there, you'll get a TypeError. And, take this docstring from pyPgSQL: cursor.execute(query[, param1[, param2, ..., paramN]) Execute a query, binding the parameters if they are passed. The binding syntax is the same as the '%' operator except that only %s [or %(name)s] should be used and the %s [or %(name)s] should not be quoted. Any necessary quoting will be performed by the execute method. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From fog at initd.org Sun May 11 10:39:09 2003 From: fog at initd.org (Federico Di Gregorio) Date: Sun May 11 05:39:11 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <02AF2FB7-8312-11D7-BD84-000393A6D442@cogdon.org> References: <02AF2FB7-8312-11D7-BD84-000393A6D442@cogdon.org> Message-ID: <1052645899.5500.10.camel@momo.initd.org> Il sab, 2003-05-10 alle 20:05, Chris Cogdon ha scritto: > On Saturday, May 10, 2003, at 10:54 US/Pacific, M.-A. Lemburg wrote: > > > The DB API says "Python extended format codes" meaning that all > > valid Python formatting codes (including their parameters) may > > be used. > > That's very interesting. Looking at some of the code I have, using, for > example %(key)d would not work, even if the parameter was an integer, > because the API turns each and every single parameter into a string > regardless of the type. > > This code snippet from pyPgSQL.PgSQL: > > if len(parms) == 1 and \ > (type(parms[0]) in [DictType, ListType, TupleType] > or \ > isinstance(parms[0], > PgResultSet)): > parms = _quoteall(parms[0]) > else: > parms = tuple(map(_quote, parms)); > self.res = self.conn.conn.query(_qstr % parms) > > Notice how the parameters are 'quoted' first (which will turn them ALL not to criticize pypgsql, but if its quoting code quotes integers, it is quite dumb and does a little bit too much work (i can understand why it does it that way, though.) other adapter are a little bit smarter... -- Federico Di Gregorio Debian GNU/Linux Developer fog@debian.org INIT.D Developer fog@initd.org Viviamo in un mondo reale, Ciccio. -- Lucy -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Questa parte del messaggio =?ISO-8859-1?Q?=E8?= firmata Url : http://mail.python.org/pipermail/db-sig/attachments/20030511/22563133/attachment.bin From andy47 at halfcooked.com Sun May 11 21:33:43 2003 From: andy47 at halfcooked.com (Andy Todd) Date: Sun May 11 15:35:53 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <3EBD3CD6.8090001@lemburg.com> References: <4E691ADF-830D-11D7-BD84-000393A6D442@cogdon.org> <3EBD3CD6.8090001@lemburg.com> Message-ID: <3EBEA597.2070705@halfcooked.com> M.-A. Lemburg wrote: > Chris Cogdon wrote: > >> >> On Saturday, May 10, 2003, at 10:23 US/Pacific, Andy Todd wrote: >> >>> Is there a definitive specification of the pyformat parameter style >>> anywhere? >>> >>> The usual sources come up dry. It is mentioned in the DB-API >>> specification (http://www.python.org/peps/pep-0249.html) but not >>> defined. >>> >>> It seems from observation that one should simply put '%(>> name>)s' in the query string and then pass a mapping with a >>> corresponding key when calling the execute method. But I'd just like >>> to make absolutely sure. >>> >>> In particular, I'm curious if the 's' actually means anything (like >>> string?). Should I be using different characters for different data >>> types, and if so, which ones? >> >> >> You just about nailed it. You should ALWAYS use 's', too, regardless >> of the data type. > > > The DB API says "Python extended format codes" meaning that all > valid Python formatting codes (including their parameters) may > be used. > Thanks both for your replies. For the curious (and the benefit of ASPN and google) the standard formatting codes are documented at; http://www.python.org/doc/current/lib/typesseq-strings.html If your db module says it supports 'pyformat' as the paramstyle then any of these format codes should be supported. It is also interesting to note that most RDBMS have native data and/or datetime data types but there is no direct conversion character because Python doesn't have a date type (until 2.3). I'm presuming that in these cases (no direct type correlation) you should use the 's' conversion type. Regards, Andy -- -------------------------------------------------------------------------------- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From jekabs.andrusaitis at tietoenator.com Mon May 12 01:04:59 2003 From: jekabs.andrusaitis at tietoenator.com (Jekabs Andrushaitis) Date: Sun May 11 18:27:42 2003 Subject: [DB-SIG] pyformat Parameter Style References: <4E691ADF-830D-11D7-BD84-000393A6D442@cogdon.org> <3EBD3CD6.8090001@lemburg.com> <3EBEA597.2070705@halfcooked.com> Message-ID: <3EBEC90B.6090006@tietoenator.com> Most of DB modules I have used understand only 's' type qualifier in pyformat argument style (ive tried it in psycopg and one of mysql modules). I do not really care much about this, but it creates alot of confusion, since normally in Python % all the qualifiers can be used... As of date fields - most DB modules use MX datetime package for date and time data types. -- Jekabs Andrushaitis Senior system analyst TietoEnator Financial Solutions 41 Lacplesa Str. Riga, LV-1011, Latvia Tel: +371 7286660 Fax: +371 7243000 E-mail: jekabs.andrusaitis@tietoenator.com From fog at initd.org Mon May 12 11:03:19 2003 From: fog at initd.org (Federico Di Gregorio) Date: Mon May 12 06:03:21 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <3EBEC90B.6090006@tietoenator.com> References: <4E691ADF-830D-11D7-BD84-000393A6D442@cogdon.org> <3EBD3CD6.8090001@lemburg.com> <3EBEA597.2070705@halfcooked.com> <3EBEC90B.6090006@tietoenator.com> Message-ID: <1052733803.934.56.camel@momo.initd.org> Il lun, 2003-05-12 alle 00:04, Jekabs Andrushaitis ha scritto: > Most of DB modules I have used understand only 's' type qualifier in > pyformat argument style (ive tried it in psycopg and one of mysql modules). if psycopg fails with %d and %f, then it is a bug. please report it. :) ciao, federico -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Questa parte del messaggio =?ISO-8859-1?Q?=E8?= firmata Url : http://mail.python.org/pipermail/db-sig/attachments/20030512/2138f3bb/attachment.bin From magnus at thinkware.se Mon May 12 22:01:17 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Mon May 12 14:59:22 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: Message-ID: <5.2.1.1.0.20030512201823.00befe08@www.thinkware.se> At Sat, 10 May 2003 19:54:30 +0200, M.-A. Lemburg wrote: >>On Saturday, May 10, 2003, at 10:23 US/Pacific, Andy Todd wrote: >>>Is there a definitive specification of the pyformat parameter style >>>anywhere? >>> >>>The usual sources come up dry. It is mentioned in the DB-API >>>specification (http://www.python.org/peps/pep-0249.html) but not defined. >>> >>>It seems from observation that one should simply put '%(>>name>)s' in the query string and then pass a mapping with a >>>corresponding key when calling the execute method. But I'd just like to >>>make absolutely sure. >>> >>>In particular, I'm curious if the 's' actually means anything (like >>>string?). Should I be using different characters for different data >>>types, and if so, which ones? > >The DB API says "Python extended format codes" meaning that all >valid Python formatting codes (including their parameters) may >be used. I've heard repeatedly that only %s should be used. This is the first time I've heard someone contradict this (however reasonable it seems). Maybe this should be clarified in the spec? (No, it's not clear if people misunderstand it reagrdless of what words in the text you can point to. ;) Is it just because of a misunderstanding that a number of drivers get this wrong, or is there some reason to turn integers and strings into floats. sqlite seems to choke on anything but %(...)s. I guess it's consistent with the Postgresql driver Gerhard works with? You either get "TypeError: an integer is required" (e.g. d, i and o) or "TypeError: bad argument type for built-in operation" (e.g. f and e). The culprit is here: def _quote(value): """_quote(value) -> string This function transforms the Python value into a string suitable to send to the SQLite database in a SQL statement. This function is automatically applied to all parameters sent with an execute() call. Because of this a SQL statement string in an execute() call should only use '%s' [or '%(name)s'] for variable substitution without any quoting.""" if value is None: return 'NULL' elif isinstance(value, StringType): return "'%s'" % value.replace("'", "''") elif isinstance(value, LongType): return str(value) elif hasattr(value, '_quote'): return value._quote() elif have_datetime and type(value) in \ (DateTime.DateTimeType, DateTime.DateTimeDeltaType): return "'%s'" % value else: return repr(value) This seems to be identical with pypgsql, wher eI found this: _quote(value) -> string This function transforms the Python value into a string suitable to send to the PostgreSQL database in a insert or update statement. This function is automatically applied to all parameter sent vis an execute() call. Because of this an update/insert statement string in an execute() call should only use '%s' [or '%(name)s'] for variable subsitution without any quoting. MySQLdb works the same (but using "format"). Everything is made a string. >>> import MySQLdb >>> conn = MySQLdb.connect(...) >>> cur = conn.cursor() >>> MySQLdb.paramstyle 'format' >>> cur.execute('SELECT name FROM whisky_brand where id=%d', 2) Traceback (most recent call last): File "", line 1, in ? File "G:\Python22\Lib\site-packages\MySQLdb\cursors.py", line 95, in execute return self._execute(query, args) File "G:\Python22\Lib\site-packages\MySQLdb\cursors.py", line 110, in _execute self.errorhandler(self, TypeError, m) File "G:\Python22\Lib\site-packages\MySQLdb\connections.py", line 33, in defaulterrorhandler raise errorclass, errorvalue TypeError: an integer is required >>> cur.execute('SELECT name FROM whisky_brand where id=%s', 2) 1L >>> -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The shortest path from thought to working program From mal at lemburg.com Tue May 13 12:31:08 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue May 13 05:31:55 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <5.2.1.1.0.20030512201823.00befe08@www.thinkware.se> References: <5.2.1.1.0.20030512201823.00befe08@www.thinkware.se> Message-ID: <3EC0BB5C.7020503@lemburg.com> Magnus Lyck? wrote: > At Sat, 10 May 2003 19:54:30 +0200, M.-A. Lemburg wrote: > >>> On Saturday, May 10, 2003, at 10:23 US/Pacific, Andy Todd wrote: >>> >>>> Is there a definitive specification of the pyformat parameter style >>>> anywhere? >>>> >>>> The usual sources come up dry. It is mentioned in the DB-API >>>> specification (http://www.python.org/peps/pep-0249.html) but not >>>> defined. >>>> >>>> It seems from observation that one should simply put '%(>>> name>)s' in the query string and then pass a mapping with a >>>> corresponding key when calling the execute method. But I'd just like >>>> to make absolutely sure. >>>> >>>> In particular, I'm curious if the 's' actually means anything (like >>>> string?). Should I be using different characters for different data >>>> types, and if so, which ones? >> >> >> The DB API says "Python extended format codes" meaning that all >> valid Python formatting codes (including their parameters) may >> be used. > > I've heard repeatedly that only %s should be used. This is the > first time I've heard someone contradict this (however reasonable > it seems). Maybe this should be clarified in the spec? (No, it's > not clear if people misunderstand it reagrdless of what words in > the text you can point to. ;) Looks like I ought to update the PEP then. Can you suggest a wording ? > Is it just because of a misunderstanding that a number of drivers > get this wrong, or is there some reason to turn integers and strings > into floats. You probably mean: floats into strings. I think the reason is that those drivers don't support binding in the sense that the SQL statement and the data are sent to the database as two separate entities. However, I don't understand why the current implementations don't use Python's wealth of format characters (and it's formatting implementation). That said, I still think that positional binding is by far the easiest to use and understand way of binding variables. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, May 13 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 42 days left From msanchez at grupoburke.com Tue May 13 10:56:17 2003 From: msanchez at grupoburke.com (Marcos =?ISO-8859-1?Q?S=E1nchez?= Provencio) Date: Tue May 13 05:56:19 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <3EC0BB5C.7020503@lemburg.com> References: <5.2.1.1.0.20030512201823.00befe08@www.thinkware.se> <3EC0BB5C.7020503@lemburg.com> Message-ID: <1052819737.4766.36.camel@cynar.proteus> El mar, 13 de 05 de 2003 a las 11:31, M.-A. Lemburg escribi?: > Magnus Lyck? wrote: > > At Sat, 10 May 2003 19:54:30 +0200, M.-A. Lemburg wrote: > > > >>> On Saturday, May 10, 2003, at 10:23 US/Pacific, Andy Todd wrote: > >>> > >>>> Is there a definitive specification of the pyformat parameter style > >>>> anywhere? > >>>> > >>>> The usual sources come up dry. It is mentioned in the DB-API > >>>> specification (http://www.python.org/peps/pep-0249.html) but not > >>>> defined. > >>>> > >>>> It seems from observation that one should simply put '%( >>>> name>)s' in the query string and then pass a mapping with a > >>>> corresponding key when calling the execute method. But I'd just like > >>>> to make absolutely sure. > >>>> > >>>> In particular, I'm curious if the 's' actually means anything (like > >>>> string?). Should I be using different characters for different data > >>>> types, and if so, which ones? > >> > >> > >> The DB API says "Python extended format codes" meaning that all > >> valid Python formatting codes (including their parameters) may > >> be used. > > > > I've heard repeatedly that only %s should be used. This is the > > first time I've heard someone contradict this (however reasonable > > it seems). Maybe this should be clarified in the spec? (No, it's > > not clear if people misunderstand it reagrdless of what words in > > the text you can point to. ;) > > Looks like I ought to update the PEP then. Can you suggest > a wording ? > > > Is it just because of a misunderstanding that a number of drivers > > get this wrong, or is there some reason to turn integers and strings > > into floats. > > You probably mean: floats into strings. > > I think the reason is that those drivers don't support binding > in the sense that the SQL statement and the data are sent to > the database as two separate entities. > > However, I don't understand why the current implementations > don't use Python's wealth of format characters (and it's formatting > implementation). > > That said, I still think that positional binding is by far > the easiest to use and understand way of binding variables. I see your point, but if you have to use your parameters several times in your sql statement it is a little awkward. For example, an optional filter parameter (imagine that with a combination of four or five parameters...): select * from xx where (yy=[param] or [param] is null) and zz=[param2] I always thought this should be encapsulated in a stored procedure until I had to use Oracle. Oracle needs writing more than a dozen lines (not the intuitive kind, if I may say) to cram that into a stored proc. From mal at lemburg.com Tue May 13 13:08:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue May 13 06:09:18 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <1052819737.4766.36.camel@cynar.proteus> References: <5.2.1.1.0.20030512201823.00befe08@www.thinkware.se> <3EC0BB5C.7020503@lemburg.com> <1052819737.4766.36.camel@cynar.proteus> Message-ID: <3EC0C429.5070102@lemburg.com> Marcos S?nchez Provencio wrote: >>That said, I still think that positional binding is by far >>the easiest to use and understand way of binding variables. > > I see your point, but if you have to use your parameters several times > in your sql statement it is a little awkward. For example, an optional > filter parameter (imagine that with a combination of four or five > parameters...): > > select * from xx > where (yy=[param] or [param] is null) > and zz=[param2] Uhm, why would you want to use "[param] is null" in SQL ? You can check that in Python. There are situations where you have to do multiple binding, but that's not really harmful, e.g. ... where col1 > [param1] and col2 < [param1] There's no additional overhead introduced by this, since the two occurences of [param1] both refer to the same Python object and the DB API modul can reuse the already converted data a second time. > I always thought this should be encapsulated in a stored procedure until > I had to use Oracle. Oracle needs writing more than a dozen lines (not > the intuitive kind, if I may say) to cram that into a stored proc. Moreover, it's nowhere near as portable to use stored procedures. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, May 13 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 42 days left From magnus at thinkware.se Tue May 13 15:02:22 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Tue May 13 08:00:18 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <3EC0BB5C.7020503@lemburg.com> References: <5.2.1.1.0.20030512201823.00befe08@www.thinkware.se> <5.2.1.1.0.20030512201823.00befe08@www.thinkware.se> Message-ID: <5.2.1.1.0.20030513124616.00bec290@www.thinkware.se> At 11:31 2003-05-13 +0200, M.-A. Lemburg wrote: >Looks like I ought to update the PEP then. Can you suggest >a wording ? I can try! :) ---- paramstyle String constant stating the type of parameter marker formatting expected by the interface. Possible values are [2]: 'qmark' Question mark style 'numeric' Numeric, positional style 'named' Named style 'format' All ANSI C printf format codes 'pyformat' All Python extended format codes See .execute below for examples. ---- IIRC there has also been discussions on whether execute("... x=%i AND y=%s", 1, 2) is correct, or whether it should be execute("... x=%i AND y=%s", (1, 2)) or perhaps either. I think a clarification on execute is good as well, and honestly, now that I look at it, I feel uncertain, but I assume it should be just .execute(s, p), where p is a sequence or a mapping, i.e. the second example above. From a BNF etc p.o.v. I guess "parameters" is *one* argument, containing all the parameters for the SQL operation. ---- .execute(operation[,parameters]) Prepare and execute a database operation (query or command). Parameters may be provided as sequence or mapping and will be bound to variables in the operation. Variables are specified in a database-specific notation (see the module's paramstyle attribute for details). [5] ... Return values are not defined. Code examples with different paramstyles: 'qmark': sql = 'SELECT * FROM table WHERE age>=? AND name=?' cur.execute(sql, (15, 'Guido')) 'numeric': sql = 'SELECT * FROM table WHERE age>=:1 AND name=:2' cur.execute(sql, (15, 'Guido')) 'named': sql = 'SELECT * FROM table WHERE age>=:min_age AND name=:name' cur.execute(sql, {'min_age': 15, 'name': 'Guido'}) 'format': sql = 'SELECT * FROM table WHERE age>=%d AND name=%s' cur.execute(sql, (15, 'Guido')) 'pyformat': sql = 'SELECT * FROM table WHERE age>%(min_age)d AND name=%(name)' cur.execute(sql, {'min_age': 15, 'name': 'Guido'}) ---- As far as I understand, sql = 'SELECT * FROM table WHERE age>=:2 AND name=:1' cur.execute(sql, (15, 'Guido')) will do exactly the same thing as the example above, and not the same thing as sql = 'SELECT * FROM table WHERE age>=:2 AND name=:1' cur.execute(sql, ('Guido', 15)) which is what I had expected. I don't know if that should be mentioned in the spec. >You probably mean: floats into strings. You are probably right. :) >I think the reason is that those drivers don't support binding >in the sense that the SQL statement and the data are sent to >the database as two separate entities. Right. They perform the "SQL % parameters" in Python after turning all parameters into strings, and that's where the exceptions come from. At least for pysqlite and the related PostgreSQL driver. >However, I don't understand why the current implementations >don't use Python's wealth of format characters (and it's formatting >implementation). Right. It's as simple as to not do anything with numeric types, rather than to call repr() on them. And that will still work if people write %s in their SQL even for numeric values. Actually, using repr() on a float will probably lead to a few surprises that %f or %s on the actual float prevents: >>> "%s %f %s" % (0.1, 0.1, repr(0.1)) '0.1 0.100000 0.10000000000000001' The backend might well be able to store an exact decimal representation, and then it's a bit silly to store the Python float approximation. >That said, I still think that positional binding is by far >the easiest to use and understand way of binding variables. A mapping is obviously useful if we have the same value several times in the SQL statement. -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The shortest path from thought to working program From magnus at thinkware.se Tue May 13 15:18:56 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Tue May 13 08:16:51 2003 Subject: [DB-SIG] Another paramstyle issue Message-ID: <5.2.1.1.0.20030513141129.02983808@www.thinkware.se> Since the issue of paramstyles is up: Today, X.paramstyle returns a string, but drivers might support several paramstyles. Should we have a module attribute that returned a list of paramestyles, or a query function? Something like this? >>> import myDBmodule as d >>> d.paramstyles ['qmark', 'named'] >>> d.supports_paramstyle('qmark') 1 >>> d.supports_paramstyle('format') 0 -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The shortest path from thought to working program From chris at cogdon.org Tue May 13 09:27:48 2003 From: chris at cogdon.org (Chris Cogdon) Date: Tue May 13 11:27:53 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <3EC0BB5C.7020503@lemburg.com> Message-ID: <742FF436-8557-11D7-B2CE-000393A6D442@cogdon.org> On Tuesday, May 13, 2003, at 02:31 US/Pacific, M.-A. Lemburg wrote: > However, I don't understand why the current implementations > don't use Python's wealth of format characters (and it's formatting > implementation). *I* understand why :) When one is passing data to the SQL backend, it should be up to the BACKEND to determine how it is passed, and by that, I mean the python interface to the backend. Take this example: Say I want to store a float into a row. The current way, which works, would be to do this: cursor.execute ( "insert into data ( %s )", my_float ) The interface sees the floating point value, turns that into the 'appropriate' representation that the SQL syntax accepts, and then passes it to the DBMS. Now, it is 'expected' that the interface knows what the 'appropriate' format is, and does what's Right. Not only should the application writer not know, and not be forced to worry about it, but he shouldn't be ALLOWED to worry about it. For example: cursor.execute ( "insert into data ( %2.2f )", my_float ) Now, why in the world would you ever WANT to strip off the precision from the value? It just doesn't make any sense. I think here's the reasoning for the mistake: THe '%' operator in python is a DISPLAY formatting operator. Ie, grab a value, turn it into a string appropriate for the particular application's DISPLAY purposes. However, in the 'pass values to the SQL backend' application, we're not displaying anything, but instead passing data to the sql backend. The '%s' and '%(keyname)s' syntax was used because it was convenient, did the job, required VERY little coding and didn't interfere with anything else. This does not mean that "Oh, since the python % operator is used in the interface, that means we can use all the OTHER python formatting codes, too" I agree that the DB-API spec as written implies that the other codes can be used too, but I do not think this was the intention and, I would argue, given the above reasons, that we do NOT code for this circumstance. Using all the python formatting operators does not make sense in this circumstance because it should NOT be up to the application writer to determine how the data is being passed to the backend. It should be up to the interface, which knows a lot more about the backend's requirements than the application writer. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From gh at ghaering.de Tue May 13 19:57:36 2003 From: gh at ghaering.de (=?ISO-8859-1?Q?Gerhard_H=E4ring?=) Date: Tue May 13 12:57:42 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <742FF436-8557-11D7-B2CE-000393A6D442@cogdon.org> References: <742FF436-8557-11D7-B2CE-000393A6D442@cogdon.org> Message-ID: <3EC12400.4030909@ghaering.de> Chris Cogdon wrote: > On Tuesday, May 13, 2003, at 02:31 US/Pacific, M.-A. Lemburg wrote: > >> However, I don't understand why the current implementations >> don't use Python's wealth of format characters (and it's formatting >> implementation). > > *I* understand why :) > > When one is passing data to the SQL backend, it should be up to the > BACKEND to determine how it is passed, and by that, I mean the python > interface to the backend. Take this example: > > Say I want to store a float into a row. The current way, which works, > would be to do this: > > cursor.execute ( "insert into data ( %s )", my_float ) > > The interface sees the floating point value, turns that into the > 'appropriate' representation that the SQL syntax accepts, and then > passes it to the DBMS. Now, it is 'expected' that the interface knows > what the 'appropriate' format is, and does what's Right. Not only should > the application writer not know, and not be forced to worry about it, > but he shouldn't be ALLOWED to worry about it. For example: > > cursor.execute ( "insert into data ( %2.2f )", my_float ) > > Now, why in the world would you ever WANT to strip off the precision > from the value? It just doesn't make any sense. > > > I think here's the reasoning for the mistake: > > THe '%' operator in python is a DISPLAY formatting operator. Ie, grab a > value, turn it into a string appropriate for the particular > application's DISPLAY purposes. However, in the 'pass values to the SQL > backend' application, we're not displaying anything, but instead passing > data to the sql backend. > > The '%s' and '%(keyname)s' syntax was used because it was convenient, > did the job, required VERY little coding and didn't interfere with > anything else. This does not mean that "Oh, since the python % operator > is used in the interface, that means we can use all the OTHER python > formatting codes, too" > > I agree that the DB-API spec as written implies that the other codes can > be used too, but I do not think this was the intention and, I would > argue, given the above reasons, that we do NOT code for this > circumstance. Using all the python formatting operators does not make > sense in this circumstance because it should NOT be up to the > application writer to determine how the data is being passed to the > backend. It should be up to the interface, which knows a lot more about > the backend's requirements than the application writer. I was going to answer in this thread, but there's nothing I could add to this brilliant post. -- Gerhard From chris at cogdon.org Tue May 13 19:29:26 2003 From: chris at cogdon.org (Chris Cogdon) Date: Tue May 13 21:29:36 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <3EC12400.4030909@ghaering.de> Message-ID: <7FFCAE93-85AB-11D7-B2CE-000393A6D442@cogdon.org> On Tuesday, May 13, 2003, at 09:57 US/Pacific, Gerhard H?ring wrote: > I was going to answer in this thread, but there's nothing I could add > to this brilliant post. Thanks for the support :) Let me add a little more fuel to the fire... Chris Cogdon wrote: > cursor.execute ( "insert into data ( %2.2f )", my_float ) > Now, why in the world would you ever WANT to strip off the precision > from the value? It just doesn't make any sense. In addition to that, stripping the precision in such a manner CHANGES THE VALUE of the data being passed to the backend, which is beyond the scope of what the % operator was intended for. If one really wanted to change the value in such a manner, I would recommend something like the following: cursor.execute ( "insert into data ( %s )", round(my_float,2) ) which is a lot more obvious what's going on. Another example.... cursor.execute ( "insert into data ( %r )", my_anything ) What in the world does THIS mean?! Currently, the backend code, on receiving a string, will make the 'appropriate' quoting changes to the string so it can be passed to the backend (this is why you don't put quotes around the %s; the DBI does it for you). For many DBMSes, the quoting rules are a little different (like, how to handle embedded special characters), or VERY different (like what type of quoting marks are allowed) than python. If we allow %r into the mix, what does this actually mean? Do we do the standard python repr() conversion, then pass it to the DB's quoting rules, or the other way around, or just one of them? (which won't work in a lot of cases). This is becoming a real mess. The ONLY valid solution I see is to very specifically state that pyformat allows %s and %(keyname)s, and nothing else, which is what many API's already do; they don't specifically check for %non-s, but it doesn't work, because the parameters are already strings by the time it gets to the python % operator (see code snippets previous in this thread). The docs also imply that 'format' is for '%s' and 'pyformat' is for '%(keyname)s', but many APIs (eg, pyPgSQL) allow both, and intelligently detect. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From rmangaliag at slu.edu.ph Wed May 14 15:29:00 2003 From: rmangaliag at slu.edu.ph (ali) Date: Wed May 14 02:18:56 2003 Subject: [DB-SIG] Re: DB-SIG Digest, Vol 2, Issue 9 References: Message-ID: <003f01c319e2$a903aec0$e019a8c0@slu.edu.ph> i'm quite a python newbie and i dont actually know if this is a valid question here... anyway, here it goes... is there an available program/software that translates a database structure/schema to python classes (code)??? example: if i have a database table created using this sql: create table sample (sid int unique, sname char(20)); the program will create a python file with this format: class sample: def __init__(self, sid, sname=None): self.sid = sid self.sname = sname def setsid(self, sid): self.sid = sid def setsname(self, sname): self.sname = sname def getsid(self): return self.sid def getsname(self): return self.sname From fog at initd.org Wed May 14 07:39:08 2003 From: fog at initd.org (Federico Di Gregorio) Date: Wed May 14 02:39:10 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <7FFCAE93-85AB-11D7-B2CE-000393A6D442@cogdon.org> References: <7FFCAE93-85AB-11D7-B2CE-000393A6D442@cogdon.org> Message-ID: <1052894326.933.180.camel@momo.initd.org> Il mer, 2003-05-14 alle 03:29, Chris Cogdon ha scritto: [snip] > This is becoming a real mess. > > The ONLY valid solution I see is to very specifically state that > pyformat allows %s and %(keyname)s, and nothing else, which is what > many API's already do; they don't specifically check for %non-s, but it > doesn't work, because the parameters are already strings by the time it > gets to the python % operator (see code snippets previous in this > thread). good points, but please, lets put this in DBAPI document. if format and pyformat only support %s and %(name)s this should be written somewhere. -- Federico Di Gregorio Debian GNU/Linux Developer fog@debian.org INIT.D Developer fog@initd.org We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. -- D.E.Knuth -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Questa parte del messaggio =?ISO-8859-1?Q?=E8?= firmata Url : http://mail.python.org/pipermail/db-sig/attachments/20030514/5b1680ce/attachment.bin From msanchez at grupoburke.com Wed May 14 11:02:58 2003 From: msanchez at grupoburke.com (=?ISO-8859-1?Q?Marcos_S=E1nchez_Provencio?=) Date: Wed May 14 04:03:37 2003 Subject: [DB-SIG] Re: DB-SIG Digest, Vol 2, Issue 9 In-Reply-To: <003f01c319e2$a903aec0$e019a8c0@slu.edu.ph> References: <003f01c319e2$a903aec0$e019a8c0@slu.edu.ph> Message-ID: <3EC1F832.7010206@grupoburke.com> Try this: http://www.thinkware.se/cgi-bin/thinki.cgi/ObjectRelationalMappersForPython ali wrote: >i'm quite a python newbie and >i dont actually know if this is a valid question here... anyway, here it >goes... > >is there an available program/software that translates a database >structure/schema to python classes (code)??? > >example: > >if i have a database table created using this sql: > >create table sample (sid int unique, sname char(20)); > >the program will create a python file with this format: > >class sample: > def __init__(self, sid, sname=None): > self.sid = sid > self.sname = sname > def setsid(self, sid): > self.sid = sid > def setsname(self, sname): > self.sname = sname > def getsid(self): return self.sid > def getsname(self): return self.sname > > > >_______________________________________________ >DB-SIG maillist - DB-SIG@python.org >http://mail.python.org/mailman/listinfo/db-sig > > From jacobs at penguin.theopalgroup.com Wed May 14 07:12:22 2003 From: jacobs at penguin.theopalgroup.com (Kevin Jacobs) Date: Wed May 14 06:12:56 2003 Subject: [DB-SIG] Re: DB-SIG Digest, Vol 2, Issue 9 In-Reply-To: <003f01c319e2$a903aec0$e019a8c0@slu.edu.ph> Message-ID: On Wed, 14 May 2003, ali wrote: > i'm quite a python newbie and > i dont actually know if this is a valid question here... anyway, here it > goes... > > is there an available program/software that translates a database > structure/schema to python classes (code)??? Yes and no. The better question is why on earth you'd want to. See my db_row module for a nicer alternative at: http://opensource.theopalgroup.com/ Regards, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From mal at lemburg.com Wed May 14 17:36:14 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed May 14 10:36:59 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <7FFCAE93-85AB-11D7-B2CE-000393A6D442@cogdon.org> References: <7FFCAE93-85AB-11D7-B2CE-000393A6D442@cogdon.org> Message-ID: <3EC2545E.6000103@lemburg.com> Chris Cogdon wrote: > The ONLY valid solution I see is to very specifically state that > pyformat allows %s and %(keyname)s, and nothing else, which is what many > API's already do; they don't specifically check for %non-s, but it > doesn't work, because the parameters are already strings by the time it > gets to the python % operator (see code snippets previous in this thread). I don't think the DB API should limit the authors in coming up with good uses of the various Python format codes. How they implement these is really up to the module author: %s could mean: "bind as string" while %i means: "bind as integer". %r could have the meaning: "bind as blob". You could even add new parameters for e.g. date, datetime and time. etc. etc. > The docs also imply that 'format' is for '%s' and 'pyformat' is for > '%(keyname)s', but many APIs (eg, pyPgSQL) allow both, and intelligently > detect. Which perfectly valid. Making the DB API docs more specific wouldn't save you from reading the DB API module docs ;-) Porting from one module to another is work nomatter how you put it. Most of it goes into rewriting the SQL because that's what's causing most of the trouble. The next big issue is that of differing data types. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, May 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 41 days left From fog at initd.org Wed May 14 15:53:04 2003 From: fog at initd.org (Federico Di Gregorio) Date: Wed May 14 10:53:05 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <3EC2545E.6000103@lemburg.com> References: <7FFCAE93-85AB-11D7-B2CE-000393A6D442@cogdon.org> <3EC2545E.6000103@lemburg.com> Message-ID: <1052923989.1093.53.camel@momo.initd.org> Il mer, 2003-05-14 alle 16:36, M.-A. Lemburg ha scritto: > Chris Cogdon wrote: > > The ONLY valid solution I see is to very specifically state that > > pyformat allows %s and %(keyname)s, and nothing else, which is what many > > API's already do; they don't specifically check for %non-s, but it > > doesn't work, because the parameters are already strings by the time it > > gets to the python % operator (see code snippets previous in this thread). > > I don't think the DB API should limit the authors in coming up > with good uses of the various Python format codes. How they > implement these is really up to the module author: > > %s could mean: "bind as string" while %i means: "bind as integer". > %r could have the meaning: "bind as blob". You could even add new > parameters for e.g. date, datetime and time. etc. etc. i am not sure i like that. using different codes is perfectly good _as long_ as the dbapi explains exactly what's the expected behaviour. -- Federico Di Gregorio Debian GNU/Linux Developer fog@debian.org INIT.D Developer fog@initd.org Beh un bacio, se ben dato, non si rifiuta. -- -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Questa parte del messaggio =?ISO-8859-1?Q?=E8?= firmata Url : http://mail.python.org/pipermail/db-sig/attachments/20030514/3b117dea/attachment.bin From chris at cogdon.org Wed May 14 09:30:39 2003 From: chris at cogdon.org (Chris Cogdon) Date: Wed May 14 11:30:44 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <3EC2545E.6000103@lemburg.com> Message-ID: <049359F2-8621-11D7-B7B7-000393A6D442@cogdon.org> On Wednesday, May 14, 2003, at 07:36 US/Pacific, M.-A. Lemburg wrote: > Chris Cogdon wrote: >> The ONLY valid solution I see is to very specifically state that >> pyformat allows %s and %(keyname)s, and nothing else, which is what >> many API's already do; they don't specifically check for %non-s, but >> it doesn't work, because the parameters are already strings by the >> time it gets to the python % operator (see code snippets previous in >> this thread). > > I don't think the DB API should limit the authors in coming up > with good uses of the various Python format codes. How they > implement these is really up to the module author: > > %s could mean: "bind as string" while %i means: "bind as integer". > %r could have the meaning: "bind as blob". You could even add new > parameters for e.g. date, datetime and time. etc. etc. I've no issue with APIs doing 'extra, non-standard' stuff as long as it doesn't interfere with the existing specification. However, I *would* like the specs clarified so that it stipulates that '%s' or '%(keyname)s' is to be used for all data types. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From magnus at thinkware.se Thu May 15 01:24:51 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Wed May 14 18:22:54 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: Message-ID: <5.2.1.1.0.20030514220347.01eb18c0@www.thinkware.se> This is a detail, but I think it's misleading to support %s but not %f etc. On the other hand, things happen with the parameters passed to SQL statements, so it might be misleading with %s etc either way. But it's the strings that are changed. The numeric values are treaded just as python programmers would expect. Why not use ? and :name instead of %s and %(name)s if the driver is to determine how the value is translated anyway. Is it *only* to make it easier to write modules? Perhaps someone could write an SQL "?" parser and donate that to the DB-API community, and we can all use qmark instead? There can't be so terribly many cases where ? occurs in an SQL statement without being a parameter marker, can there? Chris Cogdon wrote: >When one is passing data to the SQL backend, it should be up to the >BACKEND to determine how it is passed, and by that, I mean the python >interface to the backend. Take this example: > >Say I want to store a float into a row. The current way, which works, >would be to do this: > >cursor.execute ( "insert into data ( %s )", my_float ) To anyone used to Python this looks like we are passing in str(my_float) while we are in fact passing in repr(my_float). So if the backend is capable of storing exact decimal values and we try to store 0.1, we will accidentally store 0.10000000000000001. It's likely that the backend won't be configured to so many digits, and will round it to 0.1 anyway, but I don't really like this. It's not obvious as Python usually is. With cursor.execute("insert into data ( ? )", my_float) or cursor.execute("insert into data ( :1 )", my_float) I don't have such a preconception. They do't give me any "cast to string" message. After all, it *must* be the application programmer who decides what value to pass to the backend. If we have paramstyle = format, why not allow him to express this as "... x=%.2f", my_float) instead of "... x=%s", round(my_float, 2)) if that is what he wants? The problem here is not with %f or %d, but with %s and string values! See below. >The interface sees the floating point value, turns that into the >'appropriate' representation that the SQL syntax accepts, and then passes >it to the DBMS. Now, it is 'expected' that the interface knows what the >'appropriate' format is, and does what's Right. Not only should the >application writer not know, and not be forced to worry about it, but he >shouldn't be ALLOWED to worry about it. For example: I disagree. If a database application programmer is not aware of how floating point numbers work in Python and in his backend on the given platform, he is likely to cause a mess. I've cleaned up such things, both in SQL code and in C++ code in various projects. There is no way the driver can save anyone from doing stupid things like comparing floats on equality without any fuzzfactor. The programmer *has* to worry about float representations. This doesn't imply that he needs full format capability in the SQL strings, but he can't let the computer handle that in any programming language that uses floating point arithmetic in the way C and Python does. (Not that it's fully problem free with exact decimal representations either.) And it seems to me that the purpose of using %s and only %s instead of ? or :1 which is the norm in SQL is only to make life easier for the driver programmer, rather than to protect the applicaiton programmer. This is *not* without a risk for the applicaiton programmer, as we will see. >cursor.execute ( "insert into data ( %2.2f )", my_float ) To anyone used to python it should be clear what this implies. Do you mean that DB application programmers by accident enter 2.2 without knowing what it means? I find that unlikely. I suppose it's possible that someone would type %f and not realize that this is the same as %.6f though. That *is* a risk. I can appreciate that. But today, you must use %r to be able to get exactly the same behaviour across database drivers, since the drivers that are complient with the spec will use %s if you tell them to do that, and others like pysqlite will use repr() whatever you type after the %-sign. By the way, I think all the drivers in question that stops our ignorant application programmer from writing something potentially harmful as ("insert into data ( %2.2f )", 1000./3.) will gladly allow him to write something decidedly harmful such as ("insert into data ( %2.2s )", 1000./3.) and silently store 33 in the database instead of 333.33 which had been the case with %2.2f. At least that is what pysqlite does. :( Why on earth is that permitted, if the purpose is to protect the application programmer from mistakes? Obviously, it's not enough that the application programmer understands how floats work. He needs to understand how float works, how python format strings work, and the odd ways they are used in various database drivers... Of course you could say "Only use %s, don't put in any other things like %.2s, but if this is the intention, why write the code so that the user can make such a mess. Perhaps these modules should throw an exception as soon as % is followed by anything else than "s", "%" or "(word)s" ? That's where the problems really are, since the drivers escape and quote the strings, which makes them look different then they do in the scope of the client application. Also, the implementations of %s in for instance pysqlite chokes on things like "select * from x where a=%s and b like 'hello%'" You have to use 'hello%%', or you will get a "TypeError: not enough arguments for format string" In other words, it seems that a %s-limited format or %(xxx)s limited pyformat isn't there to help the application programmer, since it has dangers that ?, :1 or :xxx doesn't have, but is instead there just to make life somewhat simpler for the driver programmer. Here are other ways to do strange things with %s: >>> cur.execute('create table y ( a int, b char(50))') >>> cur.execute('insert into y values (%s, %s||%s)', (1, 'hi', 'there')) >>> cur.execute('insert into y values (%s, %.2s||%s)', (2, "'secret", "there")) >>> cur.execute('insert into y values (%s, %.1s||%.1s)', (3, "hi", "there")) >>> cur.execute('insert into y values (%s, %.4s)', (4, "hi''there")) >>> cur.execute('select * from y') >>> cur.fetchall() [(1, 'hithere'), (2, 'there'), (3, '||'), (4, 'hi')] I do think that cursor.execute("insert into data (%.2f)", my_float) might be a reasonable thing to do, and to disallow that while allowing the rubbish I did here, can hardly be for the sake of the application programmer... :) -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The shortest path from thought to working program From chris at cogdon.org Wed May 14 17:03:33 2003 From: chris at cogdon.org (Chris Cogdon) Date: Wed May 14 19:03:38 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <5.2.1.1.0.20030514220347.01eb18c0@www.thinkware.se> Message-ID: <49746883-8660-11D7-AC6D-000393A6D442@cogdon.org> On Wednesday, May 14, 2003, at 15:24 US/Pacific, Magnus Lyck? wrote: > This is a detail, but I think it's misleading to support > %s but not %f etc. It would be misleading, yes, unless it was specifically documented. > To anyone used to Python this looks like we are passing in > str(my_float) > while we are in fact passing in repr(my_float). Actually, that's not true. Python's quoting rules are NOT the same as most DMBS's quoting rules. For example the following: cursor ( "select blah from data where thingy=" + repr(thingy_value) ) would only work in some circumstances. It would NOT work if, say, thingy_value contains a ' (python would then use " as a string quoter, which means something very different to the DBMS than single quotes), or if thingy_value contains foreign characters (python 2.2 uses hex escapes, while most DMBSes require octal escapes). > So if the backend is capable of storing exact decimal values and > we try to store 0.1, we will accidentally store 0.10000000000000001. > It's likely that the backend won't be configured to so many digits, > and will round it to 0.1 anyway, but I don't really like this. That's why it's the responsibility of the interface to 'correctly translate' any values passed to it. > After all, it *must* be the application programmer who decides what > value > to pass to the backend. If we have paramstyle = format, why not allow > him > to express this as "... x=%.2f", my_float) instead of > "... x=%s", round(my_float, 2)) if that is what he wants? yes, it's the application programmers responsibility to determine what values are passed, but not HOW they're passed, since that's up to the backend syntax specification. The API writer is far more qualified to know what the requirements are than the application programmer. > I disagree. If a database application programmer is not aware of how > floating point numbers work in Python and in his backend on the given > platform, he is likely to cause a mess. I've cleaned up such things, > both in SQL code and in C++ code in various projects. There is no way > the driver can save anyone from doing stupid things like comparing > floats on equality without any fuzzfactor. The programmer *has* to > worry about float representations. This doesn't imply that he needs > full format capability in the SQL strings, but he can't let the > computer handle that in any programming language that uses floating > point arithmetic in the way C and Python does. (Not that it's fully > problem free with exact decimal representations either.) I agree that there may need to be an exception, or a clarification made, in the case of floats. However, this is also true when doing straight-out python coding. It is 'risky' to code something like the following: f = read_float_value_from_somewhere() if f = 0.2: do_something_important () This is risky because 'f' cannot ever possibly be exactly 0.2 >> cursor.execute ( "insert into data ( %2.2f )", my_float ) > > To anyone used to python it should be clear what this implies. > Do you mean that DB application programmers by accident enter 2.2 > without knowing what it means? I find that unlikely. It's 'obvious' only if the programmer has 'assumed' that his value will be treated according to the python '%' operator conventions, which is not true. Most interfaces run through a quoting function before being applied to the string containing the '%s's > But today, you must use %r to be able to get exactly the same > behaviour across database drivers, since the drivers that are > complient with the spec will use %s if you tell them to do > that, and others like pysqlite will use repr() whatever you > type after the %-sign. Well, keep in mind that 'the spec' doesn't mention ANYTHING other than '%s' and '%(key)s'. And, looking at existing interfaces, many have not coded for the possibility that anything other than those two will be used. And, again, '%r's actual MEANING is very poorly defined. repr() does not return a value that will always be accepted by the DMBS. It requires extra quoting, and then should be be quoting and THEN repr, or the other way around ? > By the way, I think all the drivers in question that stops our > ignorant application programmer from writing something potentially > harmful as ("insert into data ( %2.2f )", 1000./3.) will gladly > allow him to write something decidedly harmful such as > ("insert into data ( %2.2s )", 1000./3.) and silently store 33 > in the database instead of 333.33 which had been the case with > %2.2f. At least that is what pysqlite does. :( Most likely because the interface writer is just using the '%' operator to do the insertion of the parameters into the string, without 'stopping' the application writer from doing things like '%2.2s', and so on. There are many other instances where the behaviour outside of the specification is not well defined, but does not return an error. > "Only use %s, don't put in any other things like %.2s, but if > this is the intention, why write the code so that the user can > make such a mess. Because, trapping for this would mean writing a lot more code. Just slapping in '%' is both easy to do, and fast, because it's a 'built-in'. > Perhaps these modules should throw an exception as soon as % is > followed by anything else than "s", "%" or "(word)s" ? That's > where the problems really are, since the drivers escape and > quote the strings, which makes them look different then they > do in the scope of the client application. I agree that that would be the 'best' solution, but it would slow down the code somewhat. > Also, the implementations of %s in for instance pysqlite chokes > on things like "select * from x where a=%s and b like 'hello%'" > You have to use 'hello%%', or you will get a > "TypeError: not enough arguments for format string" Yep. That should have been part of the specification. > In other words, it seems that a %s-limited format or %(xxx)s limited > pyformat isn't there to help the application programmer, since it has > dangers that ?, :1 or :xxx doesn't have, but is instead there just to > make life somewhat simpler for the driver programmer. No argument here :) Perhaps the specification should be 'clarified' by writing the following: """ To use positional parameter parsing, use '%s' and pass your parameters as a tuple as the second parameter to the 'execute' function. To use keyword parameter parsing, use '%(key)s' and pass your parameters as a dictionary as the second parameter to the 'execute' function. If you need a '%' in your SQL syntax, use '%%'. This convention operates much like python '%' operator, and may in fact be implemented as such. However, the behaviour of using anything other than '%s' and '%(key)s' is not defined, and should be avoided. The interface is not required to trap for these conditions in this version of the specification """ -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From andy47 at halfcooked.com Thu May 15 10:44:13 2003 From: andy47 at halfcooked.com (Andy Todd) Date: Thu May 15 04:46:42 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <49746883-8660-11D7-AC6D-000393A6D442@cogdon.org> References: <49746883-8660-11D7-AC6D-000393A6D442@cogdon.org> Message-ID: <3EC3535D.4040100@halfcooked.com> Chris Cogdon wrote: > > On Wednesday, May 14, 2003, at 15:24 US/Pacific, Magnus Lyck? wrote: > [snip] > > No argument here :) > > Perhaps the specification should be 'clarified' by writing the following: > > """ > To use positional parameter parsing, use '%s' and pass your parameters > as a tuple as the second parameter to the 'execute' function. > > To use keyword parameter parsing, use '%(key)s' and pass your parameters > as a dictionary as the second parameter to the 'execute' function. > > If you need a '%' in your SQL syntax, use '%%'. > > This convention operates much like python '%' operator, and may in fact > be implemented as such. However, the behaviour of using anything other > than '%s' and '%(key)s' is not defined, and should be avoided. The > interface is not required to trap for these conditions in this version > of the specification > """ > > > +1 to placing that text in the DB-API specification. It makes things perfectly clear, even to me. Regards, Andy -- -------------------------------------------------------------------------------- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From magnus at thinkware.se Thu May 15 23:34:42 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Thu May 15 16:32:29 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <49746883-8660-11D7-AC6D-000393A6D442@cogdon.org> References: <5.2.1.1.0.20030514220347.01eb18c0@www.thinkware.se> Message-ID: <5.2.1.1.0.20030515204910.01f12578@www.thinkware.se> At 16:03 2003-05-14 -0700, Chris Cogdon wrote: >I (Magnus) wrote about %s: >>To anyone used to Python this looks like we are passing in str(my_float) >>while we are in fact passing in repr(my_float). > >Actually, that's not true. Python's quoting rules are NOT the same as most >DMBS's quoting rules. For example the following: > >cursor ( "select blah from data where thingy=" + repr(thingy_value) ) > >would only work in some circumstances. It would NOT work if, say, >thingy_value contains a ' (python would then use " as a string quoter, >which means something very different to the DBMS than single quotes), or >if thingy_value contains foreign characters (python 2.2 uses hex escapes, >while most DMBSes require octal escapes). And how often is there a ' in a float? :) This is my point. All the magic is in strings, and possibly in types like date, but there is *no* magic in the numeric types, so allowing numeric values to be passed in with %f or %i etc seems ok. Why disallow this? Practically, I'm suggesting (for pysqlite): def _quote(value): """_quote(value) -> string This function transforms the Python value into a string suitable to send to the SQLite database in a SQL statement. This function is automatically applied to all parameters sent with an execute() call. Because of this a SQL statement string in an execute() call should only use '%s' [or '%(name)s'] for variable substitution without any quoting.""" if value is None: return 'NULL' elif isinstance(value, StringType): return "'%s'" % value.replace("'", "''") - elif isinstance(value, LongType): - return str(value) + elif (isinstance(value, LongType) or isinstance(value, IntType) + or isinstance(value, FloatType)): + return value elif hasattr(value, '_quote'): return value._quote() elif have_datetime and type(value) in \ (DateTime.DateTimeType, DateTime.DateTimeDeltaType): return "'%s'" % value else: return repr(value) (One might consider using "type(value) in [long, int, float]" instead, since subclasses might do odd things on for instance __str__, but that's also a problem with the current code.) What would be the problem with this? %s will still work, but with 12 decimals instead of 17 for floats. I doubt that is a problem, and %e or %.16e or %r will still work. If the application programmer is insecure about the result, he can simply try it out with a normal formatted print. The problems exist with *strings*, not with numeric values, and *that* can't be solved by banning the use of %f etc for *numeric* values. I think it would be very useful to get the parsed SQL statements printed in some kind of debug mode, since strings can look different than one might expect. That is a feature I'd like to see. >>So if the backend is capable of storing exact decimal values and >>we try to store 0.1, we will accidentally store 0.10000000000000001. >>It's likely that the backend won't be configured to so many digits, >>and will round it to 0.1 anyway, but I don't really like this. > >That's why it's the responsibility of the interface to 'correctly >translate' any values passed to it. How on earth would the interface be able to know whether we actually want to store 0.1 or 0.10000000000000001 when a Python float can't distinguish them? For exact decimal numbers, we have a mismatch between Python and SQL, since Python is lacking such a datatype, and after some years of Python programming I feel that explicit is better than implicit. I.e. I prefer that I control this conversion rather than to let the interface do that. (The problem is still strings. I can accept that the interface escapes and quote strings, because the escape and quote stuff is not to be stored in the db, it's just a way to make the backend accept exactly what I try to store, but it's a bit annoying that things like %.5s will fail.) %f or %e formatting would give us a tool to handle the numeric type mismatch. Obviously rounding 0.1 to one decimal won't work. We will still store 0.10000000000000001 if the value is a Python float passed via repr(). If we convert it to the string '0.1' before passing it to the SQL statement, the inteface will adorn it with quotes, but maybe that will still work? (I hope so, otherwise the ? and :1 forms seem problematic.) Anyway, it seems full %-formatting could be useful, even if it's a rather small issue in my opinion. >I agree that there may need to be an exception, or a clarification made, >in the case of floats. However, this is also true when doing straight-out >python coding. It is 'risky' to code something like the following: > >f = read_float_value_from_somewhere() >if f = 0.2: > do_something_important () > >This is risky because 'f' cannot ever possibly be exactly 0.2 That is a syntax error, but f == 0.2 will work since 0.2 can't be exactly 0.2 either! :) Try it! (And try "0.1 + 0.1 == 0.2".) It more complex than this, and I won't bore you with my old "war stories", but my point is that the application programmer might get into situations where he needs to be able to have as much control as possible, and I've not heard a good reason so far to remove that control as some db interfaces completely needlessly does today. If you have a decimal(10,2) column X with 1.15 stored, and do "...WHERE X=%s", (1.15)) you will get ...WHERE X=1.1499999999999999 today. Are you sure that will match? If you did "...WHERE X=%.2f", (1.15)) and my patch was applied, you'd get ...WHERE X=1.15. That would certainly match. I imagine that if it is decimal(10,2) as I said, the float passed in will be coersed into a decimal(10,2) and it will work anyway, but I think I can show other cases where this *is* a problem if I have to. Probably rare, but why remove the simple tool to fix it when it happens? >It's 'obvious' only if the programmer has 'assumed' that his value will be >treated according to the python '%' operator conventions, which is not true. It would be much more pythonic is it *was* true. :) It's a really ugly hack as long as it's not true. One of the really great things with python is that it almost always works just as expected. I can learn something in one context and apply it in another. Other languages I use are full of "you can't do this with that" and "in that situation you will have to do it like this instead". Perl and Visual Basic for instance is full of such rubbish. >Most interfaces run through a quoting function before being applied to the >string containing the '%s's Only for strings, and that is a temporary decoration needed to get exactly the value in the original string passed to the backend! Changing the string Can't do into 'Can''t do' is just a temporary measure to make certain that we will actually use Can't do and neither more, nor less, in the actual SQL operation. I don't think that's more odd than the fact that the string "can" will have the representation 'can', and "can't" will have the representation "can't" in Python. That *is* an interface issue, and a feature for the coder. Making %f fail when it's passed 5 is not a feature. It's a bug, or at least a deviation from the spec. >Well, keep in mind that 'the spec' doesn't mention ANYTHING other than >'%s' and '%(key)s'. And, looking at existing interfaces, many have not >coded for the possibility that anything other than those two will be used. Yes it does. Read again. It only uses %s in the examples, but read the text. It seems M-A and I agree that it could be made clearer, but it *does* state "ANSI C printf format codes" and "Python extended format codes", and the tiny example is just that. Did you expect "name" being passed in as a flot or what? ;) >And, again, '%r's actual MEANING is very poorly defined. repr() does not >return a value that will always be accepted by the DMBS. It requires extra >quoting, and then should be be quoting and THEN repr, or the other way around ? I don't qute understand the precious sentences. But the interfaces in question use repr() as default. See _quote above. That's not *my* idea. >Most likely because the interface writer is just using the '%' operator to >do the insertion of the parameters into the string, without 'stopping' the >application writer from doing things like '%2.2s', and so on. There are >many other instances where the behaviour outside of the specification is >not well defined, but does not return an error. My point was mainly to point out that it's odd to block %f for no appearent good reason and allow people to write things like %.2s and thus trunctate something which is probably different than the applicaiton programmer intended. The tiny trap I see is that %f rounds to six decimals, which people might forget. By all means, suggest the use of %s, or even %e or %r for numerics, and above all, suggest that people try their code with print fmt % params, or even better make some debug function so that they can see what the SQL will look like. >Because, trapping for this would mean writing a lot more code. Just >slapping in '%' is both easy to do, and fast, because it's a 'built-in'. So it seems we agree. Things look like they do, not because it was best for the application programmer, but because it was easiest for the driver programmer. I agree that there is a value in making life easier for interface programmers. I also think that several people have misunderstood the specification, even if it's explicit about allowing the ANSI C format codes if you read it properly. But as I've shown above it is (I think) very simple to make it complient, and %s will continue to work like before unless you need ten decimals. >>Perhaps these modules should throw an exception as soon as % is >>followed by anything else than "s", "%" or "(word)s" ? That's >>where the problems really are, since the drivers escape and >>quote the strings, which makes them look different then they >>do in the scope of the client application. > >I agree that that would be the 'best' solution, but it would slow down the >code somewhat. If that is a problem, use "if __debug__:" and let people run with -O when they can't spare cycles for this check. -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The shortest path from thought to working program From chris at cogdon.org Thu May 15 16:26:01 2003 From: chris at cogdon.org (Chris Cogdon) Date: Thu May 15 18:26:13 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <5.2.1.1.0.20030515204910.01f12578@www.thinkware.se> Message-ID: <3519519C-8724-11D7-A262-000393A6D442@cogdon.org> On Thursday, May 15, 2003, at 13:34 US/Pacific, Magnus Lyck? wrote: > At 16:03 2003-05-14 -0700, Chris Cogdon wrote: >> I (Magnus) wrote about %s: >>> To anyone used to Python this looks like we are passing in >>> str(my_float) >>> while we are in fact passing in repr(my_float). >> >> Actually, that's not true. Python's quoting rules are NOT the same as >> most DMBS's quoting rules. For example the following: >> >> cursor ( "select blah from data where thingy=" + repr(thingy_value) ) >> >> would only work in some circumstances. It would NOT work if, say, >> thingy_value contains a ' (python would then use " as a string >> quoter, which means something very different to the DBMS than single >> quotes), or if thingy_value contains foreign characters (python 2.2 >> uses hex escapes, while most DMBSes require octal escapes). > > And how often is there a ' in a float? :) > > This is my point. All the magic is in strings, and possibly > in types like date, but there is *no* magic in the numeric > types, so allowing numeric values to be passed in with %f or > %i etc seems ok. Why disallow this? Practically, I'm suggesting > (for pysqlite): Actually, there IS magic in floats. You have rules for how exponents are represented, whether or not a leading 0 is required before the decimal point, whether or not a leading + is allowed, or only -, and so on. Again, the rules for a DMBS may not be the same as the rules for Python. Often, yes, they're similar enough that we can just use 'str(your_float)' and pass the output straight into the SQL query. My argument is that the only person qualified to make that statement is the interface writer. The application writer should not need to know. > (One might consider using "type(value) in [long, int, float]" > instead, since subclasses might do odd things on for instance > __str__, but that's also a problem with the current code.) Well, not necessarily. The interface knows the incoming type[1] and knows how to present it so that the backend is happy. [1]: Taking a OO-related tangent here, even in the case of a subclassed 'long' for example, the 'OO rules' state that derivitave classes must have all the properities of the base class, but are allowed to restrict the values at the base class level. For example, to take a stupid example, I may create a 'odd_long' class that is a derivitave of the 'long' class were all values must be odd. It still remains true that a 'odd_long' IS a long for all possible values of 'odd_long', and also always has a valid representation in 'long's space. > What would be the problem with this? %s will still work, but with > 12 decimals instead of 17 for floats. I doubt that is a problem, > and %e or %.16e or %r will still work. If the application programmer > is insecure about the result, he can simply try it out with a > normal formatted print. But, again, this is pushing the responsibility of 'object to string' conversion onto the application writer, which he's not supposed to worry about. That's what the interface is for. > I think it would be very useful to get the parsed SQL statements > printed in some kind of debug mode, since strings can look different > than one might expect. That is a feature I'd like to see. Most databases have a debugging mode that output the SQL statements that are being issued to the backend. I agree that it would be neat to have a 'hook' in the DB interface to do this, in the cases where this is not supplied, or you want to keep it all in python anyway. > How on earth would the interface be able to know whether we > actually want to store 0.1 or 0.10000000000000001 when a > Python float can't distinguish them? Because, the interface writer is expected to know the capabilities of both the backend, AND of Python. That's why it's called an 'interface' :) > For exact decimal numbers, we have a mismatch between Python > and SQL, since Python is lacking such a datatype, and after > some years of Python programming I feel that explicit is > better than implicit. I.e. I prefer that I control this > conversion rather than to let the interface do that. (The > problem is still strings. I can accept that the interface > escapes and quote strings, because the escape and quote stuff > is not to be stored in the db, it's just a way to make the > backend accept exactly what I try to store, but it's a bit > annoying that things like %.5s will fail.) The 'correct' solution in this case is to have a 'fixed_point' type with the capabilities you require. >> I agree that there may need to be an exception, or a clarification >> made, in the case of floats. However, this is also true when doing >> straight-out python coding. It is 'risky' to code something like the >> following: >> >> f = read_float_value_from_somewhere() >> if f = 0.2: >> do_something_important () >> >> This is risky because 'f' cannot ever possibly be exactly 0.2 > > That is a syntax error, but f == 0.2 will work since 0.2 can't > be exactly 0.2 either! :) Try it! (And try "0.1 + 0.1 == 0.2".) Sorry for the syntax error :) Yes, 0.1+0.1 == 0.2 does what's expected, but in other cases, it doesn't work. Viz: >>> 0.2 + 0.4 == 0.6 0 > It more complex than this, and I won't bore you with my old > "war stories", but my point is that the application programmer > might get into situations where he needs to be able to have as > much control as possible, and I've not heard a good reason so > far to remove that control as some db interfaces completely > needlessly does today. In that case, it's perfectly possible to form the SQL query completely yourself, and then pass it to execute without using the % syntax or parameters. > If you have a decimal(10,2) column X with 1.15 stored, and > do "...WHERE X=%s", (1.15)) you will get > ...WHERE X=1.1499999999999999 today. Are you sure that will > match? If you did "...WHERE X=%.2f", (1.15)) and my patch > was applied, you'd get ...WHERE X=1.15. That would certainly > match. I imagine that if it is decimal(10,2) as I said, the > float passed in will be coersed into a decimal(10,2) and it > will work anyway, but I think I can show other cases where > this *is* a problem if I have to. Probably rare, but why > remove the simple tool to fix it when it happens? Okay, that's a good example. The difference here is that the interface does not necessarily have access to the data types IN THE DATABASE which would be required to make the above example work as expected. I'd still like to see the other issues addressed, though (including the inconsistent use of %r). Perhaps a 'more correct' solution is to extend, in a limited fashion, which % escapes are allowed and which will return 'behaviour undefined'. (I think %r falls squarely in the behaviour undefined category) > It would be much more pythonic is it *was* true. :) > It's a really ugly hack as long as it's not true. Well, not necessarily. SQL is not python. Since we're trying to convert python concepts to non-python concepts, there's going to be a lot of give and take. The classic example is how strings are presented in both worlds. There are a lot of similarities, but they're NOT equivalent. Additionally, to me, just because a syntax uses '%s' style escapes, it does NOT necessarily mean that I get to use all the other escapes too. The documentation, even as it stands, only predicates the use of '%s' and '%(keyname)s'. > One of the really great things with python is that it > almost always works just as expected. I can learn something > in one context and apply it in another. Other languages I > use are full of "you can't do this with that" and "in that > situation you will have to do it like this instead". Perl > and Visual Basic for instance is full of such rubbish. Well, pyPgSQL does what I expect, too, because I read the docs :) >> Most interfaces run through a quoting function before being applied >> to the string containing the '%s's > > Only for strings, This is not true, and I re-cite pyPgSQL. > and that is a temporary decoration needed > to get exactly the value in the original string passed to the > backend! Changing the string > > Can't do > > into > > 'Can''t do' > > is just a temporary measure to make certain that we will > actually use > > Can't do > > and neither more, nor less, in the actual SQL operation. I don't > think that's more odd than the fact that the string "can" will > have the representation 'can', and "can't" will have the > representation "can't" in Python. That *is* an interface issue, > and a feature for the coder. This isn't totally correct, either. Python's repr() will turn 'high bit escapes' into \xnn with the newer releases of python, and this breaks the SQL, which only allowed octal escapes. Again, just because two escaping mechanisms LOOK similar, it doesn't mean they ARE :) > Yes it does. Read again. It only uses %s in the examples, > but read the text. It seems M-A and I agree that it could > be made clearer, but it *does* state "ANSI C printf format > codes" and "Python extended format codes", and the tiny > example is just that. Did you expect "name" being passed > in as a flot or what? ;) It depends on what is meant by 'python extended format codes'. Does it mean the whole gamut of %s %r %f, and so on, or does it JUST mean the capability to use the (keyname) extension to use by-name rather than by-position parameter passing. My initial interpretation, and that supported both by the pypg and pyPgSQL interfaces (not much experience with MySQL, sorry) is that the latter interpretation has been made. >> And, again, '%r's actual MEANING is very poorly defined. repr() does >> not return a value that will always be accepted by the DMBS. It >> requires extra quoting, and then should be be quoting and THEN repr, >> or the other way around ? > > I don't qute understand the precious sentences. But the > interfaces in question use repr() as default. See _quote > above. That's not *my* idea. And, perhaps they shouldn't, as repr() does not always return a value that is valid SQL syntax. >> Most likely because the interface writer is just using the '%' >> operator to do the insertion of the parameters into the string, >> without 'stopping' the application writer from doing things like >> '%2.2s', and so on. There are many other instances where the >> behaviour outside of the specification is not well defined, but does >> not return an error. > > My point was mainly to point out that it's odd to block %f for > no appearent good reason and allow people to write things like > %.2s and thus trunctate something which is probably different > than the applicaiton programmer intended. They're not blocking %f just as they're not blocking %.2s. My argument is that the python % operator is being used because it's convenient, but returns inconsistent results of you step outside the spec's predication (my interpretation of it) to only use %s and %(keyname)s. > I also think that several people have misunderstood the > specification, even if it's explicit about allowing the > ANSI C format codes if you read it properly. The DB-API 2.0 spec makes NO mention of ANSI C format codes ... that's just an implication :) -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From magnus at thinkware.se Fri May 16 04:08:08 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Thu May 15 21:05:58 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <3519519C-8724-11D7-A262-000393A6D442@cogdon.org> References: <5.2.1.1.0.20030515204910.01f12578@www.thinkware.se> Message-ID: <5.2.1.1.0.20030516004941.01ecbdc0@www.thinkware.se> I'm afraid Chris and I might end up in kill files if we prolong this more... ;) At 15:26 2003-05-15 -0700, Chris Cogdon wrote: >Actually, there IS magic in floats. You have rules for how exponents are >represented, whether or not a leading 0 is required before the decimal >point, whether or not a leading + is allowed, or only -, and so on. Can you give a real world example of this in existing databases? The SQL standard allows leading + or decimal point. At least Date/Darwen use both in ISBN 0201964260 p 84 when they describe numeric literals. >Again, the rules for a DMBS may not be the same as the rules for Python. That's not my point. The application coder must know the SQL dialect he works with. If the format codes work like a python coder would expect, with the only exception that it allways escape and quote strings, he can predict what will work for him on the given platform, and what won't. >Often, yes, they're similar enough that we can just use 'str(your_float)' >and pass the output straight into the SQL query. My argument is that the >only person qualified to make that statement is the interface writer. The >application writer should not need to know. If I am to write an application for a certain RDBMS with the DB-API I have to know how this deviates from the SQL standard on issues like string concatenation, join syntax, what aggregate functions it has, what datatypes I can use, to what extent it supports things like sub-selects or updatable views etc, but I shouldn't be burdened with knowing how to write floats so that they are understood by the database? I don't buy that. >[1]: Taking a OO-related tangent here, even in the case of a subclassed >'long' for example, the 'OO rules' state that derivitave classes must have >all the properities of the base class, but are allowed to restrict the >values at the base class level. For example, to take a stupid example, I >may create a 'odd_long' class that is a derivitave of the 'long' class >were all values must be odd. It still remains true that a 'odd_long' IS a >long for all possible values of 'odd_long', and also always has a valid >representation in 'long's space. Python doesn't enforce that. AFAIK is no real consensus among the gurus on whether subclassing means restriction or extension or both. Some programming languages lets classes declare some methods as "final" to prevent subclasses to override them, but Python doesn't. But I think we're off topic now. >But, again, this is pushing the responsibility of 'object to string' >conversion onto the application writer, which he's not supposed to worry >about. That's what the interface is for. I can see your point in theory, but do you really claim that this is a problem in the real world? I've worked extensively with at least seven different RDBMS brands in different versions in various project since 1990. I've never ever run in to a problem in passing floats or other numbers to any database. Not that I can remember at least. I never heard of anyone else who had problems with that either. I do remember a lot of other problems with databases though... Strings and nulls, and possibly dates. This is where the problems with datatypes lie in SQL, but I never heard of anyone failing with numbers. The float problem I've seen is "WHERE X = ?" with floats that suddenly didn't work after a conversion from VAX/VMS to PA-RISC/HP-UX... The old and new values didn't have the same representation... But that is another issue. >Most databases have a debugging mode that output the SQL statements that >are being issued to the backend. > >I agree that it would be neat to have a 'hook' in the DB interface to do >this, in the cases where this is not supplied, or you want to keep it all >in python anyway. I think it would be nice to see what the SQL would look like without actually trying to run it on the backend. There are situations where you definitely want to inspect your SQL before you run it, even if there is something called rollback... >>How on earth would the interface be able to know whether we >>actually want to store 0.1 or 0.10000000000000001 when a >>Python float can't distinguish them? > >Because, the interface writer is expected to know the capabilities of both >the backend, AND of Python. That's why it's called an 'interface' :) I don't think you understood me. If we store 0.1 or 0.10000000000000001 in a python float, they will be the same from Python's point of view. But it's possible that the backend could distinguish between them. The application programmer can obviously know from context that in this case it's really 0.1, and in that case it's really 0.10000000000000001, but there is no way the interface will take these contexts into consideration. It's no big deal really, you can just change cur.execute(sql, params) to cur.execute(sql % params), but then we are on our own regarding escaping/quoting. It seems confusing and silly to be forced to alternate between these modes if we are only passing in parameters in both cases. >The 'correct' solution in this case is to have a 'fixed_point' type with >the capabilities you require. Which we might have in Python 2.4... A little too far off in time for me... :) >In that case, it's perfectly possible to form the SQL query completely >yourself, and then pass it to execute without using the % syntax or parameters. As I said, I want the escape/quote done for me. :) >>If you have a decimal(10,2) column X with 1.15 stored, and >>do "...WHERE X=%s", (1.15)) you will get >>...WHERE X=1.1499999999999999 today. Are you sure that will >>match? If you did "...WHERE X=%.2f", (1.15)) and my patch >>was applied, you'd get ...WHERE X=1.15. That would certainly >>match. I imagine that if it is decimal(10,2) as I said, the >>float passed in will be coersed into a decimal(10,2) and it >>will work anyway, but I think I can show other cases where >>this *is* a problem if I have to. Probably rare, but why >>remove the simple tool to fix it when it happens? > >Okay, that's a good example. The difference here is that the interface >does not necessarily have access to the data types IN THE DATABASE which >would be required to make the above example work as expected. I'd still >like to see the other issues addressed, though (including the inconsistent >use of %r). Yes. The interface knows neither database types or application context, and it couldn't. It simply draws a conclusion on the combination of python type and backend engine, and I don't think that's always enough. >Perhaps a 'more correct' solution is to extend, in a limited fashion, >which % escapes are allowed and which will return 'behaviour undefined'. >(I think %r falls squarely in the behaviour undefined category) I think it's up to the application programmer to know what the result of a format code on a particular object will be, whether he talks to a user, a text file for further data preocessing or a database via SQL. The application programmer is after all supposed to know both how the objects he work with behave, and what SQL expects. I'd like the default behaviour to be that format codes are treated as with the %-operator, and that the exceptions are documented. That would include escape/quote for string, proper conversion for Date/Timestamp/Interval etc, and maybe for Money classes etc. I guess Unicode needs special treatment as well. But why not allow %r to work as usual? For me, one of the best things with python is that it's fairly orthogonal and transparent, and while it does make it easy to do the right thing and to understand what is happening, it rarely stops me from doing things that could be potentially harmful. I am allowed to take responsibility for my code. >>>Most interfaces run through a quoting function before being applied to >>>the string containing the '%s's >> >>Only for strings, > >This is not true, and I re-cite pyPgSQL. What other datatypes than strings are quoted in pyPgSQL? >This isn't totally correct, either. Python's repr() will turn 'high bit >escapes' into \xnn with the newer releases of python, and this breaks the >SQL, which only allowed octal escapes. That broke ReportLab, I know. I was obviously not very clear. I tried to say that I see the database interfaces fiddling with "'%s'" % value.replace("'", "''") as something similar to what repr() does to strings in a python context. I.e. it's not about changing the content of the string, but rather about presenting in particular way in a context, so that it will be able to be understood by the receiver. In other words, I feel that with the change in _quote() that I suggested pysqlite would feel much more intuitive for people who are used to the python %-operator. If you have three variables i = 5, f = 1.2 and s = "Hello ma'm", you can just do cur.execute('INSERT INTO X VALUES (%i, %f, %s)', (i, f, s)) It's not so difficult to grasp the idea that the interface will quote and escape strings for us, so we won't have to worry about that. After all, it *is* the text Hello ma'm that we want to store in the database, not 'Hello ma''m' The escaped and quoted thing is just padding during transport. Then we need to understand that as a consequence of this, doing things like %.4s won't work well due to our "padding". But when it comes to things like perhaps using %f for a field that might be an integer or None, or using %f for a very small value, we are on very familiar territory. It's just the same problems with print fmt % values. All the things we know from the %-operator will be useful here. >It depends on what is meant by 'python extended format codes'. Does it >mean the whole gamut of %s %r %f, and so on, or does it JUST mean the >capability to use the (keyname) extension to use by-name rather than >by-position parameter passing. My initial interpretation, and that >supported both by the pypg and pyPgSQL interfaces (not much experience >with MySQL, sorry) is that the latter interpretation has been made. "codes" is plural. But I realize that there have been different interpretations of this. That's what got me started. I've heard people saying "only use %s" for years, and never really thought so much about what the spec said until M-A pointed out that all codes should be working. >And, perhaps they shouldn't, as repr() does not always return a value that >is valid SQL syntax. But I don't think it's possible to create a much better generic solution. I guess that sometimes the application coder has to convert his objects beforehand. pysqlite uses "hasattr(value, '_quote')", but I don't think there is any spec support for that. >They're not blocking %f just as they're not blocking %.2s. My argument is >that the python % operator is being used because it's convenient, but >returns inconsistent results of you step outside the spec's predication >(my interpretation of it) to only use %s and %(keyname)s. They *are* blocking the *use* of %f by running all numbers though repr() for no good reason. ;) >>I also think that several people have misunderstood the >>specification, even if it's explicit about allowing the >>ANSI C format codes if you read it properly. > >The DB-API 2.0 spec makes NO mention of ANSI C format codes ... that's >just an implication :) The current version certainly does: http://www.python.org/peps/pep-0249.html : Title: Python Database API Specification v2.0 ... paramstyle String constant stating the type of parameter marker formatting expected by the interface. Possible values are [2]: 'qmark' Question mark style, e.g. '...WHERE name=?' 'numeric' Numeric, positional style, e.g. '...WHERE name=:1' 'named' Named style, e.g. '...WHERE name=:name' 'format' ANSI C printf format codes, e.g. '...WHERE name=%s' 'pyformat' Python extended format codes, e.g. '...WHERE name=%(name)s' -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The shortest path from thought to working program From chris at cogdon.org Thu May 15 20:13:27 2003 From: chris at cogdon.org (Chris Cogdon) Date: Thu May 15 22:13:32 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <5.2.1.1.0.20030516004941.01ecbdc0@www.thinkware.se> Message-ID: On Thursday, May 15, 2003, at 18:08 US/Pacific, Magnus Lyck? wrote: > I'm afraid Chris and I might end up in kill files if > we prolong this more... ;) Whee! It would be a first! :) > Can you give a real world example of this in existing > databases? The SQL standard allows leading + or decimal > point. At least Date/Darwen use both in ISBN 0201964260 > p 84 when they describe numeric literals. No, I can't. However, I'm not willing to make any kind of assumption here. The interface writer is in the best position to know if the backend requires special treatment. But... that doesn't stop them using the normal python % operator if he knows it'll accept anything that python will spit out, or making the appropriate adjustments if there are some exceptions. >> Often, yes, they're similar enough that we can just use >> 'str(your_float)' and pass the output straight into the SQL query. My >> argument is that the only person qualified to make that statement is >> the interface writer. The application writer should not need to know. > > If I am to write an application for a certain RDBMS with the > DB-API I have to know how this deviates from the SQL standard > on issues like string concatenation, join syntax, what aggregate > functions it has, what datatypes I can use, to what extent it > supports things like sub-selects or updatable views etc, but I > shouldn't be burdened with knowing how to write floats so that > they are understood by the database? I don't buy that. That would imply that ALL SQL coding decisions are best left to the application programmer, including how to quote and embed parameters. I don't think you're arguing that, though, since I think we both think the existing parameter substitution mechanism is just too damn convenient :) The existing system (IMHO) is a compromise between getting the application programmer to do everything, and defining a 'SQL-neutral' language that the interface programmer then has to turn into 'real' SQL for the backend. Ie, the existing interface says "you need to know what kind of SQL will work for your circumstance, but I'll handle the parameter passing for you". I STILL believe the better way to handle something like floats is to create a fixed_point type. However, I'm willing to concede that allowing extended formatting for the particular case of floats, to combat the precision issues of floating point, is a suitable compromise. >> [1]: Taking a OO-related tangent here, even in the case of a >> subclassed 'long' for example, the 'OO rules' state that derivitave >> classes must have all the properities of the base class, but are >> allowed to restrict the values at the base class level. For example, >> to take a stupid example, I may create a 'odd_long' class that is a >> derivitave of the 'long' class were all values must be odd. It still >> remains true that a 'odd_long' IS a long for all possible values of >> 'odd_long', and also always has a valid representation in 'long's >> space. > > Python doesn't enforce that. AFAIK is no real consensus among > the gurus on whether subclassing means restriction or extension > or both. There's a lot of things that python doesn't enforce :) But... you're right, this is off-topic. >> But, again, this is pushing the responsibility of 'object to string' >> conversion onto the application writer, which he's not supposed to >> worry about. That's what the interface is for. > > I can see your point in theory, but do you really claim that > this is a problem in the real world? No, not YET. However, when you're designing architectures, you need to play with 'non real world' examples, or you get bit later on. >> The 'correct' solution in this case is to have a 'fixed_point' type >> with the capabilities you require. > > Which we might have in Python 2.4... A little too far off > in time for me... :) Nothing stopping us writing our own types/classes. Many interfaces do this already for date/time types. > Yes. The interface knows neither database types or application > context, and it couldn't. It simply draws a conclusion on the > combination of python type and backend engine, and I don't think > that's always enough. The interface DOES know about application context, based on what types it's being passed. If you want to do something special, then you create your own type like many DBIs currently do. > But why not allow %r to work as usual? For me, one of the best > things with python is that it's fairly orthogonal and transparent, > and while it does make it easy to do the right thing and to > understand what is happening, it rarely stops me from doing > things that could be potentially harmful. I am allowed to take > responsibility for my code. But, if we're allowing exceptions for %s (ie, it quotes for you based on what the backend requires) then we're never going to achieve orthogonality, nor transparency. My argument is that the parameter passing mechanism was never INTENDED to be orthogonal to the python '%' operator, but it was just merely a convenient, and available mechanism to use. >>>> Most interfaces run through a quoting function before being applied >>>> to the string containing the '%s's >>> >>> Only for strings, >> >> This is not true, and I re-cite pyPgSQL. > > What other datatypes than strings are quoted in pyPgSQL? The _quote function does a lot more than just quote strings. In effect, it's really a 'convert python data type to what SQL expects' function, and it always returns a string, which is why, in pyPgSQL, the following fails: cursor. execute ( "select * from stuff where thingy=%d", int_value ) because 'int_value' is converted to a string (and, just happens to use repr(), but, converting a string does NOT use repr()... not directly, anyway) So... in effect... ALL data-types are _quoted, but not necessarily quoted :) >> This isn't totally correct, either. Python's repr() will turn 'high >> bit escapes' into \xnn with the newer releases of python, and this >> breaks the SQL, which only allowed octal escapes. > > That broke ReportLab, I know. I was obviously not very clear. > I tried to say that I see the database interfaces fiddling with > "'%s'" % value.replace("'", "''") as something similar to what > repr() does to strings in a python context. I.e. it's not about > changing the content of the string, but rather about presenting > in particular way in a context, so that it will be able to be > understood by the receiver. Well, yes, that's my point. :) > In other words, I feel that with the change in _quote() that I > suggested pysqlite would feel much more intuitive for people > who are used to the python %-operator. If you have three > variables i = 5, f = 1.2 and s = "Hello ma'm", you can just do > cur.execute('INSERT INTO X VALUES (%i, %f, %s)', (i, f, s)) The problem is... once you say you can use %i, %f and %s (and I've already agreed that there are good reasons to allow the flexibility in %d), then people are going to assume that you can use %r (which I've already argued doesn't make ANY sense, since repr() does not return values that are acceptable SQL), and %x (many DMBSes will only accept decimal integers). > Then we need to understand that as a consequence of this, doing > things like %.4s won't work well due to our "padding". But when > it comes to things like perhaps using %f for a field that might > be an integer or None, or using %f for a very small value, we > are on very familiar territory. It's just the same problems with > print fmt % values. All the things we know from the %-operator > will be useful here. Well, there's another problem. In python %f will raise a TypeError if you try to pass it None :) That, plus the extra quoting done with %s, are arguments against pretending that the parameter passing mechanism is orthogonal to python. It just isn't. And making it more so will create EXTRA confusion unless we change the documentation to state what is and isn't allowed. As far as DBI-2.0 is concerned, we're a little stuck because of the vagueness. I think it would be unwise to make many of the DBI's out there 'incorrect' by saying "oh, we really meant for you to support %f". For the moment, it might be wise to state that the only formatting code that is predicated by the spec is %s, but some databases will allow you to use %f as you might expect, as well as %i. The interpretation for %r I think is even more varied. Adding in support for %f formatting (specifically to cover the case of fixed-point numbers) I think is best left for 2.1 >> They're not blocking %f just as they're not blocking %.2s. My >> argument is that the python % operator is being used because it's >> convenient, but returns inconsistent results of you step outside the >> spec's predication (my interpretation of it) to only use %s and >> %(keyname)s. > > They *are* blocking the *use* of %f by running all > numbers though repr() for no good reason. ;) IF you take a particular interpretation of the specification, that ALL parameters should use %s, then running it through repr is REQUIRED. (otherwise, %s will give you str() instead of repr(), and that will return non-accurate results in some circumstances, notably floats) >> The DB-API 2.0 spec makes NO mention of ANSI C format codes ... >> that's just an implication :) > > The current version certainly does: My bad. Momentary brain fart. To summarise: I agree that either the use of %f, or additional data types, be mandated to cope with the fixed_point problem. I disagree that we should push for more orthogonality in the parameter passing, since %s does not work the same way as python we will never achieve that orthogonality. I argue that we really need to define the documentation better, specifying a 'must support' behaviour that encompasses the existing interpretations that %s should be able to handle all data types, plus the option to allow other formatting options in a 'known' behaviour. The 2.0 spec be updated to state that the only option that has 'guaranteed' support is %s, and then we work on whether we want to add in things like %f and %i for a later specification of the documentation. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From magnus at thinkware.se Fri May 16 14:46:05 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Fri May 16 07:43:51 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: References: <5.2.1.1.0.20030516004941.01ecbdc0@www.thinkware.se> Message-ID: <5.2.1.1.0.20030516111123.01eb2100@www.thinkware.se> Summary: I think the DB-API implementations should either stick to the SQL standard (? or :name) or stick to the Python standard (all %-codes work). I think it's bad to choose a third path (only %s works). Going from %s only to full %-code support is trivial. I still think the escape/quote thingie for strings should be done. I guess Chris will read the rest as well. :) At 19:13 2003-05-15 -0700, Chris Cogdon wrote: >The interface writer is in the best position to know if the backend >requires special treatment. But... that doesn't stop them using the normal >python % operator if he knows it'll accept anything that python will spit >out, or making the appropriate adjustments if there are some exceptions. Agreed. And I will claim until proven wrong :) that my three line patch makes this possible for sqlite, and that it's more or less as simple for other database drivers. Obviously, a programmer can abuse %f etc in such a way that it won't work, but I think that is the responsibility of the programmer. >>If I am to write an application for a certain RDBMS with the >>DB-API I have to know how this deviates from the SQL standard >>on issues like string concatenation, join syntax, what aggregate >>functions it has, what datatypes I can use, to what extent it >>supports things like sub-selects or updatable views etc, but I >>shouldn't be burdened with knowing how to write floats so that >>they are understood by the database? I don't buy that. > >That would imply that ALL SQL coding decisions are best left to the >application programmer, including how to quote and embed parameters. No, I'm not saying that. I'm saying that the programmer doesn't need to be "protected" from understanding how numbers work in SQL. He doesn't need to be "protected" from knowing how escaping and quoting works either, but having that done right automatically is a convenience. Getting TypeError thrown for no good reason when you use %f in the SQL is *not* a convenince, it's an inconvenience. >The existing system (IMHO) is a compromise between getting the application >programmer to do everything, and defining a 'SQL-neutral' language that >the interface programmer then has to turn into 'real' SQL for the backend. >Ie, the existing interface says "you need to know what kind of SQL will >work for your circumstance, but I'll handle the parameter passing for you". My main problem is that using both the implementation and the syntax of the %-operator does suggest the %d and %f etc will work. I want to reduce the cognitive burden on the programmer and let him enjoy a programming experience where there are as few exceptions from the general rules as possible. If we don't want the application programmer to get involved in type formatting, we should not provide the cognitive cues that this is possible. It would be much better to use ? or :name as per the SQL standard. (There are four permitted binding styles in SQL92, two (module and embedded) use :name syntax, and the other two (direct and CLI) use ?.) I'll get back to this in a separate mail. >I STILL believe the better way to handle something like floats is to >create a fixed_point type. I also want a fixed point type in Python. We agree again! :) >Nothing stopping us writing our own types/classes. Many interfaces do this >already for date/time types. But it's difficult to write database interfaces to support new types or classes written by the application programmer... I think it's a good thing that a new datetime type is added to 2.3, and I hope a good fixed point type will be in 2.4. Until then, this will be a bit shaky. >The interface DOES know about application context, based on what types >it's being passed. If you want to do something special, then you create >your own type like many DBIs currently do. But the application programmer might have different ideas about this than the DBI programmer. We can't expect the DBI programmer to be a mind reader. A way to get around this regardless of paramstyle would be to have some standard hook for custom conversion, like pysqlite has: elif hasattr(value, '_quote'): return value._quote() I guess a better name for a standard interface might be something like 'db_format()' or '__db_str__()'. >But, if we're allowing exceptions for %s (ie, it quotes for you based on >what the backend requires) then we're never going to achieve >orthogonality, nor transparency. My argument is that the parameter passing >mechanism was never INTENDED to be orthogonal to the python '%' operator, >but it was just merely a convenient, and available mechanism to use. The exceptions are not for %s in the SQL, they are for string objects in the parameter. But I agree that format and pyformat was there to be convenient, and I'd really like to see as few supported paramstyles as possible. The best thing would be is all drivers would support one of ? and :name so that we could write at least trivial SQL statements in a portable way. >The problem is... once you say you can use %i, %f and %s (and I've already >agreed that there are good reasons to allow the flexibility in %d), then >people are going to assume that you can use %r (which I've already argued >doesn't make ANY sense, since repr() does not return values that are >acceptable SQL), and %x (many DMBSes will only accept decimal integers). In general I say: Make things consistent and transparent and let the application programmer be in charge. Give him the right tools, try to make life easy for him, but don't try to tie him up in an effort to possibly avoid problems that we don't know exist in the real world. In this particular case I'm starting to lean towards suggesting that format and pyformat should be deprecated, but if the programmer can't understand when %r will give the wrong result in SQL, he won't be able to use %r correctly in a print statement to the display or to a file either. >Well, there's another problem. In python %f will raise a TypeError if you >try to pass it None :) Exatly. Just as with a print statement. You can only use %f if you are certain that you will always feed this with a float. If you can't handle that in the SQL, you can't handle that in "print fmt % params". The programmer must understand this. >As far as DBI-2.0 is concerned, we're a little stuck because of the >vagueness. I think it would be unwise to make many of the DBI's out there >'incorrect' by saying "oh, we really meant for you to support %f". For >the moment, it might be wise to state that the only formatting code that >is predicated by the spec is %s, but some databases will allow you to use >%f as you might expect, as well as %i. The interpretation for %r I think >is even more varied. They *are* incorrect since the DB-API 2.0 *is* stating that they should all be supported. I'm not in favour of changing the DB-API to restrict format and pyformat to %s. I understand that people have interpreted the DB-API differently but it seems to me that the text is actually not ambiguous. >I argue that we really need to define the documentation better, specifying >a 'must support' behaviour that encompasses the existing interpretations >that %s should be able to handle all data types, plus the option to allow >other formatting options in a 'known' behaviour. The 2.0 spec be updated >to state that the only option that has 'guaranteed' support is %s, and >then we work on whether we want to add in things like %f and %i for a >later specification of the documentation. I think it would be better if we could try to move away from format and pyformat, and try to support one of the who binding styles in the SQL standard. -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The shortest path from thought to working program From magnus at thinkware.se Fri May 16 14:57:45 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Fri May 16 07:55:58 2003 Subject: [DB-SIG] qmark to %s converter Message-ID: <5.2.1.1.0.20030516134649.00b4d5f0@www.thinkware.se> I think this code could be used to support qmark format on platforms that support format today. If we want to support *both* %s and ?, we either need to "autodetect" this, or to be able to tell the driver what to use. I haven't done that. Anyway, this is a start, and it works with the SQL statements I tried. Please try to break it. If it doesn't break, please use it! :) It would be great if we could make one paramstyle work with all drivers! #This code converts stuff like "SELECT * FROM ? WHERE X > ?" to #"SELECT * FROM %s WHERE X > %s". It should be fairly simple to #make the same thing for :this, :that. # Magnus Lyck?, 2003-05-16 import re #Cache parsed statements using #"statements.setdefault(sql, parseQmark(sql))" #to save time for repeated calls. statements = {} def parseQmark(sql): '''parseQmark(sql_with_?) -> sql_with_%s Parse SQL written with qmark parameter style (i.e. Direct SQL or SQL CLI style) so that the python %-operator can be used to pass in parameters. This means that % needs to be changed to %%, and ? needs to be changed to %s if it's not inside quotes. ''' sql = sql.replace('%', '%%') #The tricky part is to separate stuff inside quotes from stuff #outside quotes. This is done with a regular expression. We get a #list of tuples with [(not quoted, quoted), (not quoted, quoted)...]. #The present re only works if the string ends with ' though, so #we'll append '' and remove it before we return the final string. quote_separation = re.compile(r"(.*?)('.*?')", re.DOTALL) notQuoted_quoted = quote_separation.findall(sql+"''") replaced = [nq.replace('?', '%s')+q for (nq, q) in notQuoted_quoted] return "".join(replaced)[:-2] # remove extra '' # Borrowed from sqlite and patched to allow %f etc (which isn't really # needed if we use ?, since it'll be %s then anyway... :) def _quote(value): """_quote(value) -> string This function transforms the Python value into a string suitable to send to the SQLite database in a SQL statement. This function is automatically applied to all parameters sent with an execute() call. Because of this a SQL statement string in an execute() call should only use '%s' [or '%(name)s'] for variable substitution without any quoting.""" if value is None: return 'NULL' elif isinstance(value, str): return "'%s'" % value.replace("'", "''") elif type(value) in [long, int, float]: return value elif hasattr(value, '_quote'): return value._quote() # Never mind dates right now... ## elif have_datetime and type(value) in \ ## (DateTime.DateTimeType, DateTime.DateTimeDeltaType): ## return "'%s'" % value else: return repr(value) def execute(sql, params): sql = statements.setdefault(sql, parseQmark(sql)) print sql % tuple(map(_quote, params)) testdata = [ ("select * from x where i = ? and s = ?", (5, "Aren't you done?")), ("""insert into x (i, f, s) values (?, ?, ?)""", (42L, 3.14, 'Hello')), ("""SELECT 'It''s'||col2 FROM ? WHERE ? LIKE '%aren''t they cute?%' AND x = ?""", ('table1', 'col1', 5)), ("""SELECT 'It''s'||col2 FROM ? WHERE ? = 'aren''t they cute?' AND x = '?'""", ('table1', 'col1')) ] for sql, params in testdata: execute(sql, params) print -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The shortest path from thought to working program From magnus at thinkware.se Fri May 16 19:22:44 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Fri May 16 12:20:33 2003 Subject: [DB-SIG] How to know what to do with parameters... Message-ID: <5.2.1.1.0.20030516150838.01f01080@www.thinkware.se> I would like to suggest that the DB-API will stipulate that the DBinterfaces should check whether an object has a method of a certain, standardized name, and use that method to extract the stringified value to put in the SQL statement, it the object has such a method. Pysqlite has such a hook, but "_quote" might not be the best name. Maybe "_dbapi_string"? Like this: if hasattr(value, '_dbapi_string'): return value._dbapi_string() One of Tim Peters Python Zen tenets is: "In the face of ambiguity, refuse the temptation to guess." I think this is something to consider. So, how can a DBI programmer know how to convert data? Well, if he knows the type or class involved, he can know of course, and should normally handle that, but as Chris said it might not be a good thing to fall back on using repr if we don't recognize the object. If the interface can't positively indentify the type or class as something it knows for sure how to handle, it should require explicit instructions from the application programmer. Let's imagine I use a Money class, and my DBI has no support for that. I think I should be able to tell the DBI how to present my money values to SQL. I probably want it to be presented as a decimal number with two decimals, but I'm not sure that I want __str__ or __repr__ to display like that. They should probably both show the currency. Using C/python format codes is a way of doing that. It means that the application programmer has to take the responsibility that his object can properly handle the format code he chooses to use in his SQL statement. In this case I could probably implement __float__ and use %f or even %.2f. This approach has two problems though: 1. It doesn't work with ? or :name. 2. The format codes seems to suggest that it's just like the %-operator, and as we've noted it isn't quite so for strings, unicode or None at least. Trying to do things like %.1s will backfire. The approach today will be to cast to a string, and that usually works, but it seems silly to have a numeric value cast that to a string with quotes around and perform a check to see if it contains any ' that need to be escaped, and then send it to the backend which will convert it back to a number. I just tried a little in MySQL, and I imagine they put some effort into getting this right, but I still think it's a bit awkward...and I'm not sure things will always work out the way we plan. SELECT * FROM a_table WHERE 0.1 = 0.10000000000000001 will return rows. SELECT * FROM a_table WHERE 0.1 = '0.10000000000000001' will also return rows, as will SELECT * FROM a_table WHERE '0.1' = 0.10000000000000001 but not SELECT * FROM a_table WHERE '0.1' = '0.10000000000000001' or SELECT * FROM a_table WHERE 0.1 = 0.1000000000000001 which has one zero less between the ones. What if I want to do "WHERE A = B + C" and A = 0.6, B = 0.2 and C = 0.4? :) Perhaps you know that your backend will handle that by viewing the numbers as exact decimals (MySQL fails, just like Python). Passing as float will give you "WHERE 0.59999999999999998 = 0.20000000000000001 + 0.40000000000000002" which will fail, and passing as strings with rounded values will produce quotes, and you get: "WHERE '0.6'='0.2'+'0.4' which will obviously fail. I would feel much happier if I could make the DB-API send "WHERE 0.6 = 0.2 + 0.4" for me. An object with a _dbapi_string method, and an interface that recognized that would fix this for me. -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The shortest path from thought to working program From jacobs at penguin.theopalgroup.com Fri May 16 13:37:47 2003 From: jacobs at penguin.theopalgroup.com (Kevin Jacobs) Date: Fri May 16 12:38:22 2003 Subject: [DB-SIG] How to know what to do with parameters... In-Reply-To: <5.2.1.1.0.20030516150838.01f01080@www.thinkware.se> Message-ID: On Fri, 16 May 2003, Magnus Lyck? wrote: > I would like to suggest that the DB-API will stipulate > that the DBinterfaces should check whether an object has > a method of a certain, standardized name, and use that > method to extract the stringified value to put in the > SQL statement, it the object has such a method. This assumes that objects will only need to know how to encode themselves for a single known backend, or that all possible backends can rely on a single encoding. Both are very bad assumptions. (Not that most of the rest of the recent discussion on how to "fix" DB-API is not equally as flawed). If it were that easy, it would have already been done. However, it is not an intractible problem, though it does _very_ quickly escape the scope of DB-API and deserves a distinct API layer of its own. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From anthony at computronix.com Fri May 16 17:41:58 2003 From: anthony at computronix.com (Anthony Tuininga) Date: Fri May 16 12:41:59 2003 Subject: [DB-SIG] How to know what to do with parameters... In-Reply-To: <5.2.1.1.0.20030516150838.01f01080@www.thinkware.se> References: <5.2.1.1.0.20030516150838.01f01080@www.thinkware.se> Message-ID: <1053103289.15454.25.camel@localhost.localdomain> A very interesting concept. I think it needs to be extended a little, however. :-) It would be nice if any object at all could be bound without requiring constant conversion -- or rather, writing the code in one place and not have to write code like this. cursor.execute(statement, [ ConvertArg(instanceOfMoney) ] ) Perhaps the following would work instead? The object must define the following functions: _dbapi_type() which will return one of the types defined in the DB API and _dbapi_value() which will return the value in the format required by the DB API to be compatible with the type specified above Then the following would be permitted: cursor.execute(statement, [ instanceOfMoney ] ) Any thoughts on that? The names are always the hardest thing to agree on, so if you have a better name, let me know. On Fri, 2003-05-16 at 10:22, Magnus Lyck? wrote: > I would like to suggest that the DB-API will stipulate > that the DBinterfaces should check whether an object has > a method of a certain, standardized name, and use that > method to extract the stringified value to put in the > SQL statement, it the object has such a method. Pysqlite > has such a hook, but "_quote" might not be the best name. > Maybe "_dbapi_string"? Like this: > > if hasattr(value, '_dbapi_string'): > return value._dbapi_string() > > One of Tim Peters Python Zen tenets is: > "In the face of ambiguity, refuse the temptation to guess." > I think this is something to consider. > > So, how can a DBI programmer know how to convert data? > > Well, if he knows the type or class involved, he can know of > course, and should normally handle that, but as Chris said > it might not be a good thing to fall back on using repr if > we don't recognize the object. > > If the interface can't positively indentify the type or > class as something it knows for sure how to handle, it > should require explicit instructions from the application > programmer. > > Let's imagine I use a Money class, and my DBI has no > support for that. I think I should be able to tell > the DBI how to present my money values to SQL. I > probably want it to be presented as a decimal number > with two decimals, but I'm not sure that I want > __str__ or __repr__ to display like that. They should > probably both show the currency. > > Using C/python format codes is a way of doing that. It > means that the application programmer has to take the > responsibility that his object can properly handle the > format code he chooses to use in his SQL statement. > > In this case I could probably implement __float__ and > use %f or even %.2f. > > This approach has two problems though: > 1. It doesn't work with ? or :name. > 2. The format codes seems to suggest that it's just like > the %-operator, and as we've noted it isn't quite so > for strings, unicode or None at least. Trying to do > things like %.1s will backfire. > > The approach today will be to cast to a string, and that > usually works, but it seems silly to have a numeric value > cast that to a string with quotes around and perform a check > to see if it contains any ' that need to be escaped, and then > send it to the backend which will convert it back to a number. > > I just tried a little in MySQL, and I imagine they put some > effort into getting this right, but I still think it's a > bit awkward...and I'm not sure things will always work out > the way we plan. > > SELECT * FROM a_table WHERE 0.1 = 0.10000000000000001 > will return rows. > SELECT * FROM a_table WHERE 0.1 = '0.10000000000000001' > will also return rows, as will > SELECT * FROM a_table WHERE '0.1' = 0.10000000000000001 > but not > SELECT * FROM a_table WHERE '0.1' = '0.10000000000000001' > or > SELECT * FROM a_table WHERE 0.1 = 0.1000000000000001 > which has one zero less between the ones. > > What if I want to do "WHERE A = B + C" and A = 0.6, B = 0.2 > and C = 0.4? :) Perhaps you know that your backend will handle > that by viewing the numbers as exact decimals (MySQL fails, > just like Python). > > Passing as float will give you > "WHERE 0.59999999999999998 = 0.20000000000000001 + 0.40000000000000002" > which will fail, and passing as strings with rounded values will > produce quotes, and you get: "WHERE '0.6'='0.2'+'0.4' which will > obviously fail. > > I would feel much happier if I could make the DB-API send > "WHERE 0.6 = 0.2 + 0.4" for me. An object with a _dbapi_string > method, and an interface that recognized that would fix this for > me. > > > > -- > Magnus Lycka (It's really Lyckå), magnus@thinkware.se > Thinkware AB, Sweden, www.thinkware.se > I code Python ~ The shortest path from thought to working program > > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig -- Anthony Tuininga anthony@computronix.com Computronix Distinctive Software. Real People. Suite 200, 10216 - 124 Street NW Edmonton, AB, Canada T5N 4A3 Phone: (780) 454-3700 Fax: (780) 454-3838 http://www.computronix.com From anthony at computronix.com Fri May 16 17:46:31 2003 From: anthony at computronix.com (Anthony Tuininga) Date: Fri May 16 12:46:33 2003 Subject: [DB-SIG] How to know what to do with parameters... In-Reply-To: References: Message-ID: <1053103564.15454.31.camel@localhost.localdomain> I don't think Magnus is suggesting that the _dbapi_string() perform the conversion or that the _dbapi_value() method that I suggested would convert the string. That is still the domain of the DB API driver. All that is needed is some way of defining the value that is to be bound to the database driver in a way that is understandable by the driver. So if I have an instance of type Money() _dbapi_type() -> returns NUMBER _dbapi_value() -> returns a Python string that can then be manipulated by the driver as needed Or is there something else that I am missing? On Fri, 2003-05-16 at 10:37, Kevin Jacobs wrote: > On Fri, 16 May 2003, Magnus Lyck? wrote: > > I would like to suggest that the DB-API will stipulate > > that the DBinterfaces should check whether an object has > > a method of a certain, standardized name, and use that > > method to extract the stringified value to put in the > > SQL statement, it the object has such a method. > > This assumes that objects will only need to know how to encode themselves > for a single known backend, or that all possible backends can rely on a > single encoding. Both are very bad assumptions. (Not that most of the rest > of the recent discussion on how to "fix" DB-API is not equally as flawed). > > If it were that easy, it would have already been done. However, it is not > an intractible problem, though it does _very_ quickly escape the scope of > DB-API and deserves a distinct API layer of its own. > > -Kevin -- Anthony Tuininga anthony@computronix.com Computronix Distinctive Software. Real People. Suite 200, 10216 - 124 Street NW Edmonton, AB, Canada T5N 4A3 Phone: (780) 454-3700 Fax: (780) 454-3838 http://www.computronix.com From jacobs at penguin.theopalgroup.com Fri May 16 14:29:34 2003 From: jacobs at penguin.theopalgroup.com (Kevin Jacobs) Date: Fri May 16 13:30:11 2003 Subject: [DB-SIG] How to know what to do with parameters... In-Reply-To: <1053103564.15454.31.camel@localhost.localdomain> Message-ID: On 16 May 2003, Anthony Tuininga wrote: > I don't think Magnus is suggesting that the _dbapi_string() perform the > conversion or that the _dbapi_value() method that I suggested would > convert the string. That is still the domain of the DB API driver. That is certainly how I read Magnus's idea. > All that is needed is some way of defining the value that is to be bound > to the database driver in a way that is understandable by the driver. So > if I have an instance of type Money() > > _dbapi_type() -> returns NUMBER > _dbapi_value() -> returns a Python string that can then be manipulated > by the driver as needed > > Or is there something else that I am missing? The _dbapi_type is fine, although it is of little use by itself. The _dbapi_value is the part that I have deep misgivings about. Are we going to define canonical string representations for every possible database type for all possible backends as part of DB-API? -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From chris at cogdon.org Fri May 16 11:38:10 2003 From: chris at cogdon.org (Chris Cogdon) Date: Fri May 16 13:38:13 2003 Subject: [DB-SIG] How to know what to do with parameters... In-Reply-To: Message-ID: <29C5F878-87C5-11D7-A262-000393A6D442@cogdon.org> On Friday, May 16, 2003, at 10:29 US/Pacific, Kevin Jacobs wrote: > The _dbapi_type is fine, although it is of little use by itself. The > _dbapi_value is the part that I have deep misgivings about. Are we > going to > define canonical string representations for every possible database > type for > all possible backends as part of DB-API? I think this highlights one of the classical interface problems. - If we put the intelligence into the API, then it won't necessarily support new data types created by the application writer. - If we put the intelligence into the data types, then it won't necessarily support differing backends[1] This isn't meant to be arguing one way or the other. It's supposed to be +1 insightful, but -1 helpful :) [1]Yes, yes, I know that 'portability' between different DBMSes and APIs is non-existant except for trivial SQL applications. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From jlburly at yahoo.com Fri May 16 11:42:28 2003 From: jlburly at yahoo.com (jlburly) Date: Fri May 16 13:43:03 2003 Subject: [DB-SIG] pyformat Parameter Style In-Reply-To: <5.2.1.1.0.20030516111123.01eb2100@www.thinkware.se> Message-ID: <20030516174228.90901.qmail@web40808.mail.yahoo.com> --- Magnus Lyckå wrote: [snip] > >I STILL believe the better way to handle something > like floats is to > >create a fixed_point type. > > I also want a fixed point type in Python. We agree > again! :) > > >Nothing stopping us writing our own types/classes. > Many interfaces do this > >already for date/time types. > > But it's difficult to write database interfaces to > support new > types or classes written by the application > programmer... > > I think it's a good thing that a new datetime type > is added to > 2.3, and I hope a good fixed point type will be in > 2.4. Until > then, this will be a bit shaky. Until that time, it seems like Tim Peter's FixedPoint class (http://fixedpoint.sourceforge.net) would be a good candidate for incorporation in the interim. How difficult would this be to implement in the various drivers? Adding this type would allow those end users that work with sql decimal or numeric data types to have stricter enforcement of decimal arithmetric through the use of FixedPoint than they would otherwise by using float. It seems like this would also solve a lot of the issues associated with the mismatch of python float and sql decimal/numeric data types talked about recently on the list. Like mentioned above, the handling of FixedPoint could be similar to how the db api currently specifies the preference for using the mxDateTime package for sql date/time data types. The db api could encourage(?) end users that want to read/write data from/to exact numeric data types, such as sql decimal or numeric, to use FixedPoint, and for those that want to read/write data from/to approximate numeric data types, such as sql real, double precision, or float, to use python float. Jeff Lewis __________________________________ Do you Yahoo!? The New Yahoo! Search - Faster. Easier. Bingo. http://search.yahoo.com From jacobs at penguin.theopalgroup.com Fri May 16 15:22:03 2003 From: jacobs at penguin.theopalgroup.com (Kevin Jacobs) Date: Fri May 16 14:22:37 2003 Subject: [DB-SIG] How to know what to do with parameters... In-Reply-To: <29C5F878-87C5-11D7-A262-000393A6D442@cogdon.org> Message-ID: On Fri, 16 May 2003, Chris Cogdon wrote: > On Friday, May 16, 2003, at 10:29 US/Pacific, Kevin Jacobs wrote: > > The _dbapi_type is fine, although it is of little use by itself. The > > _dbapi_value is the part that I have deep misgivings about. Are we > > going to > > define canonical string representations for every possible database > > type for > > all possible backends as part of DB-API? > > I think this highlights one of the classical interface problems. > > - If we put the intelligence into the API, then it won't necessarily > support new data types created by the application writer. > - If we put the intelligence into the data types, then it won't > necessarily support differing backends[1] The first choice isn't as bad as it sounds, except when one tries to cram it all into DB-API. My own database abstraction interface can be roughly factored into four layers (from highest to lowest degree of abstraction). 1a) High level query, transaction, and error handling interfaces 2) SQL dialect translation, schema reflection, and high level data-type mapping layer 3) Backend abstraction layer (e.g., PostgreSQL, MS-SQL, Sybase, Oracle, SAPDB, etc.) 4) Backend low level drivers (basically DB-API drivers) In addition these are other systems for high level connection description (DSNs), connection management (pooling), query profiling, logging, debugging, and data manipulation. So my major concern is on how to get the DB-API to expose more of the database capabilities, and not to make them do black magic data conversions. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From magnus at thinkware.se Fri May 16 23:24:00 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Fri May 16 16:25:42 2003 Subject: [DB-SIG] How to know what to do with parameters... In-Reply-To: References: <5.2.1.1.0.20030516150838.01f01080@www.thinkware.se> Message-ID: <5.2.1.1.0.20030516192042.01ec7590@www.thinkware.se> At 12:37 2003-05-16 -0400, Kevin Jacobs wrote: >On Fri, 16 May 2003, Magnus Lyck? wrote: > > I would like to suggest that the DB-API will stipulate > > that the DBinterfaces should check whether an object has > > a method of a certain, standardized name, and use that > > method to extract the stringified value to put in the > > SQL statement, it the object has such a method. > >This assumes that objects will only need to know how to encode themselves >for a single known backend, or that all possible backends can rely on a >single encoding. Not really, although that would be good enough in many cases. I'm not claiming that this is a silver bullet that will solve all problems. Most cross platform problems in SQL doesn't have anything with parameter passing to do at all, and I can well imagine that some parameter passing problems are trickier than others... I think that simply being able to write a small method like "return "'%.2f'" % self.amt" would be useful in some instances. Apart from that, there are certainly possibilities beyond what you describe. For instance, the method in the value object might return different strings for different backends. Also, if a particular backend interface knows that certain types of data have to be handled in a particular way on that platform, it can post-process the data it got from the _dbapi_value() method call. I suppose some meta data like proposed by Anthony would be useful there. So it's not as limited as you seem to think. We're only talking about parameter passing here. It's not like we're trying to write SQL 92 code regardless of backend. That is certainly beyond the scope of my suggestion, and I'm curious to see what the Opal Group will offer in this area. Of course, with the applications I worked with that used many different backends in different installations, we didn't use all the bells and whistles of all the involved databases. We did use a least common denominator, but that worked pretty well. After all, most applications only require a fraction of the features in most backends... >If it were that easy, it would have already been done. However, it is not >an intractible problem, though it does _very_ quickly escape the scope of >DB-API and deserves a distinct API layer of its own. But I don't see that distinct API layer materializing. -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The shortest path from thought to working program From philipj at telia.com Sun May 18 00:45:51 2003 From: philipj at telia.com (Philip =?ISO-8859-1?B?SuRnZW5zdGVkdA==?=) Date: Sat May 17 17:43:37 2003 Subject: [DB-SIG] suggested change in the docs. Message-ID: <20030517234551.1d4e0001.philipj@telia.com> Hello. I discovered today that %d substitution did not work for MySQLdb (which has paramstule='format'). It took me quite some time to figure out why, and found this answer on a newgroup discussion: "because there's no %d substitution in mysqldb or any other DB adapters. There's only the %s substitution." If this is true, I suggest that the docs (DatabaseAPI-2.0.html) are updated to specifically mention this under the paramstyle section. This is important, because it's easy to be mislead by "ANSI C printf format codes, e.g. '...WHERE name=%s'" into believed that any of the regular printf substitutions should work. Thank you // Philip J?genstedt From magnus at thinkware.se Sun May 18 23:23:37 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Sun May 18 16:21:24 2003 Subject: [DB-SIG] suggested change in the docs. In-Reply-To: Message-ID: <5.2.1.1.0.20030518203218.01f174b0@www.thinkware.se> At 17 May 2003 23:45:51 +0200, Philip J?genstedt wrote: >I discovered today that %d substitution did not work for MySQLdb (which >has paramstule='format'). It took me quite some time to figure out why, >and found this answer on a newgroup discussion: >"because there's no %d substitution in mysqldb or any other DB adapters. >There's only the %s substitution." This is deeper than you thought... There was a discussion about this on the mailing list this week (last week for for people living in Asia, Australia etc ;). Look in the archives. Here are some excerpts: From: Chris Cogdon >You should ALWAYS use 's', too, regardless of the data type. From: "M.-A. Lemburg" (Who wrote the DB-API if I'm not mistaken) >The DB API says "Python extended format codes" meaning that all >valid Python formatting codes (including their parameters) may >be used. From: Andy Todd >If your db module says it supports 'pyformat' as the paramstyle then any >of these format codes should be supported. (Andy refers to http://www.python.org/doc/current/lib/typesseq-strings.html) From: Jekabs Andrushaitis >Most of DB modules I have used understand only 's' type qualifier in >pyformat argument style (ive tried it in psycopg and one of mysql modules). >I do not really care much about this, but it creates alot of confusion, >since normally in Python % all the qualifiers can be used... Is there more than one driver for MySQL? BTW, I'm not sure Jekabs is right about psycopg, since its author Federico Di Gregorio replied: >if psycopg fails with %d and %f, then it is a bug. please report it. :) and previously he had written: >not to criticize pypgsql, but if its quoting code quotes integers, it is >quite dumb and does a little bit too much work (i can understand why it >does it that way, though.) > >other adapter are a little bit smarter... >If this is true, As you might have figured out by know, there isn't a full concensus on the issue. It seems that the fact that only %s have been used in the examples have misled people. Or maybe the DB-API document was less explicit in an earlier revision? As far as I understand, the paramstyles "format" or "pyformat" are only used for backends that aren't able to pass in parameters on the server side, i.e. PostGreSQL, MySQL and SQLite. Of these, the the MySQLdb, sqlite and pyPgSQL drivers only allow %s (or %(name)s) while pgdb and psycodb follow the DB-API standard, and allow all format codes. I don't know about PoPy. >I suggest that the docs (DatabaseAPI-2.0.html) are >updated to specifically mention this under the paramstyle section. This >is important, because it's easy to be mislead by "ANSI C printf format >codes, e.g. '...WHERE name=%s'" into believed that any of the regular >printf substitutions should work. They should!!! I don't think changing the DB-API spec will do much good. People will still assume that if %s works, so should %f. As Python programmers we take this kind of orthogonality for granted, and it's one of Python's great strengths that we can do that. The fact that Python feels familiar even if we venture into a new field, and don't have a lot of odd execptions, is a big reason why it's so easy to learn. Chris Cogden and I got into a long discussion that probably bored all the others. The argument put forward for allowing only %s was that the application programmer should not be bothered with *how* the values are translated. For the other paramstyles---?, :1 and :name---you let the db interface determine how to pass the values, and there is no reason that it should be different in interfaces using 'format' or 'pyformat'. Other people should talk for themselves. My standpoint is that: * If %s is permitted, so should %d and %f etc. Otherwise, it's very misleading. Regardless of what the DB-API says, the issue will continue to will pop up here, on python-tutor, on comp.lang.python etc. * It's trivial to change the code so that all format codes work. I've presented the modified code for sqlite and pyGreSQL. * If we *don't* want to permit %d and %f etc, we should use ? and :name instead of %s and %(name)s. * I've posted code that transforms ? notation to %s, so supporting paramstyle 'qmark' in the drivers that support 'format' today is no problem. At least noone has pointed out any problems with my code yet. To support paramstyle 'named' instead of 'pyformat' is equally simple. * On changing from format/pyformat to qmark/named, we must either have some kind of autodetect or a mode setting to make the drivers backward compatible, but that's no big deal. It's something we should decide how to do though. * I think it would be great if the paramstyles format and pyformat could eventually be deprecated, and all drivers could implement qmark or named or both. Those are the only two that are in line with the SQL standard. As far as I understand, the following drivers support qmark today: odbc, mxODBC, kInterbase, pyDB2, sapdb. If we could add support for sqlite, PostgreSQL and MySQL to that list, it would be possible to support a lot of backends with the same code, as long as we use simple enough SQL statements... For Oracle, there is DCOracle2 that supports 'numeric', and cx_Oracle that supports 'named'. The driver for Sybase supports Transact-SQL's @-parameter, which is not in line with the DB-API standard at all. I hope I'm correct with these. There are a few remaining drivers (Ingres, Informix etc) that I haven't investigated. -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The shortest path from thought to working program From annapham at omnidox.com Tue May 20 15:29:28 2003 From: annapham at omnidox.com (annapham) Date: Tue May 20 15:29:30 2003 Subject: [DB-SIG] DBI module for Linux Message-ID: Greeting, I am be able to use "import dbi,odbc" in Window but not Linux. Please help. Thank you in advance. Best Regards, Anna M. Pham Software Programmer Omnidox, LC; http//www.omnidox.com "Perfect Documents, Perfect Timing" Phone: 913-239-9700 Email: annapham@omnidox.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/db-sig/attachments/20030520/9dd1fb6a/attachment.htm From chris at cogdon.org Tue May 20 13:41:41 2003 From: chris at cogdon.org (Chris Cogdon) Date: Tue May 20 15:41:45 2003 Subject: [DB-SIG] DBI module for Linux In-Reply-To: Message-ID: <14811D24-8AFB-11D7-AB68-000393A6D442@cogdon.org> On Tuesday, May 20, 2003, at 12:29 US/Pacific, annapham wrote: > Greeting, > > ? > > ???????????I am be able to use ?import dbi,odbc? in Window but not > Linux. Please help. Did you install the modules required (they're not in there by default). What kind of error messages are you receiving? (You should always answer this question straight off the bat, since we're always going to ask for it) -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From mal at lemburg.com Tue May 20 22:51:38 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue May 20 15:52:48 2003 Subject: [DB-SIG] DBI module for Linux In-Reply-To: References: Message-ID: <3ECA874A.1030907@lemburg.com> annapham wrote: > Greeting, > > I am be able to use "import dbi,odbc" in Window but not > Linux. Please help. These modules don't run on Linux. You may want to have a look at mxODBC which provide a much more advanced (and identical) interface on both Windows and Unix. http://www.egenix.com/files/python/mxODBC.html -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, May 20 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 35 days left From pythontutor at venix.com Tue May 20 17:09:40 2003 From: pythontutor at venix.com (Lloyd Kvam) Date: Tue May 20 16:10:12 2003 Subject: [DB-SIG] MySQL error 2013, 'Lost connection to MySQL server during query' Message-ID: <3ECA8B84.4010300@venix.com> OperationalError: (2013, 'Lost connection to MySQL server during query' I am getting occasional Lost Connection errors even when running against localhost. I do not have a reliable way to generate these errors, but they occur too often to ignore. I am starting to wonder if I could have made a mistake in how I use the API. I am using the MySQLdb module to connect to a MySQL database. The code uses the connection object: to get a cursor execute an SQL command save/print/use the result I have not been closing cursors, but simply leaving them to get garbage collected. I have not been reusing cursors for successive queries. Should I be closing cursors? Would I be better off reusing the same cursor where possible? Any advice would be greatly appreciated. -- Lloyd Kvam Venix Corp. 1 Court Street, Suite 378 Lebanon, NH 03766-1358 voice: 603-443-6155 fax: 801-459-9582 From chris at cogdon.org Tue May 20 15:16:57 2003 From: chris at cogdon.org (Chris Cogdon) Date: Tue May 20 17:17:06 2003 Subject: [DB-SIG] MySQL error 2013, 'Lost connection to MySQL server during query' In-Reply-To: <3ECA8B84.4010300@venix.com> Message-ID: <6394F590-8B08-11D7-B891-000393A6D442@cogdon.org> On Tuesday, May 20, 2003, at 13:09 US/Pacific, Lloyd Kvam wrote: > I have not been closing cursors, but simply leaving them to get garbage > collected. I have not been reusing cursors for successive queries. That's probably your problem there. If the DBAPI needs to keep track of which cursors are open, then there's a cycle that can only be broken by manually indicating when you dont need the cursor anymore (via a 'close') On examining the pyPgSQL API, there's no warnings about having to manually close cursors when they're done. The API you're using may have the problem, though, so if you could somehow test the fact that would go a long way to diagnosing the problem. (Perhaps subclassing the cursor object, and override the 'cursor' method of the database object to give you the subclassed object instead, and then print a message when the cursor's closed) -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From vivsar_guve at yahoo.co.uk Wed May 21 02:50:10 2003 From: vivsar_guve at yahoo.co.uk (Vivek Kumar) Date: Wed May 21 04:50:16 2003 Subject: [DB-SIG] kinterbasdb Strange problem.. PLS HELP... Message-ID: <20030521085010.74153.qmail@web11406.mail.yahoo.com> Hi all, I am trying to use Firebird Database Server for creating a Book Store (shopping cart typo).But I am facing a very strange error. The error is like this.. I have created a script called dbInterface.py in which I have a function connect def connect(): try: con=kinterbasdb.connect(dsn="C:/data/SHOP.FDB",user="shop_cart",password="shop_cart") except Exception,err: writeError("Error : %s" %str(err)) #this function writes the error to a log return None else: return con and one more function which creates a new buyer id for the visiting user.createNewBuyer... def createNewBuyer(): con=connect() if con is None: return -1 else: try: cur=con.cursor() cur.execute("""select gen_id(buyer_id_gen,1) from dummy""") # dummy(dummy char(1)) rst=cur.fetchone() if len(rst)==0: con.rollback() closeConnection(con) return 0 else: con.commit() closeConnection(con) return str(rst[0]).strip() except Exception,err: writeError("Error : %s" %str(err)) con.rollback() closeConnection(con) return 0 Now when I try to acces it through a cgi script like: import dbInterface,cgi,cgitb .... .... if buyerID is None: buyerID=dbInterface.createNewBuyer() Then it does nothing and gives an error in the connect function as: 21/5/2003 13:53:32 --> Error : (-904, 'attach_db.isc_attach_database: unavailable database. ') First I thought that there is some error with my code itself. To check this I tried these functions in the Python interprator (Command Prompt). But there it runs successfully, no error... After thinking a lot I was unable to know the cause of the problem. Please help me... The configuration I am using... Windows 2k professional Python 2.2.2 KInterbasDB 3.0.2 FirebirdSQL WI-V6.2.908 Firebird 1.0 Apache 2.0.44 TIA and Kind Regards Vivek Kumar __________________________________ Do you Yahoo!? The New Yahoo! Search - Faster. Easier. Bingo. http://search.yahoo.com From vivsar_guve at yahoo.co.uk Wed May 21 03:13:44 2003 From: vivsar_guve at yahoo.co.uk (Vivek Kumar) Date: Wed May 21 05:13:51 2003 Subject: [DB-SIG] Really Sorry for this :) (was: kinterbasdb Strage problem..) Message-ID: <20030521091344.98851.qmail@web11405.mail.yahoo.com> Hi all, I am really very sorry for my previous mail as before mailing it to the list I had not checked the forum on the sourceforge.net for the module. There is already the answer given in the "Problems with mod_python?" thread. I forgot to give the hostname in the dsn part.. But still I am somewhat confused. Why we need this hostname in dsn when we are using it in a CGI app and why not when we are using some Command Line (ie it works without the Host name)???? Kind Regards Vivek Kumar __________________________________ Do you Yahoo!? The New Yahoo! Search - Faster. Easier. Bingo. http://search.yahoo.com From davidrushby at yahoo.com Wed May 21 03:28:50 2003 From: davidrushby at yahoo.com (David Rushby) Date: Wed May 21 05:28:59 2003 Subject: [DB-SIG] kinterbasdb Strange problem.. PLS HELP... In-Reply-To: <20030521085010.74153.qmail@web11406.mail.yahoo.com> Message-ID: <20030521092850.76500.qmail@web11007.mail.yahoo.com> --- Vivek Kumar wrote: [about a problem connecting to Firebird via kinterbasdb from Apache CGI] > > The configuration I am using... > Windows 2k professional > Python 2.2.2 > KInterbasDB 3.0.2 > FirebirdSQL WI-V6.2.908 Firebird 1.0 > Apache 2.0.44 You're running Apache as a Windows service, correct? Yet you're trying to connect to the following DSN: "C:/data/SHOP.FDB" Firebird will use the LOCAL connection protocol for that connection, as opposed to the REMOTE (network) protocol that would be used if the DSN were: "localhost:C:/data/SHOP.FDB" Firebird's local protocol is implemented in such a way that any program that uses it must have access to "the desktop" (the actual implementation of the local protocal varies between 1.0 and 1.5, but local connections in both versions require access to local Windows resources, not just standard network connectivity). Unless you've specifically configured the Apache service by checking the "Allow service to interact with desktop" checkbox under the "Log On" tab of the Windows service config dialog, LOCAL Firebird connections made from within the Apache service will not be able to attach to the database. I suggest that you use the remote protocol rather than the local. Although you could change the Apache service configuration to allow it to interact with the desktop, using local connections will probably come back to bite you later (any time you write code that will run in the context of a Windows service, the local protocol causes complications), so it's not worth the trouble. ------ I also noticed the following snippet in your code: --- cur.execute("""select gen_id(buyer_id_gen,1) from dummy""") rst=cur.fetchone() if len(rst)==0: ... else: ... --- In that code, rst will always be a tuple containing a single element (the generated id); the first clause of the if statement will never be executed. If there's an error while generating/fetching the id, an exception will be raised. Also, you can use the "magic" table RDB$DATABASE as the target in arbitrary select expressions, rather than establishing a dummy table. Like this: cur.execute("select gen_id(buyer_id_gen,1) from rdb$database") __________________________________ Do you Yahoo!? The New Yahoo! Search - Faster. Easier. Bingo. http://search.yahoo.com From davidrushby at yahoo.com Wed May 21 03:44:11 2003 From: davidrushby at yahoo.com (David Rushby) Date: Wed May 21 05:44:18 2003 Subject: [DB-SIG] kinterbasdb Strange problem.. PLS HELP... In-Reply-To: <20030521085010.74153.qmail@web11406.mail.yahoo.com> Message-ID: <20030521094411.73590.qmail@web11004.mail.yahoo.com> One more note: Firebird 1.0 is much slower at creating connections than are most databases, so if you don't use some kind of persistent connection scheme in your CGIs, you'll probably be dissatisfied with the response time. Firebird 1.5 is able to create connections *much* faster, but it's not particularly stable yet (I wouldn't recommend 1.5 for production use, yet). __________________________________ Do you Yahoo!? The New Yahoo! Search - Faster. Easier. Bingo. http://search.yahoo.com From vivsar_guve at yahoo.co.uk Wed May 21 04:37:32 2003 From: vivsar_guve at yahoo.co.uk (Vivek Kumar) Date: Wed May 21 06:37:39 2003 Subject: [DB-SIG] kinterbasdb Strange problem.. PLS HELP... In-Reply-To: <20030521092850.76500.qmail@web11007.mail.yahoo.com> Message-ID: <20030521103732.88740.qmail@web11406.mail.yahoo.com> --- David Rushby wrote: > --- Vivek Kumar wrote: > [about a problem connecting to Firebird via > kinterbasdb from Apache CGI] > > > > The configuration I am using... > > Windows 2k professional > > Python 2.2.2 > > KInterbasDB 3.0.2 > > FirebirdSQL WI-V6.2.908 Firebird 1.0 > > Apache 2.0.44 .... THX a lot david, Actually my problem was solved when I visited the forum for the module on sourceforge but as I wrote in my next (ie after the one in which I stated the prob) I was confused a bit. But now, after reading ur mail I understand the point behind working of same code in a command line interprator. :-) > I also noticed the following snippet in your code: > --- > cur.execute("""select gen_id(buyer_id_gen,1) from > dummy""") > rst=cur.fetchone() > if len(rst)==0: > ... > else: > ... > --- > In that code, rst will always be a tuple containing > a single element (the > generated id); the first clause of the if statement > will never be executed. > If there's an error while generating/fetching the > id, an exception will be > raised. > > Also, you can use the "magic" table RDB$DATABASE as > the target in arbitrary > select expressions, rather than establishing a dummy > table. Like this: > cur.execute("select gen_id(buyer_id_gen,1) from > rdb$database") THX a lot more for this :). Kind Regards Vivek Kumar __________________________________ Do you Yahoo!? The New Yahoo! Search - Faster. Easier. Bingo. http://search.yahoo.com From chris at cogdon.org Sat May 24 13:55:06 2003 From: chris at cogdon.org (Chris Cogdon) Date: Sat May 24 15:55:10 2003 Subject: [DB-SIG] Python/PostgreSQL API performance comparison Message-ID: <9E2B242C-8E21-11D7-9E89-000393A6D442@cogdon.org> Hey folks! This post is more a 'for your edification' than a call for comments, but comments are welcome :) I'm in the process of rewriting all the database glue logic for my rather heavily used[1] website. In the process I discovered that the new code runs significantly slower than the old, and that worries me since the website IS rather heavily used, and I was hoping to move to dynamic generation for a lot of the pages. I believe a good deal of the slowdown was my overuse of 'elegant but slow' coding, and I intend to remedy that. But, I also decided how much was due to the change from the old 'pg' API to the dbapi-2.0 compliant PgSQL. I wrote a little program that sends through a couple of complex queries to the DB and retrieves the values using a variety of API's: - D'Arcy's 'pg' module - D'Arcy's dbapi-2.0-compliant 'pgdb' module - PgSQL - PgSQL with DECLARE cursor's off - PoPy In all cases, I ran the query three times as a 'warm up', then another 10 and timed that using os.times() The results are as follows: method [ user time, system time, child user time, child system time, real time ] pg [ 0.110, 0.010, 0.000, 0.000, 29.050 ] pgdb [ 4.490, 0.010, 0.000, 0.000, 33.230 ] PgSQL [ 3.640, 0.000, 0.000, 0.000, 40.930 ] PgSQL (nocursor) [ 3.630, 0.010, 0.000, 0.000, 32.280 ] PoPy [ 0.130, 0.010, 0.000, 0.000, 29.030 ] The python program to do this is available at http://onca.catsden.net/~chris/testtimes.py You wont be able to run this without a suitable schema in the DB, of course :) This was run on a very lightly-loaded, dual proc Pentium III/700MHz running redhat-7.2, postgresql 7.2.3. Obviously, this isn't the production system :) Observations: - For all except the C-compiled PoPy module, all the dbapi-2.0 compliant modules add around 3.5-4.5 seconds of CPU overhead. - Using PgSQL with DECLARE cursors turned on (the default) adds 7.5 seconds of real time. (probably due to increased IO and/or lack of IO streaming) - The PoPy module gives you 'pg' performance with 'dbapi-2.0' standard interface. If you have any questions, such as wanting to know details, or suggestions on extra things I can do for the tests, please do write. For myself, I'm probably going to switch to PoPy, or 'dumb down' the interface and just use Pg. 3 seconds of CPU time is not something I can afford. [1] Heavily used = 2 million hits/day, 20,000 visitors/day, 1.5TBytes outbound/month. Yes, I'm somewhat showing off ;) -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From pythontutor at venix.com Sat May 24 17:30:43 2003 From: pythontutor at venix.com (Lloyd Kvam) Date: Sat May 24 16:31:19 2003 Subject: [DB-SIG] MySQL error 2013, 'Lost connection to MySQL server during query' References: <6394F590-8B08-11D7-B891-000393A6D442@cogdon.org> Message-ID: <3ECFD673.6060509@venix.com> I have gone through my code and now carefully close all cursors. The error still recurs sporadically. There's too much code to post, and the code works most of the time. I do not have any repeatable way to reproduce the problem. In searching around the mysql.com web site I encountered a suggestion to set the thread_stack larger. I will try that, but would love to get any other suggestions. Chris Cogdon wrote: > > On Tuesday, May 20, 2003, at 13:09 US/Pacific, Lloyd Kvam wrote: > >> I have not been closing cursors, but simply leaving them to get garbage >> collected. I have not been reusing cursors for successive queries. > > > That's probably your problem there. If the DBAPI needs to keep track of > which cursors are open, then there's a cycle that can only be broken by > manually indicating when you dont need the cursor anymore (via a 'close') >..... -- Lloyd Kvam Venix Corp. 1 Court Street, Suite 378 Lebanon, NH 03766-1358 voice: 603-443-6155 fax: 801-459-9582 From chris at cogdon.org Sun May 25 13:45:04 2003 From: chris at cogdon.org (Chris Cogdon) Date: Sun May 25 15:45:05 2003 Subject: [DB-SIG] Python/PostgreSQL API performance comparison (update) Message-ID: <61479EC4-8EE9-11D7-8238-000393A6D442@cogdon.org> It was suggested to me that PgSQL would perform better if it was configured to return a simple list of tuples rather than a PgResultSet (which has some cool abilities, like being able to do cur.fetchone()['columnname'], for example). So, I've updated my performance comparison code ( http//onca.catsden.net/~chris/testtimes.py ) to include the new case, and the current results are like this: method [ user time, system time, child user time, child system time, real time ] pg [ 0.120, 0.020, 0.000, 0.000, 29.580 ] pgdb [ 4.490, 0.010, 0.000, 0.000, 33.250 ] PgSQL [ 3.650, 0.010, 0.000, 0.000, 41.150 ] PgSQL (nocursor) [ 3.620, 0.010, 0.000, 0.000, 32.320 ] PgSQL (nocursor,list) [ 3.550, 0.010, 0.000, 0.000, 32.360 ] PoPy [ 0.120, 0.000, 0.000, 0.000, 28.990 ] There is a very very small performance gain by using straight lists in pyPgSQL. Something I forgot to mention last time. The user/system times above are for the 'application' process only. Obviously, there is a lot of work being dome by the postmaster which is only factored into the above times as part of the 'real time' it takes to do the full query. The performance loss of using DECLARE cursors (shown in the PgSQL non-nocursor above) is probably extra work being done by the DMBS causing the longer response time. In short, DECLARE cursors probably save memory on the application side at the expense of extra work at the DMBS. If the application and DMBS reside on the same machine (pretty common for web applications), then there's probably no point using DECLARE cursors. Is this kind of thing useful to folk, or would you prefer me shut up about it? :) -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From msanchez at grupoburke.com Sun May 25 23:34:31 2003 From: msanchez at grupoburke.com (=?ISO-8859-1?Q?Marcos_S=E1nchez_Provencio?=) Date: Sun May 25 16:35:06 2003 Subject: [DB-SIG] Python/PostgreSQL API performance comparison (update) In-Reply-To: <61479EC4-8EE9-11D7-8238-000393A6D442@cogdon.org> References: <61479EC4-8EE9-11D7-8238-000393A6D442@cogdon.org> Message-ID: <3ED128D7.8010202@grupoburke.com> Chris Cogdon wrote: > It was suggested to me that PgSQL would perform better if it was > configured to return a simple list of tuples rather than a PgResultSet > (which has some cool abilities, like being able to do > cur.fetchone()['columnname'], for example). So, I've updated my > performance comparison code ( > http//onca.catsden.net/~chris/testtimes.py ) to include the new case, > and the current results are like this: > > method [ user time, system time, child user time, child system time, > real time ] > pg [ 0.120, 0.020, 0.000, 0.000, 29.580 ] > pgdb [ 4.490, 0.010, 0.000, 0.000, 33.250 ] > PgSQL [ 3.650, 0.010, 0.000, 0.000, 41.150 ] > PgSQL (nocursor) [ 3.620, 0.010, 0.000, 0.000, 32.320 ] > PgSQL (nocursor,list) [ 3.550, 0.010, 0.000, 0.000, 32.360 ] > PoPy [ 0.120, 0.000, 0.000, 0.000, 28.990 ] > > There is a very very small performance gain by using straight lists in > pyPgSQL. > > Something I forgot to mention last time. The user/system times above > are for the 'application' process only. Obviously, there is a lot of > work being dome by the postmaster which is only factored into the > above times as part of the 'real time' it takes to do the full query. > > The performance loss of using DECLARE cursors (shown in the PgSQL > non-nocursor above) is probably extra work being done by the DMBS > causing the longer response time. In short, DECLARE cursors probably > save memory on the application side at the expense of extra work at > the DMBS. If the application and DMBS reside on the same machine > (pretty common for web applications), then there's probably no point > using DECLARE cursors. > > Is this kind of thing useful to folk, or would you prefer me shut up > about it? :) > FWIW, I am very interested in these kind of results. One note for multithreaded appserver (webware, zope...). Due to the global lock, the DBMS and the app act funny regarding balance of load. If the app is too heavy, it can suck up one CPU while the other is idle. In this case, it would be worth to pass some load to the DBMS. From chris at cogdon.org Sun May 25 14:48:34 2003 From: chris at cogdon.org (Chris Cogdon) Date: Sun May 25 16:48:34 2003 Subject: [DB-SIG] Python/PostgreSQL API performance comparison (update) In-Reply-To: <3ED128D7.8010202@grupoburke.com> Message-ID: <4099A074-8EF2-11D7-8238-000393A6D442@cogdon.org> On Sunday, May 25, 2003, at 13:34 US/Pacific, Marcos S?nchez Provencio wrote: > One note for multithreaded appserver (webware, zope...). Due to the > global lock, the DBMS and the app act funny regarding balance of load. > If the app is too heavy, it can suck up one CPU while the other is > idle. In this case, it would be worth to pass some load to the DBMS. Unfortunately, in terms of CPU utilisation I don't think it's passing load to the DMBS, but just creating EXTRA load on the DMBS. Compare the 'user' times in both cases. The 'declare cursor' version actually consumes a small amount extra, but a LOT of extra real time. Viz: > PgSQL [ 3.650, 0.010, 0.000, 0.000, 41.150 ] > PgSQL (nocursor) [ 3.620, 0.010, 0.000, 0.000, 32.320 ] However, if memory is an issue, and you're expecting really large datasets, then I can see that being an advantage. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From jeff at jdfiles.org Wed May 28 23:03:46 2003 From: jeff at jdfiles.org (jeff@jdfiles.org) Date: Wed May 28 22:00:57 2003 Subject: [DB-SIG] dbi date Message-ID: <200305282203.46045.jeff@jdfiles.org> I'm working on a project where I extract data from one database and insert it into another. I have created functions to scrub the individual fields in various ways depending on the field type. I pull in the entire result set, then iterate over each row and column in turn I want to call the correct function for each type as it is found. The problem is that dbi.date is not a type, but an object. How can I build a select/case-style block that includes both objects and types ?