From support.dept at share1.net Mon Aug 2 00:56:31 2004 From: support.dept at share1.net (Support Dept) Date: Mon Aug 2 01:57:00 2004 Subject: [DB-SIG] A member is looking into your background. Message-ID: <200408012256.i71MuVbM025958@mail.share1.net> ******YOU HAVE BEEN POSTED ABOUT****** Someone is trying to share Opinions and Experiences about you via our website. The purpose of this email is to inform you that a submission has been made about you at our website. This is email is not commercial in nature. If this email message was delivered to your spam or bulk email folder please notify your ISP or spam filtering company regarding this mistake on their part. To see what members have posted about you at our website use this link: http://6.share4.us/lx.php?a=search&b=5&c=db-sig@python.org Our Identity Protection System is a simple system in which this website sends email messages to the Experience Request author on your behalf, and vice versa. This website will never reveal the identity of the Experience Request author to you, nor will it reveal your identity to the author of the Experience Request. IMPORTANT - If you prefer not to be notified by our website in the future when submissions are mode about you, just add your email address to our Do Not Email List. Our website will never send email to an address that appears on our Do Not Email List. To add to Do Not Email List click here: http://8.share4.info/lx.php?a=donotemail&b=db-sig@python.org Sincerely, Support Department From love at anan.servehalflife.com Mon Aug 2 16:02:12 2004 From: love at anan.servehalflife.com (love@anan.servehalflife.com) Date: Mon Aug 2 16:02:13 2004 Subject: [DB-SIG] =?utf-8?b?wpBswo3DiMKDQcKDaMKDwozCg1jCksKg?= Message-ID: mag20040802213336 *:??,??'?:*:??,? ?????? ?17??*:??,??'?:*:??,? ???????????(???)? ?????????? =========================================================== =====================???==???======================== ================????==???????=================== =========================================================== ???????????????????????? ??????????????????? ??????????????????????????????? ??????????????????????????????? ????????????????????????????????? ???1????????????10?20????????????????????????? ??????????????????????????????? ??????????????????????????? ?????????????????????????????????????? ?????????????????????????????????????? ?????????????????????????????????????? ?????????????????????????????????????? ?????????????????????????????????????????? ???????????????????????????????????? ???? ????????????????????????????????????? ??????????????????????????????????? ?????????????????????????????????? ??-------------------------------------------------???PR???-------?? ?????????????????????????????????????? ???????OK???????????????????????????????????? ????? ???????http://gannba.servepics.com/?g02 ??-------???PR???-------------------------------------------------?? ???????????????????????????? ???????????????????????? ???????http://zxcvb.servepics.com/?g02 ??------------------------------????---------------------------------???? ?????????????????????????????????????? ????????????????????????? ?????????????????????? From David.Carey at nyscot.ang.af.mil Fri Aug 6 15:04:02 2004 From: David.Carey at nyscot.ang.af.mil (Carey David SSgt 109CF/SCBJ) Date: Fri Aug 6 15:03:23 2004 Subject: [DB-SIG] MS SQL Server and odbc getting 'None' Message-ID: <200408061303.i76D3JfS026028@curly.region1.ang.af.mil> Hi, I'm using the regular odbc (not the mxODBC) with microsoft SQL Server 2000. I'm programming on windows for the platform, and switching databases is not an option. I have been using odbc with great success. I have been receiving fine from select queries, and inputting fine with insert queries, etc. I just have one problem. When I pull data from some tables, some fields come back 'None'. Now I know there is data in that field. I also know that it is a nvarchar type which is a unicode type. So some fields I have are "regular" and others are unicode. I have no problem inserting or updating to these fields, just a problem reading them. Does anyone know of a way to remedy this? Also is there a way in Python to covert a variable holding a string to a unicode type? For example: VariableXYZ = 'This is a string' u'VariableXYZ' Now I realize the above example obviously won't work, but I think it illustrates what I would like to do in some instances. Thanks -Dave From David.Carey at nyscot.ang.af.mil Fri Aug 6 15:38:40 2004 From: David.Carey at nyscot.ang.af.mil (Carey David SSgt 109CF/SCBJ) Date: Fri Aug 6 15:38:05 2004 Subject: [DB-SIG] MS SQL Server and odbc getting 'None' Message-ID: <200408061338.i76Dc1fS001276@curly.region1.ang.af.mil> Thanks for the answer on the conversion question. I'm currently using the odbc module from Mark Hammond's Win32 extensions. Thanks Dave -----Original Message----- From: Peter Mott [mailto:peter@monicol.co.uk] Sent: Friday, August 06, 2004 9:29 AM To: Carey David SSgt 109CF/SCBJ Subject: RE: [DB-SIG] MS SQL Server and odbc getting 'None' The unicode thing is I think easy: Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 Type "copyright", "credits" or "license" for more information. s = "abcd" t = unicode(s) t u'abcd Which ODBC module are you using? Regards Peter > -----Original Message----- > From: db-sig-bounces@python.org [mailto:db-sig-bounces@python.org]On > Behalf Of Carey David SSgt 109CF/SCBJ > Sent: 06 August 2004 14:04 > To: 'db-sig@python.org' > Subject: [DB-SIG] MS SQL Server and odbc getting 'None' > > > Hi, > > I'm using the regular odbc (not the mxODBC) with microsoft SQL > Server 2000. I'm programming on windows for the platform, and > switching databases is not an option. > > I have been using odbc with great success. I have been receiving fine > from select queries, and inputting fine with insert queries, etc. > > I just have one problem. When I pull data from some tables, some > fields come back 'None'. > Now I know there is data in that field. I also know that it is a > nvarchar type which is a unicode type. So some fields I have are > "regular" and others are unicode. I have no problem inserting or > updating to these fields, just a problem reading them. Does anyone > know of a way to remedy this? > > Also is there a way in Python to covert a variable holding a string to > a unicode type? > > For example: > > VariableXYZ = 'This is a string' > > u'VariableXYZ' > > > Now I realize the above example obviously won't work, but I think it > illustrates what I would like to do in some instances. > > Thanks > -Dave > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig > From s4046441 at student.uq.edu.au Fri Aug 6 04:42:00 2004 From: s4046441 at student.uq.edu.au (Ms Soo Chong) Date: Fri Aug 6 17:22:01 2004 Subject: [DB-SIG] Questions Message-ID: <3f8d3d3f9405.3f94053f8d3d@uq.edu.au> Dear Sir/Mdm, I'm currently a fourth year University student. I am currently working on a thesis topic : Development of a Database for Experimental Shock Tunnel Data and Images, which requires me to write a python program. As attached is the details of my thesis. I need to make use of python language to communicate/connect with postgreSQL and then construct a web page that display search options and results. Actually, I am very new to python and just finished reading part of 'Learning Python". Therefore, I am quite lost in terms of getting started (writing up a python program). Thus, I went to python website to look for modules/codes that connect python to postgreSQL. Under Database Topic Guide, I found DB-API spec v2.0 and www.amk.ca/python/writing/DB-API.html through some searching. I wonder if these help? Or are there any examples around which I can look through? Or are there any useful and relevant books that you can recommend? Platform: Redhat, Linux Language: Python Database: PostgreSQL Looking forward to your prompt reply. Thank you. Regards, Chong Soo Fern -------------- next part -------------- A non-text attachment was scrubbed... Name: Essence of the proposed solution.doc Type: application/octet-stream Size: 27136 bytes Desc: not available Url : http://mail.python.org/pipermail/db-sig/attachments/20040806/f39701c7/Essenceoftheproposedsolution-0001.obj From andy47 at halfcooked.com Fri Aug 6 17:50:57 2004 From: andy47 at halfcooked.com (Andy Todd) Date: Fri Aug 6 17:51:06 2004 Subject: [DB-SIG] MS SQL Server and odbc getting 'None' In-Reply-To: <200408061303.i76D3JfS026028@curly.region1.ang.af.mil> References: <200408061303.i76D3JfS026028@curly.region1.ang.af.mil> Message-ID: <4113A8E1.2090006@halfcooked.com> Carey David SSgt 109CF/SCBJ wrote: > Hi, > > I'm using the regular odbc (not the mxODBC) with microsoft SQL Server > 2000. I'm > programming on windows for the platform, and switching databases is not an > option. > > I have been using odbc with great success. I have been receiving fine from > select > queries, and inputting fine with insert queries, etc. > > I just have one problem. When I pull data from some tables, some fields come > back 'None'. > Now I know there is data in that field. I also know that it is a nvarchar > type which is a > unicode type. So some fields I have are "regular" and others are unicode. I > have no problem > inserting or updating to these fields, just a problem reading them. Does > anyone know of a > way to remedy this? > > Also is there a way in Python to covert a variable holding a string to a > unicode type? > > For example: > > VariableXYZ = 'This is a string' > > u'VariableXYZ' > > > Now I realize the above example obviously won't work, but I think it > illustrates what I would > like to do in some instances. > > Thanks > -Dave > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig The odbc module that ships with Win32All is fairly lightweight. Mark Hammond usually recommends that you don't use it for complex situations or in production type environments. I would suggest you take a look at mxODBC - http://www.egenix.com/files/python/mxODBC.html Regards, Andy -- -------------------------------------------------------------------------------- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From David.Carey at nyscot.ang.af.mil Fri Aug 6 18:28:50 2004 From: David.Carey at nyscot.ang.af.mil (Carey David SSgt 109CF/SCBJ) Date: Fri Aug 6 18:28:12 2004 Subject: [DB-SIG] MS SQL Server and odbc getting 'None' Message-ID: <200408061628.i76GS8fS024382@curly.region1.ang.af.mil> Thanks for the reply. I was hoping I wouldn't need to change my code Around, so was avoiding mxODBC. But now looking at it, it doesn't look like I would have to change much. I was thinking I wasn't seeing something right, and it was my fault instead of the odbc I was using. Well I'll give it a whirl. Thanks Dave -----Original Message----- From: Andy Todd [mailto:andy47@halfcooked.com] Sent: Friday, August 06, 2004 11:51 AM To: Carey David SSgt 109CF/SCBJ Cc: 'db-sig@python.org' Subject: Re: [DB-SIG] MS SQL Server and odbc getting 'None' Carey David SSgt 109CF/SCBJ wrote: > Hi, > > I'm using the regular odbc (not the mxODBC) with microsoft SQL > Server 2000. I'm programming on windows for the platform, and > switching databases is not an option. > > I have been using odbc with great success. I have been receiving fine > from select queries, and inputting fine with insert queries, etc. > > I just have one problem. When I pull data from some tables, some > fields come back 'None'. > Now I know there is data in that field. I also know that it is a > nvarchar type which is a unicode type. So some fields I have are > "regular" and others are unicode. I have no problem inserting or > updating to these fields, just a problem reading them. Does anyone > know of a way to remedy this? > > Also is there a way in Python to covert a variable holding a string to > a unicode type? > > For example: > > VariableXYZ = 'This is a string' > > u'VariableXYZ' > > > Now I realize the above example obviously won't work, but I think it > illustrates what I would like to do in some instances. > > Thanks > -Dave > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig The odbc module that ships with Win32All is fairly lightweight. Mark Hammond usually recommends that you don't use it for complex situations or in production type environments. I would suggest you take a look at mxODBC - http://www.egenix.com/files/python/mxODBC.html Regards, Andy -- ---------------------------------------------------------------------------- ---- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From tpanjum at hotmail.com Fri Aug 6 19:24:55 2004 From: tpanjum at hotmail.com (Anjum Naseer) Date: Fri Aug 6 19:24:57 2004 Subject: [DB-SIG] Can the cursor.execute method have a timeout specified? Message-ID: Hi, I am running a query which takes a long time and was wondering if there was any way of specifying a timeout? Snippet of code: c = adodbapi.connect("Provider=SQLOLEDB;Initial Catalog=Test;Data Source=(local);Integrated Security=SSPI") csr = c.cursor() csr.arraysize = 1000 csr.execute(someLongLastingQuery) Thanks in advance. _________________________________________________________________ Express yourself with cool new emoticons http://www.msn.co.uk/specials/myemo From misa+db-sig at redhat.com Fri Aug 6 20:04:36 2004 From: misa+db-sig at redhat.com (Mihai Ibanescu) Date: Fri Aug 6 20:04:12 2004 Subject: Fwd: Re: [DB-SIG] Can the cursor.execute method have a timeout specified? Message-ID: <20040806180436.GP23717@abulafia.devel.redhat.com> On Fri, Aug 06, 2004 at 05:24:55PM +0000, Anjum Naseer wrote: > Hi, > > I am running a query which takes a long time and was wondering if there was > any way of specifying a timeout? Although you probably fake a timeout through signal handlers, I doubt you really want to do that. A query that runs for a long time is a bad query; you should look into optimizing the query or the schema or breaking out the query into smaller chunks. Run a couple of these queries and the database backend will not be happy. Much less will your DBA. My opinion at least. Misa From misa at redhat.com Fri Aug 6 19:59:26 2004 From: misa at redhat.com (Mihai Ibanescu) Date: Fri Aug 6 21:31:54 2004 Subject: [DB-SIG] Can the cursor.execute method have a timeout specified? In-Reply-To: References: Message-ID: <20040806175926.GO23717@abulafia.devel.redhat.com> On Fri, Aug 06, 2004 at 05:24:55PM +0000, Anjum Naseer wrote: > Hi, > > I am running a query which takes a long time and was wondering if there was > any way of specifying a timeout? Although you probably fake a timeout through signal handlers, I doubt you really want to do that. A query that runs for a long time is a bad query; you should look into optimizing the query or the schema or breaking out the query into smaller chunks. Run a couple of these queries and the database backend will not be happy. Much less will your DBA. My opinion at least. Misa > c = adodbapi.connect("Provider=SQLOLEDB;Initial Catalog=Test;Data > Source=(local);Integrated Security=SSPI") > csr = c.cursor() > csr.arraysize = 1000 > csr.execute(someLongLastingQuery) > > Thanks in advance. > > _________________________________________________________________ > Express yourself with cool new emoticons http://www.msn.co.uk/specials/myemo > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig From farcepest at gmail.com Fri Aug 6 21:36:02 2004 From: farcepest at gmail.com (Andy Dustman) Date: Fri Aug 6 21:36:11 2004 Subject: [DB-SIG] Can the cursor.execute method have a timeout specified? In-Reply-To: References: Message-ID: <9826f3800408061236627081b9@mail.gmail.com> On Fri, 06 Aug 2004 17:24:55 +0000, Anjum Naseer wrote: > I am running a query which takes a long time and was wondering if there was > any way of specifying a timeout? There's not a standard (DB-API) way to do it. Your best bet is to see if your partiular driver supports a timeout option. This may have to be done at the time you create the connection, i.e. in the connect(). From mal at egenix.com Fri Aug 6 21:44:35 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Aug 6 21:44:35 2004 Subject: [DB-SIG] Can the cursor.execute method have a timeout specified? In-Reply-To: <9826f3800408061236627081b9@mail.gmail.com> References: <9826f3800408061236627081b9@mail.gmail.com> Message-ID: <4113DFA3.9070305@egenix.com> Andy Dustman wrote: > On Fri, 06 Aug 2004 17:24:55 +0000, Anjum Naseer wrote: > >>I am running a query which takes a long time and was wondering if there was >>any way of specifying a timeout? > > > There's not a standard (DB-API) way to do it. Your best bet is to see > if your partiular driver supports a timeout option. This may have to > be done at the time you create the connection, i.e. in the connect(). If you happen to use ODBC it is likely that you can configure a timeout by passing in a special parameter to the ODBC driver at connect time. See the driver documentation for how this is done. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 06 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From darcy at druid.net Mon Aug 9 18:03:28 2004 From: darcy at druid.net (D'Arcy J.M. Cain) Date: Mon Aug 9 18:03:36 2004 Subject: [DB-SIG] Questions In-Reply-To: <3f8d3d3f9405.3f94053f8d3d@uq.edu.au> References: <3f8d3d3f9405.3f94053f8d3d@uq.edu.au> Message-ID: <20040809120328.73d9ebc4.darcy@druid.net> On Fri, 06 Aug 2004 12:42:00 +1000 Ms Soo Chong wrote: > Dear Sir/Mdm, > > I'm currently a fourth year University student. I am currently working > on a thesis topic : Development of a Database for Experimental Shock > Tunnel Data and Images, which requires me to write a python program. > > As attached is the details of my thesis. > > I need to make use of python language to communicate/connect with > postgreSQL and then construct a web page that display search options > and results. Actually, I am very new to python and just finished > reading part of 'Learning Python". Therefore, I am quite lost in terms > of getting started (writing up a python program). Thus, I went to > python website to look for modules/codes that connect python to > postgreSQL. > > Under Database Topic Guide, I found DB-API spec v2.0 and > www.amk.ca/python/writing/DB-API.html through some searching. I wonder > if these help? Or are there any examples around which I can look > through? Or are there any useful and relevant books that you can > recommend? If you would like to see a simple example, check out my motorcycle rides page at http://www.druid.net/rides/ and click on the source button at the bottom to see how it is done. -- D'Arcy J.M. Cain | Democracy is three wolves http://www.druid.net/darcy/ | and a sheep voting on +1 416 425 1212 (DoD#0082) (eNTP) | what's for dinner. From wnvcole at peppermillcas.com Thu Aug 12 17:30:47 2004 From: wnvcole at peppermillcas.com (Vernon Cole) Date: Thu Aug 12 17:30:53 2004 Subject: [DB-SIG] MS SQL Server and odbc getting 'None' Message-ID: <07AF3C77A0FBD311A99F00508B65203903272C8D@SEBASTIAN> mxodbc seems to be a fine product. I have never used it because it is commercial, and my company would not be willing to cough up the bucks when there is a GPL product available. Check out http://sourceforge.net/projects/adodbapi . I use adodbapi for all my database access now. Once you get past the connection strings it can do almost anything including ODBC. (see http://www.connectionstrings.com) For those who don't know, ADO is Microsoft's new data access method. ODBC is Microsoft's former new data access method. (Perhaps ODBC became too popular, so they had to invent something more obscure??) ------------ Vernon -----Original Message----- From: Carey David SSgt 109CF/SCBJ [mailto:David.Carey@nyscot.ang.af.mil] Sent: Friday, August 06, 2004 10:29 AM To: 'Andy Todd' Cc: 'db-sig@python.org' Subject: RE: [DB-SIG] MS SQL Server and odbc getting 'None' Thanks for the reply. I was hoping I wouldn't need to change my code Around, so was avoiding mxODBC. But now looking at it, it doesn't look like I would have to change much. I was thinking I wasn't seeing something right, and it was my fault instead of the odbc I was using. Well I'll give it a whirl. Thanks Dave -----Original Message----- From: Andy Todd [mailto:andy47@halfcooked.com] Sent: Friday, August 06, 2004 11:51 AM To: Carey David SSgt 109CF/SCBJ Cc: 'db-sig@python.org' Subject: Re: [DB-SIG] MS SQL Server and odbc getting 'None' Carey David SSgt 109CF/SCBJ wrote: > Hi, > > I'm using the regular odbc (not the mxODBC) with microsoft SQL > Server 2000. I'm programming on windows for the platform, and > switching databases is not an option. > > I have been using odbc with great success. I have been receiving fine > from select queries, and inputting fine with insert queries, etc. > > I just have one problem. When I pull data from some tables, some > fields come back 'None'. > Now I know there is data in that field. I also know that it is a > nvarchar type which is a unicode type. So some fields I have are > "regular" and others are unicode. I have no problem inserting or > updating to these fields, just a problem reading them. Does anyone > know of a way to remedy this? > > Also is there a way in Python to covert a variable holding a string to > a unicode type? > > For example: > > VariableXYZ = 'This is a string' > > u'VariableXYZ' > > > Now I realize the above example obviously won't work, but I think it > illustrates what I would like to do in some instances. > > Thanks > -Dave > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig The odbc module that ships with Win32All is fairly lightweight. Mark Hammond usually recommends that you don't use it for complex situations or in production type environments. I would suggest you take a look at mxODBC - http://www.egenix.com/files/python/mxODBC.html Regards, Andy -- ---------------------------------------------------------------------------- ---- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ _______________________________________________ DB-SIG maillist - DB-SIG@python.org http://mail.python.org/mailman/listinfo/db-sig From roydobbins at earthlink.net Thu Aug 12 23:40:38 2004 From: roydobbins at earthlink.net (roy) Date: Fri Aug 13 08:28:37 2004 Subject: [DB-SIG] Modifying methods or attributes of persistent objects Message-ID: <200408121440.39013.roydobbins@earthlink.net> Hi: I am attempting to implement a small project using persistent objects (pickle). This works fine for storing a small class hierarchy, however, I am seeing a problem emerging when, as the project evolves, I see the need to say add new methods, or possibly add new attributes. for persistent objects ALREADY stored earlier, these new methods or attributes are not defined. This seems to be a major maintenance issue. What if down the road I have a well establised application, with fairly large database of objects. Now I want to create a new version, which has new methods, etc. I see no easy way to do this, other than to delete the entire database (pickled in this case) then add the whole class hierarchy and however many instances I had stored earlier. Am I missing something? Is there a cleaner, easier way to upgrade an application with enhanced objects? I think that conceptually it doesnt matter what kind of database implementation one has here, whether it is a RDBMS, a simple pickled store or whatever, the fact remains that previously created objects cannot (easily) acquire new attributes and methods. Anyone have ideas on this issue, or can you point me to something in the documentation which discusses this? Many thanks --Roy Dobbins From chris at cogdon.org Fri Aug 13 09:51:27 2004 From: chris at cogdon.org (Chris Cogdon) Date: Fri Aug 13 09:51:35 2004 Subject: [DB-SIG] Modifying methods or attributes of persistent objects In-Reply-To: <200408121440.39013.roydobbins@earthlink.net> References: <200408121440.39013.roydobbins@earthlink.net> Message-ID: <94CB3070-ECFD-11D8-99B2-000A95E3823E@cogdon.org> On Aug 12, 2004, at 14:40, roy wrote: > Hi: > > I am attempting to implement a small project using persistent objects > (pickle). This works fine for storing a small class hierarchy, > however, I am > seeing a problem emerging when, as the project evolves, I see the need > to say > add new methods, or possibly add new attributes. for persistent objects > ALREADY stored earlier, these new methods or attributes are not > defined. > > This seems to be a major maintenance issue. What if down the road I > have a > well establised application, with fairly large database of objects. > Now I > want to create a new version, which has new methods, etc. I see no > easy way > to do this, other than to delete the entire database (pickled in this > case) > then add the whole class hierarchy and however many instances I had > stored > earlier. > > Am I missing something? Is there a cleaner, easier way to upgrade an > application with enhanced objects? I think that conceptually it doesnt > matter > what kind of database implementation one has here, whether it is a > RDBMS, a > simple pickled store or whatever, the fact remains that previously > created > objects cannot (easily) acquire new attributes and methods. > > Anyone have ideas on this issue, or can you point me to something in > the > documentation which discusses this? Actually, I dont think this will be a problem. Methods get attached to the class, and not the objects... as long as the class doesnt move from the module it's defined in, once you unpickle the objects, the new methods will be available for those objects. Of course, you shouldn't try and pickle the class itself :) One thing you DO need to consider is what attributes get attached to the OBJECTS themselves. This can be solved by ensuring that the class has reasonable defaults for all the objects. Here's an example: class C: a = 0 b = 0 def fn(): pass c1 = C() c1.b= 2 In this case, when you pickle c1, what gets stored is 'this data is an instance of class C, and has an attribute 'b' with the value 2'. Since 'a' hasn't changed, it doesn't get pickled, because there's no attribute 'b' IN 'c1' at the moment. Any access to 'c1' is deferred to its class. Now, lets say your program changes, and C now looks like. class C: a = 0 b = 0 c = 1 def fn(): pass def fn2(): pass When you unpickle your 'c1'. And you query either 'a' or 'c', in both cases the result is deferred to C, while accessing 'b' will get the object itself's version. So... it should all be good, as long as you dont move 'C' from one module to another, and you have good defaults set in your class for all the attributes... or at least NEW attributes. If things get really funky, you can always define methods for picking and unpickling objects, but in most cases you wont need to. Hope this helps. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From David.Carey at nyscot.ang.af.mil Fri Aug 13 16:34:45 2004 From: David.Carey at nyscot.ang.af.mil (Carey David SSgt 109CF/SCBJ) Date: Fri Aug 13 16:34:01 2004 Subject: [DB-SIG] MS SQL Server and odbc getting 'None' Message-ID: <200408131433.i7DEXuh6025638@larry.region1.ang.af.mil> Thanks for the suggestion! I will take a look at it. mxODBC worked when pulling the information, but when I tried to update or insert records, it would act like it worked. Then when I went to the database nothing changed. So I'm using odbc to push information and mxODBC to pull it with 2 different connection strings. Maybe this will help me roll it in one. Thanks for the info! Dave -----Original Message----- From: Vernon Cole [mailto:wnvcole@peppermillcas.com] Sent: Thursday, August 12, 2004 11:31 AM To: 'Carey David SSgt 109CF/SCBJ' Cc: 'db-sig@python.org' Subject: RE: [DB-SIG] MS SQL Server and odbc getting 'None' mxodbc seems to be a fine product. I have never used it because it is commercial, and my company would not be willing to cough up the bucks when there is a GPL product available. Check out http://sourceforge.net/projects/adodbapi . I use adodbapi for all my database access now. Once you get past the connection strings it can do almost anything including ODBC. (see http://www.connectionstrings.com) For those who don't know, ADO is Microsoft's new data access method. ODBC is Microsoft's former new data access method. (Perhaps ODBC became too popular, so they had to invent something more obscure??) ------------ Vernon -----Original Message----- From: Carey David SSgt 109CF/SCBJ [mailto:David.Carey@nyscot.ang.af.mil] Sent: Friday, August 06, 2004 10:29 AM To: 'Andy Todd' Cc: 'db-sig@python.org' Subject: RE: [DB-SIG] MS SQL Server and odbc getting 'None' Thanks for the reply. I was hoping I wouldn't need to change my code Around, so was avoiding mxODBC. But now looking at it, it doesn't look like I would have to change much. I was thinking I wasn't seeing something right, and it was my fault instead of the odbc I was using. Well I'll give it a whirl. Thanks Dave -----Original Message----- From: Andy Todd [mailto:andy47@halfcooked.com] Sent: Friday, August 06, 2004 11:51 AM To: Carey David SSgt 109CF/SCBJ Cc: 'db-sig@python.org' Subject: Re: [DB-SIG] MS SQL Server and odbc getting 'None' Carey David SSgt 109CF/SCBJ wrote: > Hi, > > I'm using the regular odbc (not the mxODBC) with microsoft SQL > Server 2000. I'm programming on windows for the platform, and > switching databases is not an option. > > I have been using odbc with great success. I have been receiving fine > from select queries, and inputting fine with insert queries, etc. > > I just have one problem. When I pull data from some tables, some > fields come back 'None'. > Now I know there is data in that field. I also know that it is a > nvarchar type which is a unicode type. So some fields I have are > "regular" and others are unicode. I have no problem inserting or > updating to these fields, just a problem reading them. Does anyone > know of a way to remedy this? > > Also is there a way in Python to covert a variable holding a string to > a unicode type? > > For example: > > VariableXYZ = 'This is a string' > > u'VariableXYZ' > > > Now I realize the above example obviously won't work, but I think it > illustrates what I would like to do in some instances. > > Thanks > -Dave > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig The odbc module that ships with Win32All is fairly lightweight. Mark Hammond usually recommends that you don't use it for complex situations or in production type environments. I would suggest you take a look at mxODBC - http://www.egenix.com/files/python/mxODBC.html Regards, Andy -- ---------------------------------------------------------------------------- ---- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ _______________________________________________ DB-SIG maillist - DB-SIG@python.org http://mail.python.org/mailman/listinfo/db-sig From wnvcole at peppermillcas.com Fri Aug 13 18:01:52 2004 From: wnvcole at peppermillcas.com (Vernon Cole) Date: Fri Aug 13 18:02:35 2004 Subject: [DB-SIG] MS SQL Server and odbc getting 'None' Message-ID: <07AF3C77A0FBD311A99F00508B65203903272C91@SEBASTIAN> Dave: The problem you may be having is that you must "commit" the transaction(s) using the connection object after you perform them using the cursor object. Look just above the "except ValueError" in the enclosed script. I was looking for a snippet to send you, but this program is not much longer and is in production at my site, so I know it is complete and works. The following script is stored on a public share on the SQL server. It allows a user to interactively select a row from a table, check its value, and optionally modify a field. vvvvvvvvvvvvvvvvvv cut here vvvvvvvvvvvvvvvvvvvvvvvvvv """ pbOK800.py -- program to set/clear "OK800" flags in the Casino phonebook """ import sys import adodbapi #use the ADO data access toolkit _OK800 = False _c = None def get_ext(e): global _OK800 # ########################## # SQL command SQLcmd = """ select extension, Casino, Department, PositionLine, OK800 from tbldepartment where extension = '%s' """ _c.execute(SQLcmd % e) rec = _c.fetchone() if not rec: print 'Extension',e, 'not found in PhoneBook' return None else: ext = rec[0] if rec[4].upper() == 'Y': _OK800 = True else: _OK800 = False print 'Extension=', ext, rec[1], rec[2], rec[3] print ' OK800=', _OK800 return ext def bool_input(prompt): while True: s = raw_input(prompt) if len(s) > 0: c = str() c = s[0].upper() if c == 'Y' \ or c == '1' \ or c == 'T' \ or c == 'S': return True elif c=='N' \ or c == '0' \ or c == 'F': return False else: print 'Incorrect input. Try again...' def OK(): global _OK800 global _c _computername="FRANKLIN" #name of computer with Data Warehouse SQL Server _databasename="PhoneBook" #name of database connStrSQLServer = r"Initial Catalog=%s; Data Source=%s; Provider=SQLOLEDB.1; Integrated Security=SSPI" %(_databasename, _computername) # print connStrSQLServer print 'Connecting to:',_computername,_databasename s = adodbapi.connect(connStrSQLServer) #connect to datamine server "s" _c = s.cursor() # create a cursor to send commands to "s" print """ You may enter 1, Y, T or S for YES, enter 0, N or F for NO. """ keep_trying = True while keep_trying: try: ex = raw_input('Examine which Extension number? :') ext = get_ext(ex) if (ext): go = bool_input('Change OK800 for this extension? [Y/N]:') if go: if _OK800: NewValue = 'N' else: NewValue = 'Y' u = "Update tblDepartment "\ "set OK800 = '%s' where extension = '%s'" % \ (NewValue, ext) _c.execute(u) s.commit() check = get_ext(ext) except ValueError: print 'Incorrect input. Try again.' keep_trying = bool_input('Continue? [Y/N]:') _c.close() s.close() if __name__ == '__main__': OK() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ end of clip ^^^^^^^^^^^^^^^^^^^^^^^^ This is called by executing the following which is actually stored as an icon on the users desktop. vvvvvvvvvvvvvvvvvvvvvvvvvvvv begin call script vvvvvvvvvvvvvvvvvvvvvvvv p = "//franklin/PhoneBook/Utilities" import sys if p not in sys.path: sys.path.append(p) from PbOK800 import OK OK() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ end call script ^^^^^^^^^^^^^^^^^^^^^^^^^ Good luck, ---- Vernon -----Original Message----- From: Carey David SSgt 109CF/SCBJ [mailto:David.Carey@nyscot.ang.af.mil] Sent: Friday, August 13, 2004 8:35 AM To: 'Vernon Cole' Cc: 'db-sig@python.org' Subject: RE: [DB-SIG] MS SQL Server and odbc getting 'None' Thanks for the suggestion! I will take a look at it. mxODBC worked when pulling the information, but when I tried to update or insert records, it would act like it worked. Then when I went to the database nothing changed. So I'm using odbc to push information and mxODBC to pull it with 2 different connection strings. Maybe this will help me roll it in one. Thanks for the info! Dave From mal at egenix.com Sat Aug 14 12:53:47 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Sat Aug 14 14:53:51 2004 Subject: [DB-SIG] MS SQL Server and odbc getting 'None' In-Reply-To: <200408131433.i7DEXuh6025638@larry.region1.ang.af.mil> References: <200408131433.i7DEXuh6025638@larry.region1.ang.af.mil> Message-ID: <411DEF3B.4030609@egenix.com> Carey David SSgt 109CF/SCBJ wrote: > Thanks for the suggestion! I will take a look at it. mxODBC worked when > pulling the information, but when I tried to update or insert records, it > would act like it worked. Then when I went to the database nothing changed. You most probably forgot to commit your changes (call .commit() on the connection object). Unlike most ODBC command line tools, mxODBC does not default to auto-commit mode, since using transactions is always better than making permanent changes to the database right away. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From s4046441 at student.uq.edu.au Fri Aug 13 17:48:54 2004 From: s4046441 at student.uq.edu.au (Ms Soo Chong) Date: Mon Aug 16 12:43:53 2004 Subject: [DB-SIG] Questions on PostgreSQL module Message-ID: Hi all, Thanks Alan Gauld and Lee Harr for the prompt reply for my previous questions. As you guys know, I'm just got to know python and is trying hard to learn and apply it now. I have some questions on the PostgreSQL module. What's the difference between pygreSQL : http://www.pygresql.org/README.txt and DB-API 2.0 : http://www.python.org/peps/pep-0249.html? I noticed that one of them uses cursor objects and the other don't. Can someone tell me the differences and other than the two websites I mentioned earlier, are there any good and simple examples (showing the connection between python, postgreSQL and CGI) around? The ones that I'm reading it now are pretty complicated for me now. Thank you. Looking forward to hear from your guys soon... cheers SF From David.Carey at nyscot.ang.af.mil Fri Aug 13 18:41:20 2004 From: David.Carey at nyscot.ang.af.mil (Carey David SSgt 109CF/SCBJ) Date: Mon Aug 16 12:43:53 2004 Subject: [DB-SIG] MS SQL Server and odbc getting 'None' Message-ID: <200408131640.i7DGeUh6019086@larry.region1.ang.af.mil> Ok, I believe I see. If I'm doing multiple updates/inserts inside a loop, is it better to commit inside the loop with each execute, or outside the loop and commit just once? I would assume that committing outside the loop would be the best. I hope I made that clear. Thanks Dave -----Original Message----- From: Vernon Cole [mailto:wnvcole@peppermillcas.com] Sent: Friday, August 13, 2004 12:02 PM To: 'Carey David SSgt 109CF/SCBJ' Cc: 'db-sig@python.org' Subject: RE: [DB-SIG] MS SQL Server and odbc getting 'None' Dave: The problem you may be having is that you must "commit" the transaction(s) using the connection object after you perform them using the cursor object. Look just above the "except ValueError" in the enclosed script. I was looking for a snippet to send you, but this program is not much longer and is in production at my site, so I know it is complete and works. The following script is stored on a public share on the SQL server. It allows a user to interactively select a row from a table, check its value, and optionally modify a field. vvvvvvvvvvvvvvvvvv cut here vvvvvvvvvvvvvvvvvvvvvvvvvv """ pbOK800.py -- program to set/clear "OK800" flags in the Casino phonebook """ import sys import adodbapi #use the ADO data access toolkit _OK800 = False _c = None def get_ext(e): global _OK800 # ########################## # SQL command SQLcmd = """ select extension, Casino, Department, PositionLine, OK800 from tbldepartment where extension = '%s' """ _c.execute(SQLcmd % e) rec = _c.fetchone() if not rec: print 'Extension',e, 'not found in PhoneBook' return None else: ext = rec[0] if rec[4].upper() == 'Y': _OK800 = True else: _OK800 = False print 'Extension=', ext, rec[1], rec[2], rec[3] print ' OK800=', _OK800 return ext def bool_input(prompt): while True: s = raw_input(prompt) if len(s) > 0: c = str() c = s[0].upper() if c == 'Y' \ or c == '1' \ or c == 'T' \ or c == 'S': return True elif c=='N' \ or c == '0' \ or c == 'F': return False else: print 'Incorrect input. Try again...' def OK(): global _OK800 global _c _computername="FRANKLIN" #name of computer with Data Warehouse SQL Server _databasename="PhoneBook" #name of database connStrSQLServer = r"Initial Catalog=%s; Data Source=%s; Provider=SQLOLEDB.1; Integrated Security=SSPI" %(_databasename, _computername) # print connStrSQLServer print 'Connecting to:',_computername,_databasename s = adodbapi.connect(connStrSQLServer) #connect to datamine server "s" _c = s.cursor() # create a cursor to send commands to "s" print """ You may enter 1, Y, T or S for YES, enter 0, N or F for NO. """ keep_trying = True while keep_trying: try: ex = raw_input('Examine which Extension number? :') ext = get_ext(ex) if (ext): go = bool_input('Change OK800 for this extension? [Y/N]:') if go: if _OK800: NewValue = 'N' else: NewValue = 'Y' u = "Update tblDepartment "\ "set OK800 = '%s' where extension = '%s'" % \ (NewValue, ext) _c.execute(u) s.commit() check = get_ext(ext) except ValueError: print 'Incorrect input. Try again.' keep_trying = bool_input('Continue? [Y/N]:') _c.close() s.close() if __name__ == '__main__': OK() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ end of clip ^^^^^^^^^^^^^^^^^^^^^^^^ This is called by executing the following which is actually stored as an icon on the users desktop. vvvvvvvvvvvvvvvvvvvvvvvvvvvv begin call script vvvvvvvvvvvvvvvvvvvvvvvv p = "//franklin/PhoneBook/Utilities" import sys if p not in sys.path: sys.path.append(p) from PbOK800 import OK OK() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ end call script ^^^^^^^^^^^^^^^^^^^^^^^^^ Good luck, ---- Vernon -----Original Message----- From: Carey David SSgt 109CF/SCBJ [mailto:David.Carey@nyscot.ang.af.mil] Sent: Friday, August 13, 2004 8:35 AM To: 'Vernon Cole' Cc: 'db-sig@python.org' Subject: RE: [DB-SIG] MS SQL Server and odbc getting 'None' Thanks for the suggestion! I will take a look at it. mxODBC worked when pulling the information, but when I tried to update or insert records, it would act like it worked. Then when I went to the database nothing changed. So I'm using odbc to push information and mxODBC to pull it with 2 different connection strings. Maybe this will help me roll it in one. Thanks for the info! Dave From andy47 at halfcooked.com Tue Aug 17 18:01:21 2004 From: andy47 at halfcooked.com (Andy Todd) Date: Tue Aug 17 18:01:29 2004 Subject: [DB-SIG] MS SQL Server and odbc getting 'None' In-Reply-To: <200408131640.i7DGeUh6019086@larry.region1.ang.af.mil> References: <200408131640.i7DGeUh6019086@larry.region1.ang.af.mil> Message-ID: <41222BD1.4070607@halfcooked.com> Carey David SSgt 109CF/SCBJ wrote: > Ok, I believe I see. If I'm doing multiple updates/inserts inside a loop, > is it better to commit inside the loop with each execute, or outside the > loop and commit just once? I would assume that committing outside the loop > would be the best. > > I hope I made that clear. > > Thanks > Dave > [snip] The answer is, of course, it depends. If each of your updates/inserts is independent then commit within the loop. If they all have to make it to the database or none do commit after the loop. Essentially you commit after each 'atomic' piece of work. People worry about the processing overhead of a commit. It does, after all, add at least a couple of processor cycles. But unless your throughput is of the order of many thousands of records a second you are unlikely to notice the extra effort of a commit after each operation. Modern relational databases are quite good at this kind of thing after all. Regards, Andy -- -------------------------------------------------------------------------------- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From David.Carey at nyscot.ang.af.mil Wed Aug 18 14:19:12 2004 From: David.Carey at nyscot.ang.af.mil (Carey David SSgt 109CF/SCBJ) Date: Wed Aug 18 14:18:24 2004 Subject: [DB-SIG] MS SQL Server and odbc getting 'None' Message-ID: <200408181218.i7ICILh6008279@larry.region1.ang.af.mil> Thanks for the information. I was wondering a little bit about the overhead of the commit like you said, but figured even if it was a lot, it wouldn't be too bad for me because it would only be going for a few minutes at a time. I'd only be doing a few hundred per shot at most. Its also good to hear that it doesn't use all that much overhead. Thanks Dave -----Original Message----- From: Andy Todd [mailto:andy47@halfcooked.com] Sent: Tuesday, August 17, 2004 12:01 PM To: Carey David SSgt 109CF/SCBJ Cc: Python DB-SIG Mailing List Subject: Re: [DB-SIG] MS SQL Server and odbc getting 'None' Carey David SSgt 109CF/SCBJ wrote: > Ok, I believe I see. If I'm doing multiple updates/inserts inside a > loop, is it better to commit inside the loop with each execute, or > outside the loop and commit just once? I would assume that committing > outside the loop would be the best. > > I hope I made that clear. > > Thanks > Dave > [snip] The answer is, of course, it depends. If each of your updates/inserts is independent then commit within the loop. If they all have to make it to the database or none do commit after the loop. Essentially you commit after each 'atomic' piece of work. People worry about the processing overhead of a commit. It does, after all, add at least a couple of processor cycles. But unless your throughput is of the order of many thousands of records a second you are unlikely to notice the extra effort of a commit after each operation. Modern relational databases are quite good at this kind of thing after all. Regards, Andy -- ---------------------------------------------------------------------------- ---- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From randall at tnr.cc Wed Aug 18 22:22:43 2004 From: randall at tnr.cc (Randall Smith) Date: Wed Aug 18 22:22:48 2004 Subject: [DB-SIG] db module wrapper Message-ID: <4123BA93.1040604@tnr.cc> I have created a db module wrapper for hiding the differences between the DB API 2 implementations. I currently testing it on psycopg and cx_Oracle. Some of the things I have done are: * Standardize time objects to use Python datetime.datetime and datetime.date classes for both input and output. The module converts between types transparently. This is working on both psycopg and cx_Oracle. * Standardize the params input to use the ? operator and a list. The module translates input params and query into the one appropriate for the native module. For example with cx_Oracle: "Select foo from table where goo > ?" with params [myparam] converts to "Select foo from table where goo > :var1" with params {'var1':myparam}. This is working with both psycopg and cx_Oracle. * In addition, I also used Kevin Jacob's db_row module to give access by name to the rows returned from a query. Features can be turned off. First of all, am I reinventing the wheel here? Does a wrapper already exist that hides API implementation differences? Second, do you have any suggestions for standardizing? I used the ? operator for params. Is there a better option? How do you feel about using the datetime module? I know it is not available in earlier versions of Python, but I want to stay in the Python standard library if possible. I would love feedback. Randall From ianb at colorstudy.com Thu Aug 19 02:04:20 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Thu Aug 19 02:04:24 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4123BA93.1040604@tnr.cc> References: <4123BA93.1040604@tnr.cc> Message-ID: <4123EE84.1080608@colorstudy.com> Randall Smith wrote: > I have created a db module wrapper for hiding the differences between > the DB API 2 implementations. I currently testing it on psycopg and > cx_Oracle. Some of the things I have done are: > > * Standardize time objects to use Python datetime.datetime and > datetime.date classes for both input and output. The module converts > between types transparently. This is working on both psycopg and > cx_Oracle. That sounds like a good idea. It would be important that you can also make it uniformly use mxDateTime or Zope's DateTime, as there are many legacy applications that expect those. And I think there are some advantages of mxDateTime over datetime, though I don't know what they are. It would be nice to have a more extendable way to decode and encode types. Psycopg has this, but there's no standard. > * Standardize the params input to use the ? operator and a list. The > module translates input params and query into the one appropriate for > the native module. For example with cx_Oracle: "Select foo from table > where goo > ?" with params [myparam] converts to "Select foo from table > where goo > :var1" with params {'var1':myparam}. This is working with > both psycopg and cx_Oracle. Does this work with all the cases? This can be hard, especially the (Postgres) cases: SELECT * FROM questions WHERE answer = 'maybe?' AND user_id = ? SELECT * FROM questions WHERE answer = '''maybe?''' AND user_id = ? SELECT * FROM questions WHERE answer = '\'maybe?\'' AND user_id = ? SELECT * FROM questions WHERE answer = ' maybe ? ' AND user_id = ? SELECT * FROM "questions?" WHERE answer = 'maybe' AND user_id = ? For MySQL support, you'd need to recognize double quotes in addition to single quotes, with the backquote for quoted columns. > * In addition, I also used Kevin Jacob's db_row module to give access by > name to the rows returned from a query. > > Features can be turned off. That's good. db_row is definitely nice if you are dealing directly with results, but if you have some wrapper it seems unnecessary. > First of all, am I reinventing the wheel here? Does a wrapper already > exist that hides API implementation differences? Second, do you have > any suggestions for standardizing? I used the ? operator for params. Is > there a better option? How do you feel about using the datetime > module? I know it is not available in earlier versions of Python, but I > want to stay in the Python standard library if possible. I would love > feedback. Those are all good features, and there's definitely a desire for this out there -- people bring this up both on this list and elsewhere. There's other people that have various solutions, but nothing that seems to be actively released, promoted, and maintained. The other big issue is exceptions, so that you can catch an exception (e.g, ProgrammingError) no matter which database it came from. People have mentioned low-impact ways of doing this, but I don't know the mechanics. It would certainly fit with the rest of what you are doing. Ian From mcolosimo at mitre.org Thu Aug 19 15:22:07 2004 From: mcolosimo at mitre.org (Marc Colosimo) Date: Thu Aug 19 15:22:13 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4123EE84.1080608@colorstudy.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> Message-ID: On Aug 18, 2004, at 8:04 PM, Ian Bicking wrote: > Randall Smith wrote: >> I have created a db module wrapper for hiding the differences between >> the DB API 2 implementations. I currently testing it on psycopg and >> cx_Oracle. Some of the things I have done are: >> * Standardize time objects to use Python datetime.datetime and >> datetime.date classes for both input and output. The module converts >> between types transparently. This is working on both psycopg and >> cx_Oracle. > > That sounds like a good idea. It would be important that you can also > make it uniformly use mxDateTime or Zope's DateTime, as there are many > legacy applications that expect those. And I think there are some > advantages of mxDateTime over datetime, though I don't know what they > are. > > It would be nice to have a more extendable way to decode and encode > types. Psycopg has this, but there's no standard. > >> * Standardize the params input to use the ? operator and a list. The >> module translates input params and query into the one appropriate for >> the native module. For example with cx_Oracle: "Select foo from >> table where goo > ?" with params [myparam] converts to "Select foo >> from table where goo > :var1" with params {'var1':myparam}. This is >> working with both psycopg and cx_Oracle. > > Does this work with all the cases? This can be hard, especially the > (Postgres) cases: > > SELECT * FROM questions WHERE answer = 'maybe?' AND user_id = ? > SELECT * FROM questions WHERE answer = '''maybe?''' AND user_id = ? > SELECT * FROM questions WHERE answer = '\'maybe?\'' AND user_id = ? > SELECT * FROM questions WHERE answer = ' > maybe > ? > ' AND user_id = ? > SELECT * FROM "questions?" WHERE answer = 'maybe' AND user_id = ? > > > For MySQL support, you'd need to recognize double quotes in addition > to single quotes, with the backquote for quoted columns. > I would like to make the suggestion of adding something like quoteparams (the name in pgdb) to the DB API 3 spec. I have a hack which I use this directly to quote params when I write them out to a file to be loaded in later. By exporting this, the above wrapper can call that and, hopefully, get the correct result back no mater what the DB is. Marc From mal at egenix.com Thu Aug 19 15:36:59 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Thu Aug 19 15:37:07 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> Message-ID: <4124ACFB.3020405@egenix.com> Marc Colosimo wrote: > > On Aug 18, 2004, at 8:04 PM, Ian Bicking wrote: > >> Randall Smith wrote: >> >>> I have created a db module wrapper for hiding the differences between >>> the DB API 2 implementations. I currently testing it on psycopg and >>> cx_Oracle. Some of the things I have done are: >>> * Standardize time objects to use Python datetime.datetime and >>> datetime.date classes for both input and output. The module converts >>> between types transparently. This is working on both psycopg and >>> cx_Oracle. >> >> >> That sounds like a good idea. It would be important that you can also >> make it uniformly use mxDateTime or Zope's DateTime, as there are many >> legacy applications that expect those. And I think there are some >> advantages of mxDateTime over datetime, though I don't know what they >> are. >> >> It would be nice to have a more extendable way to decode and encode >> types. Psycopg has this, but there's no standard. >> >>> * Standardize the params input to use the ? operator and a list. The >>> module translates input params and query into the one appropriate for >>> the native module. For example with cx_Oracle: "Select foo from >>> table where goo > ?" with params [myparam] converts to "Select foo >>> from table where goo > :var1" with params {'var1':myparam}. This is >>> working with both psycopg and cx_Oracle. >> >> >> Does this work with all the cases? This can be hard, especially the >> (Postgres) cases: >> >> SELECT * FROM questions WHERE answer = 'maybe?' AND user_id = ? >> SELECT * FROM questions WHERE answer = '''maybe?''' AND user_id = ? >> SELECT * FROM questions WHERE answer = '\'maybe?\'' AND user_id = ? >> SELECT * FROM questions WHERE answer = ' >> maybe >> ? >> ' AND user_id = ? >> SELECT * FROM "questions?" WHERE answer = 'maybe' AND user_id = ? >> >> >> For MySQL support, you'd need to recognize double quotes in addition >> to single quotes, with the backquote for quoted columns. >> > > I would like to make the suggestion of adding something like quoteparams > (the name in pgdb) to the DB API 3 spec. I have a hack which I use this > directly to quote params when I write them out to a file to be loaded in > later. By exporting this, the above wrapper can call that and, > hopefully, get the correct result back no mater what the DB is. How would you standardize this parameter ? There are way too many different quoting rules out these - MySQL is probably the worst example of them all, mostly because they threw in all kinds of quoting and commenting rules extending way beyond the SQL standard. Note that standard SQL quoting only knows about using double single quotes to quote a single quote, e.g. '' comes out as literal ' in string literals. Same for comments: only '--' is accepted as comment. Things get even more complicated when you start to think about different ways to write float literals, date and time. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 19 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mcolosimo at mitre.org Thu Aug 19 16:05:46 2004 From: mcolosimo at mitre.org (Marc Colosimo) Date: Thu Aug 19 16:05:55 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4124ACFB.3020405@egenix.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <4124ACFB.3020405@egenix.com> Message-ID: On Aug 19, 2004, at 9:36 AM, M.-A. Lemburg wrote: > Marc Colosimo wrote: >> On Aug 18, 2004, at 8:04 PM, Ian Bicking wrote: >>> Randall Smith wrote: >>> >>>> I have created a db module wrapper for hiding the differences >>>> between the DB API 2 implementations. I currently testing it on >>>> psycopg and cx_Oracle. Some of the things I have done are: >>>> * Standardize time objects to use Python datetime.datetime and >>>> datetime.date classes for both input and output. The module >>>> converts between types transparently. This is working on both >>>> psycopg and cx_Oracle. >>> >>> >>> That sounds like a good idea. It would be important that you can >>> also make it uniformly use mxDateTime or Zope's DateTime, as there >>> are many legacy applications that expect those. And I think there >>> are some advantages of mxDateTime over datetime, though I don't know >>> what they are. >>> >>> It would be nice to have a more extendable way to decode and encode >>> types. Psycopg has this, but there's no standard. >>> >>>> * Standardize the params input to use the ? operator and a list. >>>> The module translates input params and query into the one >>>> appropriate for the native module. For example with cx_Oracle: >>>> "Select foo from table where goo > ?" with params [myparam] >>>> converts to "Select foo from table where goo > :var1" with params >>>> {'var1':myparam}. This is working with both psycopg and cx_Oracle. >>> >>> For MySQL support, you'd need to recognize double quotes in addition >>> to single quotes, with the backquote for quoted columns. >>> >> I would like to make the suggestion of adding something like >> quoteparams (the name in pgdb) to the DB API 3 spec. I have a hack >> which I use this directly to quote params when I write them out to a >> file to be loaded in later. By exporting this, the above wrapper can >> call that and, hopefully, get the correct result back no mater what >> the DB is. > > How would you standardize this parameter ? > > There are way too many different quoting rules out these - MySQL is > probably the worst example of them all, mostly because they > threw in all kinds of quoting and commenting rules extending > way beyond the SQL standard. > > Note that standard SQL quoting only knows about using double > single quotes to quote a single quote, e.g. '' comes out as > literal ' in string literals. Same for comments: only '--' is > accepted as comment. > > Things get even more complicated when you start to think > about different ways to write float literals, date and > time. > > The adaptor modules will deal with this just as they do now (psycopg, pgdb, and cx_Oracle and others), which is why I said the DB API. The developers for each of these modules already had to deal with this. I don't want to think about the insanity between quoting. And as you just pointed out, this is why I want to export it in the API. The execute query function is already standardize, all I am asking for is a function that just returns the string and doesn't execute it. sql = """Select foo from table where goo > %s and foo = %s""" param1 = 10 param2 = "bar" cursor.execute(sql, (param1, param2)) # quotes and executes sql statement st = cursor.quoteparams(sql, (param1, param2)) # returns the quoted sql statement , goo > 10 and foo = 'bar' cursor.execute(st) # executes already quoted sql statement marc From mal at egenix.com Thu Aug 19 17:57:59 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Thu Aug 19 17:58:02 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <4124ACFB.3020405@egenix.com> Message-ID: <4124CE07.1040606@egenix.com> Marc Colosimo wrote: >>> I would like to make the suggestion of adding something like >>> quoteparams (the name in pgdb) to the DB API 3 spec. I have a hack >>> which I use this directly to quote params when I write them out to a >>> file to be loaded in later. By exporting this, the above wrapper can >>> call that and, hopefully, get the correct result back no mater what >>> the DB is. >> >> >> How would you standardize this parameter ? >> >> There are way too many different quoting rules out these - MySQL is >> probably the worst example of them all, mostly because they >> threw in all kinds of quoting and commenting rules extending >> way beyond the SQL standard. >> >> Note that standard SQL quoting only knows about using double >> single quotes to quote a single quote, e.g. '' comes out as >> literal ' in string literals. Same for comments: only '--' is >> accepted as comment. >> >> Things get even more complicated when you start to think >> about different ways to write float literals, date and >> time. >> >> > > The adaptor modules will deal with this just as they do now (psycopg, > pgdb, and cx_Oracle and others), which is why I said the DB API. The > developers for each of these modules already had to deal with this. I > don't want to think about the insanity between quoting. And as you just > pointed out, this is why I want to export it in the API. The execute > query function is already standardize, all I am asking for is a > function that just returns the string and doesn't execute it. > > sql = """Select foo from table where goo > %s and foo = %s""" > param1 = 10 > param2 = "bar" > cursor.execute(sql, (param1, param2)) # quotes and executes > sql statement > > st = cursor.quoteparams(sql, (param1, param2)) # returns the > quoted sql statement , goo > 10 and foo = 'bar' > cursor.execute(st) # executes already quoted > sql statement I don't understand why you'd want to access the quoted SQL string with embedded parameters. After all binding parameters exist to make all this quoting unnecessary and indeed, many drivers pass the SQL string and the parameters well separated to the database. The database then builds an access path using the SQL string (with binding parameters) and runs the access path against the parameters that are specified as separate part of the request. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 19 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From randall at tnr.cc Thu Aug 19 18:31:36 2004 From: randall at tnr.cc (Randall Smith) Date: Thu Aug 19 18:31:42 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4123BA93.1040604@tnr.cc> References: <4123BA93.1040604@tnr.cc> Message-ID: <4124D5E8.6070802@tnr.cc> One thing I'm surprised nobody mentioned is a uniform login method. Does anyone have a suggestion for this? I thought about using the jdbc standard. Randall Randall Smith wrote: > I have created a db module wrapper for hiding the differences between > the DB API 2 implementations. I currently testing it on psycopg and > cx_Oracle. Some of the things I have done are: > > * Standardize time objects to use Python datetime.datetime and > datetime.date classes for both input and output. The module converts > between types transparently. This is working on both psycopg and > cx_Oracle. > > * Standardize the params input to use the ? operator and a list. The > module translates input params and query into the one appropriate for > the native module. For example with cx_Oracle: "Select foo from table > where goo > ?" with params [myparam] converts to "Select foo from table > where goo > :var1" with params {'var1':myparam}. This is working with > both psycopg and cx_Oracle. > > * In addition, I also used Kevin Jacob's db_row module to give access by > name to the rows returned from a query. > > Features can be turned off. > > First of all, am I reinventing the wheel here? Does a wrapper already > exist that hides API implementation differences? Second, do you have > any suggestions for standardizing? I used the ? operator for params. Is > there a better option? How do you feel about using the datetime > module? I know it is not available in earlier versions of Python, but I > want to stay in the Python standard library if possible. I would love > feedback. > > Randall > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig From chris at cogdon.org Thu Aug 19 18:33:21 2004 From: chris at cogdon.org (Chris Cogdon) Date: Thu Aug 19 18:33:31 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4124CE07.1040606@egenix.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <4124ACFB.3020405@egenix.com> <4124CE07.1040606@egenix.com> Message-ID: <7C097220-F1FD-11D8-82FE-000A95E3823E@cogdon.org> On Aug 19, 2004, at 08:57, M.-A. Lemburg wrote: > I don't understand why you'd want to access the quoted > SQL string with embedded parameters. After all binding > parameters exist to make all this quoting unnecessary and > indeed, many drivers pass the SQL string and the parameters > well separated to the database. The database then builds > an access path using the SQL string (with binding parameters) > and runs the access path against the parameters that are > specified as separate part of the request. I can certainly understand why, because I've asked for this kind of thing in the past. If I was writing a program that can, programatically piece together 'where' criteria based on multiple, highly-varying external sources (such as tens of checkboxes a user on the other side of a web browser might select), I would want to have a structure that could return 'where snippets' (which would be later joined by ANDs, and stuck onto the query). The EASIEST way to do this is to be able to get access to the quoting mechanism (and different databases have different ways of quoting, for example, postgresql uses single quotes, and mysql uses double quotes (unless you put it into compatability mode)) and then pass plain old strings back into the section of code that actually issues the query. One program that does exactly this is ViewCVS. in query.cgi there's a bunch of inputs and checkboxes. Each input, when used, generates a where snippet (i made up that name just now), plus a list of tables that needs to participate in the query. Once each input component is polled, all the where snippets and table requirements are collected, duplicates removed, and the query is pieced together. The only reason it works as simply as it does is because ViewCVS has access to the quoting mechanism. Unfortunately, because there is no 'standard' for getting access to the quoting mechanism, I cannot drop in another database, or even another database connector, unless it happens to have the same, non-standard, function for quoting attributes or I made modifications to the program. One thing I learned when I pushed to have such a function included as part of the spec (even if it was OPTIONAL) is that many databases do NOT quote variable objects, and do NOT pass the entire query as a string to the backend, but instead pass a mixture of a string, and various binary objects, as a 'tuple'. Since the binary objects are passed 'raw', there is no need to conver them to a string only to have them converted back to binary again in the parser. Overall, saving time. (unfortunately, this doesn't take into account that some binary types need to be converted ANYWAY, such as handling endianness) (In fact, the alternative to passing where snippets around is to emulate this behaviour... passing a format string and a list of parameters as a tuples around - which is awfully more complicated) So... to Marc... that's why I think the DB-API 'community' is reluctant to standardise a way of getting access to the quoting mechanism: some databases (eg, oracle) don't actually 'do' quoting. But... to the DB-API community, I still hope that you can see that, at least with databases that *DO* do quoting, that there is a good argument for making access to the quoting mechanism an optional but at least STANDARDISED part of the specification. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From mcolosimo at mitre.org Thu Aug 19 18:48:31 2004 From: mcolosimo at mitre.org (Marc Colosimo) Date: Thu Aug 19 18:48:38 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4124CE07.1040606@egenix.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <4124ACFB.3020405@egenix.com> <4124CE07.1040606@egenix.com> Message-ID: <9A70CBEC-F1FF-11D8-8726-000A95A5D8B2@mitre.org> On Aug 19, 2004, at 11:57 AM, M.-A. Lemburg wrote: > Marc Colosimo wrote: >>>> I would like to make the suggestion of adding something like >>>> quoteparams (the name in pgdb) to the DB API 3 spec. I have a hack >>>> which I use this directly to quote params when I write them out to >>>> a file to be loaded in later. By exporting this, the above wrapper >>>> can call that and, hopefully, get the correct result back no mater >>>> what the DB is. >>> >>> >>> How would you standardize this parameter ? >>> >>> There are way too many different quoting rules out these - MySQL is >>> probably the worst example of them all, mostly because they >>> threw in all kinds of quoting and commenting rules extending >>> way beyond the SQL standard. >>> >>> Note that standard SQL quoting only knows about using double >>> single quotes to quote a single quote, e.g. '' comes out as >>> literal ' in string literals. Same for comments: only '--' is >>> accepted as comment. >>> >>> Things get even more complicated when you start to think >>> about different ways to write float literals, date and >>> time. >>> >>> >> The adaptor modules will deal with this just as they do now (psycopg, >> pgdb, and cx_Oracle and others), which is why I said the DB API. The >> developers for each of these modules already had to deal with this. I >> don't want to think about the insanity between quoting. And as you >> just pointed out, this is why I want to export it in the API. The >> execute query function is already standardize, all I am asking for >> is a function that just returns the string and doesn't execute it. >> sql = """Select foo from table where goo > %s and foo = %s""" >> param1 = 10 >> param2 = "bar" >> cursor.execute(sql, (param1, param2)) # quotes and >> executes sql statement >> st = cursor.quoteparams(sql, (param1, param2)) # returns the >> quoted sql statement , goo > 10 and foo = 'bar' >> cursor.execute(st) # executes already >> quoted sql statement > > I don't understand why you'd want to access the quoted > SQL string with embedded parameters. Gee, the second line of my original post (see above)! > After all binding > parameters exist to make all this quoting unnecessary and > indeed, many drivers pass the SQL string and the parameters > well separated to the database. not pgdb (_quoteparams) or cx_oracle (Cursor_SetBindVariables). I couldn't check psycopg because the links from freshmeat are dead. > The database then builds > an access path using the SQL string (with binding parameters) > and runs the access path against the parameters that are > specified as separate part of the request. > > Marc From randall at tnr.cc Thu Aug 19 19:27:08 2004 From: randall at tnr.cc (Randall Smith) Date: Thu Aug 19 19:27:19 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4123EE84.1080608@colorstudy.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> Message-ID: <4124E2EC.1010001@tnr.cc> Ian Bicking wrote: > > The other big issue is exceptions, so that you can catch an exception > (e.g, ProgrammingError) no matter which database it came from. People > have mentioned low-impact ways of doing this, but I don't know the > mechanics. It would certainly fit with the rest of what you are doing. > > Ian I'm definitely doing this. I first must decide which exception should be raised, then map the native module's exception to the wrapper's exception. Is there some majority opinion on which exceptions should be raised for each occurance? What I'm really doing is defining a new API. I would like it to be in line with the majority's opinion. Randall From dieter at handshake.de Thu Aug 19 19:25:31 2004 From: dieter at handshake.de (Dieter Maurer) Date: Thu Aug 19 20:46:27 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4123BA93.1040604@tnr.cc> References: <4123BA93.1040604@tnr.cc> Message-ID: <16676.57995.851107.259406@gargle.gargle.HOWL> Randall Smith wrote at 2004-8-18 15:22 -0500: > ... >* Standardize the params input to use the ? operator and a list. The >module translates input params and query into the one appropriate for >the native module. For example with cx_Oracle: "Select foo from table >where goo > ?" with params [myparam] converts to "Select foo from table >where goo > :var1" with params {'var1':myparam}. This is working with >both psycopg and cx_Oracle. Why are you replacing a readable and reliable syntax by one that is difficult to read and unreliable? As soon as you have a couple of "?", it becomes quite difficult to grasp... -- Dieter From wnvcole at peppermillcas.com Thu Aug 19 21:19:14 2004 From: wnvcole at peppermillcas.com (Vernon Cole) Date: Thu Aug 19 21:21:12 2004 Subject: [DB-SIG] db module wrapper Message-ID: <1DE30EB3ECE266409FDE6E7032F178C41397F5@pcimail1s.peppermillcas.com> Marc-Andre is right, as usual. The other reason we pass parameters to the SQL engine, rather than just plug them in using a normal python string substitution, is that the SQL server can run a repeated query faster (in some cases MUCH faster) if we pass it parameters. If it were not for this performance benefit, we could use the existing language tools to do the job. The given example can be run in standard python as: sql = """Select foo from table where goo > %d and foo = '%s'""" param1 = 10 param2 = "bar" cursor.execute(sql % (param1, param2)) and the requested feature done like: st = sql % (param1, param2) cursor.execute(st) Okay, everybody knew that already, but I want to be sure we are all working from the same page here... The simple case like this is not the problem. The problem is that we are using a languge designed by IBM as a human interface tool, and trying to make it work as a data retrieval engine. SQL is clearly the wrong tool for the job. Unfortunatly, it is the only tool we have. So lets examine the complex case... Our computer holds lots of structures containing binary data of various kinds. It is held in efficient internal representations. The SQL server contains binary data of similar kinds, but the internal representation may be different. Using the SQL language, in any of its various dialects, demands that the API programmer must find a way to communicate between these two binary representations using text strings. (I almost said ASCII strings, but even THAT is not standard. DB2 uses EBCDIC strings and MsSQL uses unicode!) The application programmer will pass in a control string and a (possibly very complex) structure of binary data. So Marc C. is right, also. We need a way -- if only for debugging and error handling -- to reach in and retrieve the string which the API ultimatly sends to the SQL engine to make the magic happen. (This string should not be fed back into a query -- the SQL will be a different dialect from the API input syntax.) Here is my suggestion for the way to specify that method: cursor.__lastQuery__() and for the raw result data of a query: cursor.__lastResult__() The double-underline syntax is intended to show that the returned result is highly implementation dependent. Everybody keep in mind the goal here. DbAPI 2.0 gave a unified way for a programmer to deduce from the API how he could BUILD a query string for any given engine. 3.0 should make it possible to write a moderatly complex query -- in a standard syntax -- which will EXECUTE correctly on any SQL engine. That's a different kind of animal entirely. ---------- Vernon -----Original Message----- From: M.-A. Lemburg [mailto:mal@egenix.com] Sent: Thursday, August 19, 2004 9:58 AM To: Marc Colosimo Cc: db-sig@python.org; Ian Bicking Subject: Re: [DB-SIG] db module wrapper Marc Colosimo wrote: >>> I would like to make the suggestion of adding something like >>> quoteparams (the name in pgdb) to the DB API 3 spec. I have a hack [[ much text clipped out ]] > function that just returns the string and doesn't execute it. > > sql = """Select foo from table where goo > %s and foo = %s""" > param1 = 10 > param2 = "bar" > cursor.execute(sql, (param1, param2)) # quotes and executes > sql statement > > st = cursor.quoteparams(sql, (param1, param2)) # returns the > quoted sql statement , goo > 10 and foo = 'bar' > cursor.execute(st) # executes already quoted > sql statement I don't understand why you'd want to access the quoted [[more clipping]] indeed, many drivers pass the SQL string and the parameters well separated to the database. The database then builds an access path using the SQL string (with binding parameters) and runs the access path against the parameters that are specified as separate part of the request. -- Marc-Andre Lemburg From mal at egenix.com Thu Aug 19 22:13:29 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Thu Aug 19 22:13:33 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <16676.57995.851107.259406@gargle.gargle.HOWL> References: <4123BA93.1040604@tnr.cc> <16676.57995.851107.259406@gargle.gargle.HOWL> Message-ID: <412509E9.8070700@egenix.com> Dieter Maurer wrote: > Randall Smith wrote at 2004-8-18 15:22 -0500: > >>... >>* Standardize the params input to use the ? operator and a list. The >>module translates input params and query into the one appropriate for >>the native module. For example with cx_Oracle: "Select foo from table >>where goo > ?" with params [myparam] converts to "Select foo from table >>where goo > :var1" with params {'var1':myparam}. This is working with >>both psycopg and cx_Oracle. > > > Why are you replacing a readable and reliable syntax by one > that is difficult to read and unreliable? > > As soon as you have a couple of "?", it becomes quite difficult > to grasp... You get used to it rather quickly ;-) (and it's easy to parse and convert into the other formats). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 19 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From wnvcole at peppermillcas.com Thu Aug 19 22:59:15 2004 From: wnvcole at peppermillcas.com (Vernon Cole) Date: Thu Aug 19 22:59:23 2004 Subject: [DB-SIG] db module wrapper Message-ID: <1DE30EB3ECE266409FDE6E7032F178C41397F6@pcimail1s.peppermillcas.com> Ah, the games begin! Here is my suggestion for how to encode parameters into an cursor.execute call: DON'T! Use an arbitrary argument list, formatted like a PRINT statement: cursor.execute("Select foo from table where goo >",foo,"and bar =",bar) Now THAT is easy to read and reliable. "BUT," you say: "the other way is easy to parse and convert into other formats." I quote from Guido on this list last January 2nd: vvvvv (begin clip) vvvvvv For Python itself, making the life of the developers harder is usually considered okay if it makes the life of the users easier. I'm curious if the life of the users isn't made *considerably* harder by some gratuitous differences between DB-API implementations, when a user needs to switch database vendors or (a slightly different case!) needs to support multiple vendors simultaneously. Yes, you can write all your code using the facilities to inquire DB-API features, but in practice nobody does that until they are forced to consider multiple versions and realize just how different they are. In a few months on this list I've seen many folks post about this. (Of course, you can argue that the database vendors are a major source of the differences -- the differences in SQL syntax are indeed staggering! But I'm not sure if that gives DB-API developers an excuse -- otherwise we might as well have no standard at all.) ^^^^^ (end clip) ^^^^^^ Let's go for the gold. We do not need to copy ANY given dialect of SQL. We need to find OR CREATE a dialect which will give our application programmers the best possible productivity. ----------- Vernon -----Original Message----- From: M.-A. Lemburg [mailto:mal@egenix.com] Sent: Thursday, August 19, 2004 2:13 PM To: Dieter Maurer Cc: db-sig@python.org Subject: Re: [DB-SIG] db module wrapper Dieter Maurer wrote: > Randall Smith wrote at 2004-8-18 15:22 -0500: > >>... >>* Standardize the params input to use the ? operator and a list. The >>module translates input params and query into the one appropriate for >>the native module. For example with cx_Oracle: "Select foo from table >>where goo > ?" with params [myparam] converts to "Select foo from table >>where goo > :var1" with params {'var1':myparam}. This is working with >>both psycopg and cx_Oracle. > > > Why are you replacing a readable and reliable syntax by one > that is difficult to read and unreliable? > > As soon as you have a couple of "?", it becomes quite difficult > to grasp... You get used to it rather quickly ;-) (and it's easy to parse and convert into the other formats). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 19 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________ DB-SIG maillist - DB-SIG@python.org http://mail.python.org/mailman/listinfo/db-sig From wnvcole at peppermillcas.com Thu Aug 19 23:08:22 2004 From: wnvcole at peppermillcas.com (Vernon Cole) Date: Thu Aug 19 23:08:29 2004 Subject: [DB-SIG] db module wrapper Message-ID: <1DE30EB3ECE266409FDE6E7032F178C41397F7@pcimail1s.peppermillcas.com> Randall: Is a standard even possible? Consult www.connectionstrings.com to see how many variations exist even when the client computer is restricted to a MsWindows system. Nevertheless, if someone else has already visited this Pandora's box, let's look at what they came up with. Where do we find info on the "jdbc standard"? --------- Vernon -----Original Message----- From: Randall Smith [mailto:randall@tnr.cc] Sent: Thursday, August 19, 2004 10:32 AM To: Python DB-SIG Mailing List Subject: Re: [DB-SIG] db module wrapper One thing I'm surprised nobody mentioned is a uniform login method. Does anyone have a suggestion for this? I thought about using the jdbc standard. Randall From randall at tnr.cc Fri Aug 20 03:44:43 2004 From: randall at tnr.cc (Randall Smith) Date: Fri Aug 20 03:44:51 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <16676.57995.851107.259406@gargle.gargle.HOWL> References: <4123BA93.1040604@tnr.cc> <16676.57995.851107.259406@gargle.gargle.HOWL> Message-ID: <4125578B.5040000@tnr.cc> Dieter Maurer wrote: > Randall Smith wrote at 2004-8-18 15:22 -0500: > >>... >>* Standardize the params input to use the ? operator and a list. The >>module translates input params and query into the one appropriate for >>the native module. For example with cx_Oracle: "Select foo from table >>where goo > ?" with params [myparam] converts to "Select foo from table >>where goo > :var1" with params {'var1':myparam}. This is working with >>both psycopg and cx_Oracle. > > > Why are you replacing a readable and reliable syntax by one > that is difficult to read and unreliable? > > As soon as you have a couple of "?", it becomes quite difficult > to grasp... > Out of the format, named, numeric, pyformat, and qmark paramstyles, I find qmark the most simple. What is difficult to grasp? I'm certainly not set on this and open to reasons to use another paramstyle, but my objective is to settle on one. Randall From randall at tnr.cc Fri Aug 20 06:43:17 2004 From: randall at tnr.cc (Randall Smith) Date: Fri Aug 20 06:43:34 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4123EE84.1080608@colorstudy.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> Message-ID: <41258165.2040006@tnr.cc> Ian Bicking wrote: > > Does this work with all the cases? This can be hard, especially the > (Postgres) cases: > > SELECT * FROM questions WHERE answer = 'maybe?' AND user_id = ? > SELECT * FROM questions WHERE answer = '''maybe?''' AND user_id = ? > SELECT * FROM questions WHERE answer = '\'maybe?\'' AND user_id = ? > SELECT * FROM questions WHERE answer = ' > maybe > ? > ' AND user_id = ? > SELECT * FROM "questions?" WHERE answer = 'maybe' AND user_id = ? > > > For MySQL support, you'd need to recognize double quotes in addition to > single quotes, with the backquote for quoted columns. > I thought about this allot today. I finally came up with a surprisingly simple solution that I coded < 30 lines. Performance is good also. At first, I was thinking of ways to identify the quoted strings. But the scope of this task only requires that I know where NOT to look for qmarks. The only oddity is non-standard escapes like Postgresql's '\'. '\' is not a problem to handle, I just have to know when to treat it as an escape character and when not to. Do you know of any other databases that use '\' as an escape character or any other non-standard escape characters in use? Randall From ianb at colorstudy.com Fri Aug 20 07:10:36 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Fri Aug 20 07:10:40 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <41258165.2040006@tnr.cc> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <41258165.2040006@tnr.cc> Message-ID: <412587CC.9090606@colorstudy.com> Randall Smith wrote: > The only oddity is non-standard escapes like Postgresql's '\'. '\' is > not a problem to handle, I just have to know when to treat it as an > escape character and when not to. Do you know of any other databases > that use '\' as an escape character or any other non-standard escape > characters in use? MySQL also uses \', as well as strings like "maybe?", "\'maybe?\'", or "\"maybe?\"". SQLite and Firebird -- other databases I am familiar with -- don't use backslash quoting. Also note 'maybe\\' and 'maybe\\\'?' (and so on). Backslash quoting can be a pain. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From farcepest at gmail.com Fri Aug 20 07:30:33 2004 From: farcepest at gmail.com (Andy Dustman) Date: Fri Aug 20 07:30:35 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <41258165.2040006@tnr.cc> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <41258165.2040006@tnr.cc> Message-ID: <9826f38004081922306be78eac@mail.gmail.com> On Thu, 19 Aug 2004 23:43:17 -0500, Randall Smith wrote: > The only oddity is non-standard escapes like Postgresql's '\'. '\' is > not a problem to handle, I just have to know when to treat it as an > escape character and when not to. Do you know of any other databases > that use '\' as an escape character or any other non-standard escape > characters in use? Non-standard escapes and features aren't a problem; the problem is that people write code that use those non-standard features. Here's a fun fact: MySQLdb-1..x uses %s for the value placeholder. MySQLdb-2.x will use ?, but only because the implementation (and API) is changing; 4.1 and newer have a real parameter binding system which uses ?. Only problem is, it also supports the old API, and I suspect you have to use the old API if you are talking to an <4.1 server, even if you have a >=4.1 client, because the protocol is different too. -- Computer interfaces should never be made of meat. From ianb at colorstudy.com Fri Aug 20 07:51:37 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Fri Aug 20 07:51:40 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <9826f38004081922306be78eac@mail.gmail.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <41258165.2040006@tnr.cc> <9826f38004081922306be78eac@mail.gmail.com> Message-ID: <41259169.3080209@colorstudy.com> Andy Dustman wrote: > Non-standard escapes and features aren't a problem; the problem is > that people write code that use those non-standard features. > > Here's a fun fact: MySQLdb-1..x uses %s for the value placeholder. > MySQLdb-2.x will use ?, but only because the implementation (and API) > is changing; 4.1 and newer have a real parameter binding system which > uses ?. Only problem is, it also supports the old API, and I suspect > you have to use the old API if you are talking to an <4.1 server, even > if you have a >=4.1 client, because the protocol is different too. And all the more reason for an interface that hides those incompatibilities and annoyances. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From randall at tnr.cc Fri Aug 20 07:51:49 2004 From: randall at tnr.cc (Randall Smith) Date: Fri Aug 20 07:51:53 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <412587CC.9090606@colorstudy.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <41258165.2040006@tnr.cc> <412587CC.9090606@colorstudy.com> Message-ID: <41259175.8080605@tnr.cc> Ian Bicking wrote: > Randall Smith wrote: > >> The only oddity is non-standard escapes like Postgresql's '\'. '\' is >> not a problem to handle, I just have to know when to treat it as an >> escape character and when not to. Do you know of any other databases >> that use '\' as an escape character or any other non-standard escape >> characters in use? > > > MySQL also uses \', as well as strings like "maybe?", "\'maybe?\'", or > "\"maybe?\"". SQLite and Firebird -- other databases I am familiar with > -- don't use backslash quoting. > > Also note 'maybe\\' and 'maybe\\\'?' (and so on). Backslash quoting can > be a pain. > Thanks. You obviously have experience with this issue. The 'maybe\\' example would currently break my program. I can fix this by assuming an even number of '\' will not escape a quote. I've been collecting your examples using them as test cases. I think I'm close to completing the parameter abstraction feature. On a different topic, a new question has come to mind. What is a good way for the wrapper to know what module it is using? It needs to know if it is psycopg, pgdb, cx_Oracle, mx.ODBC.Windows, or whatever for some module specific behavior. I can get dbmodule.__version__, but dbmodule.__name__ is not reliable. I currently rely on a string passed in at initialization. Randall From randall at tnr.cc Fri Aug 20 08:04:25 2004 From: randall at tnr.cc (Randall Smith) Date: Fri Aug 20 08:04:36 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <9826f38004081922306be78eac@mail.gmail.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <41258165.2040006@tnr.cc> <9826f38004081922306be78eac@mail.gmail.com> Message-ID: <41259469.4040005@tnr.cc> Andy Dustman wrote: > > > Non-standard escapes and features aren't a problem; the problem is > that people write code that use those non-standard features. > Agreed. And because the objective of the module wrapper in discussion is to be a standard, it would be advisable to avoid non-standard escapes when using it. But that's not sufficient reason not to deal with them. I don't think they'll be a problem though. Ian has pointed out most if not all of the pitfalls to me, and the wrapper successfully deals with all of them in its current state. Randall From mal at egenix.com Fri Aug 20 13:06:25 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Aug 20 13:06:29 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <1DE30EB3ECE266409FDE6E7032F178C41397F6@pcimail1s.peppermillcas.com> References: <1DE30EB3ECE266409FDE6E7032F178C41397F6@pcimail1s.peppermillcas.com> Message-ID: <4125DB31.4000506@egenix.com> Vernon Cole wrote: > Ah, the games begin! Before yuo head off, may I suggest that you create a layer on top of the DB API compatible drivers (much like you have in Perl) instead of forcing incompatible changes onto the various DB interface modules ?! The user can then use the abstraction layer and the module authors can continue to craft modules that provide interfaces which adhere to the DB API interface standard. > Here is my suggestion for how to encode parameters into an cursor.execute > call: DON'T! > > Use an arbitrary argument list, formatted like a PRINT statement: > > cursor.execute("Select foo from table where goo >",foo,"and bar =",bar) > > Now THAT is easy to read and reliable. > > "BUT," you say: "the other way is easy to parse and convert into other > formats." > > I quote from Guido on this list last January 2nd: > vvvvv (begin clip) vvvvvv > > For Python itself, making the life of the developers harder is > usually considered okay if it makes the life of the users easier. > > I'm curious if the life of the users isn't made *considerably* harder > by some gratuitous differences between DB-API implementations, when > a user needs to switch database vendors or (a slightly different > case!) needs to support multiple vendors simultaneously. > > Yes, you can write all your code using the facilities to inquire > DB-API features, but in practice nobody does that until they are > forced to consider multiple versions and realize just how different > they are. In a few months on this list I've seen many folks post > about this. > > (Of course, you can argue that the database vendors are a major source > of the differences -- the differences in SQL syntax are indeed > staggering! But I'm not sure if that gives DB-API developers an > excuse -- otherwise we might as well have no standard at all.) > ^^^^^ (end clip) ^^^^^^ > > Let's go for the gold. We do not need to copy ANY given dialect of SQL. We > need to find OR CREATE a dialect which will give our application programmers > the best possible productivity. Hmm, ODBC tried to go for gold, but didn't get very far. I have a feeling that your quest is not going to have a different outcome :-/ The typical approach to the problem (if at all) that Guido mentions is to have an application specific abstraction layer which then gets adapted for the intended list of database backends. This layer has all the knowledge about the application needs as well as the ways in which these needs can be mapped to the databases (both in terms of Python database module interface and database SQL dialect). In my experience trying to generalize these abstraction layers always fails at some point: either the database SQL dialects and offered features are too different, which let's you end up only supporting a very low-tech common subset, or the abstraction has to go through enormous lengths at trying to create a common behaviour among the database backends (eg. on how multiple result sets are handled, array processing, result set scrolling, etc.etc.). I think that it would be more helpful to application writers to have a database of common database (and interface module) specific gotchas, so that it becomes easier to write the application specific abstraction layer. Even if you only use one Python database interface module, e.g. mxODBC, to access all your supported database backends, the situation does not get much better. You still have to deal with different supported data types, ways of writing schemas, indexes, triggers, etc. and the different database drivers usually use differing error handling too (some raise errors, others don't, error codes usually don't match between database beckends). So if you really have the need to support multiple backends, the situation is not made a lot easier by having a single interface to write applications against. PS: I'm off to EuroFooCamp now... anybody from the list attending that event ? Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 20 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From rosendo.martinez at valdisme.com Fri Aug 20 14:06:45 2004 From: rosendo.martinez at valdisme.com (rosendo) Date: Fri Aug 20 14:02:01 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4125DB31.4000506@egenix.com> Message-ID: <20040820120200.43C3A1E4003@bag.python.org> Would be great if i could go to the event!!!..... But the job, like always :-( I hope some day!...jajajajajja See you soon!. Rosendo. -----Mensaje original----- De: db-sig-bounces@python.org [mailto:db-sig-bounces@python.org] En nombre de M.-A. Lemburg Enviado el: viernes, 20 de agosto de 2004 13:06 Para: Vernon Cole CC: db-sig@python.org Asunto: Re: [DB-SIG] db module wrapper Vernon Cole wrote: > Ah, the games begin! Before yuo head off, may I suggest that you create a layer on top of the DB API compatible drivers (much like you have in Perl) instead of forcing incompatible changes onto the various DB interface modules ?! The user can then use the abstraction layer and the module authors can continue to craft modules that provide interfaces which adhere to the DB API interface standard. > Here is my suggestion for how to encode parameters into an cursor.execute > call: DON'T! > > Use an arbitrary argument list, formatted like a PRINT statement: > > cursor.execute("Select foo from table where goo >",foo,"and bar =",bar) > > Now THAT is easy to read and reliable. > > "BUT," you say: "the other way is easy to parse and convert into other > formats." > > I quote from Guido on this list last January 2nd: > vvvvv (begin clip) vvvvvv > > For Python itself, making the life of the developers harder is > usually considered okay if it makes the life of the users easier. > > I'm curious if the life of the users isn't made *considerably* harder > by some gratuitous differences between DB-API implementations, when > a user needs to switch database vendors or (a slightly different > case!) needs to support multiple vendors simultaneously. > > Yes, you can write all your code using the facilities to inquire > DB-API features, but in practice nobody does that until they are > forced to consider multiple versions and realize just how different > they are. In a few months on this list I've seen many folks post > about this. > > (Of course, you can argue that the database vendors are a major source > of the differences -- the differences in SQL syntax are indeed > staggering! But I'm not sure if that gives DB-API developers an > excuse -- otherwise we might as well have no standard at all.) > ^^^^^ (end clip) ^^^^^^ > > Let's go for the gold. We do not need to copy ANY given dialect of SQL. We > need to find OR CREATE a dialect which will give our application programmers > the best possible productivity. Hmm, ODBC tried to go for gold, but didn't get very far. I have a feeling that your quest is not going to have a different outcome :-/ The typical approach to the problem (if at all) that Guido mentions is to have an application specific abstraction layer which then gets adapted for the intended list of database backends. This layer has all the knowledge about the application needs as well as the ways in which these needs can be mapped to the databases (both in terms of Python database module interface and database SQL dialect). In my experience trying to generalize these abstraction layers always fails at some point: either the database SQL dialects and offered features are too different, which let's you end up only supporting a very low-tech common subset, or the abstraction has to go through enormous lengths at trying to create a common behaviour among the database backends (eg. on how multiple result sets are handled, array processing, result set scrolling, etc.etc.). I think that it would be more helpful to application writers to have a database of common database (and interface module) specific gotchas, so that it becomes easier to write the application specific abstraction layer. Even if you only use one Python database interface module, e.g. mxODBC, to access all your supported database backends, the situation does not get much better. You still have to deal with different supported data types, ways of writing schemas, indexes, triggers, etc. and the different database drivers usually use differing error handling too (some raise errors, others don't, error codes usually don't match between database beckends). So if you really have the need to support multiple backends, the situation is not made a lot easier by having a single interface to write applications against. PS: I'm off to EuroFooCamp now... anybody from the list attending that event ? Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 20 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________ DB-SIG maillist - DB-SIG@python.org http://mail.python.org/mailman/listinfo/db-sig From andy47 at halfcooked.com Fri Aug 20 16:34:32 2004 From: andy47 at halfcooked.com (Andy Todd) Date: Fri Aug 20 16:34:38 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <9A70CBEC-F1FF-11D8-8726-000A95A5D8B2@mitre.org> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <4124ACFB.3020405@egenix.com> <4124CE07.1040606@egenix.com> <9A70CBEC-F1FF-11D8-8726-000A95A5D8B2@mitre.org> Message-ID: <41260BF8.80807@halfcooked.com> Marc Colosimo wrote: > > On Aug 19, 2004, at 11:57 AM, M.-A. Lemburg wrote: > >> Marc Colosimo wrote: >> [snip] > >> After all binding >> parameters exist to make all this quoting unnecessary and >> indeed, many drivers pass the SQL string and the parameters >> well separated to the database. > > > not pgdb (_quoteparams) or cx_oracle (Cursor_SetBindVariables). I > couldn't check psycopg because the links from freshmeat are dead. > Call me Mr Picky, but that is not true for cx_Oracle. It uses a feature of Oracle called bind variables. For an explanation see; http://www.rittman.net/archives/000832.html The function Cursor_SetBindVariables in Cursor.c of cx_Oracle simply unpacks the values passed to the execute statement. Quoting and other esoteric conversion requirements and performed by the Oracle software. >> The database then builds >> an access path using the SQL string (with binding parameters) >> and runs the access path against the parameters that are >> specified as separate part of the request. >> > > As ever, Marc-Andre is correct. The key reason for this is performance as parsing *can* have quite a big overhead. Regards, Andy -- -------------------------------------------------------------------------------- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From mcolosimo at mitre.org Fri Aug 20 19:01:14 2004 From: mcolosimo at mitre.org (Marc Colosimo) Date: Fri Aug 20 19:01:26 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <41260BF8.80807@halfcooked.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <4124ACFB.3020405@egenix.com> <4124CE07.1040606@egenix.com> <9A70CBEC-F1FF-11D8-8726-000A95A5D8B2@mitre.org> <41260BF8.80807@halfcooked.com> Message-ID: <8B4A5E82-F2CA-11D8-8726-000A95A5D8B2@mitre.org> On Aug 20, 2004, at 10:34 AM, Andy Todd wrote: > Marc Colosimo wrote: >> On Aug 19, 2004, at 11:57 AM, M.-A. Lemburg wrote: >>> Marc Colosimo wrote: >>> > [snip] >>> After all binding >>> parameters exist to make all this quoting unnecessary and >>> indeed, many drivers pass the SQL string and the parameters >>> well separated to the database. >> not pgdb (_quoteparams) or cx_oracle (Cursor_SetBindVariables). I >> couldn't check psycopg because the links from freshmeat are dead. > > Call me Mr Picky, but that is not true for cx_Oracle. It uses a > feature of Oracle called bind variables. For an explanation see; > > http://www.rittman.net/archives/000832.html > > The function Cursor_SetBindVariables in Cursor.c of cx_Oracle simply > unpacks the values passed to the execute statement. Quoting and other > esoteric conversion requirements and performed by the Oracle software. > Okay, I was wrong about BindVariables, which does something completely different than quoting ( I don't fully understand how cx_Oracle does it and at this time I really don't care or have time to look into). The way cx_Oracle handles things would throw a wrench into what I want (my original request, which also has been smashed around), but wasn't fully explained why (I didn't fully know how some modules work). However call me Picky Jr, I'm still standing my ground on pgdb (PyGreSQL), which is what I based my original message on and use, and stand corrected on the other. Therefore, the generalization, like so many, doesn't cover all bases (especially in the world of SQL and databases). Finally, I like some of the previous ideas and would like to see something like what Chris talked about, but I wont hold my breath. Marc From wnvcole at peppermillcas.com Fri Aug 20 19:31:04 2004 From: wnvcole at peppermillcas.com (Vernon Cole) Date: Fri Aug 20 19:32:53 2004 Subject: [DB-SIG] db module wrapper Message-ID: <1DE30EB3ECE266409FDE6E7032F178C41397F8@pcimail1s.peppermillcas.com> I confess, and beg pardon. I let myself wander off from Randall's topic -- which is creation of a wrapper for various DB API 2.0 packages -- onto the topic of creation of DB API 3.0. Nevertheless, I think that Randall's work may help to define 3.0 in some ways. I feel very strongly that the eventual 3.0 must be able to run simple to medium querys, and simple adds or updates, using a completly standardized syntax. When one gets to the more complex stuff -- indexing and triggers for example -- standardization is out the window. Way out. I think the goal of the wrapper is roughly that, too. Having said all that, is it not true that the role of cursor.execute() in the wrapper is not to PARSE an SQL statement, but to BUILD one? The four ways now used to encode parameters are all based loosly on the principle of an fprintf() function in C. A string with some type of escape patterns is passed, followed by a list of parameters. I am trying to suggest that we can simplify things by using the power of python. We are using a language with strongly typed parameters. A module implementor can tell at run time the data type of the contents of an arbitrary parameter. If the cursor.execute() method accepts an arbitrary parameter list, then we have no need of escapes. The application programmer simply breaks his sql statement at that point, puts in the python variable he has in mind, and then continues the SQL as another string literal. The implementor will inject the appropriate escape into the string she is building and prepare the paramater for passing as needed by the target SQL engine. Am I missing something? Why would this not work? ----------- Vernon -----Original Message----- From: M.-A. Lemburg [mailto:mal@egenix.com] Sent: Friday, August 20, 2004 5:06 AM [clip] Before yuo head off, may I suggest that you create a layer on top of the DB API compatible drivers (much like you have in Perl) instead of forcing incompatible changes onto the various DB interface modules ?! The user can then use the abstraction layer and the module authors can continue to craft modules that provide interfaces which adhere to the DB API interface standard. The typical approach to the problem (if at all) that Guido mentions is to have an application specific abstraction layer which then gets adapted for the intended list of database backends. This layer has all the knowledge about the application needs as well as the ways in which these needs can be mapped to the databases (both in terms of Python database module interface and database SQL dialect). In my experience trying to generalize these abstraction layers always fails at some point: either the database SQL dialects and offered features are too different, which let's you end up only supporting a very low-tech common subset, or the abstraction has to go through enormous lengths at trying to create a common behaviour among the database backends (eg. on how multiple result sets are handled, array processing, result set scrolling, etc.etc.). [clip] From terekhov at emc.com Fri Aug 20 20:04:53 2004 From: terekhov at emc.com (Mikhail Terekhov) Date: Fri Aug 20 20:05:00 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <1DE30EB3ECE266409FDE6E7032F178C41397F8@pcimail1s.peppermillcas.com> References: <1DE30EB3ECE266409FDE6E7032F178C41397F8@pcimail1s.peppermillcas.com> Message-ID: <41263D45.9060909@emc.com> Vernon Cole wrote: > I confess, and beg pardon. I let myself wander off from Randall's topic -- > which is creation of a wrapper for various DB API 2.0 packages -- onto the > topic of creation of DB API 3.0. Nevertheless, I think that Randall's work > may help to define 3.0 in some ways. > I feel very strongly that the eventual 3.0 must be able to run simple to > medium querys, and simple adds or updates, using a completly standardized > syntax. When one gets to the more complex stuff -- indexing and triggers > for example -- standardization is out the window. Way out. I think the goal > of the wrapper is roughly that, too. > Having said all that, is it not true that the role of cursor.execute() in > the wrapper is not to PARSE an SQL statement, but to BUILD one? I would say that the role of the cursor.execute() is EXECUTE SQL statement, not to parse or build one. > The four ways now used to encode parameters are all based loosly on the > principle of an fprintf() function in C. A string with some type of escape > patterns is passed, followed by a list of parameters. I am trying to suggest > that we can simplify things by using the power of python. We are using a > language with strongly typed parameters. A module implementor can tell at > run time the data type of the contents of an arbitrary parameter. > If the cursor.execute() method accepts an arbitrary parameter list, then we > have no need of escapes. The application programmer simply breaks his sql > statement at that point, puts in the python variable he has in mind, and > then continues the SQL as another string literal. The implementor will > inject the appropriate escape into the string she is building and prepare > the paramater for passing as needed by the target SQL engine. That is impossible in general without full SQL parsing capability. For example, string quoting may be different (' vs. ") depending on where in the statement it is used (table/schema/column name vs. literal in a WHERE clause). This is overkill IMHO as well as connection/cursor and execute/fetchXXX 'overabstractions'. It may be very misleading sometimes (especially execute/fetchXXX). Regards, -- Mikhail Terekhov From randall at tnr.cc Fri Aug 20 20:32:49 2004 From: randall at tnr.cc (Randall Smith) Date: Fri Aug 20 20:33:44 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <1DE30EB3ECE266409FDE6E7032F178C41397F8@pcimail1s.peppermillcas.com> References: <1DE30EB3ECE266409FDE6E7032F178C41397F8@pcimail1s.peppermillcas.com> Message-ID: <412643D1.1060404@tnr.cc> What about executemany? Then you'll have to pass in a list of lists as paramaters. l1 = ['harry', 'sam', 'charley'] l2 = [1, 2, 3] query = "select * from tname where col1 = ", l1, " and col2 = ", l2 cursor.execute(query) Is this what you're talking about? It's actually pretty cool, but a big break from API 2. Maybe it could be an added feature b/c I want the wrapper to be DBAPI2 compliant. It wouldn't be that hard to code. The more I think about it, the more I like it. This is probably what would happen in practice. query = [] query.append('select * from tname where col1 = ') query.append(l1) query.append('and col2 = ') query.append(l2) cursor.execute(query) That's actually quite readable. Am I getting the right idea? Is this what you are intending? Is this in anyway disadvantageous to the API2 way? You're certainly on track about the API 3.0 issue. This IS a new API. The wrapper is implementing a more strict API than 2.0 does. I'm definitely giving this some thought. It seems very nice. Randall Vernon Cole wrote: > I confess, and beg pardon. I let myself wander off from Randall's topic -- > which is creation of a wrapper for various DB API 2.0 packages -- onto the > topic of creation of DB API 3.0. Nevertheless, I think that Randall's work > may help to define 3.0 in some ways. > I feel very strongly that the eventual 3.0 must be able to run simple to > medium querys, and simple adds or updates, using a completly standardized > syntax. When one gets to the more complex stuff -- indexing and triggers > for example -- standardization is out the window. Way out. I think the goal > of the wrapper is roughly that, too. > Having said all that, is it not true that the role of cursor.execute() in > the wrapper is not to PARSE an SQL statement, but to BUILD one? > The four ways now used to encode parameters are all based loosly on the > principle of an fprintf() function in C. A string with some type of escape > patterns is passed, followed by a list of parameters. I am trying to suggest > that we can simplify things by using the power of python. We are using a > language with strongly typed parameters. A module implementor can tell at > run time the data type of the contents of an arbitrary parameter. > If the cursor.execute() method accepts an arbitrary parameter list, then we > have no need of escapes. The application programmer simply breaks his sql > statement at that point, puts in the python variable he has in mind, and > then continues the SQL as another string literal. The implementor will > inject the appropriate escape into the string she is building and prepare > the paramater for passing as needed by the target SQL engine. > Am I missing something? > Why would this not work? > ----------- > Vernon > > -----Original Message----- > From: M.-A. Lemburg [mailto:mal@egenix.com] > Sent: Friday, August 20, 2004 5:06 AM > [clip] > Before yuo head off, may I suggest that you create a layer on > top of the DB API compatible drivers (much like you have in Perl) > instead of forcing incompatible changes onto the various DB interface > modules ?! > > The user can then use the abstraction layer and the module > authors can continue to craft modules that provide interfaces > which adhere to the DB API interface standard. > The typical approach to the problem (if at all) that > Guido mentions is to have an application specific > abstraction layer which then gets adapted for the > intended list of database backends. > > This layer has all the knowledge about the application > needs as well as the ways in which these needs can be > mapped to the databases (both in terms of Python database > module interface and database SQL dialect). > > In my experience trying to generalize these abstraction > layers always fails at some point: either the database > SQL dialects and offered features are too different, > which let's you end up only supporting a very low-tech > common subset, or the abstraction has to go through > enormous lengths at trying to create a common behaviour > among the database backends (eg. on how multiple result > sets are handled, array processing, result set scrolling, > etc.etc.). > [clip] > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig From dieter at handshake.de Fri Aug 20 20:10:59 2004 From: dieter at handshake.de (Dieter Maurer) Date: Fri Aug 20 20:58:25 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4125578B.5040000@tnr.cc> References: <4123BA93.1040604@tnr.cc> <16676.57995.851107.259406@gargle.gargle.HOWL> <4125578B.5040000@tnr.cc> Message-ID: <16678.16051.417360.828462@gargle.gargle.HOWL> Randall Smith wrote at 2004-8-19 20:44 -0500: > ... >> As soon as you have a couple of "?", it becomes quite difficult >> to grasp... >> >Out of the format, named, numeric, pyformat, and qmark paramstyles, I >find qmark the most simple. What is difficult to grasp? The relationship between the "?"s and the corresponding arguments. You have to count "?"s and arguments and this can be difficult for non-trivial SQL statements with many "?". Furthermore, the "?" provide no glue to the meaning of value. -- Dieter From randall at tnr.cc Fri Aug 20 21:26:55 2004 From: randall at tnr.cc (Randall Smith) Date: Fri Aug 20 21:27:11 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <16678.16051.417360.828462@gargle.gargle.HOWL> References: <4123BA93.1040604@tnr.cc> <16676.57995.851107.259406@gargle.gargle.HOWL> <4125578B.5040000@tnr.cc> <16678.16051.417360.828462@gargle.gargle.HOWL> Message-ID: <4126507F.20505@tnr.cc> Dieter Maurer wrote: > Randall Smith wrote at 2004-8-19 20:44 -0500: > >>... >> >>> As soon as you have a couple of "?", it becomes quite difficult >>> to grasp... >>> >> >>Out of the format, named, numeric, pyformat, and qmark paramstyles, I >>find qmark the most simple. What is difficult to grasp? > > > The relationship between the "?"s and the corresponding arguments. > You have to count "?"s and arguments and this can be difficult > for non-trivial SQL statements with many "?". > > Furthermore, the "?" provide no glue to the meaning of value. > I see. It would be possible to support more than 1 paramstyle by examining the type of the param input or allowing an explicit function parameter in execute as in execute(query, params, pstyle='named'). Randall From randall at tnr.cc Fri Aug 20 21:52:16 2004 From: randall at tnr.cc (Randall Smith) Date: Fri Aug 20 21:52:22 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <41265174.3060502@colorstudy.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <41258165.2040006@tnr.cc> <9826f38004081922306be78eac@mail.gmail.com> <41259169.3080209@colorstudy.com> <41264EC9.8090403@tnr.cc> <41265174.3060502@colorstudy.com> Message-ID: <41265670.9060504@tnr.cc> Assume that the first item in the list is always sql. Is that a valid assumption? Randall Ian Bicking wrote: > Just thought about it, how would you deal with: > > name = 'bob' > query = ['select * from tname where col = ', name] > > The query is just two strings -- the first is a SQL expression, the > second some data. But there's no way to tell that just from looking at > the query list. > > > > Randall Smith wrote: > >> Ian, >> >> What do you think about Vernon Cole's idea with passing a query in as >> a list. I thought about it and I'm starting to like it. Check this out. >> >> l1 = ['harry', 'sam', 'charley'] >> l2 = [1, 2, 3] >> >> query = "select * from tname where col1 = ", l1, " and col2 = ", l2 >> cursor.execute(query) >> >> query = [] >> query.append('select * from tname where col1 = ') >> query.append(l1) >> query.append('and col2 = ') >> query.append(l2) >> cursor.execute(query) >> >> Note: This is like executemany. >> >> There is no need for parsing at all and the line breaking encouraged >> by using a list makes for readable code. >> >> What do you think? >> >> Randall > > From ianb at colorstudy.com Fri Aug 20 21:57:01 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Fri Aug 20 21:58:13 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <41265670.9060504@tnr.cc> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <41258165.2040006@tnr.cc> <9826f38004081922306be78eac@mail.gmail.com> <41259169.3080209@colorstudy.com> <41264EC9.8090403@tnr.cc> <41265174.3060502@colorstudy.com> <41265670.9060504@tnr.cc> Message-ID: <4126578D.5060902@colorstudy.com> Randall Smith wrote: > Assume that the first item in the list is always sql. Is that a valid > assumption? It would be really easy to make mistakes that way. Or to compose statements. For instance, given two query fragments for a where clause, you might want to compose them like: new_query = ['('] + query1 + [') AND ('] + query2 + [')'] But then you'd have to worry about how query1 and query2 were constructed, so that you concatenated the strings of any SQL expressions, but didn't concatenate to any literals. And there's a lot of even simpler mistakes that could be made, and the errors could be hard to find or cause security holes. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From randall at tnr.cc Fri Aug 20 22:45:01 2004 From: randall at tnr.cc (Randall Smith) Date: Fri Aug 20 22:45:17 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <1DE30EB3ECE266409FDE6E7032F178C41397F9@pcimail1s.peppermillcas.com> References: <1DE30EB3ECE266409FDE6E7032F178C41397F9@pcimail1s.peppermillcas.com> Message-ID: <412662CD.3000109@tnr.cc> Vernon Cole wrote: > Ian: > I think that would be an SQL syntax error. My SQL query analyser (MsSQL) > replies to > "select * from tname where col = bob" > with an error message "invalid column name 'bob'". > > I think you would have to phrase it like: > query = ("select * from tname where col = '", name, "'") > The single tics are part of the query syntax. You don't have to manually quote 'bob'. The database module knows 'name' is a string and quotes it properly using the parameter functionality that already exists in all DB API2 modules. > > MsSQL accepts that construct whether 'col' is a numeric or a string valued > column. > YMMV > > I'm still thinking about how .executemany() might work... > ----- Easy. Look at my previous posts. l1 = ['harry', 'sam', 'charley'] l2 = [1, 2, 3] query = "select * from tname where col1 = ", l1, " and col2 = ", l2 cursor.execute(query) query = [] query.append('select * from tname where col1 = ') query.append(l1) query.append('and col2 = ') query.append(l2) cursor.execute(query) > Vernon > > -----Original Message----- > From: Randall Smith [mailto:randall@tnr.cc] > Sent: Friday, August 20, 2004 1:52 PM > To: Ian Bicking > Cc: db >> Python DB-SIG Mailing List > Subject: Re: [DB-SIG] db module wrapper > > > Assume that the first item in the list is always sql. Is that a valid > assumption? > > Randall > > Ian Bicking wrote: > >>Just thought about it, how would you deal with: >> >>name = 'bob' >>query = ['select * from tname where col = ', name] >> >>The query is just two strings -- the first is a SQL expression, the >>second some data. But there's no way to tell that just from looking at >>the query list. >> >> >> >>Randall Smith wrote: >> >> >>>Ian, >>> >>>What do you think about Vernon Cole's idea with passing a query in as >>>a list. I thought about it and I'm starting to like it. Check this out. >>> >>>l1 = ['harry', 'sam', 'charley'] >>>l2 = [1, 2, 3] >>> >>>query = "select * from tname where col1 = ", l1, " and col2 = ", l2 >>>cursor.execute(query) >>> >>>query = [] >>>query.append('select * from tname where col1 = ') >>>query.append(l1) >>>query.append('and col2 = ') >>>query.append(l2) >>>cursor.execute(query) >>> >>>Note: This is like executemany. >>> >>>There is no need for parsing at all and the line breaking encouraged >>>by using a list makes for readable code. >>> >>>What do you think? >>> >>>Randall >> >> > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig From randall at tnr.cc Fri Aug 20 22:59:27 2004 From: randall at tnr.cc (Randall Smith) Date: Fri Aug 20 22:59:35 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4126578D.5060902@colorstudy.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <41258165.2040006@tnr.cc> <9826f38004081922306be78eac@mail.gmail.com> <41259169.3080209@colorstudy.com> <41264EC9.8090403@tnr.cc> <41265174.3060502@colorstudy.com> <41265670.9060504@tnr.cc> <4126578D.5060902@colorstudy.com> Message-ID: <4126662F.6020405@tnr.cc> Ian Bicking wrote: > Randall Smith wrote: > >> Assume that the first item in the list is always sql. Is that a valid >> assumption? > > > It would be really easy to make mistakes that way. Or to compose > statements. For instance, given two query fragments for a where clause, > you might want to compose them like: > > new_query = ['('] + query1 + [') AND ('] + query2 + [')'] > > But then you'd have to worry about how query1 and query2 were > constructed, so that you concatenated the strings of any SQL > expressions, but didn't concatenate to any literals. > > And there's a lot of even simpler mistakes that could be made, and the > errors could be hard to find or cause security holes. > Your point is well taken. Anyhow, since it is simple to code, I'm going to put it in as an extra. If passed only a list, the execute wrapper will treat it as an alternating sql/param list that must begin with sql. Nobody will be forced to use it, but it will be there and consistently supported across all dbs since there is no db-specific functionality (a nice benefit). If it proves to have some unavoidable dangerous characteristic, I'll drop it. On a separate topic, got any ideas on how to abstract exceptions? If someone has already figured out an easy way to do this, I sure would appreciate the knowledge. Randall From ianb at colorstudy.com Fri Aug 20 23:06:41 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Fri Aug 20 23:08:06 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <412662CD.3000109@tnr.cc> References: <1DE30EB3ECE266409FDE6E7032F178C41397F9@pcimail1s.peppermillcas.com> <412662CD.3000109@tnr.cc> Message-ID: <412667E1.7060102@colorstudy.com> Randall Smith wrote: >> I'm still thinking about how .executemany() might work... >> ----- > > Easy. Look at my previous posts. > > l1 = ['harry', 'sam', 'charley'] > l2 = [1, 2, 3] > > query = "select * from tname where col1 = ", l1, " and col2 = ", l2 > cursor.execute(query) > > query = [] > query.append('select * from tname where col1 = ') > query.append(l1) > query.append('and col2 = ') > query.append(l2) > cursor.execute(query) I think you'd still want cursor.executemany(query), as you don't want to simple infer executemany by the presence of a list. In essence, what I think you are suggesting is that this executemany: cursor.executemany( "SELECT * FROM tname WHERE col = ? AND col2 = ?", [('harry', 1), ('sam', 2), ('charley', 3)]) Becomes: cursor.executemany( "SELECT * FROM tname WHERE col = ", ['harry', 'sam', 'charley'], " AND col 2 = ", [1, 2, 3]) -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From randall at tnr.cc Fri Aug 20 23:53:20 2004 From: randall at tnr.cc (Randall Smith) Date: Fri Aug 20 23:53:28 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4123BA93.1040604@tnr.cc> References: <4123BA93.1040604@tnr.cc> Message-ID: <412672D0.7060807@tnr.cc> About exceptions. I'm just throwing this out. Tell me what you think. By default, translate a native_module.Exception into a module_wrapper.Exception unless specified by module_wrapper.config that Exception1 should be converted to Exception2. module_wrapper method will be able to catch exceptions and either claim them or transform them. module_wrapper.config will have some option like {'pyscopg_1.1.15': 'exc_trans execute DatebaseError ProgrammingError'} Which means: When the execute method raises a Database error, transform it to a Programming error. Anything not specified in the config would simply transform from psycopg.Error to wrapper.Error How does this approach sound? Randall From ianb at colorstudy.com Sat Aug 21 00:01:06 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Sat Aug 21 00:02:18 2004 Subject: [DB-SIG] Unifying exceptions (was: db module wrapper) In-Reply-To: <412672D0.7060807@tnr.cc> References: <4123BA93.1040604@tnr.cc> <412672D0.7060807@tnr.cc> Message-ID: <412674A2.6010607@colorstudy.com> There might be a clever/tricky solution that would be a little easier to implement and more reliable. Hmm... you could try changing the exceptions, like: import psycopg psycopg.DatabaseError = MyDatabaseError I don't know if this will always stick, or if it's always mutable in this way. Another way might be: class MyDatabaseError(Exception): pass psycopg.DatabaseError.__bases__ = (MyDatabaseError,) Then you can catch MyDatabaseError, and it will catch psycopg.DatabaseError (since it's a subclass). This might be more likely to stick, and work better with the C modules. (Maybe, I don't know...?) -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From jacobs at theopalgroup.com Sat Aug 21 00:28:06 2004 From: jacobs at theopalgroup.com (Kevin Jacobs) Date: Sat Aug 21 00:28:06 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4126662F.6020405@tnr.cc> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <41258165.2040006@tnr.cc> <9826f38004081922306be78eac@mail.gmail.com> <41259169.3080209@colorstudy.com> <41264EC9.8090403@tnr.cc> <41265174.3060502@colorstudy.com> <41265670.9060504@tnr.cc> <4126578D.5060902@colorstudy.com> <4126662F.6020405@tnr.cc> Message-ID: <41267AF6.4040608@theopalgroup.com> Randall Smith wrote: > On a separate topic, got any ideas on how to abstract exceptions? If > someone has already figured out an easy way to do this, I sure would > appreciate the knowledge. > Yes. I've posted my method for completely virtualizing DB-API exceptions here no less than 4 times over the years. Do yourself a favor and spend some time in the archives. You'll be pleasantly surprised at how many of your ideas will have been previously addressed, many with working code, important test cases, and useful discussion. -Kevin From ianb at colorstudy.com Sat Aug 21 00:33:45 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Sat Aug 21 00:34:58 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <41267AF6.4040608@theopalgroup.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <41258165.2040006@tnr.cc> <9826f38004081922306be78eac@mail.gmail.com> <41259169.3080209@colorstudy.com> <41264EC9.8090403@tnr.cc> <41265174.3060502@colorstudy.com> <41265670.9060504@tnr.cc> <4126578D.5060902@colorstudy.com> <4126662F.6020405@tnr.cc> <41267AF6.4040608@theopalgroup.com> Message-ID: <41267C49.4050801@colorstudy.com> Kevin Jacobs wrote: > Randall Smith wrote: > >> On a separate topic, got any ideas on how to abstract exceptions? If >> someone has already figured out an easy way to do this, I sure would >> appreciate the knowledge. >> > Yes. I've posted my method for completely virtualizing > DB-API exceptions here no less than 4 times over the > years. Do yourself a favor and spend some time in the > archives. You'll be pleasantly surprised at how many > of your ideas will have been previously addressed, > many with working code, important test cases, and > useful discussion. And we will continue to do this on the list until we leave behind better artifacts than some mailing list archives and assorted recipes. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From jacobs at theopalgroup.com Sat Aug 21 00:45:27 2004 From: jacobs at theopalgroup.com (Kevin Jacobs) Date: Sat Aug 21 00:45:28 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <41267C49.4050801@colorstudy.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <41258165.2040006@tnr.cc> <9826f38004081922306be78eac@mail.gmail.com> <41259169.3080209@colorstudy.com> <41264EC9.8090403@tnr.cc> <41265174.3060502@colorstudy.com> <41265670.9060504@tnr.cc> <4126578D.5060902@colorstudy.com> <4126662F.6020405@tnr.cc> <41267AF6.4040608@theopalgroup.com> <41267C49.4050801@colorstudy.com> Message-ID: <41267F07.80205@theopalgroup.com> Ian Bicking wrote: > Kevin Jacobs wrote: > >> Randall Smith wrote: >> >>> On a separate topic, got any ideas on how to abstract exceptions? >>> If someone has already figured out an easy way to do this, I sure >>> would appreciate the knowledge. >>> >> Yes. I've posted my method for completely virtualizing >> DB-API exceptions here no less than 4 times over the >> years. Do yourself a favor and spend some time in the >> archives. You'll be pleasantly surprised at how many >> of your ideas will have been previously addressed, >> many with working code, important test cases, and >> useful discussion. > > > And we will continue to do this on the list until we leave behind > better artifacts than some mailing list archives and assorted recipes. > I wasn't suggesting taking the discussion off-list -- just that the archives are a good place to start. Without being too cynical, most of the current discussion has been raised before, discussed, and in most cases found to be lacking for a general consensus API (like DB-API). After all, there isn't really much else to discuss on the DB-SIG mailing list. -Kevin PS: For those disinclined to review the archives, here is a link to my "recipe": http://mail.python.org/pipermail/db-sig/2003-April/003345.html From randall at tnr.cc Sat Aug 21 00:58:16 2004 From: randall at tnr.cc (Randall Smith) Date: Sat Aug 21 00:58:53 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <41267F07.80205@theopalgroup.com> References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <41258165.2040006@tnr.cc> <9826f38004081922306be78eac@mail.gmail.com> <41259169.3080209@colorstudy.com> <41264EC9.8090403@tnr.cc> <41265174.3060502@colorstudy.com> <41265670.9060504@tnr.cc> <4126578D.5060902@colorstudy.com> <4126662F.6020405@tnr.cc> <41267AF6.4040608@theopalgroup.com> <41267C49.4050801@colorstudy.com> <41267F07.80205@theopalgroup.com> Message-ID: <41268208.7070600@tnr.cc> Kevin Jacobs wrote: > Ian Bicking wrote: > >> Kevin Jacobs wrote: >> >>> Randall Smith wrote: >>> >>>> On a separate topic, got any ideas on how to abstract exceptions? >>>> If someone has already figured out an easy way to do this, I sure >>>> would appreciate the knowledge. You're right to assume that I haven't checked the archives. I should as I might find ideas not discussed here. However, it has been fun and encouraging discussing the subject and listening to ideas and opinions, even if they have been expressed before. From the number of responses, I think there is enough interest to go forward and, based on the last few days of discussion, I believe we have a solution to most issues. >>>> >>> Yes. I've posted my method for completely virtualizing >>> DB-API exceptions here no less than 4 times over the >>> years. Do yourself a favor and spend some time in the >>> archives. You'll be pleasantly surprised at how many >>> of your ideas will have been previously addressed, >>> many with working code, important test cases, and >>> useful discussion. >> >> >> >> And we will continue to do this on the list until we leave behind >> better artifacts than some mailing list archives and assorted recipes. >> > I wasn't suggesting taking the discussion off-list -- just that > the archives are a good place to start. Without being too > cynical, most of the current discussion has been raised > before, discussed, and in most cases found to be lacking > for a general consensus API (like DB-API). After all, > there isn't really much else to discuss on the DB-SIG > mailing list. > > -Kevin > > PS: For those disinclined to review the archives, here is > a link to my "recipe": > > http://mail.python.org/pipermail/db-sig/2003-April/003345.html Thanks > Randall From Chris.Clark at ca.com Sat Aug 21 01:06:29 2004 From: Chris.Clark at ca.com (Clark, Chris M) Date: Sat Aug 21 08:12:56 2004 Subject: [DB-SIG] db module wrapper Message-ID: <1A47908AA9FC9A439863408923A9BFD7C04F59@usilms26.ca.com> > -----Original Message----- > From: db-sig-bounces+clach04=ca.com@python.org > [mailto:db-sig-bounces+clach04=ca.com@python.org]On Behalf Of Randall > Smith > Sent: Friday, August 20, 2004 2:53 PM > To: Python DB-SIG Mailing List > Subject: Re: [DB-SIG] db module wrapper > > > About exceptions. > > I'm just throwing this out. Tell me what you think. > .... Cut > {'pyscopg_1.1.15': 'exc_trans execute DatebaseError ProgrammingError'} > Which means: When the execute method raises a Database > error, transform > it to a Programming error. > (Possibly naive) Question: what problem would be solved by translating/renaming exceptions? I've had a look at Kevin Jacobs exception virtualization code and it hides (very well) WHICH driver raised the exception which seems to be the bug problem for portable applications. I'm trying to figure out why one would want to rename the exception type. If the issue is that a driver isn't returning a sensible (or consistent with other drivers) exception this sounds like a weakness in the driver ( or possibly the 2.0 DBI spec. it depends on the circumstances). Do you have an example of a specific scenario when the Database error occurs for PostgreSQL when it makes more sense to raise a ProgrammingError? It may make more sense for the Postgres DBI driver to be updated rather than introducing another layer (I'm conveniently ignoring backwards compatibility for now :-p). On the subject of the 2.0 spec and exceptions; I've been working with a colleague to create a (replacement) driver to Ingres http://opensource.ca.com/projects/ingres/ and one of the gotchas I had (I'm willing to accept I don't full comprehend 2.0 spec :-p) was that I wasn't clear under exactly what circumstances various exceptions I should be raised. http://www.python.org/peps/pep-0249.html lists the exceptions and some guides of when but it wasn't fully specified. E.g. instead of generating a Database exception whenever the DBMS sends an error back I check for various errors (duplicate key on insert, foreign key failures, etc) so that an IntegrityError is raised instead. I think it would be useful if a more explicit list of examples where available in the spec (such as "duplicate key on insert"). I'm still working out what exceptions we are going to raise and when so I don't have such a definitive list myself but we are working on something like that. Any comments? Have I misunderstood the problem? Is this something that would be a candidate for say a 2.1 DBI spec (or 3.0, etc.). > Anything not specified in the config would simply transform from > psycopg.Error to wrapper.Error As per above, I'd rather the DBI drivers return consistent exceptions. Chris From dieter at handshake.de Sat Aug 21 23:46:02 2004 From: dieter at handshake.de (Dieter Maurer) Date: Sun Aug 22 00:36:09 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <4126507F.20505@tnr.cc> References: <4123BA93.1040604@tnr.cc> <16676.57995.851107.259406@gargle.gargle.HOWL> <4125578B.5040000@tnr.cc> <16678.16051.417360.828462@gargle.gargle.HOWL> <4126507F.20505@tnr.cc> Message-ID: <16679.49818.840837.279135@gargle.gargle.HOWL> Randall Smith wrote at 2004-8-20 14:26 -0500: > ... >It would be possible to support more than 1 paramstyle by >examining the type of the param input or allowing an explicit function >parameter in execute as in execute(query, params, pstyle='named'). I would be happy with the second option. -- Dieter From dieter at handshake.de Sat Aug 21 23:50:40 2004 From: dieter at handshake.de (Dieter Maurer) Date: Sun Aug 22 00:36:10 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <412672D0.7060807@tnr.cc> References: <4123BA93.1040604@tnr.cc> <412672D0.7060807@tnr.cc> Message-ID: <16679.50096.726106.459427@gargle.gargle.HOWL> Randall Smith wrote at 2004-8-20 16:53 -0500: > ... >{'pyscopg_1.1.15': 'exc_trans execute DatebaseError ProgrammingError'} >Which means: When the execute method raises a Database error, transform >it to a Programming error. > >Anything not specified in the config would simply transform from >psycopg.Error to wrapper.Error > >How does this approach sound? The version number in the configuration is a bit worrying... The exception transformation will rarely change with a version change. Therefore, it would be nice, when a module upgrade would not for a configuration change. -- Dieter From randall at tnr.cc Sun Aug 22 02:20:47 2004 From: randall at tnr.cc (Randall Smith) Date: Sun Aug 22 02:20:53 2004 Subject: [DB-SIG] db module wrapper In-Reply-To: <16679.50096.726106.459427@gargle.gargle.HOWL> References: <4123BA93.1040604@tnr.cc> <412672D0.7060807@tnr.cc> <16679.50096.726106.459427@gargle.gargle.HOWL> Message-ID: <4127E6DF.8030900@tnr.cc> Development Updates: I having a working module with wrapping for qmark paramstyle, Python datetime objects (input and output), Exceptions (Kevin Jacob's solution), and date related module constructors. I also moved database specific files into config files. Modules that fully support DB API2 should not need config files. In testing, both cx_Oracle and psycopg work without config files, but have some caveats. Without configs, cx_Oracle dates are converted via mx.DateTime's string parser (not optimal) and a backslash escape in a query will fail with psycopg (not psycopg's fault). MySQLdb requires a config to set the DATETIME type. I've set threadsafety = 0 and just passing description through. I need to do something with description. Should I standardize the description type_code? Extras: db_row - If you have db_row in your path, you can have results formatted with it. I also had some other ideas. I'm going to make module.paramstyle and module.DATETIME writable. For example, module.paramestyle = 'named' will allow you to use named parameters and so on supporting all 5 paramstyles. For DATETIME, I will support mx.DateTime and Zope date objects. Changing the DATETIME attribute will change both input and output of dates. The input being paramaters in param queries. Note that all these things are still 'ideas', but I think they are feasible. The code is very unfinished, so I'm not going to make a release now, but if you want to have a look and/or test it out, let me know and I'll send it to you directly. It is a single module, dball, plus configs. Randall From jfranz at neurokode.com Sun Aug 22 04:15:19 2004 From: jfranz at neurokode.com (Jonathan M. Franz) Date: Sun Aug 22 04:16:18 2004 Subject: [DB-SIG] Re: db module wrapper Message-ID: <412801B7.401@neurokode.com> Well, I'm late to this conversation - so I apologize if my points have already been made. What Randal is doing is very similar to what we're working towards for PDO 2.0 - PDO _is_ actively maintained and promoted, btw. Heck, back in the 1.0.2 days we even were thrown on some CDs that linux enterprise magazine in europe distributed. Frankly, we've always been surprised by the cold reception we've gotten on the db-sig mailing list whenever we raise a concern or mention our module. Perhaps we've been too argumentative about things we feel the dbapi is lacking? Some of the arguments earlier this year when Guido spoke up were things we took very seriously - but the conversation about what todo for dbapi 3.0 died for no apparent (to me) reason. Anyway, some thoughts on ways we do things now in pdo, and how they intersect with Randall's ideas: (cross pollination of ideas is always a good thing) - connection strings We use a single string to hold connection data, with the core parameters standardized (dbapi-module,username, password, database), and the rest given in a module specific way, but all formated into a single string thusly: aConnection = pdo.connect("module=someModule;user=foobar;passwd=mypass;db=myDB;host=localhost") This should be _very_ familar to ADO users. - parameter formating Currently we just use the underlying dbapi module's quoting method - but for 2.0 (which we hope to beta soon!) we're moving towards a :named style, using a modified form of the recipe for style unification at ASPN. http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/278612 We'd considered a positional ('?') style, but found it too limiting. The first time you do a insert or update with 20 ?s in it, you get the feeling that it is rather awkward. Of course, you could allow the user to switch styles on the fly - it'd just be an extra set of conversion functions. - exceptions We already have implemented a unified exception system - PDO contains it's own exceptions, and the textual description contains the underlying exception and the underlying exception's text - allowing for users to capture one set of exceptions and yet giving them all the detail they need to find out why the dbapi module. It's easy to implement, just take a look and steal away. ============== Things to ponder: One thing that made my life rather painful when working on PDO 2.0 was that the underlying type systems in dbapi modules are so different. The implementation of editable resultsets (no more update queries!) required some interesting juggling to bring the underlying types into the current namespace in a usable way. Do you plan on editable resultsets in your module? More soon as I catch up further with the thread. ~Jon Franz NeuroKode Labs, LLC From randall at tnr.cc Sun Aug 22 05:19:09 2004 From: randall at tnr.cc (Randall Smith) Date: Sun Aug 22 05:19:23 2004 Subject: [DB-SIG] Re: db module wrapper In-Reply-To: <412801B7.401@neurokode.com> References: <412801B7.401@neurokode.com> Message-ID: <412810AD.4010609@tnr.cc> Jonathan M. Franz wrote: > Well, I'm late to this conversation - so I apologize if my points have > already been made. > > What Randal is doing is very similar to what we're working towards for > PDO 2.0 - PDO _is_ actively maintained and promoted, btw. Heck, back in Developers are certainly spread thin and when I saw your post, my first thought was maybe I should drop what I'm doing and use PDO. It is similar in that they both try to hide differences in underlying modules. There is a difference in the scopes of the projects. PDO is ADO for Python. The scope of my project is to maintain a DB API V2 interface while hiding module differences. Although we do have some duplicated effort, there are several differences. I just want my DB API2 code to work with all modules. You want an ADO interface. > the 1.0.2 days we even were thrown on some CDs that linux enterprise > magazine in europe distributed. Frankly, we've always been surprised by > the cold reception we've gotten on the db-sig mailing list whenever we > raise a concern or mention our module. Perhaps we've been too > argumentative about things we feel the dbapi is lacking? > Some of the arguments earlier this year when Guido spoke up were things > we took very seriously - but the conversation about what todo for dbapi > 3.0 died for no apparent (to me) reason. > > Anyway, some thoughts on ways we do things now in pdo, and how they > intersect with Randall's ideas: > (cross pollination of ideas is always a good thing) > > - connection strings > We use a single string to hold connection data, with the core parameters > standardized (dbapi-module,username, password, database), and the rest > given in a module specific way, but all formated into a single string > thusly: > aConnection = > pdo.connect("module=someModule;user=foobar;passwd=mypass;db=myDB;host=localhost") I don't plan to modify connection parameters. The thought being that when you switch databases, you have to change this anyway. > > This should be _very_ familar to ADO users. > > - parameter formating > Currently we just use the underlying dbapi module's quoting method - but > for 2.0 (which we hope to beta soon!) we're moving towards a :named > style, using a modified form of the recipe for style unification at > ASPN. http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/278612 > We'd considered a positional ('?') style, but found it too limiting. > The first time you do a insert or update with 20 ?s in it, you get the > feeling that it is rather awkward. > Of course, you could allow the user to switch styles on the fly - it'd > just be an extra set of conversion functions. When discussing this, people had different opinions. Therefore, I'm enabling all 5 paramstyles. > > - exceptions > We already have implemented a unified exception system - PDO contains > it's own exceptions, and the textual description contains the underlying > exception and the underlying exception's text - allowing for users to > capture one set of exceptions and yet giving them all the detail they > need to find out why the dbapi module. It's easy to implement, just > take a look and steal away. The popular opinion was to use Kevin Jacob's idea: http://mail.python.org/pipermail/db-sig/2003-April/003345.html I say popular. Ian and Kevin suggested it. > > ============== > Things to ponder: > One thing that made my life rather painful when working on PDO 2.0 was > that the underlying type systems in dbapi modules are so different. The > implementation of editable resultsets (no more update queries!) required > some interesting juggling to bring the underlying types into the current > namespace in a usable way. Yea. That does sound difficult. Editable resultsets are nice. > > Do you plan on editable resultsets in your module? Definitely not. To much work. I'm mainly programming to API 2. Config files are used to adapt modules that do not completely comply. In theory, a module that is API2 compliant needs no config. > > More soon as I catch up further with the thread. > > ~Jon Franz > NeuroKode Labs, LLC > > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig I'm all about not duplicating effort. I'm not a glutton for needless work. Whether or not my project is made public and maintained depends on public interest. Are our projects similar enough that I should drop mine? I don't know. I'm not looking for an ADO interface, so I plan on using what I have created. Others may think there is too much duplication. In that case, I'll use it, but not publish/maintain it. Though, as judged from the list, there seems to be interest. So my current plans are to put it in Sourceforge CVS. Randall From jfranz at neurokode.com Sun Aug 22 05:57:28 2004 From: jfranz at neurokode.com (Jonathan M. Franz) Date: Sun Aug 22 05:58:27 2004 Subject: [DB-SIG] Re: db module wrapper In-Reply-To: <412810AD.4010609@tnr.cc> References: <412801B7.401@neurokode.com> <412810AD.4010609@tnr.cc> Message-ID: <412819A8.2050901@neurokode.com> Randall Smith wrote: > Developers are certainly spread thin and when I saw your post, my > first thought was maybe I should drop what I'm doing and use PDO. It > is similar in that they both try to hide differences in underlying > modules. There is a difference in the scopes of the projects. PDO is > ADO for Python. The scope of my project is to maintain a DB API V2 > interface while hiding module differences. Although we do have some > duplicated effort, there are several differences. I just want my DB > API2 code to work with all modules. You want an ADO interface. Not really ADO for python, just more ADO/JDBC/delphi-data-objects like :) No worries on keeping yours separate - I was just referencing your reinventing the wheel statement, just very late :) , >> aConnection = >> pdo.connect("module=someModule;user=foobar;passwd=mypass;db=myDB;host=localhost") > > > I don't plan to modify connection parameters. The thought being that > when you switch databases, you have to change this anyway. We found that we almost had to - some modules accept their arguments by name, and can use the **kwargs trick, some don't and thus can't :( Except for the name of the primary arguments (module, user, passwd, db), we don't modify the args in any way versus the underlying module - and even then we're just changing the names. >> - exceptions > > The popular opinion was to use Kevin Jacob's idea: > http://mail.python.org/pipermail/db-sig/2003-April/003345.html That looks good - for PDO, we didn't need to keep the original API's exception names, hence our wrapper method. >> Do you plan on editable resultsets in your module? > > Definitely not. To much work. I know the feeling :) I'm back around to where it's not so bad now - but that was after ripping things out and restructuring things, adding query objects, etc - so the solution isn't so bad, but the work to get to the sane solution and get everything needed for it to not be insane was a bear. Note to Marc: the clarifications as to the descriptor fields and their meanings we discussed on the python list a long time ago was a big help - but those clarifications are still outstanding in the spec itself. Should I submit a doc patch? > I'm mainly programming to API 2. Config files are used to adapt > modules that do not completely comply. In theory, a module that is > API2 compliant needs no config. We're pretty much at the same point. A word of warning - watch out for assumed behaviors. For example, some modules return the number of rows affected by an execute as the return value of the execute; Others do not. The standard just states that return values are undefined - hence the ambiguity and different behavior. > Others may think there is too much duplication. In that case, I'll > use it, but not publish/maintain it. Though, as judged from the list, > there seems to be interest. So my current plans are to put it in > Sourceforge CVS. Sounds like a plan, good luck with it, and I'll await the posting of the code! ~Jon Franz NeuroKode Labs, LLC From plblists at iotk.com Sun Aug 22 14:36:54 2004 From: plblists at iotk.com (Peter L. Buschman) Date: Sun Aug 22 14:37:10 2004 Subject: [DB-SIG] Re: db module wrapper In-Reply-To: <412810AD.4010609@tnr.cc> References: <412801B7.401@neurokode.com> <412810AD.4010609@tnr.cc> Message-ID: <6.0.1.1.2.20040822142802.02ff3cc8@127.0.0.1> Hi Randall: I've been following your discussion avidly. I've been working gradually on the same thing, a DBAPI 2.0 compliant wrapper to abstract the differences between drivers as part of a larger database abstraction layer I've been working on. The final project will have lots more intelligence about SQL dialects and the capabilities / supported features of individual RDBMS, but the initial step is just to get this small piece working. I was wondering how far you'd come with paramstyle conversion? I've attached some of my conversion code that allows me to convert from any paramstyle to any other paramstyle, but it does not yet support escaped placeholders. Once I have this working, I plan to incorporate Kevin Jacob's exception wrapping trick to create a thin layer on top of any DBAPI 2.0 module I choose to use. The final piece necessary is to abstract the exception arguments of different drivers so they are returned consistently. Kindest Regards, Peter Buschman At 05:19 AM 8/22/2004, Randall Smith wrote: >Jonathan M. Franz wrote: >>Well, I'm late to this conversation - so I apologize if my points have >>already been made. >>What Randal is doing is very similar to what we're working towards for >>PDO 2.0 - PDO _is_ actively maintained and promoted, btw. Heck, back in >Developers are certainly spread thin and when I saw your post, my first >thought was maybe I should drop what I'm doing and use PDO. It is similar >in that they both try to hide differences in underlying modules. There is >a difference in the scopes of the projects. PDO is ADO for Python. The >scope of my project is to maintain a DB API V2 interface while hiding >module differences. Although we do have some duplicated effort, there are >several differences. I just want my DB API2 code to work with all >modules. You want an ADO interface. >>the 1.0.2 days we even were thrown on some CDs that linux enterprise >>magazine in europe distributed. Frankly, we've always been surprised by >>the cold reception we've gotten on the db-sig mailing list whenever we >>raise a concern or mention our module. Perhaps we've been too >>argumentative about things we feel the dbapi is lacking? >>Some of the arguments earlier this year when Guido spoke up were things >>we took very seriously - but the conversation about what todo for dbapi >>3.0 died for no apparent (to me) reason. >>Anyway, some thoughts on ways we do things now in pdo, and how they >>intersect with Randall's ideas: >>(cross pollination of ideas is always a good thing) >>- connection strings >>We use a single string to hold connection data, with the core parameters >>standardized (dbapi-module,username, password, database), and the rest >>given in a module specific way, but all formated into a single string thusly: >>aConnection = >>pdo.connect("module=someModule;user=foobar;passwd=mypass;db=myDB;host=localhost") >> >I don't plan to modify connection parameters. The thought being that when >you switch databases, you have to change this anyway. >>This should be _very_ familar to ADO users. >>- parameter formating >>Currently we just use the underlying dbapi module's quoting method - but >>for 2.0 (which we hope to beta soon!) we're moving towards a :named >>style, using a modified form of the recipe for style unification at >>ASPN. http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/278612 >>We'd considered a positional ('?') style, but found it too limiting. >>The first time you do a insert or update with 20 ?s in it, you get the >>feeling that it is rather awkward. >>Of course, you could allow the user to switch styles on the fly - it'd >>just be an extra set of conversion functions. >When discussing this, people had different opinions. Therefore, I'm >enabling all 5 paramstyles. >>- exceptions >>We already have implemented a unified exception system - PDO contains >>it's own exceptions, and the textual description contains the underlying >>exception and the underlying exception's text - allowing for users to >>capture one set of exceptions and yet giving them all the detail they >>need to find out why the dbapi module. It's easy to implement, just take >>a look and steal away. >The popular opinion was to use Kevin Jacob's idea: >http://mail.python.org/pipermail/db-sig/2003-April/003345.html > >I say popular. Ian and Kevin suggested it. >>============== >>Things to ponder: >>One thing that made my life rather painful when working on PDO 2.0 was >>that the underlying type systems in dbapi modules are so different. The >>implementation of editable resultsets (no more update queries!) required >>some interesting juggling to bring the underlying types into the current >>namespace in a usable way. >Yea. That does sound difficult. Editable resultsets are nice. >>Do you plan on editable resultsets in your module? >Definitely not. To much work. I'm mainly programming to API 2. Config >files are used to adapt modules that do not completely comply. In theory, >a module that is API2 compliant needs no config. >>More soon as I catch up further with the thread. >>~Jon Franz >>NeuroKode Labs, LLC >> >>_______________________________________________ >>DB-SIG maillist - DB-SIG@python.org >>http://mail.python.org/mailman/listinfo/db-sig > >I'm all about not duplicating effort. I'm not a glutton for needless >work. Whether or not my project is made public and maintained depends on >public interest. Are our projects similar enough that I should drop >mine? I don't know. I'm not looking for an ADO interface, so I plan on >using what I have created. Others may think there is too much >duplication. In that case, I'll use it, but not publish/maintain it. >Though, as judged from the list, there seems to be interest. So my >current plans are to put it in Sourceforge CVS. > >Randall >_______________________________________________ >DB-SIG maillist - DB-SIG@python.org >http://mail.python.org/mailman/listinfo/db-sig > -------------- next part -------------- import string import re PLACEHOLDER_TOKENS = { 'qmark' : '?', 'format' : '%s', } PLACEHOLDER_EXPS = { 'qmark' : re.compile('(\?)'), 'numeric' : re.compile('(:\d+)'), 'named' : re.compile('(:\w+)'), 'format' : re.compile('(%s)'), 'pyformat' : re.compile('(%\(\w+\)s)'), } PARAMNAME_EXPS = { 'named' : re.compile(':(\w+)'), 'pyformat' : re.compile('%\((\w+)\)s'), } PLACEHOLDER_SUBS = { 'qmark' : lambda param_number : '?', 'numeric' : lambda param_number : ':%d' % (param_number), 'named' : lambda param_number : ':param%d' % (param_number), 'format' : lambda param_number : '%s', 'pyformat' : lambda param_number : '%%(param%d)s' % (param_number), } PARAMSTYLES = { 'all' : [ 'qmark', 'numeric', 'named', 'format', 'pyformat' ], 'sequence' : [ 'qmark', 'numeric', 'format' ], 'dict' : [ 'named', 'pyformat' ], 'token' : [ 'qmark', 'format' ], } ## ## ## def token_to_token( from_paramstyle, to_paramstyle, query, params ): from_token = PLACEHOLDER_TOKENS[from_paramstyle] to_token = PLACEHOLDER_TOKENS[to_paramstyle] new_params = params new_query = string.replace(query, from_token, to_token) return new_query, new_params ## ## ## def sequence_to_sequence( from_paramstyle, to_paramstyle, query, params ): placeholder_exp = PLACEHOLDER_EXPS[from_paramstyle] placeholder_sub = PLACEHOLDER_SUBS[to_paramstyle] new_query = query new_params = params param_num = 0 for placeholder in placeholder_exp.findall(query): param_num += 1 new_placeholder = placeholder_sub(param_num) new_query = string.replace(new_query, placeholder, new_placeholder, 1) return new_query, new_params ## ## ## def sequence_to_dict( from_paramstyle, to_paramstyle, query, params ): placeholder_exp = PLACEHOLDER_EXPS[from_paramstyle] placeholder_sub = PLACEHOLDER_SUBS[to_paramstyle] new_query = query new_params = {} param_num = 0 for placeholder in placeholder_exp.findall(query): param_num += 1 param_name = 'param%d' % (param_num) new_placeholder = placeholder_sub(param_num) new_query = string.replace(new_query, placeholder, new_placeholder, 1) new_params[param_name] = params[param_num-1] return new_query, new_params ## ## ## def dict_to_sequence( from_paramstyle, to_paramstyle, query, params ): placeholder_exp = PLACEHOLDER_EXPS[from_paramstyle] placeholder_sub = PLACEHOLDER_SUBS[to_paramstyle] paramname_exp = PARAMNAME_EXPS[from_paramstyle] new_query = query new_params = [] param_num = 0 for placeholder in placeholder_exp.findall(query): param_num += 1 param_name = paramname_exp.findall(placeholder)[0] new_placeholder = placeholder_sub(param_num) new_query = string.replace(new_query, placeholder, new_placeholder, 1) new_params.append(params[param_name]) return new_query, new_params #def paramstyle_convert( from_paramstyle, to_paramstyle, query, params ): # placeholder_exp = placeholder_exps[from_paramstyle] # placeholder_sub = placeholder_subs[to_paramstyle] # new_query = query # if from_paramstyle in seq_paramstyles: # from_is_sequence = 1 # from_is_dict = 0 # else: # from_is_sequence = 0 # from_is_dict = 1 # if to_paramstyle in seq_paramstyles: # to_is_sequence = 1 # to_is_dict = 0 # new_params = [] # else: # to_is_sequence = 0 # to_is_dict = 1 # new_params = {} # param_number = 0 # for placeholder in placeholder_exp.findall(query): # param_number += 1 # new_placeholder = placeholder_sub(param_number) # if from_is_dict: # paramname_exp = paramname_exps[to_paramstyle] # paramname = paramname_exp.findall(placeholder) if __name__ == '__main__': sequence_params = ['a', 'b', 'c', 'd'] dict_params = { 'foo' : 'a', 'bar' : 'b', 'baz' : 'c', 'quux' : 'd', } tests = { 'qmark' : [ 'SELECT * FROM ? WHERE ? > ? OR ? IS NOT NULL', sequence_params ], 'numeric' : [ 'SELECT * FROM :1 WHERE :2 > :3 OR :4 IS NOT NULL', sequence_params ], 'named' : [ 'SELECT * FROM :foo WHERE :bar > :baz OR :quux IS NOT NULL', dict_params ], 'format' : [ 'SELECT * FROM %s WHERE %s > %s OR %s IS NOT NULL', sequence_params ], 'pyformat' : [ 'SELECT * FROM %(foo)s WHERE %(bar)s > %(baz)s OR %(quux)s IS NOT NULL', dict_params ], } print '' print '[ PLACEHOLDER MATCH TESTS ]' print '' for paramstyle in tests.keys(): query = tests[paramstyle][0] regexp = PLACEHOLDER_EXPS[paramstyle] print regexp.findall(query) indent = 4 width = 16 print '' print '[ TOKEN TO TOKEN TESTS ]' print '' for from_paramstyle in PARAMSTYLES['token']: from_query = tests[from_paramstyle][0] from_params = tests[from_paramstyle][1] print "%s[ %s ] '%s' '%s'" % ( ' ' * indent, from_paramstyle, from_query, from_params ) print '' for to_paramstyle in PARAMSTYLES['token']: to_query, to_params = sequence_to_sequence(from_paramstyle, to_paramstyle, from_query, from_params) print '%s%s%s: %s' % ( ' ' * indent * 2, to_paramstyle, '.' * (width + indent - len(to_paramstyle)), to_query ) print '%s%s: %s' % ( ' ' * indent * 2, ' ' * (width + indent), to_params ) print '' print '' print '[ SEQUENCE TO SEQUENCE TESTS ]' print '' for from_paramstyle in PARAMSTYLES['sequence']: from_query = tests[from_paramstyle][0] from_params = tests[from_paramstyle][1] print "%s[ %s ] '%s' '%s'" % ( ' ' * indent, from_paramstyle, from_query, from_params ) print '' for to_paramstyle in PARAMSTYLES['sequence']: to_query, to_params = sequence_to_sequence(from_paramstyle, to_paramstyle, from_query, from_params) print '%s%s%s: %s' % ( ' ' * indent * 2, to_paramstyle, '.' * (width + indent - len(to_paramstyle)), to_query ) print '%s%s: %s' % ( ' ' * indent * 2, ' ' * (width + indent), to_params ) print '' print '' print '[ SEQUENCE TO DICT TESTS ]' print '' for from_paramstyle in PARAMSTYLES['sequence']: from_query = tests[from_paramstyle][0] from_params = tests[from_paramstyle][1] print "%s[ %s ] '%s' '%s'" % ( ' ' * indent, from_paramstyle, from_query, from_params ) print '' for to_paramstyle in PARAMSTYLES['dict']: to_query, to_params = sequence_to_dict(from_paramstyle, to_paramstyle, from_query, from_params) print '%s%s%s: %s' % ( ' ' * indent * 2, to_paramstyle, '.' * (width + indent - len(to_paramstyle)), to_query ) print '%s%s: %s' % ( ' ' * indent * 2, ' ' * (width + indent), to_params ) print '' print '' print '[ DICT TO SEQUENCE TESTS ]' print '' for from_paramstyle in PARAMSTYLES['dict']: from_query = tests[from_paramstyle][0] from_params = tests[from_paramstyle][1] print "%s[ %s ] '%s' '%s'" % ( ' ' * indent, from_paramstyle, from_query, from_params ) print '' for to_paramstyle in PARAMSTYLES['sequence']: to_query, to_params = dict_to_sequence(from_paramstyle, to_paramstyle, from_query, from_params) print '%s%s%s: %s' % ( ' ' * indent * 2, to_paramstyle, '.' * (width + indent - len(to_paramstyle)), to_query ) print '%s%s: %s' % ( ' ' * indent * 2, ' ' * (width + indent), to_params ) print '' # print '' # print '[ PARAMSTYLE TRANSLATIONS ]' # for paramstyle, query, params in tests: # print '' # from_paramstyle = paramstyle # print '%s[ %s ]' % (' ' * indent, from_paramstyle.upper()) # print '' # label = 'query' # print '%s%s%s: %s' % (' ' * indent, label, '.' * (width + indent - len(label)), query) # label = 'paramstyle' # print '%s%s%s: %s' % (' ' * indent, label, '.' * (width + indent - len(label)), paramstyle) # print '' # for to_paramstyle in paramstyles: # paramstyle_convert(from_paramstyle, to_paramstyle, query, params) # translate_paramstyle = paramstyle_translations[from_paramstyle][to_paramstyle] # translated_query, translated_params = translate_paramstyle(query, *params) # label = '%s_query' % (to_paramstyle) # print '%s%s%s: %s' % (' ' * indent * 2, label, '.' * (width - len(label)), translated_query) # label = '%s_params' % (to_paramstyle) # print '%s%s%s: %s' % (' ' * indent * 2, label, '.' * (width - len(label)), translated_params) # print '' From randall at tnr.cc Sun Aug 22 20:05:33 2004 From: randall at tnr.cc (Randall Smith) Date: Sun Aug 22 20:05:39 2004 Subject: [DB-SIG] Re: db module wrapper In-Reply-To: <6.0.1.1.2.20040822142802.02ff3cc8@127.0.0.1> References: <412801B7.401@neurokode.com> <412810AD.4010609@tnr.cc> <6.0.1.1.2.20040822142802.02ff3cc8@127.0.0.1> Message-ID: <4128E06D.5020007@tnr.cc> Peter L. Buschman wrote: > > Hi Randall: > > I've been following your discussion avidly. I've been working gradually > on the same thing, a DBAPI 2.0 compliant wrapper to abstract the > differences between drivers as part of a larger database abstraction > layer I've been working on. The final project will have lots more > intelligence > about SQL dialects and the capabilities / supported features of > individual RDBMS, but the initial step is just to get this small piece > working. > > I was wondering how far you'd come with paramstyle conversion? I've I've got all 5 paramstyles working with escaping. However, it is an ugly solution. Yours is much nicer. Look in the QueryFormatter class of the attachment I send you. > attached some of my conversion code that allows me to convert from > any paramstyle to any other paramstyle, but it does not yet support > escaped placeholders. Once I have this working, I plan to incorporate > Kevin Jacob's exception wrapping trick to create a thin layer on top of Did this. It is nice. > any DBAPI 2.0 module I choose to use. The final piece necessary is > to abstract the exception arguments of different drivers so they are > returned consistently. I asked about this on this list. Nobody seemed interested. It would be nice to have a list of which modules raise which exceptions for which events. > > Kindest Regards, > > Peter Buschman Peter. Looks like we have the same objective except for the 'intelligence about SQL dialects'. I have no intention of interpreting SQL. I think it would make allot of sense for us to join forces. Case in point: I spent many hours yesterday on paramstyle conversion. Do you have a project website or CVS? Project name? I've got CVS on Sourceforge. If we decide to join code we could use yours or my CVS. If not, we should at least share code and ideas. I'll send you an attachment in another email with my work. This conversation should be on DB-SIG, so I'm not attaching it here. > > > > At 05:19 AM 8/22/2004, Randall Smith wrote: > > >> Jonathan M. Franz wrote: >> >>> Well, I'm late to this conversation - so I apologize if my points >>> have already been made. >>> What Randal is doing is very similar to what we're working towards >>> for PDO 2.0 - PDO _is_ actively maintained and promoted, btw. Heck, >>> back in >> >> Developers are certainly spread thin and when I saw your post, my >> first thought was maybe I should drop what I'm doing and use PDO. It >> is similar in that they both try to hide differences in underlying >> modules. There is a difference in the scopes of the projects. PDO is >> ADO for Python. The scope of my project is to maintain a DB API V2 >> interface while hiding module differences. Although we do have some >> duplicated effort, there are several differences. I just want my DB >> API2 code to work with all modules. You want an ADO interface. >> >>> the 1.0.2 days we even were thrown on some CDs that linux enterprise >>> magazine in europe distributed. Frankly, we've always been surprised >>> by the cold reception we've gotten on the db-sig mailing list >>> whenever we raise a concern or mention our module. Perhaps we've >>> been too argumentative about things we feel the dbapi is lacking? >>> Some of the arguments earlier this year when Guido spoke up were >>> things we took very seriously - but the conversation about what todo >>> for dbapi 3.0 died for no apparent (to me) reason. >>> Anyway, some thoughts on ways we do things now in pdo, and how they >>> intersect with Randall's ideas: >>> (cross pollination of ideas is always a good thing) >>> - connection strings >>> We use a single string to hold connection data, with the core >>> parameters standardized (dbapi-module,username, password, database), >>> and the rest given in a module specific way, but all formated into a >>> single string thusly: >>> aConnection = >>> pdo.connect("module=someModule;user=foobar;passwd=mypass;db=myDB;host=localhost") >>> >> >> I don't plan to modify connection parameters. The thought being that >> when you switch databases, you have to change this anyway. >> >>> This should be _very_ familar to ADO users. >>> - parameter formating >>> Currently we just use the underlying dbapi module's quoting method - >>> but for 2.0 (which we hope to beta soon!) we're moving towards a >>> :named style, using a modified form of the recipe for style >>> unification at ASPN. >>> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/278612 >>> We'd considered a positional ('?') style, but found it too limiting. >>> The first time you do a insert or update with 20 ?s in it, you get >>> the feeling that it is rather awkward. >>> Of course, you could allow the user to switch styles on the fly - >>> it'd just be an extra set of conversion functions. >> >> When discussing this, people had different opinions. Therefore, I'm >> enabling all 5 paramstyles. >> >>> - exceptions >>> We already have implemented a unified exception system - PDO contains >>> it's own exceptions, and the textual description contains the >>> underlying exception and the underlying exception's text - allowing >>> for users to capture one set of exceptions and yet giving them all >>> the detail they need to find out why the dbapi module. It's easy to >>> implement, just take a look and steal away. >> >> The popular opinion was to use Kevin Jacob's idea: >> http://mail.python.org/pipermail/db-sig/2003-April/003345.html >> >> I say popular. Ian and Kevin suggested it. >> >>> ============== >>> Things to ponder: >>> One thing that made my life rather painful when working on PDO 2.0 >>> was that the underlying type systems in dbapi modules are so >>> different. The implementation of editable resultsets (no more update >>> queries!) required some interesting juggling to bring the underlying >>> types into the current namespace in a usable way. >> >> Yea. That does sound difficult. Editable resultsets are nice. >> >>> Do you plan on editable resultsets in your module? >> >> Definitely not. To much work. I'm mainly programming to API 2. >> Config files are used to adapt modules that do not completely comply. >> In theory, a module that is API2 compliant needs no config. >> >>> More soon as I catch up further with the thread. >>> ~Jon Franz >>> NeuroKode Labs, LLC >>> >>> _______________________________________________ >>> DB-SIG maillist - DB-SIG@python.org >>> http://mail.python.org/mailman/listinfo/db-sig >> >> >> I'm all about not duplicating effort. I'm not a glutton for needless >> work. Whether or not my project is made public and maintained depends >> on public interest. Are our projects similar enough that I should >> drop mine? I don't know. I'm not looking for an ADO interface, so I >> plan on using what I have created. Others may think there is too much >> duplication. In that case, I'll use it, but not publish/maintain it. >> Though, as judged from the list, there seems to be interest. So my >> current plans are to put it in Sourceforge CVS. >> >> Randall >> _______________________________________________ >> DB-SIG maillist - DB-SIG@python.org >> http://mail.python.org/mailman/listinfo/db-sig >> > > ------------------------------------------------------------------------ > > > import string > import re > > PLACEHOLDER_TOKENS = { > 'qmark' : '?', > 'format' : '%s', > } > > PLACEHOLDER_EXPS = { > 'qmark' : re.compile('(\?)'), > 'numeric' : re.compile('(:\d+)'), > 'named' : re.compile('(:\w+)'), > 'format' : re.compile('(%s)'), > 'pyformat' : re.compile('(%\(\w+\)s)'), > } > > PARAMNAME_EXPS = { > 'named' : re.compile(':(\w+)'), > 'pyformat' : re.compile('%\((\w+)\)s'), > } > > PLACEHOLDER_SUBS = { > 'qmark' : lambda param_number : '?', > 'numeric' : lambda param_number : ':%d' % (param_number), > 'named' : lambda param_number : ':param%d' % (param_number), > 'format' : lambda param_number : '%s', > 'pyformat' : lambda param_number : '%%(param%d)s' % (param_number), > } > > PARAMSTYLES = { > 'all' : [ 'qmark', 'numeric', 'named', 'format', 'pyformat' ], > 'sequence' : [ 'qmark', 'numeric', 'format' ], > 'dict' : [ 'named', 'pyformat' ], > 'token' : [ 'qmark', 'format' ], > } > > > ## > ## > ## > def token_to_token( from_paramstyle, to_paramstyle, query, params ): > from_token = PLACEHOLDER_TOKENS[from_paramstyle] > to_token = PLACEHOLDER_TOKENS[to_paramstyle] > new_params = params > new_query = string.replace(query, from_token, to_token) > return new_query, new_params > > > ## > ## > ## > def sequence_to_sequence( from_paramstyle, to_paramstyle, query, params ): > placeholder_exp = PLACEHOLDER_EXPS[from_paramstyle] > placeholder_sub = PLACEHOLDER_SUBS[to_paramstyle] > new_query = query > new_params = params > param_num = 0 > for placeholder in placeholder_exp.findall(query): > param_num += 1 > new_placeholder = placeholder_sub(param_num) > new_query = string.replace(new_query, placeholder, new_placeholder, 1) > return new_query, new_params > > > ## > ## > ## > def sequence_to_dict( from_paramstyle, to_paramstyle, query, params ): > placeholder_exp = PLACEHOLDER_EXPS[from_paramstyle] > placeholder_sub = PLACEHOLDER_SUBS[to_paramstyle] > new_query = query > new_params = {} > param_num = 0 > for placeholder in placeholder_exp.findall(query): > param_num += 1 > param_name = 'param%d' % (param_num) > new_placeholder = placeholder_sub(param_num) > new_query = string.replace(new_query, placeholder, new_placeholder, 1) > new_params[param_name] = params[param_num-1] > return new_query, new_params > > > ## > ## > ## > def dict_to_sequence( from_paramstyle, to_paramstyle, query, params ): > placeholder_exp = PLACEHOLDER_EXPS[from_paramstyle] > placeholder_sub = PLACEHOLDER_SUBS[to_paramstyle] > paramname_exp = PARAMNAME_EXPS[from_paramstyle] > new_query = query > new_params = [] > param_num = 0 > for placeholder in placeholder_exp.findall(query): > param_num += 1 > param_name = paramname_exp.findall(placeholder)[0] > new_placeholder = placeholder_sub(param_num) > new_query = string.replace(new_query, placeholder, new_placeholder, 1) > new_params.append(params[param_name]) > return new_query, new_params > > > #def paramstyle_convert( from_paramstyle, to_paramstyle, query, params ): > # placeholder_exp = placeholder_exps[from_paramstyle] > # placeholder_sub = placeholder_subs[to_paramstyle] > # new_query = query > # if from_paramstyle in seq_paramstyles: > # from_is_sequence = 1 > # from_is_dict = 0 > # else: > # from_is_sequence = 0 > # from_is_dict = 1 > # if to_paramstyle in seq_paramstyles: > # to_is_sequence = 1 > # to_is_dict = 0 > # new_params = [] > # else: > # to_is_sequence = 0 > # to_is_dict = 1 > # new_params = {} > # param_number = 0 > # for placeholder in placeholder_exp.findall(query): > # param_number += 1 > # new_placeholder = placeholder_sub(param_number) > # if from_is_dict: > # paramname_exp = paramname_exps[to_paramstyle] > # paramname = paramname_exp.findall(placeholder) > > if __name__ == '__main__': > sequence_params = ['a', 'b', 'c', 'd'] > dict_params = { > 'foo' : 'a', > 'bar' : 'b', > 'baz' : 'c', > 'quux' : 'd', > } > tests = { > 'qmark' : [ 'SELECT * FROM ? WHERE ? > ? OR ? IS NOT NULL', sequence_params ], > 'numeric' : [ 'SELECT * FROM :1 WHERE :2 > :3 OR :4 IS NOT NULL', sequence_params ], > 'named' : [ 'SELECT * FROM :foo WHERE :bar > :baz OR :quux IS NOT NULL', dict_params ], > 'format' : [ 'SELECT * FROM %s WHERE %s > %s OR %s IS NOT NULL', sequence_params ], > 'pyformat' : [ 'SELECT * FROM %(foo)s WHERE %(bar)s > %(baz)s OR %(quux)s IS NOT NULL', dict_params ], > } > print '' > print '[ PLACEHOLDER MATCH TESTS ]' > print '' > for paramstyle in tests.keys(): > query = tests[paramstyle][0] > regexp = PLACEHOLDER_EXPS[paramstyle] > print regexp.findall(query) > indent = 4 > width = 16 > print '' > print '[ TOKEN TO TOKEN TESTS ]' > print '' > for from_paramstyle in PARAMSTYLES['token']: > from_query = tests[from_paramstyle][0] > from_params = tests[from_paramstyle][1] > print "%s[ %s ] '%s' '%s'" % ( ' ' * indent, from_paramstyle, from_query, from_params ) > print '' > for to_paramstyle in PARAMSTYLES['token']: > to_query, to_params = sequence_to_sequence(from_paramstyle, to_paramstyle, from_query, from_params) > print '%s%s%s: %s' % ( ' ' * indent * 2, to_paramstyle, '.' * (width + indent - len(to_paramstyle)), to_query ) > print '%s%s: %s' % ( ' ' * indent * 2, ' ' * (width + indent), to_params ) > print '' > print '' > print '[ SEQUENCE TO SEQUENCE TESTS ]' > print '' > for from_paramstyle in PARAMSTYLES['sequence']: > from_query = tests[from_paramstyle][0] > from_params = tests[from_paramstyle][1] > print "%s[ %s ] '%s' '%s'" % ( ' ' * indent, from_paramstyle, from_query, from_params ) > print '' > for to_paramstyle in PARAMSTYLES['sequence']: > to_query, to_params = sequence_to_sequence(from_paramstyle, to_paramstyle, from_query, from_params) > print '%s%s%s: %s' % ( ' ' * indent * 2, to_paramstyle, '.' * (width + indent - len(to_paramstyle)), to_query ) > print '%s%s: %s' % ( ' ' * indent * 2, ' ' * (width + indent), to_params ) > print '' > print '' > print '[ SEQUENCE TO DICT TESTS ]' > print '' > for from_paramstyle in PARAMSTYLES['sequence']: > from_query = tests[from_paramstyle][0] > from_params = tests[from_paramstyle][1] > print "%s[ %s ] '%s' '%s'" % ( ' ' * indent, from_paramstyle, from_query, from_params ) > print '' > for to_paramstyle in PARAMSTYLES['dict']: > to_query, to_params = sequence_to_dict(from_paramstyle, to_paramstyle, from_query, from_params) > print '%s%s%s: %s' % ( ' ' * indent * 2, to_paramstyle, '.' * (width + indent - len(to_paramstyle)), to_query ) > print '%s%s: %s' % ( ' ' * indent * 2, ' ' * (width + indent), to_params ) > print '' > print '' > print '[ DICT TO SEQUENCE TESTS ]' > print '' > for from_paramstyle in PARAMSTYLES['dict']: > from_query = tests[from_paramstyle][0] > from_params = tests[from_paramstyle][1] > print "%s[ %s ] '%s' '%s'" % ( ' ' * indent, from_paramstyle, from_query, from_params ) > print '' > for to_paramstyle in PARAMSTYLES['sequence']: > to_query, to_params = dict_to_sequence(from_paramstyle, to_paramstyle, from_query, from_params) > print '%s%s%s: %s' % ( ' ' * indent * 2, to_paramstyle, '.' * (width + indent - len(to_paramstyle)), to_query ) > print '%s%s: %s' % ( ' ' * indent * 2, ' ' * (width + indent), to_params ) > print '' > > # print '' > # print '[ PARAMSTYLE TRANSLATIONS ]' > # for paramstyle, query, params in tests: > # print '' > # from_paramstyle = paramstyle > # print '%s[ %s ]' % (' ' * indent, from_paramstyle.upper()) > # print '' > # label = 'query' > # print '%s%s%s: %s' % (' ' * indent, label, '.' * (width + indent - len(label)), query) > # label = 'paramstyle' > # print '%s%s%s: %s' % (' ' * indent, label, '.' * (width + indent - len(label)), paramstyle) > # print '' > # for to_paramstyle in paramstyles: > # paramstyle_convert(from_paramstyle, to_paramstyle, query, params) > # translate_paramstyle = paramstyle_translations[from_paramstyle][to_paramstyle] > # translated_query, translated_params = translate_paramstyle(query, *params) > # label = '%s_query' % (to_paramstyle) > # print '%s%s%s: %s' % (' ' * indent * 2, label, '.' * (width - len(label)), translated_query) > # label = '%s_params' % (to_paramstyle) > # print '%s%s%s: %s' % (' ' * indent * 2, label, '.' * (width - len(label)), translated_params) > # print '' > > From pf_moore at yahoo.co.uk Mon Aug 23 21:52:09 2004 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Mon Aug 23 21:52:04 2004 Subject: [DB-SIG] Re: db module wrapper References: <4123BA93.1040604@tnr.cc> <4123EE84.1080608@colorstudy.com> <4124ACFB.3020405@egenix.com> <4124CE07.1040606@egenix.com> <9A70CBEC-F1FF-11D8-8726-000A95A5D8B2@mitre.org> <41260BF8.80807@halfcooked.com> <8B4A5E82-F2CA-11D8-8726-000A95A5D8B2@mitre.org> Message-ID: <4qmtfx92.fsf@yahoo.co.uk> Marc Colosimo writes: > On Aug 20, 2004, at 10:34 AM, Andy Todd wrote: > >> Marc Colosimo wrote: >>> On Aug 19, 2004, at 11:57 AM, M.-A. Lemburg wrote: >>>> Marc Colosimo wrote: >>>> >> [snip] >>>> After all binding >>>> parameters exist to make all this quoting unnecessary and >>>> indeed, many drivers pass the SQL string and the parameters >>>> well separated to the database. >>> not pgdb (_quoteparams) or cx_oracle (Cursor_SetBindVariables). I >>> couldn't check psycopg because the links from freshmeat are dead. >> >> Call me Mr Picky, but that is not true for cx_Oracle. It uses a >> feature of Oracle called bind variables. For an explanation see; >> >> http://www.rittman.net/archives/000832.html >> >> The function Cursor_SetBindVariables in Cursor.c of cx_Oracle simply >> unpacks the values passed to the execute statement. Quoting and >> other esoteric conversion requirements and performed by the Oracle >> software. >> > > Okay, I was wrong about BindVariables, which does something completely > different than quoting ( I don't fully understand how cx_Oracle does > it and at this time I really don't care or have time to look into). > The way cx_Oracle handles things would throw a wrench into what I want The key point with Oracle bind variables is that you pass a SQL statement select col1, col2 from tab where col3 > :var1 and col4 between :var2 and :var3 to the database, plus a series of parameters var1, var2, var3. The database parses the SQL *once only* and reuses the parsed statement whenever it can, *even if the values of the variables change*. This is a massive performance bonus, as parsing is a serious overhead (in Oracle, at least). Think of it as SQL + bind vars == function + arguments. You compile the function once and call multiple times with different arguments. One of the biggest performance mistakes you can make when building an Oracle system is to not use bind variables. Paul. -- Accomplishing the impossible means only that the boss will add it to your regular duties. -- Doug Larson From mal at egenix.com Tue Aug 24 12:38:01 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Tue Aug 24 12:38:05 2004 Subject: [DB-SIG] Re: db module wrapper In-Reply-To: <412819A8.2050901@neurokode.com> References: <412801B7.401@neurokode.com> <412810AD.4010609@tnr.cc> <412819A8.2050901@neurokode.com> Message-ID: <412B1A89.30902@egenix.com> Jonathan M. Franz wrote: > Note to Marc: the clarifications as to the descriptor fields and their > meanings we discussed on the python list a long time ago was a big help > - but those clarifications are still outstanding in the spec itself. > Should I submit a doc patch? Yes, please. >> I'm mainly programming to API 2. Config files are used to adapt >> modules that do not completely comply. In theory, a module that is >> API2 compliant needs no config. > > > We're pretty much at the same point. A word of warning - watch out for > assumed behaviors. For example, some modules return the number of rows > affected by an execute as the return value of the execute; Others do > not. The standard just states that return values are undefined - hence > the ambiguity and different behavior. It says "Return values are not defined". You should use .rowcount to access the number of affected rows. The reason is that getting this information from the database can be an expensive operation; hence you don't want to do this for every single call to .execute(). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 24 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at egenix.com Tue Aug 24 12:42:14 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Tue Aug 24 12:42:18 2004 Subject: [DB-SIG] Unifying exceptions In-Reply-To: <412674A2.6010607@colorstudy.com> References: <4123BA93.1040604@tnr.cc> <412672D0.7060807@tnr.cc> <412674A2.6010607@colorstudy.com> Message-ID: <412B1B86.3010506@egenix.com> Ian Bicking wrote: > There might be a clever/tricky solution that would be a little easier to > implement and more reliable. > > Hmm... you could try changing the exceptions, like: > > import psycopg > psycopg.DatabaseError = MyDatabaseError > > I don't know if this will always stick, or if it's always mutable in > this way. Another way might be: > > class MyDatabaseError(Exception): > pass > psycopg.DatabaseError.__bases__ = (MyDatabaseError,) > > Then you can catch MyDatabaseError, and it will catch > psycopg.DatabaseError (since it's a subclass). This might be more > likely to stick, and work better with the C modules. (Maybe, I don't > know...?) The way I usually cope with this is to bind the module exceptions to the connection abstraction: # Assure that self.Error catches all database related errors. if hasattr(module, 'Error'): Error = module.Error elif hasattr(module, 'error'): Error = module.error else: Error = exceptions.StandardError self.Error = Error # Localize available common constants for database independence for symbol in ('InterfaceError', 'InternalError', 'OperationalError', 'Warning', 'ProgrammingError', 'IntegrityError', 'DataError', 'NotSupportedError', ): obj = getattr(module, symbol, NotGiven) if obj is not NotGiven: setattr(self, symbol, obj) elif symbol[-5:] == 'Error' or \ symbol[-7:] == 'Warning': # All DB-API 2 error classes should be available for # compatibility reasons setattr(self, symbol, Error) Works great (and is compatible to the DB API 2.0 extension for connection objects). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 24 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From randall at tnr.cc Tue Aug 24 17:20:23 2004 From: randall at tnr.cc (Randall Smith) Date: Tue Aug 24 17:20:28 2004 Subject: [DB-SIG] Re: db module wrapper In-Reply-To: <412B1A89.30902@egenix.com> References: <412801B7.401@neurokode.com> <412810AD.4010609@tnr.cc> <412819A8.2050901@neurokode.com> <412B1A89.30902@egenix.com> Message-ID: <412B5CB7.8050203@tnr.cc> M.-A. Lemburg wrote: > Jonathan M. Franz wrote: > >> Note to Marc: the clarifications as to the descriptor fields and their >> meanings we discussed on the python list a long time ago was a big >> help - but those clarifications are still outstanding in the spec >> itself. Should I submit a doc patch? > > > Yes, please. > >>> I'm mainly programming to API 2. Config files are used to adapt >>> modules that do not completely comply. In theory, a module that is >>> API2 compliant needs no config. >> >> >> >> We're pretty much at the same point. A word of warning - watch out >> for assumed behaviors. For example, some modules return the number of >> rows affected by an execute as the return value of the execute; Others >> do not. The standard just states that return values are undefined - >> hence the ambiguity and different behavior. > > > It says "Return values are not defined". You should use .rowcount to > access the number of affected rows. The reason is that getting this > information from the database can be an expensive operation; hence > you don't want to do this for every single call to .execute(). > Unless someone suggests otherwise, the wrapper's execute method will return None. Randall From floydophone at gmail.com Wed Aug 25 06:15:27 2004 From: floydophone at gmail.com (Peter Hunt) Date: Wed Aug 25 06:15:31 2004 Subject: [DB-SIG] My SQLString class Message-ID: <6654eac404082421152ed69df1@mail.gmail.com> Here's a simple class which escapes SQL parameters as a format string, perhaps someone can improve upon it or use it? -------------- next part -------------- import types class SQLString(types.StringType): ESCAPE_CHARS = r"""'\"""" def _escape_string(self, s): for c in self.ESCAPE_CHARS: s = s.replace(c, "\\" + c) return s def __mod__(self, value_list): if type(value_list) == types.ListType or type(value_list) == types.TupleType: params = () for value in value_list: if type(value) in types.StringTypes: value = "'%s'" % self._escape_string(value) params = params + (value,) return str.__mod__(self,params) elif type(value_list) == types.DictType: for k in value_list: value = value_list[k] if type(value) in types.StringTypes: value_list[k] = "\"%s\"" % self._escape_string(value) return str.__mod__(self,value_list) elif type(value_list) in types.StringTypes: return str.__mod__(self,"\"%s\"" % self._escape_string(value_list)) else: return str.__mod__(self,value_list) if __name__ == "__main__": s = SQLString("you %s really %s times %d....!") print s % ("are","'cool'",10000) From clarion at awalnet.net.sa Fri Aug 27 01:14:24 2004 From: clarion at awalnet.net.sa (Clarion) Date: Fri Aug 27 08:58:36 2004 Subject: [DB-SIG] Table Oriented Programming Message-ID: <005e01c48bc2$706574b0$1500000a@D1LV760J> >As to your actual question, I have not seen any publications about table >oriented programming for several years. Bruce Johnson, to whom I am also >addressing this, is the guru of RDM. Perhaps he has seen or produced >something recently. [Bruce: are you listening?] Hi, If you are referring to the same person that I know, then he is quite active in database applications development. His website is at http://www.capesoft.com Regards, Ubaidullah Nubar. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/db-sig/attachments/20040827/f0197de4/attachment.htm From randall at tnr.cc Sun Aug 29 21:53:22 2004 From: randall at tnr.cc (Randall Smith) Date: Sun Aug 29 21:53:26 2004 Subject: [DB-SIG] db wrapper naming Message-ID: <41323432.8000704@tnr.cc> Since our discussion here about a dbapi2 wrapper, Peter Hunt and I have been working on it. We have it mostly working with the ability to work with Python datetime and mx.DateTime types and changeable paramstyles. We have been discussing names and are leaning on dal, an acronym for Database Abstraction Layer. This package will have a subpackage named dbapi that will contain all of our current work. Future work outside of the dbapi wrapper scope will reside in other packages within dal. I'm posting here to get opinions/suggestions about these naming conventions so that we can identify any problems that may arise from it. We would appreciate your opinions and suggestions. Randall From mal at egenix.com Mon Aug 30 21:50:50 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Mon Aug 30 21:50:54 2004 Subject: [DB-SIG] db wrapper naming In-Reply-To: <41323432.8000704@tnr.cc> References: <41323432.8000704@tnr.cc> Message-ID: <4133851A.9080808@egenix.com> Randall Smith wrote: > Since our discussion here about a dbapi2 wrapper, Peter Hunt and I have > been working on it. We have it mostly working with the ability to work > with Python datetime and mx.DateTime types and changeable paramstyles. > We have been discussing names and are leaning on dal, an acronym for > Database Abstraction Layer. This package will have a subpackage named > dbapi that will contain all of our current work. Future work outside of > the dbapi wrapper scope will reside in other packages within dal. > > I'm posting here to get opinions/suggestions about these naming > conventions so that we can identify any problems that may arise from it. > We would appreciate your opinions and suggestions. PyDBI ? After all, what you're doing here is rather similar to what the Perl folks did for their DBI interface. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 30 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From wnvcole at peppermillcas.com Mon Aug 30 22:47:51 2004 From: wnvcole at peppermillcas.com (Vernon Cole) Date: Mon Aug 30 22:48:00 2004 Subject: [DB-SIG] db wrapper naming Message-ID: <1DE30EB3ECE266409FDE6E7032F178C4139803@pcimail1s.peppermillcas.com> I second the motion. The computer business is too full of TLA's. (Three letter acronyms). Any TLA is bound to conflict with some other TLA. PyDBI is a name that even I would be able to remember. -------- Vernon -----Original Message----- From: db-sig-bounces@python.org [mailto:db-sig-bounces@python.org]On Behalf Of M.-A. Lemburg Sent: Monday, August 30, 2004 1:51 PM To: Randall Smith Cc: Python DB-SIG Mailing List Subject: Re: [DB-SIG] db wrapper naming Randall Smith wrote: > Since our discussion here about a dbapi2 wrapper, Peter Hunt and I have > been working on it. We have it mostly working with the ability to work > with Python datetime and mx.DateTime types and changeable paramstyles. > We have been discussing names and are leaning on dal, an acronym for > Database Abstraction Layer. This package will have a subpackage named > dbapi that will contain all of our current work. Future work outside of > the dbapi wrapper scope will reside in other packages within dal. > > I'm posting here to get opinions/suggestions about these naming > conventions so that we can identify any problems that may arise from it. > We would appreciate your opinions and suggestions. PyDBI ? After all, what you're doing here is rather similar to what the Perl folks did for their DBI interface. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 30 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________ DB-SIG maillist - DB-SIG@python.org http://mail.python.org/mailman/listinfo/db-sig From ianb at colorstudy.com Mon Aug 30 22:54:33 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Mon Aug 30 22:54:57 2004 Subject: [DB-SIG] db wrapper naming In-Reply-To: <4133851A.9080808@egenix.com> References: <41323432.8000704@tnr.cc> <4133851A.9080808@egenix.com> Message-ID: <41339409.7060102@colorstudy.com> M.-A. Lemburg wrote: > Randall Smith wrote: > >> Since our discussion here about a dbapi2 wrapper, Peter Hunt and I >> have been working on it. We have it mostly working with the ability >> to work with Python datetime and mx.DateTime types and changeable >> paramstyles. We have been discussing names and are leaning on dal, an >> acronym for Database Abstraction Layer. This package will have a >> subpackage named dbapi that will contain all of our current work. >> Future work outside of the dbapi wrapper scope will reside in other >> packages within dal. >> >> I'm posting here to get opinions/suggestions about these naming >> conventions so that we can identify any problems that may arise from >> it. We would appreciate your opinions and suggestions. > > > PyDBI ? > > After all, what you're doing here is rather similar to what > the Perl folks did for their DBI interface. Why the "Py"? Isn't it kind of implied? I like dbi, though. Maybe the project name would be PyDBI, with the module name being dbi? -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From mal at egenix.com Tue Aug 31 10:25:04 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Tue Aug 31 10:25:06 2004 Subject: [DB-SIG] db wrapper naming In-Reply-To: <41339409.7060102@colorstudy.com> References: <41323432.8000704@tnr.cc> <4133851A.9080808@egenix.com> <41339409.7060102@colorstudy.com> Message-ID: <413435E0.8050105@egenix.com> Ian Bicking wrote: > M.-A. Lemburg wrote: > >> Randall Smith wrote: >> >>> Since our discussion here about a dbapi2 wrapper, Peter Hunt and I >>> have been working on it. We have it mostly working with the ability >>> to work with Python datetime and mx.DateTime types and changeable >>> paramstyles. We have been discussing names and are leaning on dal, an >>> acronym for Database Abstraction Layer. This package will have a >>> subpackage named dbapi that will contain all of our current work. >>> Future work outside of the dbapi wrapper scope will reside in other >>> packages within dal. >>> >>> I'm posting here to get opinions/suggestions about these naming >>> conventions so that we can identify any problems that may arise from >>> it. We would appreciate your opinions and suggestions. >> >> >> >> PyDBI ? >> >> After all, what you're doing here is rather similar to what >> the Perl folks did for their DBI interface. > > > Why the "Py"? Isn't it kind of implied? I like dbi, though. Maybe the > project name would be PyDBI, with the module name being dbi? The name "dbi" is already taken (by DB API 1.0 which required such a module, see e.g. the win32 odbc module). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 31 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From jfranz at neurokode.com Tue Aug 31 19:32:02 2004 From: jfranz at neurokode.com (jfranz@neurokode.com) Date: Tue Aug 31 19:32:13 2004 Subject: [DB-SIG] RE: module name Message-ID: <3554.66.144.8.194.1093973522.squirrel@www.neurokode.com> I don't think pyDBI is a good name - it implies that its a library with the same api as the perl and/or ruby DBI - which, it doesn't do. The abstraction of the 'driver' module is similar - but thats about it. Just my two cents. *shrug* ~Jon Franz NeuroKode Labs, LLC