From anthra.norell at tiscalinet.ch Sun Jun 1 00:36:30 2003 From: anthra.norell at tiscalinet.ch (Anthra Norell) Date: Sat May 31 17:39:04 2003 Subject: [DB-SIG] MySQL doc Message-ID: <001301c327bc$b31c0a00$0201a8c0@mcuf7> I downloaded 0.9.2 from the Source Forge and managed to install it on Windows ME., but have no clue what to do next. I was more successful with an image library that comes with a manual. I would greatly appreciate any hint on where I can find docs, manuals, etc. (MySQL-python-0.9.2.win32-py2.2.exe) Frederic -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/db-sig/attachments/20030531/1478a314/attachment.htm From gh at ghaering.de Sun Jun 1 05:16:11 2003 From: gh at ghaering.de (Gerhard =?unknown-8bit?Q?H=E4ring?=) Date: Sat May 31 22:16:17 2003 Subject: [DB-SIG] MySQL doc In-Reply-To: <001301c327bc$b31c0a00$0201a8c0@mcuf7> References: <001301c327bc$b31c0a00$0201a8c0@mcuf7> Message-ID: <20030601021611.GB7831@mephisto.ghaering.de> * Anthra Norell [2003-05-31 23:36 +0200]: > I downloaded 0.9.2 from the Source Forge and managed to install it on > Windows ME., but have no clue what to do next. I was more successful > with an image library that comes with a manual. I would greatly > appreciate any hint on where I can find docs, manuals, etc. > (MySQL-python-0.9.2.win32-py2.2.exe) Download the source distribution as well. It contains the docs. Gerhard -- http://ghaering.de/ From paul at dubois.ws Sun Jun 1 12:50:17 2003 From: paul at dubois.ws (Paul DuBois) Date: Sun Jun 1 12:50:24 2003 Subject: [DB-SIG] MySQL doc In-Reply-To: <20030601021611.GB7831@mephisto.ghaering.de> References: <001301c327bc$b31c0a00$0201a8c0@mcuf7> <20030601021611.GB7831@mephisto.ghaering.de> Message-ID: At 4:16 +0200 6/1/03, Gerhard =?unknown-8bit?Q?H=E4ring?= wrote: >* Anthra Norell [2003-05-31 23:36 +0200]: >> I downloaded 0.9.2 from the Source Forge and managed to install it on >> Windows ME., but have no clue what to do next. I was more successful >> with an image library that comes with a manual. I would greatly >> appreciate any hint on where I can find docs, manuals, etc. >> (MySQL-python-0.9.2.win32-py2.2.exe) > >Download the source distribution as well. It contains the docs. > >Gerhard >-- >http://ghaering.de/ If you happen to have MySQL Cookbook, it covers DB-API. Even if you don't have the book, you can get the software distribution that accompanies it, which has a number of example scripts that use DB-API. http://www.kitebird.com/mysql-cookbook/ Look under the "api" directory of the recipes distribution for the most basic scripts that use DB-API. Disclosure: I wrote the book. From fog at initd.org Mon Jun 2 18:42:42 2003 From: fog at initd.org (Federico Di Gregorio) Date: Mon Jun 2 13:42:43 2003 Subject: [DB-SIG] Python/PostgreSQL API performance comparison In-Reply-To: <9E2B242C-8E21-11D7-9E89-000393A6D442@cogdon.org> References: <9E2B242C-8E21-11D7-9E89-000393A6D442@cogdon.org> Message-ID: <1054575754.22312.28.camel@localhost> Il sab, 2003-05-24 alle 21:55, Chris Cogdon ha scritto: > Hey folks! This post is more a 'for your edification' than a call for > comments, but comments are welcome :) > > I'm in the process of rewriting all the database glue logic for my > rather heavily used[1] website. In the process I discovered that the > new code runs significantly slower than the old, and that worries me > since the website IS rather heavily used, and I was hoping to move to > dynamic generation for a lot of the pages. I believe a good deal of the > slowdown was my overuse of 'elegant but slow' coding, and I intend to > remedy that. But, I also decided how much was due to the change from > the old 'pg' API to the dbapi-2.0 compliant PgSQL. > > I wrote a little program that sends through a couple of complex queries > to the DB and retrieves the values using a variety of API's: > > - D'Arcy's 'pg' module > - D'Arcy's dbapi-2.0-compliant 'pgdb' module > - PgSQL > - PgSQL with DECLARE cursor's off > - PoPy > > In all cases, I ran the query three times as a 'warm up', then another > 10 and timed that using os.times() The results are as follows: > > method [ user time, system time, child user time, child system time, > real time ] > pg [ 0.110, 0.010, 0.000, 0.000, 29.050 ] > pgdb [ 4.490, 0.010, 0.000, 0.000, 33.230 ] > PgSQL [ 3.640, 0.000, 0.000, 0.000, 40.930 ] > PgSQL (nocursor) [ 3.630, 0.010, 0.000, 0.000, 32.280 ] > PoPy [ 0.130, 0.010, 0.000, 0.000, 29.030 ] would you add psycopg to the tests, please. just eager to know how well it does. -- Federico Di Gregorio Debian GNU/Linux Developer fog@debian.org INIT.D Developer fog@initd.org Debian. The best software from the best people [see above] -- brought to you by One Line Spam -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Questa parte del messaggio =?ISO-8859-1?Q?=E8?= firmata Url : http://mail.python.org/pipermail/db-sig/attachments/20030602/08bef1fb/attachment.bin From chris at cogdon.org Mon Jun 2 12:43:12 2003 From: chris at cogdon.org (Chris Cogdon) Date: Mon Jun 2 14:43:22 2003 Subject: [DB-SIG] Python/PostgreSQL API performance comparison In-Reply-To: <1054575754.22312.28.camel@localhost> Message-ID: <107E3DAC-952A-11D7-9EC2-000393B658A2@cogdon.org> On Monday, Jun 2, 2003, at 10:42 US/Pacific, Federico Di Gregorio wrote: > would you add psycopg to the tests, please. just eager to know how well > it does. Ah, thanks for reminding me about your module :) Here's the results: pg [ 0.180, 0.010, 0.000, 0.000, 29.610 ] pgdb [ 4.570, 0.000, 0.000, 0.000, 33.520 ] PgSQL [ 4.480, 0.010, 0.000, 0.000, 42.450 ] PgSQL (nocursor) [ 4.480, 0.010, 0.000, 0.000, 33.530 ] PgSQL (nocursor,list) [ 3.620, 0.020, 0.000, 0.000, 32.570 ] PoPy [ 0.200, 0.010, 0.000, 0.000, 29.280 ] psycopg [ 0.300, 0.010, 0.000, 0.000, 29.330 ] Vigilant readers will note that the figures are a little larger than those I posted last time. A fair bit of data has been added to the database since. Yes, having a 'consistent framework' would be nice, but... I never said these were 'laboratory quality' values :) So... psycopg performs pretty well... but... I'm sure you could squeeze that 0.1 out, couldn't you? :) Reading up on the docs for psycopg, it appears it's designed to perform well under heavy-load applications with multiple, persistant connections to the database. I might see if I can design a test to shake out that particular metric, since that's very important for my application. I've also noticed that some database connectors do not handle 8-byte integer values properly. For example, a select sum(file_size) would return 2^31-1 rather than the correct, larger value (as a python bigint). Yes, I have the source, and could probably fix it myself, but it's an interesting metric nevertheless. The results are: pg/pgdb: Fails on my linux box, but works on a netbsd box. Needs investigation. PgSQL: Works PoPy: Fails psycopg: Works -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From fog at initd.org Mon Jun 2 19:48:40 2003 From: fog at initd.org (Federico Di Gregorio) Date: Mon Jun 2 14:48:41 2003 Subject: [DB-SIG] Python/PostgreSQL API performance comparison In-Reply-To: <107E3DAC-952A-11D7-9EC2-000393B658A2@cogdon.org> References: <107E3DAC-952A-11D7-9EC2-000393B658A2@cogdon.org> Message-ID: <1054579776.22312.42.camel@localhost> Il lun, 2003-06-02 alle 20:43, Chris Cogdon ha scritto: > On Monday, Jun 2, 2003, at 10:42 US/Pacific, Federico Di Gregorio wrote: > > > would you add psycopg to the tests, please. just eager to know how well > > it does. > > Ah, thanks for reminding me about your module :) Here's the results: > > pg [ 0.180, 0.010, 0.000, 0.000, 29.610 ] > pgdb [ 4.570, 0.000, 0.000, 0.000, 33.520 ] > PgSQL [ 4.480, 0.010, 0.000, 0.000, 42.450 ] > PgSQL (nocursor) [ 4.480, 0.010, 0.000, 0.000, 33.530 ] > PgSQL (nocursor,list) [ 3.620, 0.020, 0.000, 0.000, 32.570 ] > PoPy [ 0.200, 0.010, 0.000, 0.000, 29.280 ] > psycopg [ 0.300, 0.010, 0.000, 0.000, 29.330 ] > > Vigilant readers will note that the figures are a little larger than > those I posted last time. A fair bit of data has been added to the > database since. Yes, having a 'consistent framework' would be nice, > but... I never said these were 'laboratory quality' values :) > > So... psycopg performs pretty well... but... I'm sure you could squeeze > that 0.1 out, couldn't you? :) Reading up on the docs for psycopg, it i is already pretty good, considered that psycopg does a lot more stuff than popy (custom type-casters, etc.) > appears it's designed to perform well under heavy-load applications > with multiple, persistant connections to the database. I might see if I > can design a test to shake out that particular metric, since that's > very important for my application. psycopg is very aggressive on calling Py_*_ALLOW_THREADS macros so you'll get the best performance on multithreaded applications. thank you for the tests, federico -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Questa parte del messaggio =?ISO-8859-1?Q?=E8?= firmata Url : http://mail.python.org/pipermail/db-sig/attachments/20030602/586d3795/attachment.bin From chris at cogdon.org Mon Jun 2 13:00:10 2003 From: chris at cogdon.org (Chris Cogdon) Date: Mon Jun 2 15:00:15 2003 Subject: [DB-SIG] Python/PostgreSQL API performance comparison In-Reply-To: <1054579776.22312.42.camel@localhost> Message-ID: <6F4A4E7A-952C-11D7-9EC2-000393B658A2@cogdon.org> On Monday, Jun 2, 2003, at 11:49 US/Pacific, Federico Di Gregorio wrote: > thank you for the tests, You're welcome! I've added as a low-priority ToDo to create a simple framework, including a database schema and some data, so folk can replicate the tests on their own systems and send back changes for additional tests to try. This will include a query that takes very little DMBS processing, (such as a straight table select), as well as one that takes a lot (such as what I've got at the moment). I doubt there'd be any difference in the DB-API performance... but, you never know :) It really is low priority for me... but if someone really would like it, I can up the priority with the appropriate encouragements :) I'm still happy to take suggestions for changes or additions, though. If they're easy, I'll add them straight in. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From gh at ghaering.de Mon Jun 2 22:28:24 2003 From: gh at ghaering.de (=?ISO-8859-1?Q?Gerhard_H=E4ring?=) Date: Mon Jun 2 15:28:07 2003 Subject: [DB-SIG] Python/PostgreSQL API performance comparison In-Reply-To: <6F4A4E7A-952C-11D7-9EC2-000393B658A2@cogdon.org> References: <6F4A4E7A-952C-11D7-9EC2-000393B658A2@cogdon.org> Message-ID: <3EDBA558.2000800@ghaering.de> Chris Cogdon wrote: > On Monday, Jun 2, 2003, at 11:49 US/Pacific, Federico Di Gregorio wrote: > >> thank you for the tests, > > > You're welcome! I've added as a low-priority ToDo to create a simple > framework, including a database schema and some data, so folk can > replicate the tests on their own systems and send back changes for > additional tests to try. This will include a query that takes very > little DMBS processing, (such as a straight table select), as well as > one that takes a lot (such as what I've got at the moment). I doubt > there'd be any difference in the DB-API performance... but, you never > know :) > > It really is low priority for me... but if someone really would like it, > I can up the priority with the appropriate encouragements :) > > I'm still happy to take suggestions for changes or additions, though. If > they're easy, I'll add them straight in. It's been on my TODO list as well. Cos without publicly available schema *and* test data, such benchmarks are of limited value. Thanks for your information, though. It's been the first time I've heard pyPgSQL having too much overhead in a *real-life* case. -- Gerhard From jeff at jdfiles.org Mon Jun 2 18:28:49 2003 From: jeff at jdfiles.org (jeff@jdfiles.org) Date: Mon Jun 2 17:25:48 2003 Subject: [DB-SIG] dbiDate - help urgently needed Message-ID: <200306021728.49414.jeff@jdfiles.org> How can I check if an element in a list is a dbiDate object? I am having no luck with type() or isinstance(). My project is falling behind schedule and I can find no explanations anywhere online. From chris at cogdon.org Mon Jun 2 15:39:00 2003 From: chris at cogdon.org (Chris Cogdon) Date: Mon Jun 2 17:39:03 2003 Subject: [DB-SIG] dbiDate - help urgently needed In-Reply-To: <200306021728.49414.jeff@jdfiles.org> Message-ID: <9F9210AE-9542-11D7-9EC2-000393B658A2@cogdon.org> On Monday, Jun 2, 2003, at 14:28 US/Pacific, jeff@jdfiles.org wrote: > How can I check if an element in a list is a dbiDate object? > > I am having no luck with type() or isinstance(). My project is falling > behind > schedule and I can find no explanations anywhere online. I don't know on point, but I can tell you how to find out 'in general'. Do print `type(known_dbiDate_object)`, and that will print enough information to figure out what type it actually is. If you're having difficulty with type and isinstance, then there's a good chance you're not really looking at dbiDate objects. If you don't get any answers on point, then perhaps reduce your code down to a snippet that doesn't work, and post that sample here, including any output from 'debug stubs' like the above. We might be able to help you even if we don't have the stuff installed. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From jeff at jdfiles.org Mon Jun 2 19:23:38 2003 From: jeff at jdfiles.org (Jeff Sacksteder) Date: Mon Jun 2 18:24:25 2003 Subject: [DB-SIG] dbiDate - help urgently needed In-Reply-To: <9F9210AE-9542-11D7-9EC2-000393B658A2@cogdon.org> References: <9F9210AE-9542-11D7-9EC2-000393B658A2@cogdon.org> Message-ID: <3EDBCE6A.1050006@jdfiles.org> > Do print `type(known_dbiDate_object)`, and that will print enough > information to figure out what type it actually is. If you're having > difficulty with type and isinstance, then there's a good chance you're > not really looking at dbiDate objects. for item in my_list: print type(item) returns the following- I want to do something like- for item in my_list: if type(item) is StringType: print 'Foo' elif type(item) is DbiDate: print 'Bar' else: print 'Baz' But DbiDate is an object, not a type. I thought is instance might be the ticket, since it can compare classes _and_ types. No luck so far. Apparently I am the first person in the history of DBI to want to do this. From chris at cogdon.org Mon Jun 2 16:44:34 2003 From: chris at cogdon.org (Chris Cogdon) Date: Mon Jun 2 18:44:37 2003 Subject: [DB-SIG] dbiDate - help urgently needed In-Reply-To: <3EDBCE15.9010800@jdfiles.org> Message-ID: On Monday, Jun 2, 2003, at 15:22 US/Pacific, Jeff Sacksteder wrote: >> Do print `type(known_dbiDate_object)`, and that will print enough >> information to figure out what type it actually is. If you're having >> difficulty with type and isinstance, then there's a good chance >> you're not really looking at dbiDate objects. > > > for item in my_list: > print type(item) > > returns the following- > > > > > > I want to do something like- > > for item in my_list: > if type(item) is StringType: > print 'Foo' > elif type(item) is DbiDate: > print 'Bar' > else: > print 'Baz' > > But DbiDate is an object, not a type. I thought is instance might be > the ticket, since it can compare classes _and_ types. No luck so far. > Apparently I am the first person in the history of DBI to want to do > this. I'm REALLY surprised that doing a type() on an instance of DbiDate is returning . If DbiDate is a 'classic class', then it should return If DbiDate is a 'new class', then it should return something like In either case, isinstance should return something sensible. Ie, isinstance( item, DbiDate ) should return '1' as appropriate, but 'type' may not work as expected. What output are you getting for the second 'for' loop? -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From tjenkins at devis.com Mon Jun 2 19:45:36 2003 From: tjenkins at devis.com (Tom Jenkins) Date: Mon Jun 2 18:48:24 2003 Subject: [DB-SIG] dbiDate - help urgently needed In-Reply-To: <200306021728.49414.jeff@jdfiles.org> References: <200306021728.49414.jeff@jdfiles.org> Message-ID: <3EDBD390.5090804@devis.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 jeff@jdfiles.org wrote: | How can I check if an element in a list is a dbiDate object? | | I am having no luck with type() or isinstance(). My project is falling behind | schedule and I can find no explanations anywhere online. | | http://aspn.activestate.com/ASPN/Mail/Message/python-Tutor/1524370 is the only relevant thing i can find with google. - -- Tom Jenkins devIS - Development Infostructure http://www.devis.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.0-nr2 (Windows XP) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQE+29OQV7Yk9/McDYURAs4NAKDCey9ZL5Blwu+j5vrea/XBRC/qkwCglQDU Eolw8tJJf0yV6qoq7X1LyAM= =4Zjk -----END PGP SIGNATURE----- From pythontutor at venix.com Mon Jun 2 19:59:09 2003 From: pythontutor at venix.com (Lloyd Kvam) Date: Mon Jun 2 19:00:14 2003 Subject: [DB-SIG] dbiDate - help urgently needed References: <9F9210AE-9542-11D7-9EC2-000393B658A2@cogdon.org> <3EDBCE6A.1050006@jdfiles.org> Message-ID: <3EDBD6BD.9040006@venix.com> dbiDate is form the old 1.0 API http://www.python.org/topics/database/DatabaseAPI-2.0.html Python Database API v2.0 It specifies using mx.DateTime for date time values. The Python language just recently evolved to unify tpyes and classes. >>> isinstance(1, int) 1 >>> import mx.DateTime as DT >>> d = DT.Date(2003,6,2) >>> d >>> isinstance(d, DT.DateTime) Traceback (most recent call last): File "", line 1, in ? TypeError: isinstance() arg 2 must be a class or type >>> isinstance(d, DT.DateTimeType) 1 Not all of the modules support isinstance in the way you would expect. So we will probably be using type(obj_reference) for a while longer. Jeff Sacksteder wrote: >> Do print `type(known_dbiDate_object)`, and that will print enough >> information to figure out what type it actually is. If you're having >> difficulty with type and isinstance, then there's a good chance you're >> not really looking at dbiDate objects. > > > > for item in my_list: > print type(item) > > returns the following- > > > > > > I want to do something like- > > for item in my_list: > if type(item) is StringType: > print 'Foo' > elif type(item) is DbiDate: > print 'Bar' > else: > print 'Baz' > > But DbiDate is an object, not a type. I thought is instance might be the > ticket, since it can compare classes _and_ types. No luck so far. > Apparently I am the first person in the history of DBI to want to do this. > > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig > -- Lloyd Kvam Venix Corp. 1 Court Street, Suite 378 Lebanon, NH 03766-1358 voice: 603-443-6155 fax: 801-459-9582 From chris at cogdon.org Mon Jun 2 17:13:17 2003 From: chris at cogdon.org (Chris Cogdon) Date: Mon Jun 2 19:13:26 2003 Subject: [DB-SIG] dbiDate - help urgently needed In-Reply-To: <3EDBD6BD.9040006@venix.com> Message-ID: On Monday, Jun 2, 2003, at 15:59 US/Pacific, Lloyd Kvam wrote: > dbiDate is form the old 1.0 API > > http://www.python.org/topics/database/DatabaseAPI-2.0.html > Python Database API v2.0 > > It specifies using mx.DateTime for date time values. > > > The Python language just recently evolved to unify tpyes and classes. > >>> isinstance(1, int) > 1 > >>> import mx.DateTime as DT > >>> d = DT.Date(2003,6,2) > >>> d > > >>> isinstance(d, DT.DateTime) > Traceback (most recent call last): > File "", line 1, in ? > TypeError: isinstance() arg 2 must be a class or type > >>> isinstance(d, DT.DateTimeType) > 1 > > Not all of the modules support isinstance in the way you would expect. > So > we will probably be using type(obj_reference) for a while longer. Oh... I see what's happening there. The 'Date' as used above is actually a function which returns an instance of a DateTimeType, and not a class, as you might expect. DateTime is another such function. The output from 'd', which is obviously module-generated, is returning a misleading value. Jeff... was the output you got from your code REALLY the output you saw? -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From jeff at jdfiles.org Mon Jun 2 20:29:27 2003 From: jeff at jdfiles.org (Jeff Sacksteder) Date: Mon Jun 2 19:30:15 2003 Subject: [DB-SIG] dbiDate - help urgently needed In-Reply-To: References: Message-ID: <3EDBDDD7.6020102@jdfiles.org> > What output are you getting for the second 'for' loop? Traceback (most recent call last): File "C:\Python21\Pythonwin\pywin\framework\scriptutils.py", line 301, in RunScript exec codeObject in __main__.__dict__ File "C:\Documents and Settings\jwsacksteder.RAM_PRECISION\Desktop\etl.py", line 120, in ? process_table(move_table) File "C:\Documents and Settings\jwsacksteder.RAM_PRECISION\Desktop\etl.py", line 72, in process_table elif isinstance(row_item,DbiDate): NameError: global name 'DbiDate' is not defined From jeff at jdfiles.org Mon Jun 2 20:31:08 2003 From: jeff at jdfiles.org (Jeff Sacksteder) Date: Mon Jun 2 19:32:02 2003 Subject: [DB-SIG] dbiDate - help urgently needed In-Reply-To: References: Message-ID: <3EDBDE3C.3090806@jdfiles.org> > Jeff... was the output you got from your code REALLY the output you saw? Yes. cut-n-paste. From chris at cogdon.org Mon Jun 2 17:38:57 2003 From: chris at cogdon.org (Chris Cogdon) Date: Mon Jun 2 19:39:00 2003 Subject: [DB-SIG] dbiDate - help urgently needed In-Reply-To: <3EDBDDD7.6020102@jdfiles.org> Message-ID: <60F3BB75-9553-11D7-9EC2-000393B658A2@cogdon.org> On Monday, Jun 2, 2003, at 16:29 US/Pacific, Jeff Sacksteder wrote: >> What output are you getting for the second 'for' loop? > > > Traceback (most recent call last): > File "C:\Python21\Pythonwin\pywin\framework\scriptutils.py", line > 301, in RunScript > exec codeObject in __main__.__dict__ > File "C:\Documents and > Settings\jwsacksteder.RAM_PRECISION\Desktop\etl.py", line 120, in ? > process_table(move_table) > File "C:\Documents and > Settings\jwsacksteder.RAM_PRECISION\Desktop\etl.py", line 72, in > process_table > elif isinstance(row_item,DbiDate): > NameError: global name 'DbiDate' is not defined Ah, right. And that makes sense given Lloyd's recent response. 'DbiDate' is not a real type, or a real anything. I suspect that 'typing' the object is returning something misleading. Two morals from this story: 1. If you're getting error messages, always tell us what the error message is straight away. If it's long and wordy, then perhaps just the last few stack entries. Having error messages saves us from going down unnecessary paths. 2. Don't 'fake' output by, say, hand transposing it from your screen into your mail program. There are nuances and subtleties in the output, sometimes that if mis-copied will lead an 'expert' down the wrong path. That's frustrating. If you've created code to do something, then cut and paste that code and output exactly. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From chris at cogdon.org Mon Jun 2 17:40:11 2003 From: chris at cogdon.org (Chris Cogdon) Date: Mon Jun 2 19:40:14 2003 Subject: [DB-SIG] dbiDate - help urgently needed In-Reply-To: <60F3BB75-9553-11D7-9EC2-000393B658A2@cogdon.org> Message-ID: <8D4E9BD8-9553-11D7-9EC2-000393B658A2@cogdon.org> On Monday, Jun 2, 2003, at 16:38 US/Pacific, Chris Cogdon wrote: > 2. Don't 'fake' output by, say, hand transposing it from your screen > into your mail program. There are nuances and subtleties in the > output, sometimes that if mis-copied will lead an 'expert' down the > wrong path. That's frustrating. If you've created code to do > something, then cut and paste that code and output exactly. Public retraction on this second part. Sorry :) -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From jeff at jdfiles.org Mon Jun 2 20:40:19 2003 From: jeff at jdfiles.org (Jeff Sacksteder) Date: Mon Jun 2 19:41:07 2003 Subject: [DB-SIG] dbiDate - help urgently needed In-Reply-To: <200306021728.49414.jeff@jdfiles.org> References: <200306021728.49414.jeff@jdfiles.org> Message-ID: <3EDBE063.2060600@jdfiles.org> This is working but seems suboptimal. Is this the most correct way to go about it? for row_item in row: if type(row_item) is StringType: clean_item = clean_str(row_item) elif type(row_item) == type(dbi.dbiDate(0)): clean_item = clean_date(row_item) else: clean_item = clean_misc(row_item) From chris at cogdon.org Mon Jun 2 17:56:43 2003 From: chris at cogdon.org (Chris Cogdon) Date: Mon Jun 2 19:56:46 2003 Subject: [DB-SIG] dbiDate - help urgently needed In-Reply-To: <3EDBE063.2060600@jdfiles.org> Message-ID: On Monday, Jun 2, 2003, at 16:40 US/Pacific, Jeff Sacksteder wrote: > This is working but seems suboptimal. Is this the most correct way to > go about it? > > for row_item in row: > if type(row_item) is StringType: > clean_item = clean_str(row_item) > elif type(row_item) == type(dbi.dbiDate(0)): Try using > elif type(row_item) == dbi.DateTimeType instead (not tried myself. This is coming from Lloyd's information) -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From jeff at jdfiles.org Mon Jun 2 21:15:51 2003 From: jeff at jdfiles.org (Jeff Sacksteder) Date: Mon Jun 2 20:16:38 2003 Subject: [DB-SIG] dbiDate - help urgently needed In-Reply-To: References: Message-ID: <3EDBE8B7.8040003@jdfiles.org> Chris Cogdon wrote: > > On Monday, Jun 2, 2003, at 16:40 US/Pacific, Jeff Sacksteder wrote: > >> This is working but seems suboptimal. Is this the most correct way to >> go about it? >> >> for row_item in row: >> if type(row_item) is StringType: >> clean_item = clean_str(row_item) >> elif type(row_item) == type(dbi.dbiDate(0)): > > > Try using > >> elif type(row_item) == dbi.DateTimeType > > > instead (not tried myself. This is coming from Lloyd's information) > > Seems more logical. Thank you all for your assistance! From pythontutor at venix.com Mon Jun 2 23:08:31 2003 From: pythontutor at venix.com (Lloyd Kvam) Date: Mon Jun 2 22:09:33 2003 Subject: [DB-SIG] dbiDate - help urgently needed References: <200306021728.49414.jeff@jdfiles.org> <3EDBE063.2060600@jdfiles.org> Message-ID: <3EDC031F.2050203@venix.com> Presumably you only need to determine the functions for each column once. However, that would risk passing None as a row value (from NULLs) into the functions. It also requires using a row with no NULLs. Alternatively, you can use the information that comes back in the cursor.description to determine column types. (I have not used the Version 1 API, so I can't guarantee that description has the type info.) http://www.python.org/topics/database/DatabaseAPI-2.0.html#cursor Lists the values for each column returned in cursor.description. def choose_clean(dbtype): if dbtype in (??? some list based on what your DB gives you ???): return clean_str elif dbtype in (?? date, timestamp type things ??): return clean_date elif ???? and so on ???? clean_funcs = [choose_clean(dbtype) for dbname,dbtype,x1,x2,x3,x4,x5 in cursor.description] for row_item in row: clean_items = map(apply, clean_funcs, zip(row_item)) ??? do what you do with clean_items ??? < end funny pseudo code > This may not be much of an improvement depending on just what you are trying to accomplish. It avoids making the same decision over and over again for each column. However the map(apply... is rather strange looking. It also requires that the functions can deal with a None. I've only used the dbAPI-2. Your references to dbiDate (so far as I know) suggest you are using dbAPI-1. Using the builtin type function is reasonable. Look at the types module. You might really want StringTypes rather than StringType. Jeff Sacksteder wrote: > This is working but seems suboptimal. Is this the most correct way to go > about it? > > for row_item in row: > if type(row_item) is StringType: > clean_item = clean_str(row_item) > elif type(row_item) == type(dbi.dbiDate(0)): > clean_item = clean_date(row_item) > else: clean_item = > clean_misc(row_item) > > > > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig > -- Lloyd Kvam Venix Corp. 1 Court Street, Suite 378 Lebanon, NH 03766-1358 voice: 603-443-6155 fax: 801-459-9582 From chris at cogdon.org Mon Jun 2 21:31:29 2003 From: chris at cogdon.org (Chris Cogdon) Date: Mon Jun 2 23:31:34 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method Message-ID: I'd like to make the following proposal for the next revision of the API specfication: New method to 'database' object: quote. It has exactly the same parameter parsing semantics as the 'execute' method, but instead of sending the statement to the DBMS, it simply returns the string after the parameters have been quoted. In effect, assuming the following: cur = db.cursor () the following two statements would be equivalent: cur.execute ( "some sql statement", parameters ) cur.execute ( db.quote ( "some sql statement", parameters ) ) Reasoning: - There are many instances where you want to 'build up' the SQL statement in pieces. For example, I have an application where I create one object representing the interface to the database. To search for records of a particular class, you create a 'criteria' object, and pass that to the method of the database object. The criteria object is responsibile for building the SQL 'WHERE" clause necessary to find the required objects. The database method is responsible for building the tables necessary for the where to function correctly. This would be VERY hard if I had to organise the SQL, and the list of parameters independantly before finally passing it to the 'execute' method which binds the two together. - Some API's make their quoting functions available (pg has 'quote', pyPgSQL has '_quote'), but some do not (eg PoPy and psycopg) mostly because they're written in C, and requires explicit efforts to expose functions (as it should be, but it's inconvienient in this case). - It would be best to have it as a method of a database object, even though typical implementation would not require knowing anything about the connection. This way, I can write code that is 'database agnostic', but ensure that the right version of 'quote' is called when required. So... any support for this change? Any comments? -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From thierry at nekhem-labs.org Tue Jun 3 07:52:04 2003 From: thierry at nekhem-labs.org (Thierry MICHEL) Date: Tue Jun 3 02:52:06 2003 Subject: [DB-SIG] Python/PostgreSQL API performance comparison In-Reply-To: <1054579776.22312.42.camel@localhost> References: <107E3DAC-952A-11D7-9EC2-000393B658A2@cogdon.org> <1054579776.22312.42.camel@localhost> Message-ID: <1054622837.19988.50.camel@haru.xtensive.com> Hi all, Thank you very much for the tests Chris! On Mon, 2003-06-02 at 20:49, Federico Di Gregorio wrote: > Il lun, 2003-06-02 alle 20:43, Chris Cogdon ha scritto: > > On Monday, Jun 2, 2003, at 10:42 US/Pacific, Federico Di Gregorio wrote: > > > > > would you add psycopg to the tests, please. just eager to know how well > > > it does. > > > > Ah, thanks for reminding me about your module :) Here's the results: > > > > pg [ 0.180, 0.010, 0.000, 0.000, 29.610 ] > > pgdb [ 4.570, 0.000, 0.000, 0.000, 33.520 ] > > PgSQL [ 4.480, 0.010, 0.000, 0.000, 42.450 ] > > PgSQL (nocursor) [ 4.480, 0.010, 0.000, 0.000, 33.530 ] > > PgSQL (nocursor,list) [ 3.620, 0.020, 0.000, 0.000, 32.570 ] > > PoPy [ 0.200, 0.010, 0.000, 0.000, 29.280 ] > > psycopg [ 0.300, 0.010, 0.000, 0.000, 29.330 ] > > > > Vigilant readers will note that the figures are a little larger than > > those I posted last time. A fair bit of data has been added to the > > database since. Yes, having a 'consistent framework' would be nice, > > but... I never said these were 'laboratory quality' values :) > > > > So... psycopg performs pretty well... but... I'm sure you could squeeze > > that 0.1 out, couldn't you? :) Reading up on the docs for psycopg, it > > i is already pretty good, considered that psycopg does a lot more stuff > than popy (custom type-casters, etc.) > > appears it's designed to perform well under heavy-load applications > > with multiple, persistant connections to the database. I might see if I > > can design a test to shake out that particular metric, since that's > > very important for my application. > > psycopg is very aggressive on calling Py_*_ALLOW_THREADS macros so > you'll get the best performance on multithreaded applications. As popy do too. The use of this macro is recommanded each time you want to access external C functions of postgresql library not fully threaded. > thank you for the tests, > federico > > > ______________________________________________________________________ > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig These tests will help us the define better the design of our new driver Popy3 we are still developping. We hope give you the first alpha release of Popy3 as soon as possible. Thanks again. -- Thierry MICHEL Software Designer & Developer 19 rue de l'ail 67000 Strasbourg France -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/db-sig/attachments/20030603/cabc5e9b/attachment-0001.bin From mal at lemburg.com Tue Jun 3 10:23:43 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jun 3 03:23:56 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: References: Message-ID: <3EDC4CFF.9010007@lemburg.com> Chris Cogdon wrote: > I'd like to make the following proposal for the next revision of the API > specfication: > > New method to 'database' object: quote. > > It has exactly the same parameter parsing semantics as the 'execute' > method, but instead of sending the statement to the DBMS, it simply > returns the string after the parameters have been quoted. > > In effect, assuming the following: > > cur = db.cursor () > > the following two statements would be equivalent: > > cur.execute ( "some sql statement", parameters ) > cur.execute ( db.quote ( "some sql statement", parameters ) ) > > Reasoning: > > - There are many instances where you want to 'build up' the SQL > statement in pieces. For example, I have an application where I create > one object representing the interface to the database. To search for > records of a particular class, you create a 'criteria' object, and pass > that to the method of the database object. The criteria object is > responsibile for building the SQL 'WHERE" clause necessary to find the > required objects. The database method is responsible for building the > tables necessary for the where to function correctly. > > This would be VERY hard if I had to organise the SQL, and the list of > parameters independantly before finally passing it to the 'execute' > method which binds the two together. I don't buy that. By "quoting" you actually mean "binding" and you thus lose all the benefits of having the SQL separated from the data. The good thing about binding parameters is that the user does *not* have to worry about quoting (plus it usually buys a performance gain for next to nothing). Of course, if some APIs think it's worth exposing such a quote function, they are free to do so. It shouldn't be required, though. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Jun 03 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 21 days left From magnus at thinkware.se Tue Jun 3 10:42:36 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Tue Jun 3 03:39:34 2003 Subject: [DB-SIG] dbiDateType = type(dbi.dbiDate(0)) In-Reply-To: Message-ID: <5.2.1.1.0.20030603093113.01f33ec0@www.thinkware.se> At 19:00 2003-06-02 -0400, Jeff wrote: >>for item in my_list: >> if type(item) is StringType: >> print 'Foo' >> elif type(item) is DbiDate: >> print 'Bar' >> else: >> print 'Baz' >> >>But DbiDate is an object, not a type. Everything is an object... As you already showed, the object in question is not of instance type, but of DbiDate type. ( ) >>I thought is instance might be the ticket, since it can compare classes >>_and_ types. No luck so far. Apparently I am the first person in the >>history of DBI to want to do this. No, it's quite simple. You need to get a type object! It's probably directly available somewhere, but why search for that when you can make one---type() returns type objects! dbiDateType = type(dbi.dbiDate(0)) if type(item) == dbiDateType: ... It seems you are using the old ODBC interface in win32all. It's not really supported as far as I understand, and not complient with DB-API 2. mxODBC or adodbapi might be a better choice. -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The shortest path from thought to working program From fog at debian.org Tue Jun 3 11:34:53 2003 From: fog at debian.org (Federico Di Gregorio) Date: Tue Jun 3 04:16:28 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <3EDC4CFF.9010007@lemburg.com> References: <3EDC4CFF.9010007@lemburg.com> Message-ID: <20030603083453.GA28034@initd.org> Scavenging the mail folder uncovered M.-A. Lemburg's letter: > Chris Cogdon wrote: > >This would be VERY hard if I had to organise the SQL, and the list of > >parameters independantly before finally passing it to the 'execute' > >method which binds the two together. > > I don't buy that. By "quoting" you actually mean "binding" and you > thus lose all the benefits of having the SQL separated from the > data. agreed. > The good thing about binding parameters is that the user > does *not* have to worry about quoting (plus it usually buys > a performance gain for next to nothing). > > Of course, if some APIs think it's worth exposing such a quote > function, they are free to do so. It shouldn't be required, though. i sometimes log all queries while debugging an app. having quote will make much easier to log queries with parameters and discover binding problems et al. thus, yes, quote is usefull to me. -- Federico Di Gregorio Debian GNU/Linux Developer & Italian Press Contact fog@debian.org Those who do not study Lisp are doomed to reimplement it. Poorly. -- from Karl M. Hegbloom .signature From jacobs at penguin.theopalgroup.com Tue Jun 3 09:45:16 2003 From: jacobs at penguin.theopalgroup.com (Kevin Jacobs) Date: Tue Jun 3 08:46:12 2003 Subject: [DB-SIG] Python/PostgreSQL API performance comparison In-Reply-To: <107E3DAC-952A-11D7-9EC2-000393B658A2@cogdon.org> Message-ID: Due to the recent interest in PostgreSQL DB-API driver comparisons, I though I'd jump on the proverbial bandwagon. As some of you already know, my company has developed very substantial financial applications using Python, and frequently PostgreSQL. We've been using the PsycoPg driver for the past two years, because we found it to be the most solid and least flawed of all the available drivers at the time. I've updated our driver suite to include a recent version PoPy, and run one of our very substantial test suites. The suite constitutes a mix the following tasks: 1) Many simple OLTP queries. 2) Many simple and complex OLAP queries, some returning as many as a hundred thousand rows of data. 3) Data-cube construction and manipulation 4) Business report generation The application server and database run on the same (otherwise quiescent) test server, the working data set is all in-core (though quite large), and only serial requests (i.e., only a single active worker thread) are issued. Results running Python CVS on the same application server with only the driver setting changed: Driver / version Wall Time (average of 4 runs, though little variation was ---------------- --------- observed) PsycoPg 1.0.13 10m41s PoPy 2.0.8 11m33s I'll run the suite with more drivers over the next week, as time permits. Regards, -Kevin [*] Unfortunately, other than for this simple test suite, PoPy is basically unusable for us. This is because it does not return proper PostgreSQL type codes, only vague type strings ('NUMBER','DATETIME','MISSING'?!). Thus, it does not provide enough information to distinguish, e.g., booleans from numbers, numeric from floating-point values, dates from datetimes, etc. As previously reported, it does not translate large integers correctly, and it mangles some date interval types. These deficiencies may not affect simpler or less-demanding applications, but to us they qualify as unacceptable information loss. -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From chris at cogdon.org Tue Jun 3 11:39:24 2003 From: chris at cogdon.org (Chris Cogdon) Date: Tue Jun 3 13:39:30 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <3EDC4CFF.9010007@lemburg.com> Message-ID: <514B8FDE-95EA-11D7-8BD4-000393B658A2@cogdon.org> On Tuesday, Jun 3, 2003, at 00:23 US/Pacific, M.-A. Lemburg wrote: > I don't buy that. By "quoting" you actually mean "binding" and you > thus lose all the benefits of having the SQL separated from the > data. > > The good thing about binding parameters is that the user > does *not* have to worry about quoting (plus it usually buys > a performance gain for next to nothing). > > Of course, if some APIs think it's worth exposing such a quote > function, they are free to do so. It shouldn't be required, though. Okay... let me paraphrase your argument to see if I got things straight. Exposing the quoting mechanism would make sense on APIs for DBMSes that require a single SQL string be passed. However, some DMBSes use a different query passing mechanism, where the SQL query with embedded '?' marks (or whatever) and parameters in binary format are passed separately. While somewhat more complicated, this is a performance gain as the DMBS query passer does not need to parse the parameters, nor does the application need to convert them into a string. For these types of DMBSs, quoting and binding are different operations. For things like PostgreSQL, they're the same. Is that correct? Now, my counter-argument :) I would suspect that these DMBSes still have a way of representing any literal value inside the SQL string itself. For example: cur.execute ( "select * from employee where name like ? and type='manager'", name_criteria ) If not, I would then have to break out 'manager' into a parameter just to be passed to the DMBS. Now, if this is true, then the DMBS has a quoting convention that I believe should be exposed. There are many applications, such as the one I described previously, where this would be useful. Yes, it lowers performance somewhat, but the alternative is to jump through a lot of hoops to get the separated string and parameters to the 'execute' method, or use a quoting function external to the API (which, being separate from the API layer, is error-prone). I'm suggesting that we make this part of the API (2.1, perhaps) so that there's something there my application can depend on. I believe it's an important part of dealing with the DMBS, and should be part of the specs/requirements. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From jacobs at penguin.theopalgroup.com Tue Jun 3 15:21:11 2003 From: jacobs at penguin.theopalgroup.com (Kevin Jacobs) Date: Tue Jun 3 14:21:58 2003 Subject: [DB-SIG] Python/PostgreSQL API performance comparison In-Reply-To: Message-ID: [Updated to include results from pgdb and pyPgSQL] Due to the recent interest in PostgreSQL DB-API driver comparisons, I thought I'd jump on the proverbial bandwagon. As some of you already know, my company has developed very substantial financial applications using Python, and frequently PostgreSQL. We've been using the PsycoPg driver for the past two years, because we found it to be the most solid and least flawed of all the available drivers at the time. I've updated our driver suite to include a recent versions PoPy, pgdb, and pyPgSQL, and then run one of our very substantial test suites. The suite constitutes a mix the following tasks: 1) Many simple OLTP queries. 2) Many simple and complex OLAP queries, some returning as many as a hundred thousand rows of data. 3) Data-cube construction and manipulation 4) Business report generation The application server and database run on the same (otherwise quiescent) test server, the working data set is all in-core (though quite large), and only serial requests (i.e., only a single active worker thread) are issued. Results running Python CVS on the same application server with only the driver setting changed: Driver / version Wall Time (average of 4 runs, though little variation was ---------------- --------- observed) PsycoPg 1.0.13 10m41s PoPy 2.0.8 11m33s [1] pgdb 3.3 14m40s pyPgSQL 2.3 >15m51s [2] Did not complete all tests! Notes: [1] Unfortunately, other than for this simple test suite, PoPy is basically unusable for us. This is because it does not return proper PostgreSQL type codes, only vague type strings ('NUMBER','DATETIME','MISSING'?!). Thus, it does not provide enough information to distinguish, e.g., booleans from numbers, numeric from floating-point values, dates from datetimes, etc. As previously reported, it does not translate large integers correctly, and it mangles some date interval types. These deficiencies may not affect simpler or less-demanding applications, but to us they qualify as unacceptable information loss. [2] pyPgSQL required several minor modifications to work properly. First, the compile failed if LONG_LONG was not defined/detected. Second, it returns a non-standard cursor.description with an extra element indicating array status. While a potentially useful thing, it is very very non-standard. Finally, the cursor descriptions use non-hashable type codes, which caused problems for our OR-mapper. I modified the type objects, adding an appropriate __hash__ method, and all was well. Once it was running, several test vectors failed deterministically with the following error: /usr/local/python-cvs/lib/python2.3/site-packages/pyPgSQL/PgSQL.py: OperationalError: RelationForgetRelation: relation xxxxxxx is still open Regards, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From jacobs at penguin.theopalgroup.com Tue Jun 3 15:33:05 2003 From: jacobs at penguin.theopalgroup.com (Kevin Jacobs) Date: Tue Jun 3 14:33:41 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <514B8FDE-95EA-11D7-8BD4-000393B658A2@cogdon.org> Message-ID: On Tue, 3 Jun 2003, Chris Cogdon wrote: > I'm suggesting that we make this part of the API (2.1, perhaps) so that > there's something there my application can depend on. I believe it's an > important part of dealing with the DMBS, and should be part of the > specs/requirements. Why not implement it yourself for your own application? Implementing generic quoting properly often requires that a fair amount of type-metadata is available. It is more the job of a object/relational mapper than a low-level DB-API driver, although a DB-API driver can do it with some help from the backend. Please do not be mislead by some of the simplistic (and wrong) implementations that you may find laying around. Specifically, quoting strings (char and varchar), integers, and floats is usually fairly easy -- just borrow the code or read the web docs. The pain starts when you want to quote times, dates, booleans, numerics, money, intervals, network types, etc. without knowing the schema type it will be bound to beforehand. For more insight into these complications, study ODBC for a while. It is a good example of a broader and richer (but more painful) API than DB-API. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From davidrushby at yahoo.com Tue Jun 3 12:45:21 2003 From: davidrushby at yahoo.com (David Rushby) Date: Tue Jun 3 14:45:26 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <514B8FDE-95EA-11D7-8BD4-000393B658A2@cogdon.org> Message-ID: <20030603184521.23065.qmail@web11002.mail.yahoo.com> --- Chris Cogdon wrote: > > On Tuesday, Jun 3, 2003, at 00:23 US/Pacific, M.-A. Lemburg wrote: > > > I don't buy that. By "quoting" you actually mean "binding" and you > > thus lose all the benefits of having the SQL separated from the > > data. > > > > The good thing about binding parameters is that the user > > does *not* have to worry about quoting (plus it usually buys > > a performance gain for next to nothing). > > > > Of course, if some APIs think it's worth exposing such a quote > > function, they are free to do so. It shouldn't be required, though. > > Okay... let me paraphrase your argument to see if I got things straight. > > Exposing the quoting mechanism would make sense on APIs for DBMSes that > require a single SQL string be passed. However, some DMBSes use a > different query passing mechanism, where the SQL query with embedded > '?' marks (or whatever) and parameters in binary format are passed > separately. While somewhat more complicated, this is a performance gain > as the DMBS query passer does not need to parse the parameters, nor > does the application need to convert them into a string. > > For these types of DMBSs, quoting and binding are different operations. > For things like PostgreSQL, they're the same. > > I would suspect that these DMBSes still have a way of representing any > literal value inside the SQL string itself. They ought to, but some don't allow literal representation of certain data types--blobs and database arrays are frequent offenders. > Now, if this is true, then the DMBS has a quoting convention that > I believe should be exposed. Again, "I believe should be exposed" != "actually is exposed in the client API". > Yes, it lowers performance somewhat, but the alternative is to jump > through a lot of hoops to get the separated string and parameters > to the 'execute' method, or use a quoting function external to the > API (which, being separate from the API layer, is error-prone). > > I'm suggesting that we make this part of the API (2.1, perhaps) so that > there's something there my application can depend on. I believe it's an > important part of dealing with the DMBS, and should be part of the > specs/requirements. I'm not opposed to adding this method to the spec, as long as: a) it's optional--some DBMSes just can't support it b) the performance penalty applies *only* to those who explicitly choose to use it __________________________________ Do you Yahoo!? Yahoo! Calendar - Free online calendar with sync to Outlook(TM). http://calendar.yahoo.com From chris at cogdon.org Tue Jun 3 12:50:07 2003 From: chris at cogdon.org (Chris Cogdon) Date: Tue Jun 3 14:50:11 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: Message-ID: <3224A884-95F4-11D7-8BD4-000393B658A2@cogdon.org> On Tuesday, Jun 3, 2003, at 11:33 US/Pacific, Kevin Jacobs wrote: > On Tue, 3 Jun 2003, Chris Cogdon wrote: >> I'm suggesting that we make this part of the API (2.1, perhaps) so >> that >> there's something there my application can depend on. I believe it's >> an >> important part of dealing with the DMBS, and should be part of the >> specs/requirements. > > Why not implement it yourself for your own application? Because, the quoting requirements will differ from DMBS to DMBS. Putting it into the application will unnecessarily tie the application to the DMBS. > Implementing > generic quoting properly often requires that a fair amount of > type-metadata > is available. It is more the job of a object/relational mapper than a > low-level DB-API driver, although a DB-API driver can do it with some > help > from the backend. Please do not be mislead by some of the simplistic > (and > wrong) implementations that you may find laying around. The API is responsible for knowing the requirements of the DMBS. That's what it's *FOR* :) -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From chris at cogdon.org Tue Jun 3 12:51:56 2003 From: chris at cogdon.org (Chris Cogdon) Date: Tue Jun 3 14:51:59 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <20030603184521.23065.qmail@web11002.mail.yahoo.com> Message-ID: <72C2BE5B-95F4-11D7-8BD4-000393B658A2@cogdon.org> On Tuesday, Jun 3, 2003, at 11:45 US/Pacific, David Rushby wrote: >> Now, if this is true, then the DMBS has a quoting convention that >> I believe should be exposed. > > Again, "I believe should be exposed" != "actually is exposed in the > client > API". I know that, which is why I'm making the suggestion that we turn the != into a == > I'm not opposed to adding this method to the spec, as long as: > a) it's optional--some DBMSes just can't support it > b) the performance penalty applies *only* to those who explicitly > choose to > use it I'm good with that. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From jacobs at penguin.theopalgroup.com Tue Jun 3 15:56:45 2003 From: jacobs at penguin.theopalgroup.com (Kevin Jacobs) Date: Tue Jun 3 14:57:20 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <3224A884-95F4-11D7-8BD4-000393B658A2@cogdon.org> Message-ID: On Tue, 3 Jun 2003, Chris Cogdon wrote: > > Implementing generic quoting properly often requires that a fair amount > > of type-metadata is available. It is more the job of a > > object/relational mapper than a low-level DB-API driver, although a > > DB-API driver can do it with some help from the backend. Please do not > > be mislead by some of the simplistic (and wrong) implementations that > > you may find laying around. > > The API is responsible for knowing the requirements of the DMBS. That's > what it's *FOR* :) Yes, and it works when binding because there is enough context to map each parameter to a particular SQL type. Generic quoting without that context is simply not possible without significant infrastructure or meta-data. It is a great deal harder than exposing an internal method that already exists within a driver. I still suggest that you try writing a generic quoting library for your application. At minimum you'll find out first hand how tricky it is. At best, you'll end up with a nice tool that can then be shared with others. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From Harald.Meland at usit.uio.no Tue Jun 3 22:08:18 2003 From: Harald.Meland at usit.uio.no (Harald Meland) Date: Tue Jun 3 15:08:44 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <514B8FDE-95EA-11D7-8BD4-000393B658A2@cogdon.org> (Chris Cogdon's message of "Tue, 3 Jun 2003 10:39:24 -0700") References: <514B8FDE-95EA-11D7-8BD4-000393B658A2@cogdon.org> Message-ID: [Chris Cogdon] > I would suspect that these DMBSes still have a way of representing any > literal value inside the SQL string itself. For example: > > cur.execute ( "select * from employee where name like ? and > type='manager'", name_criteria ) Spot on. Exposing such a quoting mechanism in the standard API does *not* imply that binding should go away/be deprecated/whatever. As I see it, there's not much benefit in having various driver modules expose such quoting mechanisms in their own way; the benefit lies in exposing the mechanism in the same manner across all the driver modules. Hence, definition of how this mechanism should be exposed should go into (the next version of) the DB-API -- as a strong requirement for compliance. > I'm suggesting that we make this part of the API (2.1, perhaps) so > that there's something there my application can depend on. I believe > it's an important part of dealing with the DMBS, and should be part of > the specs/requirements. +1 -- Harald From chris at cogdon.org Tue Jun 3 13:10:10 2003 From: chris at cogdon.org (Chris Cogdon) Date: Tue Jun 3 15:10:14 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: Message-ID: On Tuesday, Jun 3, 2003, at 11:56 US/Pacific, Kevin Jacobs wrote: > On Tue, 3 Jun 2003, Chris Cogdon wrote: > >>> Implementing generic quoting properly often requires that a fair >>> amount >>> of type-metadata is available. It is more the job of a >>> object/relational mapper than a low-level DB-API driver, although a >>> DB-API driver can do it with some help from the backend. Please do >>> not >>> be mislead by some of the simplistic (and wrong) implementations that >>> you may find laying around. >> >> The API is responsible for knowing the requirements of the DMBS. >> That's >> what it's *FOR* :) > > Yes, and it works when binding because there is enough context to map > each > parameter to a particular SQL type. Generic quoting without that > context is > simply not possible without significant infrastructure or meta-data. > It is > a great deal harder than exposing an internal method that already > exists > within a driver. I believe my suggestion was to expose the quoting mechanism in a way that EXACTLY REPLICATES the mechanism inside the API. Reiterating my first email on the subject, the following two examples would be equivalent: cur.execute ( "some SQL statement", parameters ) cur.execute ( db.quote ( "some SQL statement", parameters ) ) In the first example, the SQL and the parameters are passed to execute, in the second, only the string, but the SQL statement and parameters are passed to the API's 'quote' method in **exactly the same way** as it's passed to 'execute' in the first example. I don't see the problem here. > I still suggest that you try writing a generic quoting library for your > application. At minimum you'll find out first hand how tricky it is. > At > best, you'll end up with a nice tool that can then be shared with > others. I can just lift them from the native-Python PostgreSQL libraries, for example, which is what I'm doing now. Unfortunately, the C-native versions don't expose the quoting method, meaning that my 'piecemeal SQL' application has to either do a lot of work getting separated SQL and parameters to the 'execute' method, or write my own quoter. The former is a lot of work, the latter is 'error-prone', especially if I want to change APIs or even DBMSes. This is why I'm suggesting making it part of the next specification revision. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From magnus at thinkware.se Wed Jun 4 01:06:46 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Tue Jun 3 18:03:52 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: Message-ID: <5.2.1.1.0.20030603230906.01f4dea8@www.thinkware.se> I don't see a need for this quote function for building SQL statements, and I don't see how it could work cross platform as presented, but I think it could be a good way to see what is happening under the hood. I would like to have that for pure debugging and logging purposes. Chris Cogdon wrote: >I believe my suggestion was to expose the quoting mechanism in a way that >EXACTLY REPLICATES the mechanism inside the API. Reiterating my first >email on the subject, the following two examples would be equivalent: > >cur.execute ( "some SQL statement", parameters ) >cur.execute ( db.quote ( "some SQL statement", parameters ) ) > >In the first example, the SQL and the parameters are passed to execute, in >the second, only the string, but the SQL statement and parameters are >passed to the API's 'quote' method in **exactly the same way** as it's >passed to 'execute' in the first example. I don't think your example can work reliably for all platforms quite like that. But this might work: cur.execute( *db.quote( "some SQL statement", parameters) ) Not that it would be of much use though... Let's take a concrete example. Let's assume that we use either mxODBC or adodbapi on Windows, and connect to a System data source called "databaseX". It's just an ODBC source, and we don't know what kind of database it is. What do we care--it speaks ODBC. Our SQL is "SELECT * FROM T WHERE D = ?" and we send in a single mxDateTime object as a parameter. What these interfaces will actually do is to send the SQL string just as it is, ? and all, down to the ODBC level at least. It will convert the mxDateTime object to some suitable format for the platform, but it will never worry about how to get it into the SQL string, because it never will. After all, both Dynamic SQL and SQL CLI mandates that parameter passing with ? should work, so as far as I know, there will never be an SQL string with the date value stuffed in. The database server will read directly from the parameter list when is finds a ? in the SQL, instead of doing the labour of extracting a string literal that it perhaps needs to unquote and unescape. Seems like less work. For every server, there *is* at least one potential SQL string which would work in an equivalent way to the SQL with the parameters we sent, but there is no reason for any part of the chain between the application and the db server to know what that might look like. In this particular case it will vary depending on the data source. If the database happens to use the MS Jet engine, (i.e. MS Access) the SQL could look like "SELECT * FROM T WHERE D = #1999-12-31#", but for most other backends it should look like "SELECT * FROM T WHERE D = '1999-12-31'". The #-version would only work on Jet, and it's the only one that works on Jet. Depending on national seetings, what kind of database you have etc, the date string might look different, regardless of quote-style. Passed as a separate parameter, it might work to send it as a COM Date object if we're in Windows. If the DB-API would try to pass in the parameter into the SQL string, it could just guess how to do that, and it is competely beyond its "duties". So, I don't see that db.quote on adodbapi could actually return anything more intelligent than the input you gave it. A string and a tuple with parameters. Actually, what would be different here is the types of the parameters. A passed in mxDateTime object might be converted to a COM Date object, or to a string with a date in ISO 8601 format etc. That could be useful I guess. Not even this is trivial though. ADO doesn't seem to be so easy to play with, so adodbapi tries different strategies in case of error. It will for instance try both COM Date objects and ISO 8601 strings for dates. If an interface is implemented in C or C++, it's not certain that the actual data type being passed to the next step can be described in Python at all? M-A? What about mxODBC for instance? In other words, I would like such a quote function to return what it would actually send to the next step in the chain, be that ADO or the MySQL client libs. Either just a string, or a tuple containing a string and a converted parameter tuple---but I'm not sure it's a reasonable thing to ask for in all interfaces. -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The Agile Programming Language From david at sundayta.com Wed Jun 4 00:17:05 2003 From: david at sundayta.com (David Warnock) Date: Tue Jun 3 18:16:57 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <5.2.1.1.0.20030603230906.01f4dea8@www.thinkware.se> References: <5.2.1.1.0.20030603230906.01f4dea8@www.thinkware.se> Message-ID: <3EDD1E61.2080007@sundayta.com> Magnus, > I don't see a need for this quote function for building SQL > statements, and I don't see how it could work cross platform > as presented, but I think it could be a good way to see what > is happening under the hood. I would like to have that for > pure debugging and logging purposes. In java there is a JDBC Driver called p6spy which installs as a standard jdbc driver in your application between it and the real jdbc driver. It then logs every call to the jdbc driver. For example it shows prepared statements before and after binding. This is very useful for debugging sql (eg to make sure that proxies are working). However, my limited understanding of the dbi-api is that it is not yet complete enough as a layer for this to be possible. So far as I can see it is not yet so easy to write cross dbms code in python as it is in java. I would like to be proved wrong. Dave -- David Warnock, Sundayta Ltd. http://www.sundayta.com iDocSys for Document Management. VisibleResults for Fundraising. Development and Hosting of Web Applications and Sites. From Harald.Meland at usit.uio.no Wed Jun 4 01:58:49 2003 From: Harald.Meland at usit.uio.no (Harald Meland) Date: Tue Jun 3 18:58:55 2003 Subject: [DB-SIG] Asymmetry in DB-API 2.0 types Message-ID: Hi, as the current DB-API stands, the types it mentions can be constructed via standard calls, but there's no standard module-independent way to decipher such types. An example: It's reasonably easy to pass a DateTime object as a bind argument to .execute() in a driver-independent way. However, if I'd like to get the date part of a DateTime object returned from a query, I'll have to jump through module-specific hoops. Is this still another case of "this can't or shouldn't be handled by the driver, but by a (hypothetical) separate layer on top of the drivers", or is there interest in extending the type definitions in the DB-API to include deciphering features? -- Harald From Harald.Meland at usit.uio.no Wed Jun 4 01:45:01 2003 From: Harald.Meland at usit.uio.no (Harald Meland) Date: Tue Jun 3 19:04:08 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <5.2.1.1.0.20030603230906.01f4dea8@www.thinkware.se> ( =?iso-8859-1?q?Magnus_Lyck=E5's_message_of?= "Wed, 04 Jun 2003 00:06:46 +0200") References: <5.2.1.1.0.20030603230906.01f4dea8@www.thinkware.se> Message-ID: [Magnus Lyck?] > Chris Cogdon wrote: >>cur.execute ( "some SQL statement", parameters ) >>cur.execute ( db.quote ( "some SQL statement", parameters ) ) >> > > I don't think your example can work reliably for all platforms > quite like that. But this might work: > > cur.execute( *db.quote( "some SQL statement", parameters) ) > > Not that it would be of much use though... [...] > In other words, I would like such a quote function to return > what it would actually send to the next step in the chain, be > that ADO or the MySQL client libs. Either just a string, or a > tuple containing a string and a converted parameter tuple---but > I'm not sure it's a reasonable thing to ask for in all interfaces. I don't get it; what kind of problem is such a "maybe-quote" function intended to solve? -- Harald From stuart.b at commonground.com.au Wed Jun 4 10:53:44 2003 From: stuart.b at commonground.com.au (Stuart Bishop) Date: Tue Jun 3 19:54:09 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: Message-ID: <9C3EF241-961E-11D7-870A-000A95A06FC6@commonground.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >> I'm suggesting that we make this part of the API (2.1, perhaps) so >> that there's something there my application can depend on. I believe >> it's an important part of dealing with the DMBS, and should be part of >> the specs/requirements. > > +1 +1 It is needed for dumping generated SQL to error logs and such. I can't think of other tasks that absolutely require this, but there are plenty of tasks that are simplified by having it. Here are implementations of this, and more I'd be interested in, for PostgreSQL (I think based on code I ripped off from Psycopg). In particular, not all backends will allow binding of table or column names, so if you use case sensitive or non-alphanumeric table or column names you need to know the quoting rules for them. def quote(value): ''' Quote a value, such as a string or an integer, suitable for inclusion in an SQL command ''' # Currently barfs on dates. Numbers quoted due to lazyness # but justified to ensure we don't lose precision during transport. quote_dict = {"\'": "''", "\\": "\\\\"} for dkey in quote_dict.keys(): if value.find(dkey) >= 0: value=quote_dict[dkey].join(value.split(dkey)) return "'%s'" % value def quoteIdentifier(identifier): ''' Quote a database object name, such as a table or a view ''' quote_dict = {'\"': '""', "\\": "\\\\"} for dkey in quote_dict.keys(): if identifier.find(dkey) >= 0: identifier=quote_dict[dkey].join(identifier.split(dkey)) return '"%s"' % identifier cols = ['id','ISBN Number','DOI'] table = 'Demo House Publishers' match_col = 'ISBN Number' matches = ['1234567890','189481522X'] print 'SELECT (%s)\nFROM %s\nWHERE %s in (%s)' % ( ', '.join([quoteIdentifier(i) for i in cols]), quoteIdentifier(table), quoteIdentifier(match_col), ', '.join([quote(m) for m in matches]), ) Generates: SELECT ("id", "ISBN Number", "DOI") FROM "Demo House Publishers" WHERE "ISBN Number" in ('1234567890', '189481522X') - -- Stuart Bishop http://shangri-la.dropbear.id.au/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (Darwin) iD8DBQE+3TUOh8iUz1x5geARAgTTAJ9Ng8xhONbZSwDZ3z8nwpRb2rMCIgCbBZo1 xLKqgfDsdgrnrajVWlfWGcI= =+pfO -----END PGP SIGNATURE----- From davidrushby at yahoo.com Tue Jun 3 17:54:45 2003 From: davidrushby at yahoo.com (David Rushby) Date: Tue Jun 3 19:54:49 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <72C2BE5B-95F4-11D7-8BD4-000393B658A2@cogdon.org> Message-ID: <20030603235445.74991.qmail@web11003.mail.yahoo.com> --- Chris Cogdon wrote: > > On Tuesday, Jun 3, 2003, at 11:45 US/Pacific, David Rushby wrote: > >> Now, if this is true, then the DMBS has a quoting convention that > >> I believe should be exposed. > > > > Again, "I believe should be exposed" != "actually is exposed in the > > client API". > > I know that, which is why I'm making the suggestion that we turn the != > into a == I phrased that poorly. I meant that some DBMSes "have a quoting convention", yet don't expose it in their *native* C clients APIs, which makes exposure in the Python driver immaterial (unless the author of the Python driver were to write his own quoting code, which would be error-prone and likely to break or not be extended to new data types across DBMS versions). __________________________________ Do you Yahoo!? Yahoo! Calendar - Free online calendar with sync to Outlook(TM). http://calendar.yahoo.com From chris at cogdon.org Tue Jun 3 18:07:08 2003 From: chris at cogdon.org (Chris Cogdon) Date: Tue Jun 3 20:07:13 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <20030603235445.74991.qmail@web11003.mail.yahoo.com> Message-ID: <7BBD2F0A-9620-11D7-8BD4-000393B658A2@cogdon.org> On Tuesday, Jun 3, 2003, at 16:54 US/Pacific, David Rushby wrote: > I phrased that poorly. I meant that some DBMSes "have a quoting > convention", > yet don't expose it in their *native* C clients APIs, Actually, I don't expect any DBMS client library to expose a quoting function. They tend to be very minimalist :) > which makes exposure in > the Python driver immaterial (unless the author of the Python driver > were to > write his own quoting code, which would be error-prone and likely to > break or > not be extended to new data types across DBMS versions). That's the python driver writer's JOB, to support changes in the underlying DBMS, making those changes invisible to the application writer. Inside the python driver is EXACTLY the right place for this function. Yes, if something breaks, the application writer can always substitute new, working code, but that's not the ideal solution. It belongs in the python driver. If the DMBS client library is nice enough to supply a quoting function (either detecting or not detecting types), then the python driver writer could very well use the function if he or she wished, of course. Additionally, if new data types were added, then applications using the driver would simply not have access to these types until the driver was changed. What's the problem, here? -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From chris at cogdon.org Tue Jun 3 18:24:38 2003 From: chris at cogdon.org (Chris Cogdon) Date: Tue Jun 3 20:24:44 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <5.2.1.1.0.20030603230906.01f4dea8@www.thinkware.se> Message-ID: On Tuesday, Jun 3, 2003, at 15:06 US/Pacific, Magnus Lyck? wrote: > I don't see a need for this quote function for building SQL > statements, and I don't see how it could work cross platform > as presented, but I think it could be a good way to see what > is happening under the hood. I would like to have that for > pure debugging and logging purposes. Actually, I don't think it would work that well for logging, but... :) I most certainly have an application where this would make things easier for me (in fact, I'm already using it, but that's only because the python drivers I'm using have a callable 'quote' function). Can we take it as read that there are applications where this would be useful? No? Oh, okay... here's one. I have an XML interface into my database so that XML queries on the information can be made. For any of three data types, a number of optional criteria can be specified to restrict the information that's queried and sent back to the client. My function turns XML elements into individual parts of a 'where' clause. For example. CZZ gets turned into this 'where fragment' displayname<='CZZ' There could be zero or more of these fragments. Each of them is joined together with 'AND's and then added in as the WHERE clause of the SQL. Now, this only works because I can call the quote method to build up a valid fragment. Otherwise, I would have to write my own quoter (even more error prone than having it in the python driver), or somehow put ?'s in there, and then pass all the parameters I receive down the chain to be joined up with the full select statement. I don't think it would take much to realise that this is MUCH messier than being able to 'quote' as the where parts are being built up. Now... all this only works if the DBMSs that use the '? convention ALSO support passing literal values (such as 'CZZ') inside the SQL. And, if that's the case, then these DBMSs also have quoting conventions, and if that's the case, then I think there should be a function INSIDE THE DRIVER that does the quoting for us just like most of the PostgreSQL drivers do. Sure, in the case of the '?' DBMSes the quoting function wont be used for most applications, but I've just shown you one that does, and it's the most efficient way of doing it. > I don't think your example can work reliably for all platforms > quite like that. But this might work: > > cur.execute( *db.quote( "some SQL statement", parameters) ) My intention is that the 'quote' function returns a single string, with all the parameters quoted and embedded into the SQL. It could also work like this: def trivial_where_generator (): return db.quote ( "displayname = ?", displayname_value ) cur.execute ( "select * from artists where " + trivial_where_generator() ) > What these interfaces will actually do is to send the SQL string > just as it is, ? and all, down to the ODBC level at least. It will > convert the mxDateTime object to some suitable format for the > platform, but it will never worry about how to get it into the SQL > string, because it never will. After all, both Dynamic SQL and SQL > CLI mandates that parameter passing with ? should work, so as far > as I know, there will never be an SQL string with the date value > stuffed in. The database server will read directly from the parameter > list when is finds a ? in the SQL, instead of doing the labour of > extracting a string literal that it perhaps needs to unquote and > unescape. Seems like less work. Since that matches with my paraphrase a few emails ago, I agree with that. > For every server, there *is* at least one potential SQL string > which would work in an equivalent way to the SQL with the parameters > we sent, but there is no reason for any part of the chain between > the application and the db server to know what that might look like. Unless, of course, in the particular application its much more efficient to deal with full SQL strings. Again, I'm not suggesting that this is a 'good thing' for all applications, only some :) > In this particular case it will vary depending on the data source. > If the database happens to use the MS Jet engine, (i.e. MS Access) > the SQL could look like "SELECT * FROM T WHERE D = #1999-12-31#", > but for most other backends it should look like > "SELECT * FROM T WHERE D = '1999-12-31'". The #-version would only > work on Jet, and it's the only one that works on Jet. Depending on > national seetings, what kind of database you have etc, the date > string might look different, regardless of quote-style. Passed as > a separate parameter, it might work to send it as a COM Date object > if we're in Windows. Which is why I'm asking for it to be a part of the python driver. The driver knows what type of database it is talking to, so it is perfectly capable of generating the correct 'string-equivalent SQL' for the database. For example the following 'proposed' statement: db.quote ( "select * from T where D=?", date ) would generate a different string depending on what DMBS it is talking to. > If the DB-API would try to pass in the parameter into the SQL > string, it could just guess how to do that, and it is competely > beyond its "duties". That's EXACTLY what all of the PostgreSQL APIs do right now. It's their JOB to figure out how to get those parameters into the SQL string to send them to the DBMS. I appreciate that other DBMSes do not need to worry about quoting, since they can use a different mechanism to get the parameters to the DBMS. However, I claim that there are applications where 'pre-quoting' is considerably more efficient for the application, and I'm asking that the next revision of the API include a method to make a 'embed into SQL string' function available to the application writer. > In other words, I would like such a quote function to return > what it would actually send to the next step in the chain, be > that ADO or the MySQL client libs. Either just a string, or a > tuple containing a string and a converted parameter tuple---but > I'm not sure it's a reasonable thing to ask for in all interfaces. I'm asking that it be a straight string, since that's easiest to manipulate in the application. If for some reason the DMBS does NOT support some literals being passed as a straight string, then an error be returned. After all, if it can't be 'quoted', then there's no way of achieving what you want anyway. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From davidrushby at yahoo.com Tue Jun 3 18:47:26 2003 From: davidrushby at yahoo.com (David Rushby) Date: Tue Jun 3 20:47:31 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <7BBD2F0A-9620-11D7-8BD4-000393B658A2@cogdon.org> Message-ID: <20030604004726.33545.qmail@web11006.mail.yahoo.com> --- Chris Cogdon wrote: > > On Tuesday, Jun 3, 2003, at 16:54 US/Pacific, David Rushby wrote: > > > I phrased that poorly. I meant that some DBMSes "have a quoting > > convention", yet don't expose it in their *native* C clients APIs, > > Actually, I don't expect any DBMS client library to expose a quoting > function. They tend to be very minimalist :) > > > which makes exposure in the Python driver immaterial (unless the > > author of the Python driver were to write his own quoting code, > > which would be error-prone and likely to break or not be extended > > to new data types across DBMS versions). > > That's the python driver writer's JOB, to support changes in the > underlying DBMS, making those changes invisible to the application > writer. Inside the python driver is EXACTLY the right place for this > function. > > If the DMBS client library is nice enough to supply a quoting function > (either detecting or not detecting types), then the python driver > writer could very well use the function if he or she wished, of course. > > Additionally, if new data types were added, then applications using the > driver would simply not have access to these types until the driver was > changed. What's the problem, here? "If the DMBS client library is nice enough to supply a quoting function (either detecting or not detecting types)"? How would the quoting facility be of any value if it didn't detect types? As Kevin said in a previous mail: """ Yes, and it works when binding because there is enough context to map each parameter to a particular SQL type. Generic quoting without that context is simply not possible without significant infrastructure or meta-data. It is a great deal harder than exposing an internal method that already exists within a driver. """ Without support from the native database client library, the Python driver would have to fully parse the SQL statement, determine the internal types of all the fields its parameters refer to (by looking up metadata in the system tables), and then perform the quoting. I say that's well outside "the python driver writer's JOB". __________________________________ Do you Yahoo!? Yahoo! Calendar - Free online calendar with sync to Outlook(TM). http://calendar.yahoo.com From chris at cogdon.org Tue Jun 3 19:06:44 2003 From: chris at cogdon.org (Chris Cogdon) Date: Tue Jun 3 21:06:48 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <20030604004726.33545.qmail@web11006.mail.yahoo.com> Message-ID: On Tuesday, Jun 3, 2003, at 17:47 US/Pacific, David Rushby wrote: > "If the DMBS client library is nice enough to supply a quoting function > (either detecting or not detecting types)"? How would the quoting > facility > be of any value if it didn't detect types? What I'm proposing DOES detect types. Ie, it works exactly the same way the 'execute' function does (on a string-only DBMS), except it returns the string, rather than passing it to the DBMS. > Without support from the native database client library, the Python > driver > would have to fully parse the SQL statement, determine the internal > types of > all the fields its parameters refer to (by looking up metadata in the > system > tables), and then perform the quoting. > > I say that's well outside "the python driver writer's JOB". pyPgSQL and pgdb already do quoting, already parse the SQL statement looking for the %s markers, and already do type detection, all without support from the database client library. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From zen at shangri-la.dropbear.id.au Wed Jun 4 12:15:22 2003 From: zen at shangri-la.dropbear.id.au (Stuart Bishop) Date: Tue Jun 3 21:15:45 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <20030604004726.33545.qmail@web11006.mail.yahoo.com> Message-ID: <03CB8ECF-962A-11D7-870A-000A95A06FC6@shangri-la.dropbear.id.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, June 4, 2003, at 10:47 AM, David Rushby wrote: > "If the DMBS client library is nice enough to supply a quoting function > (either detecting or not detecting types)"? How would the quoting > facility > be of any value if it didn't detect types? > > As Kevin said in a previous mail: > """ > Yes, and it works when binding because there is enough context to map > each > parameter to a particular SQL type. Generic quoting without that > context is > simply not possible without significant infrastructure or meta-data. > It is > a great deal harder than exposing an internal method that already > exists > within a driver. > """ In what cases is this true, and can these cases be represented as an SQL command? > Without support from the native database client library, the Python > driver > would have to fully parse the SQL statement, determine the internal > types of > all the fields its parameters refer to (by looking up metadata in the > system > tables), and then perform the quoting. > > I say that's well outside "the python driver writer's JOB". I know that *I* can write 'SELECT a FROM b WHERE c = 1' without knowing that column b in table a has a numeric type, or even caring. You simply have to quote based on the type of the parameters given to the quote function. If it is passed as a string, quote it as a string. If it is a dbiDate, quote it as a date. As far as I can see, this trivial approach works happily for generating anything that can actually be represented as an SQL command (so if I call quote(blob) where blob is a 2GB string, I would expect an exception telling me I'm being stupid. But if blob happens to be < 2000 characters and I'm talking to an Oracle driver I'd expect a nicely quoted ASCII representation). - -- Stuart Bishop http://shangri-la.dropbear.id.au/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (Darwin) iD8DBQE+3Ugxh8iUz1x5geARAqvEAJ4lwAATN5FAwBD3Tqgrus+QsLSNpQCg6Vb0 8PE5ncngDnbGf30/MnjvUSU= =djd5 -----END PGP SIGNATURE----- From chris at cogdon.org Tue Jun 3 19:46:20 2003 From: chris at cogdon.org (Chris Cogdon) Date: Tue Jun 3 21:46:23 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <03CB8ECF-962A-11D7-870A-000A95A06FC6@shangri-la.dropbear.id.au> Message-ID: <56E404C0-962E-11D7-8BD4-000393B658A2@cogdon.org> I'm certainly getting a lot more 'aggressive' feedback from this proposal than I was expecting. I was hoping for more of a 'good idea', or 'good idea, but a lot of work in these circumstances X, Y, Z', or a 'you're doing it the wrong way, try ABC' instead. Have I really gotten on people's nerves so much that they wont even LOOK at the application I'm trying to write? Perhaps suggesting alternatives, or something else? Anyway... I'd like to consolidate some of the arguments I've received: Doing what I'm proposing with a DMBS that requires pure-string statements (like PostgreSQL and MySQL) is relatively easy, since the driver has to quote anyway: making it visible in a standard form is little work. However, for databases that handle all the 'binding' on the DB site is hard because there's NO current code in the driver that does this functionality. Also, as I understand it, finding the relevant '?' marks (for example) in a SQL statement is non-trivial, since '?' may appear in places other than binding points (For example, inside a literal string). (Now, I had to go figure that one out for myself... someone could have just mentioned that). Because of the above issue, the proposal, useful or not, is non-trivial. Perhaps it would be better to write a library that allowed piecemeal construction of SQL Statements without binding the parameters into the string statement. But, instead, keeping them together until being passed into the 'execute' function. I'll omit the example code for brevity... but I think I could come up with something that demonstrates this point nicely. Write me if you want to see it. On other points... I still maintain that writing code to make a particular DB client library meet the DB-API spec *IS* the driver writer's job. Of course supplying a quoting function isn't part of that job yet, because it's not part of the specification. My proposal was to make the quoting function part of the next spec revision and therefore make it the driver writers job to get the quoting function correct. If the difficulty in doing so outweighs the utility of having this method, then please... just say so. I think it would be useful, but I can see alternatives that are difficult. However, it might not be as difficult as supporting quoting for a DMBS that does not currently require quoting to support db-api 2.0. If that's the case, then please, just say so. If someone could go through the example applications, the ones that I think a quoting method would be useful to use, and suggest another way of doing it, or perhaps just saying "do what you were suggesting as an alternate; write a few functions to let a string and parameters be joined together without quoting" then I would be appreciative. Regards, -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From magnus at thinkware.se Wed Jun 4 05:40:32 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Tue Jun 3 22:37:31 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: References: <5.2.1.1.0.20030603230906.01f4dea8@www.thinkware.se> Message-ID: <5.2.1.1.0.20030604034307.01f85bf0@www.thinkware.se> At 17:24 2003-06-03 -0700, Chris Cogdon wrote: >>In this particular case it will vary depending on the data source. >>If the database happens to use the MS Jet engine, (i.e. MS Access) >>the SQL could look like "SELECT * FROM T WHERE D = #1999-12-31#", >>but for most other backends it should look like >>"SELECT * FROM T WHERE D = '1999-12-31'". The #-version would only >>work on Jet, and it's the only one that works on Jet. Depending on >>national seetings, what kind of database you have etc, the date >>string might look different, regardless of quote-style. Passed as >>a separate parameter, it might work to send it as a COM Date object >>if we're in Windows. > >Which is why I'm asking for it to be a part of the python driver. The >driver knows what type of database it is talking to, so it is perfectly >capable of generating the correct 'string-equivalent SQL' for the database. The adodbapi *doesn't* know what's behind that ODBC data source. It might be PostgreSQL, MySQL, DB2, Sybase, Oracle, SAP DB or Jet etc. They can all behave differently. The Python driver has no clue, and even if it might be able to detect it, you can't honestly expect it to contain the code to handle all possible databases? I don't think there is a single SQL database out there that can run on Windows but lacks an ODBC driver. An ADO or ODBC driver would need to know every single database syntax out there, and it would only use that knowledge in this particular case, since it passes the parameters as is on execute. >That's EXACTLY what all of the PostgreSQL APIs do right now. It's their >JOB to figure out how to get those parameters into the SQL string to send >them to the DBMS. Only because PostrgreSQL fails to support Dynamic SQL and CLI. All the PostgreSQL drivers have to cover for this shortcoming, just as drivers for MySQL. That is not a feature, it's a shortcoming in those servers. (At least PostgreSQL supports embedded SQL, right?) >I appreciate that other DBMSes do not need to worry about quoting, since >they can use a different mechanism to get the parameters to the DBMS. >However, I claim that there are applications where 'pre-quoting' is >considerably more efficient for the application, and I'm asking that the >next revision of the API include a method to make a 'embed into SQL >string' function available to the application writer. It's always possible to build up entire SQL strings without the involvement of the DB-API. At least the application programmer can decide what backends and features he wants, and learn them. The db-api coder for an ODBC driver will have a hell of a job trying to predict what the application coder might want to do... I doubt your statement about efficiency. In what way? One could obviously imagine a quote function only for PostgreSQL and MySQL drivers, but does that have to be in the DB-API? It seems to me that this is what you are asking for. Naturally, one could put such logic in the drivers for Oracle or DB2 for instance, but why? It doesn't really belong there. In MySQL and PostgreSQL it's already there: It's part of the execute call today. From a debugging perspective, "quote" is certainly more useful in the cases where the Python driver does the quoting, but in general, I do feel it could be useful to see exactly what I would have sent to the next step in the message chain if I did an execute. -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The Agile Programming Language From bill.allie at mug.org Tue Jun 3 23:42:23 2003 From: bill.allie at mug.org (Billy G. Allie) Date: Tue Jun 3 22:42:28 2003 Subject: [DB-SIG] Python/PostgreSQL API performance comparison In-Reply-To: References: Message-ID: <3EDD5C8F.9040700@mug.org> Kevin Jacobs wrote: >[Updated to include results from pgdb and pyPgSQL] > >Due to the recent interest in PostgreSQL DB-API driver comparisons, I >thought I'd jump on the proverbial bandwagon. As some of you already know, >my company has developed very substantial financial applications using >Python, and frequently PostgreSQL. > >We've been using the PsycoPg driver for the past two years, because we found >it to be the most solid and least flawed of all the available drivers at the >time. I've updated our driver suite to include a recent versions PoPy, >pgdb, and pyPgSQL, and then run one of our very substantial test suites. >The suite constitutes a mix the following tasks: > > 1) Many simple OLTP queries. > 2) Many simple and complex OLAP queries, some returning as many as a > hundred thousand rows of data. > 3) Data-cube construction and manipulation > 4) Business report generation > >The application server and database run on the same (otherwise quiescent) >test server, the working data set is all in-core (though quite large), and >only serial requests (i.e., only a single active worker thread) are issued. > >Results running Python CVS on the same application server with only the >driver setting changed: > >Driver / version Wall Time (average of 4 runs, though little variation was >---------------- --------- observed) >PsycoPg 1.0.13 10m41s >PoPy 2.0.8 11m33s [1] >pgdb 3.3 14m40s >pyPgSQL 2.3 >15m51s [2] Did not complete all tests! > > >Notes: > >[1] Unfortunately, other than for this simple test suite, PoPy is basically > unusable for us. This is because it does not return proper PostgreSQL > type codes, only vague type strings ('NUMBER','DATETIME','MISSING'?!). > Thus, it does not provide enough information to distinguish, e.g., > booleans from numbers, numeric from floating-point values, dates from > datetimes, etc. As previously reported, it does not translate large > integers correctly, and it mangles some date interval types. These > deficiencies may not affect simpler or less-demanding applications, but > to us they qualify as unacceptable information loss. > >[2] pyPgSQL required several minor modifications to work properly. First, > the compile failed if LONG_LONG was not defined/detected. Second, it > returns a non-standard cursor.description with an extra element > indicating array status. While a potentially useful thing, it is very > very non-standard. Finally, the cursor descriptions use non-hashable > type codes, which caused problems for our OR-mapper. I modified the > type objects, adding an appropriate __hash__ method, and all was well. > Kevin, what system did you compile pyPgSQL on? Setup.py tries to detect if long long is supported on the system it is compiled on. If it is detected, then the LONG_LONG definition from pyport.h (via Python.h) is used. If long long support isn't detected, then it doesn't compile in pgint8 support. I'm curious as to your configuration. In regards to the description, pyPgSQL adds a array indicator as an eigth item. The symantics of the first seven items have not changed, they match the DB-API specification. How is this /very/ non-standard and how would this break code using the descriptor (descriptor[x][y] where x is the field (column) index and x is 0 through 6 will return what the DB-API 2.0 specification mandates)? You're right. PgTypes is not hashable. I never had a reason to have it hashable before. It will be hashable in the next release (the hash of a PgTypes will return the hash of the underlying type OID). > Once it was running, several test vectors failed deterministically with > the following error: > > /usr/local/python-cvs/lib/python2.3/site-packages/pyPgSQL/PgSQL.py: > OperationalError: RelationForgetRelation: relation xxxxxxx is still open > > > I've never run across that error before. Were there perhaps multiple threads sharing a connection? PostgreSQL's connections are not thread safe and pyPgSQL does not support threads sharing a connection (threadsafety == 1). -- ___________________________________________________________________________ ____ | Billy G. Allie | Domain....: Bill.Allie@mug.org | /| | 7436 Hartwell | MSN.......: B_G_Allie@email.msn.com |-/-|----- | Dearborn, MI 48126| |/ |LLIE | (313) 582-1540 | -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/db-sig/attachments/20030603/7a0df2b3/attachment-0001.htm From chris at cogdon.org Tue Jun 3 21:20:14 2003 From: chris at cogdon.org (Chris Cogdon) Date: Tue Jun 3 23:20:20 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <5.2.1.1.0.20030604034307.01f85bf0@www.thinkware.se> Message-ID: <750A8CA7-963B-11D7-8BD4-000393B658A2@cogdon.org> On Tuesday, Jun 3, 2003, at 19:40 US/Pacific, Magnus Lyck? wrote: > The adodbapi *doesn't* know what's behind that ODBC data source. It > might be PostgreSQL, MySQL, DB2, Sybase, Oracle, SAP DB or Jet etc. > They can all behave differently. > > The Python driver has no clue, and even if it might be able to detect > it, you can't honestly expect it to contain the code to handle all > possible databases? I don't think there is a single SQL database out > there that can run on Windows but lacks an ODBC driver. An ADO or ODBC > driver would need to know every single database syntax out there, and > it would only use that knowledge in this particular case, since it > passes the parameters as is on execute. Point conceded. >> That's EXACTLY what all of the PostgreSQL APIs do right now. It's >> their JOB to figure out how to get those parameters into the SQL >> string to send them to the DBMS. > > Only because PostrgreSQL fails to support Dynamic SQL and CLI. > All the PostgreSQL drivers have to cover for this shortcoming, > just as drivers for MySQL. That is not a feature, it's a > shortcoming in those servers. (At least PostgreSQL supports > embedded SQL, right?) Clue me in here. What's Dynamic SQL, CLI, and Embedded SQL ? > It's always possible to build up entire SQL strings without > the involvement of the DB-API. At least the application programmer > can decide what backends and features he wants, and learn them. Granted. However, the application programmer would also need to know how to correctly represent particular datatypes as a string, and that's something that 'string-only' APIs do right now. In those cases, it 'makes sense' to be able to call exactly the same function the API uses to convert those values to strings. Even now, that's driver-dependant (pg uses quote, PgSQL uses _quote, and it's not available at all on PoPy nor psycopg). But... I DO understand the difficulty in exposing a function that doesn't currently exist (such as the ODBC driver). And it may NOT be worth the difficulty in adding that functionality as a requirement for a later API. (Perhaps optional? At least if it's doable, we could get a consistent interface). > The db-api coder for an ODBC driver will have a hell of a job > trying to predict what the application coder might want to do... Point conceded. > I doubt your statement about efficiency. In what way? I've cited examples before, but... if they're not clear enough, I think I'd need to cite a complete working model. Can we take it as read that there may be SOME applications where this ability would be useful ? > One could obviously imagine a quote function only for PostgreSQL > and MySQL drivers, but does that have to be in the DB-API? My argument is yes, because the API knows more about the database it is connecting to than anything else... in some circumstances. In the others, perhaps it could remain unimplemented, and the application writer is on their own. > It > seems to me that this is what you are asking for. Naturally, one > could put such logic in the drivers for Oracle or DB2 for instance, > but why? It doesn't really belong there. In MySQL and PostgreSQL > it's already there: It's part of the execute call today. Yes, but the interface is non-standard, or not reachable. If it was in a consistent place (as I'm suggesting), then I could swap between drivers or databases at will. I concede that, given that some DBMSes do not need to quote internally at all, that perhaps the solution to the application, to convert to strings before reaching the execute method, is the wrong solution. How about I try and reduce my application down to a simple example using quote, and then rewrite it using my somewhat more complicated alternative, and you can all pick over that. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From philippe.entzmann at laposte.net Wed Jun 4 10:18:05 2003 From: philippe.entzmann at laposte.net (Philippe ENTZMANN) Date: Wed Jun 4 03:32:40 2003 Subject: [DB-SIG] MSSQL "proxy" access Message-ID: Hi, I'm prototyping an application with Python and MSSQL. I use odbc module for data access. I would like to be able to run my app both on Windows and Linux. Is there a way to access directly an MSSQL database from a python program running on Linux ? Is it a good idea to use some XMLRPC stuff to "proxy" the data requests from the clients (Linux and Windows) to the MSSQL server thru a python server (Windows) ? Is there already a module for doing such things ? (db proxiing via XMLRPC) Thanks. From mal at lemburg.com Wed Jun 4 10:38:56 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jun 4 03:39:21 2003 Subject: [DB-SIG] Asymmetry in DB-API 2.0 types In-Reply-To: References: Message-ID: <3EDDA210.8020807@lemburg.com> Harald Meland wrote: > Hi, > > as the current DB-API stands, the types it mentions can be constructed > via standard calls, but there's no standard module-independent way to > decipher such types. > > An example: It's reasonably easy to pass a DateTime object as a bind > argument to .execute() in a driver-independent way. However, if I'd > like to get the date part of a DateTime object returned from a query, > I'll have to jump through module-specific hoops. True. > Is this still another case of "this can't or shouldn't be handled by > the driver, but by a (hypothetical) separate layer on top of the > drivers", or is there interest in extending the type definitions in > the DB-API to include deciphering features? The driver output values are driver dependent and there's nothing much the API spec could do about this because not all databases support all different kinds of data types and it is not even clear for which result columns a specific data type is needed or desired. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Jun 04 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 20 days left From mal at lemburg.com Wed Jun 4 11:02:22 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jun 4 04:02:57 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <750A8CA7-963B-11D7-8BD4-000393B658A2@cogdon.org> References: <750A8CA7-963B-11D7-8BD4-000393B658A2@cogdon.org> Message-ID: <3EDDA78E.5070102@lemburg.com> Chris Cogdon wrote: > But... I DO understand the difficulty in exposing a function that > doesn't currently exist (such as the ODBC driver). And it may NOT be > worth the difficulty in adding that functionality as a requirement for a > later API. (Perhaps optional? At least if it's doable, we could get a > consistent interface). That we could do, but note that quoting would be requiring type information in database specific ways, plus it may sometimes also need encoding or code page information for string values, e.g. a database running with UTF-8 code page will want to see UTF-8 in string literals. BTW, ODBC has a function called SQLNativeSql() which returns the rewritten SQL statement in the database backend SQL dialect. Not all drivers implement this and those that do only deal with ODBC quoted SQL functions or joins. If it were an easy to have API, believe me, Microsoft would have put that into the standard as well :-) -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Jun 04 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 20 days left From mal at lemburg.com Wed Jun 4 10:56:46 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jun 4 04:12:17 2003 Subject: [DB-SIG] MSSQL "proxy" access In-Reply-To: References: Message-ID: <3EDDA63E.6060400@lemburg.com> Philippe ENTZMANN wrote: > Hi, > > I'm prototyping an application with Python and MSSQL. I use odbc module > for data access. > > I would like to be able to run my app both on Windows and Linux. > > Is there a way to access directly an MSSQL database from a python > program running on Linux ? > > Is it a good idea to use some XMLRPC stuff to "proxy" the data requests > from the clients (Linux and Windows) to the MSSQL server thru a python > server (Windows) ? > > Is there already a module for doing such things ? (db proxiing via XMLRPC) mxODBC via FreeTDS should work. They both run on Unix. As for proxying, eGenix has a project in the queue which will make this kind of thing easy to setup and with good performance for ODBC drivers in general (not using XMLRPC, BTW, since that introduces too much overhead). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Jun 04 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 20 days left From mal at lemburg.com Wed Jun 4 11:17:34 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jun 4 04:18:09 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <5.2.1.1.0.20030603230906.01f4dea8@www.thinkware.se> References: <5.2.1.1.0.20030603230906.01f4dea8@www.thinkware.se> Message-ID: <3EDDAB1E.80903@lemburg.com> Magnus Lyck? wrote: > cur.execute( *db.quote( "some SQL statement", parameters) ) > > Not that it would be of much use though... > > Let's take a concrete example. Let's assume that we use either > mxODBC or adodbapi on Windows, and connect to a System data source > called "databaseX". It's just an ODBC source, and we don't know > what kind of database it is. What do we care--it speaks ODBC. > > Our SQL is "SELECT * FROM T WHERE D = ?" and we send in a single > mxDateTime object as a parameter. > > What these interfaces will actually do is to send the SQL string > just as it is, ? and all, down to the ODBC level at least. It will > convert the mxDateTime object to some suitable format for the > platform, but it will never worry about how to get it into the SQL > string, because it never will. After all, both Dynamic SQL and SQL > CLI mandates that parameter passing with ? should work, so as far > as I know, there will never be an SQL string with the date value > stuffed in. The database server will read directly from the parameter > list when is finds a ? in the SQL, instead of doing the labour of > extracting a string literal that it perhaps needs to unquote and > unescape. Seems like less work. Even better: the database server can prepare the data access paths from the abstract SQL statement and cache that access path for subsequent queries -- that's hardly possible if you have the arguments quoted and embedded directly into the SQL. In the latter case, you'd also lose the DB-API feature of caching the last used SQL statement on a database cursor. Another argument against using embedded data types is that of precision loss, e.g. ODBC interfaces C types directly to the driver which assures that you don't lose precision by converting them to a string representation. This happens for Python floats, integers and even date/time values. > If an interface is implemented in C or C++, it's not certain that > the actual data type being passed to the next step can be described > in Python at all? M-A? What about mxODBC for instance? This depends on the support from the ODBC driver for querying data types of input parameters: Most drivers can tell the interface which data types they would like to see and mxODBC then converts the Python types accordingly. Some drivers don't provide this information, so mxODBC has to do an educated guess based on the Python type passed in by the programmer -- if it doesn't work, then the interface will generate an exception and the programmer will have to adapt the program to pass in more suitable types. Quoting of data types only rarely becomes visible in the ODBC layer interaction and most of the times you only need to quote strings for e.g. table names etc. (note that these are not string literals !). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Jun 04 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 20 days left From msanchez at grupoburke.com Wed Jun 4 11:31:55 2003 From: msanchez at grupoburke.com (=?ISO-8859-1?Q?Marcos_S=E1nchez_Provencio?=) Date: Wed Jun 4 04:32:56 2003 Subject: [DB-SIG] MSSQL "proxy" access In-Reply-To: References: Message-ID: <3EDDAE7B.3000609@grupoburke.com> Philippe ENTZMANN escribi?: > Hi, > > I'm prototyping an application with Python and MSSQL. I use odbc module > for data access. > > I would like to be able to run my app both on Windows and Linux. > > Is there a way to access directly an MSSQL database from a python > program running on Linux ? > > Is it a good idea to use some XMLRPC stuff to "proxy" the data requests > from the clients (Linux and Windows) to the MSSQL server thru a python > server (Windows) ? > > Is there already a module for doing such things ? (db proxiing via XMLRPC) > > Thanks. > The Sybase module http://object-craft.com.au/projects/sybase/ + FreeTDS can access MSSQL. Parameter binding is not ready yet for FreeTDS, you'll have to write your own quoting code :-\ From philippe.entzmann at laposte.net Wed Jun 4 12:51:16 2003 From: philippe.entzmann at laposte.net (Philippe ENTZMANN) Date: Wed Jun 4 05:53:26 2003 Subject: [DB-SIG] Re: MSSQL "proxy" access In-Reply-To: <3EDDA63E.6060400@lemburg.com> References: <3EDDA63E.6060400@lemburg.com> Message-ID: M.-A. Lemburg wrote: > Philippe ENTZMANN wrote: > >> Hi, >> >> I'm prototyping an application with Python and MSSQL. I use odbc >> module for data access. >> >> I would like to be able to run my app both on Windows and Linux. >> >> Is there a way to access directly an MSSQL database from a python >> program running on Linux ? >> >> Is it a good idea to use some XMLRPC stuff to "proxy" the data >> requests from the clients (Linux and Windows) to the MSSQL server thru >> a python server (Windows) ? >> >> Is there already a module for doing such things ? (db proxiing via >> XMLRPC) > > > mxODBC via FreeTDS should work. They both run on Unix. > Thanks for this info. mxODBC is a commercial product. Is there any free working alternative ? From Harald.Meland at usit.uio.no Wed Jun 4 13:05:46 2003 From: Harald.Meland at usit.uio.no (Harald Meland) Date: Wed Jun 4 06:05:55 2003 Subject: [DB-SIG] Asymmetry in DB-API 2.0 types In-Reply-To: <3EDDA210.8020807@lemburg.com> (mal@lemburg.com's message of "Wed, 04 Jun 2003 09:38:56 +0200") References: <3EDDA210.8020807@lemburg.com> Message-ID: [M.-A. Lemburg] > The driver output values are driver dependent and there's nothing > much the API spec could do about this because not all databases > support all different kinds of data types and it is not even > clear for which result columns a specific data type is needed > or desired. Wouldn't it be possible to extend the current specification for e.g. DATETIME: DATETIME This type object is used to describe date/time columns in a database. to include an API for deciphering such type objects, e.g. like this (we'll obviously need to work on the exact behaviour of the various methods, this is just an example to get the discussion started): Objects of type DATETIME must support the following methods: date() Return a string on the form "YYYY-MM-DD" representing the date part of the object's value. time() Return a string on the form "HH:MM:SS" representing the time part of the object's value. ticks() Return the object's value as seconds since epoch; raises ValueError if object's value is not representable as ticks. year instance variable containing the (4-digit) year, as an integer, of the object's value. ... The NUMBER type could be required to expose information on e.g. precision in a standard way, etc. After all, every driver has to use a *single* type for each value a query returns; if the value doesn't fit into one of the standard types mentioned in the DB-API, then the driver is of course free to use some other type for that. -- Harald From magnus at thinkware.se Wed Jun 4 13:10:53 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Wed Jun 4 06:07:50 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <750A8CA7-963B-11D7-8BD4-000393B658A2@cogdon.org> References: <5.2.1.1.0.20030604034307.01f85bf0@www.thinkware.se> Message-ID: <5.2.1.1.0.20030604103716.01e91a28@www.thinkware.se> At 20:20 2003-06-03 -0700, Chris Cogdon wrote: >Clue me in here. What's Dynamic SQL, CLI, and Embedded SQL ? Phew... I'm not so good at short explanations. :) Among the things described in the SQL standard, are standardized ways to call the database from within your program. You then need to pass data from C/C++/COBOL/ADA/whatever variables to the SQL statements and back. This is called binding. Over the years, different binding styles have been invented, and the SQL/92 standard identifies four different. I don't know if any more has been added in later versions of the SQL Standards. (Something for Java?) The four in SQL/92 are: module, embedded SQL (ESQL), Call-Level Interface (CLI) and direct SQL. Module is the oldest, and I don't know how much it's used outside COBOL or PL/I. For instance, you might write... PROCEDURE DELETE_PART ( SQLSTATE, :PNO_PARAM CHAR(6) ) ; DELETE FROM P WHERE P.PNO = :PNO_PARAM ; ...in the module language, and as far as I understand, you compile this with a specific compiler, and link it with your PL/I program where you do... CALL DELETE_PART ( RETCODE, PNO_ARG ) ; ...to call this procedure. I think Embedded SQL (ESQL) is what most programmers who write SQL from C or C++ have used. You place SQL statements inside special sections in your C code, and run a pre-processor that turns this into RDBMS specific function calls, before you compile your program. Depending on your backend, different things might happen here. In DB2 your SQL statements will end up in a bind-file that you store in the database more or less like if they were stored procedures, and the C/C++ code that came from the ESQL pre-processor will contain function calls that call these prodedures, and map parameters to your local variables. It's up to the RDBMS to make your ESQL turn into something that works. At least ESQL comes in two "Flavours": Static and Dynamic. (I don't know if there is dynamic SQL in module. I doubt it.) In static SQL, your entire SQL statement is determined at compile time, and it might look like this in the middle of your C code: EXEC SQL DELETE FROM P WHERE P.PNO = :PNO_PARAM ; PNO_PARAM has been declared as a C variable of suitable type inside a special "DECLARE SECTION" before it's used. For selects etc that might return many rows, you will need to perform an EXECUTE, and then FETCH in a loop and check for error codes etc. It's not too different from the Python DB-API in principle, but much more verbose and complicated, since it can't rely on high level features like Python's lists, tuples and exceptions. With dynamic SQL, which isn't used as often, you can't use the :PNO_PARAM approach above. That approach is used to bind a C variable to the SQL call att compile time, which won't work if the SQL string is to be built dynamically. It's not Python you know... :) Instead, you use '?' as a placeholder in your SQL string, and pass the parameters separately using a rather complicated approach. Let's leave that. CLI is a much better than Dynamic SQL, but the syntax of the SQL strings are the same, it's the surrounding junk that differs. The Command-Level Interface (CLI) is the most recent interface and binding style. Like Dynamic SQL, '?' is used for parameter passing. While Embedded SQL defines a source code standard, CLI defines an API. With CLI, we don't need to bother with Embedded SQL and preprocessors at all. Instead, we call functions with standardized names, passing in SQL as character arrays or whatever our programming language supports. Binding variables is still much, much more verbose than in Python, but at least it's possible to use the same binary to connect to different backends if you use a least common denominator for SQL. Direct SQL is what you run from an interactive SQL monitor, such as psql. That you know... >Granted. However, the application programmer would also need to know how >to correctly represent particular datatypes as a string, and that's >something that 'string-only' APIs do right now. In those cases, it 'makes >sense' to be able to call exactly the same function the API uses to >convert those values to strings. Even now, that's driver-dependant (pg >uses quote, PgSQL uses _quote, and it's not available at all on PoPy nor >psycopg). I agree with that. >But... I DO understand the difficulty in exposing a function that doesn't >currently exist (such as the ODBC driver). And it may NOT be worth the >difficulty in adding that functionality as a requirement for a later API. >(Perhaps optional? At least if it's doable, we could get a consistent >interface). Agree here too. >I've cited examples before, but... if they're not clear enough, I think >I'd need to cite a complete working model. Can we take it as read that >there may be SOME applications where this ability would be useful ? Do you mean that the SQL would run faster in the database, or do you mean that it would be faster for you to code the SQL? I don't doubt that the latter can be true. >Yes, but the interface is non-standard, or not reachable. If it was in a >consistent place (as I'm suggesting), then I could swap between drivers or >databases at will. I agree with that. -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The Agile Programming Language From jacobs at penguin.theopalgroup.com Wed Jun 4 07:20:44 2003 From: jacobs at penguin.theopalgroup.com (Kevin Jacobs) Date: Wed Jun 4 06:21:20 2003 Subject: [DB-SIG] MSSQL "proxy" access In-Reply-To: Message-ID: On Wed, 4 Jun 2003, Philippe ENTZMANN wrote: > > I'm prototyping an application with Python and MSSQL. I use odbc module > for data access. > > I would like to be able to run my app both on Windows and Linux. > > Is there a way to access directly an MSSQL database from a python > program running on Linux ? We have been very happy using David Cole's Sybase module compiled with FreeTDS (which can speak MSSQL). -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From jacobs at penguin.theopalgroup.com Wed Jun 4 07:21:59 2003 From: jacobs at penguin.theopalgroup.com (Kevin Jacobs) Date: Wed Jun 4 06:22:34 2003 Subject: [DB-SIG] MSSQL "proxy" access In-Reply-To: <3EDDA63E.6060400@lemburg.com> Message-ID: On Wed, 4 Jun 2003, M.-A. Lemburg wrote: > As for proxying, eGenix has a project in the queue which will make > this kind of thing easy to setup and with good performance for ODBC > drivers in general (not using XMLRPC, BTW, since that introduces too > much overhead). You're working on an ODBC-ODBC bridge? -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From magnus at thinkware.se Wed Jun 4 13:32:59 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Wed Jun 4 06:29:58 2003 Subject: [DB-SIG] Re: MSSQL "proxy" access In-Reply-To: Message-ID: <5.2.1.1.0.20030604122036.01f4e818@www.thinkware.se> Philippe ENTZMANN wrote: >mxODBC is a commercial product. Is there any free working alternative ? SQL Relay might be useful for you: http://sqlrelay.sourceforge.net/ But it also relies on FreeTDS for access to MS SQL Server, so it might not bring any advantage above the ordinary Sybase driver. It is another server process to run... For access on Windows, you might consider using ADO or the adodbapi package as alternatives to the odbc module in win32all. The odbc package is fairly old, and not complient with DB-API version 2 as far as I understand. -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The Agile Programming Language From mal at lemburg.com Wed Jun 4 13:31:10 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jun 4 06:31:45 2003 Subject: [DB-SIG] MSSQL "proxy" access In-Reply-To: References: Message-ID: <3EDDCA6E.4020202@lemburg.com> Kevin Jacobs wrote: > On Wed, 4 Jun 2003, M.-A. Lemburg wrote: > >>As for proxying, eGenix has a project in the queue which will make >>this kind of thing easy to setup and with good performance for ODBC >>drivers in general (not using XMLRPC, BTW, since that introduces too >>much overhead). > > You're working on an ODBC-ODBC bridge? Well, sort of. It's actually an mxODBC-mxODBC bridge making use of the higher level interface we have in mxODBC (more available information means less network traffic :-). Ideally it should then be possible to separate the ODBC driver installation from the application using the mxODBC interface,e.g. you can have the mxODBC application running on Linux, talking to the Excel ODBC driver on a Windows box, or the MS SQL ODBC driver on the MS SQL Server machine. That should get rid off the problems you sometimes have in finding suitable ODBC drivers for the platform running the application. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Jun 04 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 20 days left From andy47 at halfcooked.com Wed Jun 4 16:31:18 2003 From: andy47 at halfcooked.com (Andy Todd) Date: Wed Jun 4 10:32:21 2003 Subject: [DB-SIG] Asymmetry in DB-API 2.0 types In-Reply-To: References: <3EDDA210.8020807@lemburg.com> Message-ID: <3EDE02B6.6060801@halfcooked.com> Harald Meland wrote: > [M.-A. Lemburg] > > >>The driver output values are driver dependent and there's nothing >>much the API spec could do about this because not all databases >>support all different kinds of data types and it is not even >>clear for which result columns a specific data type is needed >>or desired. > > > Wouldn't it be possible to extend the current specification for > e.g. DATETIME: > > DATETIME > This type object is used to describe date/time columns in a > database. > > to include an API for deciphering such type objects, e.g. like this > (we'll obviously need to work on the exact behaviour of the various > methods, this is just an example to get the discussion started): > > Objects of type DATETIME must support the following methods: > > date() > Return a string on the form "YYYY-MM-DD" representing the date > part of the object's value. > > time() > Return a string on the form "HH:MM:SS" representing the time > part of the object's value. > > ticks() > Return the object's value as seconds since epoch; raises > ValueError if object's value is not representable as ticks. > > year > instance variable containing the (4-digit) year, as an integer, > of the object's value. > > ... > > The NUMBER type could be required to expose information on > e.g. precision in a standard way, etc. > > After all, every driver has to use a *single* type for each value a > query returns; if the value doesn't fit into one of the standard types > mentioned in the DB-API, then the driver is of course free to use some > other type for that. A standard way to access date, time and datetime objects is, in my opinion, a good idea. Most of the drivers I use work with mxDateTime but that can cause issues for some people because of the license. There is an alternative. Its only in beta at the moment, but Python 2.3 has a datetime module in the standard library; http://www.python.org/doc/2.3b1/lib/module-datetime.html So if we're going to pick a standard I'd be going with that one. I don't know if there are any plans to back port this module to prior releases of Python but I wouldn't be surprised. Regards, Andy -- -------------------------------------------------------------------------------- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From msanchez at grupoburke.com Wed Jun 4 18:37:54 2003 From: msanchez at grupoburke.com (=?ISO-8859-1?Q?Marcos_S=E1nchez_Provencio?=) Date: Wed Jun 4 11:38:32 2003 Subject: [DB-SIG] Asymmetry in DB-API 2.0 types In-Reply-To: <3EDE02B6.6060801@halfcooked.com> References: <3EDDA210.8020807@lemburg.com> <3EDE02B6.6060801@halfcooked.com> Message-ID: <3EDE1252.5040908@grupoburke.com> ?What is the problem with the license? I believe it is more or less the same as Python's. I want to thank Mr Lemburg for this and other contributions he has given to the Python community. Andy Todd escribi?: > Harald Meland wrote: > >> [M.-A. Lemburg] >> >> >>> The driver output values are driver dependent and there's nothing >>> much the API spec could do about this because not all databases >>> support all different kinds of data types and it is not even >>> clear for which result columns a specific data type is needed >>> or desired. >> >> >> >> Wouldn't it be possible to extend the current specification for >> e.g. DATETIME: >> >> DATETIME >> This type object is used to describe date/time columns in a >> database. >> >> to include an API for deciphering such type objects, e.g. like this >> (we'll obviously need to work on the exact behaviour of the various >> methods, this is just an example to get the discussion started): >> >> Objects of type DATETIME must support the following methods: >> >> date() >> Return a string on the form "YYYY-MM-DD" representing the date >> part of the object's value. >> >> time() >> Return a string on the form "HH:MM:SS" representing the time >> part of the object's value. >> >> ticks() >> Return the object's value as seconds since epoch; raises >> ValueError if object's value is not representable as ticks. >> >> year >> instance variable containing the (4-digit) year, as an integer, >> of the object's value. >> >> ... >> >> The NUMBER type could be required to expose information on >> e.g. precision in a standard way, etc. >> >> After all, every driver has to use a *single* type for each value a >> query returns; if the value doesn't fit into one of the standard types >> mentioned in the DB-API, then the driver is of course free to use some >> other type for that. > > > A standard way to access date, time and datetime objects is, in my > opinion, a good idea. Most of the drivers I use work with mxDateTime but > that can cause issues for some people because of the license. > > There is an alternative. Its only in beta at the moment, but Python 2.3 > has a datetime module in the standard library; > > http://www.python.org/doc/2.3b1/lib/module-datetime.html > > So if we're going to pick a standard I'd be going with that one. I don't > know if there are any plans to back port this module to prior releases > of Python but I wouldn't be surprised. > > Regards, > Andy From andy47 at halfcooked.com Wed Jun 4 17:56:30 2003 From: andy47 at halfcooked.com (Andy Todd) Date: Wed Jun 4 11:57:38 2003 Subject: [DB-SIG] Asymmetry in DB-API 2.0 types In-Reply-To: <3EDE1252.5040908@grupoburke.com> References: <3EDDA210.8020807@lemburg.com> <3EDE02B6.6060801@halfcooked.com> <3EDE1252.5040908@grupoburke.com> Message-ID: <3EDE16AE.4080508@halfcooked.com> Marcos S?nchez Provencio wrote: > ?What is the problem with the license? I believe it is more or less the > same as Python's. I want to thank Mr Lemburg for this and other > contributions he has given to the Python community. > > Andy Todd escribi?: > >> Harald Meland wrote: >> >>> [M.-A. Lemburg] >>> >>> [snippage] Sorry, my mistake. It is mxODBC that isn't free, not mxDateTime. Apologies to all and I'd also like to join in the fulsome praise of MAL's contributions. Regards, Andy -- -------------------------------------------------------------------------------- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From chris at cogdon.org Wed Jun 4 18:06:18 2003 From: chris at cogdon.org (Chris Cogdon) Date: Wed Jun 4 20:06:30 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <3EDDA78E.5070102@lemburg.com> Message-ID: <880CBAE9-96E9-11D7-8BD4-000393B658A2@cogdon.org> On Wednesday, Jun 4, 2003, at 01:02 US/Pacific, M.-A. Lemburg wrote: > Chris Cogdon wrote: >> But... I DO understand the difficulty in exposing a function that >> doesn't currently exist (such as the ODBC driver). And it may NOT be >> worth the difficulty in adding that functionality as a requirement >> for a later API. (Perhaps optional? At least if it's doable, we could >> get a consistent interface). > > That we could do, but note that quoting would be requiring type > information in database specific ways, plus it may sometimes also > need encoding or code page information for string values, e.g. > a database running with UTF-8 code page will want to see UTF-8 > in string literals. I believe the python driver ALREADY has access to this information, since it needs it to determine how to correctly pass this information to the DBMS, be it in a string or binary format. I do agree, however, that converting to string format is a) non-trivial, especially if you're not doing it already, and b) prone to precision loss in the case of floating point numbers. Both of those issues are not insurmountable, and do not make the feature particularly less useful. I DO appreciate the amount of work that the driver writers have done in interpreting the DMBS specifications, requirements and APIs, and turning them into a python driver that is useful. Thank you! That doesn't preclude me asking or suggesting for extensions to the API specification to make the application writer's job easier... otherwise, we would be all dealing with 'trivial wrappers' geared to make the driver-writer's life easier. My initial request was born from a feature that I thought would be useful, and would like to have seen in a 'standard interface', be it a required or optional part of the next specification. (If it's any consolation, I spend 20-30 hours each week on the community artwork site I run, and sometimes I'm feeling pretty damn under-appreciated as well. So, rest assured, I'm certainly 'passing the favour on'... if you're not using my website yourself, perhaps you're using the fruits of the efforts of someone who is) -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From chris at cogdon.org Wed Jun 4 18:12:26 2003 From: chris at cogdon.org (Chris Cogdon) Date: Wed Jun 4 20:12:31 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <3EDDAB1E.80903@lemburg.com> Message-ID: <639EFD1C-96EA-11D7-8BD4-000393B658A2@cogdon.org> On Wednesday, Jun 4, 2003, at 01:17 US/Pacific, M.-A. Lemburg wrote: > Even better: the database server can prepare the data access > paths from the abstract SQL statement and cache that access > path for subsequent queries -- that's hardly possible if you > have the arguments quoted and embedded directly into the > SQL. In the latter case, you'd also lose the DB-API feature > of caching the last used SQL statement on a database cursor. > > Another argument against using embedded data types is that > of precision loss, e.g. ODBC interfaces C types directly > to the driver which assures that you don't lose precision > by converting them to a string representation. This happens > for Python floats, integers and even date/time values. All of that is exceptionally useful. But there are circumstances where it's MUCH easier for the application to be able to deal with 'string only' SQL fragments. Yes, these cases are rare, and there are workarounds, making the request of a 'particular DMBS-compatible quoting function' non-compulsory. However, that does not mean it does not have SOME utility. If precision-loss is not an issue (perhaps all the required types are strings or integers), and performance loss is not an issue (perhaps there's more work in using the 'workaround'), then why would this not be useful? Saying "It's too hard for the python driver writer" IS a useful argument, and one that I'm not contesting, but that argument has not really been used here. Everyone seems to be saying "I don't see why you would want this", or "there are disadvantages to this, such as performance loss" which can EASILY be argued against by showing you an application, and claiming that the performance loss is insignificant in comparison to the simplicity of coding in other parts. (after all, that's why you'd write a python driver in python rather than C, right?) -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From chris at cogdon.org Wed Jun 4 18:18:57 2003 From: chris at cogdon.org (Chris Cogdon) Date: Wed Jun 4 20:19:02 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <5.2.1.1.0.20030604103716.01e91a28@www.thinkware.se> Message-ID: <4C687268-96EB-11D7-8BD4-000393B658A2@cogdon.org> On Wednesday, Jun 4, 2003, at 03:10 US/Pacific, Magnus Lyck? wrote: > At 20:20 2003-06-03 -0700, Chris Cogdon wrote: >> Clue me in here. What's Dynamic SQL, CLI, and Embedded SQL ? > > Phew... I'm not so good at short explanations. :) [explaination omitted] Okay, will need a few re-reads to digest that. If it's okay with you, I'll post any questions directly, and send a summary/paraphrase back to the list. >> I've cited examples before, but... if they're not clear enough, I >> think I'd need to cite a complete working model. Can we take it as >> read that there may be SOME applications where this ability would be >> useful ? > > Do you mean that the SQL would run faster in the database, > or do you mean that it would be faster for you to code the > SQL? I don't doubt that the latter can be true. Definitely the latter. However, I've ALREADY written a small module that lets you piecemeal a SQL statement together, but keeping the parameters separate. I'll flesh it out to cover all the little side cases and post a link to it when it's done. Ie, it wasn't THAT hard, and not that messy to use, either. But, I have an older application that needs to 'store' the query used to get the information presented. And, in that case, it is still useful to be able to store the SQL directly. (Again, I could probably make my own workaround, but just being able to string-store the WHERE clause is a LOT easier) So, perhaps what I really should be asking for is to include a specification for an interface, such as what I've suggested before, but make that a OPTIONAL part of the specification. Ie, if it's easy enough to expose, then the driver writers should expose it in the standard form. Does THAT sound reasonable ? -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From magnus at thinkware.se Thu Jun 5 12:04:07 2003 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Thu Jun 5 05:01:04 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <4C687268-96EB-11D7-8BD4-000393B658A2@cogdon.org> References: <5.2.1.1.0.20030604103716.01e91a28@www.thinkware.se> Message-ID: <5.2.1.1.0.20030605110221.01e84318@www.thinkware.se> At 17:18 2003-06-04 -0700, Chris Cogdon wrote: >So, perhaps what I really should be asking for is to include a >specification for an interface, such as what I've suggested before, but >make that a OPTIONAL part of the specification. Ie, if it's easy enough to >expose, then the driver writers should expose it in the standard form. > >Does THAT sound reasonable ? Certainly. Many drivers implement things beyond the mandatory standard, and if it's possible to get the extras compatible across drivers, that's a big bonus in my opinion. -- Magnus Lycka (It's really Lyckå), magnus@thinkware.se Thinkware AB, Sweden, www.thinkware.se I code Python ~ The Agile Programming Language From mal at lemburg.com Thu Jun 5 12:40:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jun 5 05:41:05 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <5.2.1.1.0.20030605110221.01e84318@www.thinkware.se> References: <5.2.1.1.0.20030604103716.01e91a28@www.thinkware.se> <5.2.1.1.0.20030605110221.01e84318@www.thinkware.se> Message-ID: <3EDF100C.6040903@lemburg.com> Magnus Lyck? wrote: > At 17:18 2003-06-04 -0700, Chris Cogdon wrote: > >> So, perhaps what I really should be asking for is to include a >> specification for an interface, such as what I've suggested before, >> but make that a OPTIONAL part of the specification. Ie, if it's easy >> enough to expose, then the driver writers should expose it in the >> standard form. >> >> Does THAT sound reasonable ? > > Certainly. Many drivers implement things beyond the mandatory > standard, and if it's possible to get the extras compatible > across drivers, that's a big bonus in my opinion. No objection to adding a note to the spec about this. I still think that you have to flesh out a reasonable API for this, though, e.g. quoting should implemented on a per data type basis rather than on a per SQL statement basis. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Jun 05 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 19 days left From mal at lemburg.com Thu Jun 5 13:08:31 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jun 5 06:09:06 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <639EFD1C-96EA-11D7-8BD4-000393B658A2@cogdon.org> References: <639EFD1C-96EA-11D7-8BD4-000393B658A2@cogdon.org> Message-ID: <3EDF169F.4000508@lemburg.com> Chris Cogdon wrote: > Saying "It's too hard for the python driver writer" IS a useful > argument, and one that I'm not contesting, but that argument has not > really been used here. Everyone seems to be saying "I don't see why you > would want this", or "there are disadvantages to this, such as > performance loss" which can EASILY be argued against by showing you an > application, and claiming that the performance loss is insignificant in > comparison to the simplicity of coding in other parts. (after all, > that's why you'd write a python driver in python rather than C, right?) That argument doesn't need to get mentioned, since it is implicit for all changes/additions to the DB API spec. It's also the reason why we're so conservative about changes to the spec (and I think that the large number of existing DB API interfaces has proven that this is indeed a good way to proceed). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Jun 05 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 19 days left From gh at ghaering.de Thu Jun 5 13:03:30 2003 From: gh at ghaering.de (=?ISO-8859-1?Q?Gerhard_H=E4ring?=) Date: Thu Jun 5 06:11:31 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <3EDF100C.6040903@lemburg.com> References: <5.2.1.1.0.20030604103716.01e91a28@www.thinkware.se> <5.2.1.1.0.20030605110221.01e84318@www.thinkware.se> <3EDF100C.6040903@lemburg.com> Message-ID: <3EDF1572.7060900@ghaering.de> M.-A. Lemburg wrote: > Magnus Lyck? wrote: > >> At 17:18 2003-06-04 -0700, Chris Cogdon wrote: >> >>> So, perhaps what I really should be asking for is to include a >>> specification for an interface, such as what I've suggested before, >>> but make that a OPTIONAL part of the specification. Ie, if it's easy >>> enough to expose, then the driver writers should expose it in the >>> standard form. >>> >>> Does THAT sound reasonable ? >> >> Certainly. Many drivers implement things beyond the mandatory >> standard, and if it's possible to get the extras compatible >> across drivers, that's a big bonus in my opinion. > > No objection to adding a note to the spec about this. I still > think that you have to flesh out a reasonable API for this, > though, e.g. quoting should implemented on a per data type > basis rather than on a per SQL statement basis. The pyPgSQL, PySQLite and psycopg developers already agreed on an API for the three projects (the newest versions implement this API): If you want to add support for a new datatype for the DB-API module, this data type should implement a __quote__ method that returns the appropriate string. PySQLite and pyPgSQL already have a _quote method like the module-level quote method Chris was arguing for. But it's currently only an implementation detail, not part of the public APIs. -- Gerhard< From mal at lemburg.com Thu Jun 5 13:16:10 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jun 5 06:16:47 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <880CBAE9-96E9-11D7-8BD4-000393B658A2@cogdon.org> References: <880CBAE9-96E9-11D7-8BD4-000393B658A2@cogdon.org> Message-ID: <3EDF186A.3080806@lemburg.com> Chris Cogdon wrote: > > On Wednesday, Jun 4, 2003, at 01:02 US/Pacific, M.-A. Lemburg wrote: > >> Chris Cogdon wrote: >> >>> But... I DO understand the difficulty in exposing a function that >>> doesn't currently exist (such as the ODBC driver). And it may NOT be >>> worth the difficulty in adding that functionality as a requirement >>> for a later API. (Perhaps optional? At least if it's doable, we could >>> get a consistent interface). >> >> >> That we could do, but note that quoting would be requiring type >> information in database specific ways, plus it may sometimes also >> need encoding or code page information for string values, e.g. >> a database running with UTF-8 code page will want to see UTF-8 >> in string literals. > > I believe the python driver ALREADY has access to this information, > since it needs it to determine how to correctly pass this information to > the DBMS, be it in a string or binary format. Not necessarily. In ODBC the database can tell the driver to send the data using a certain data type, but that information is only known for SQL statements that you actually plan to actually execute (and not even in all cases). I wonder whether you've seen a module I've been working on a few years ago: http://www.egenix.com/files/python/dbinfo.py This tries to take care of quoting among other things. Note that the information about how quoting is done can easily be separated out of the DB-API layer and sometimes that makes more sense than trying to put more load on the DB API implementors. We use a much more advanced version of dbinfo.py in our commercial software -- works great :-) We use it mostly to generate schemas for various database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Jun 05 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 19 days left From mal at lemburg.com Thu Jun 5 13:26:12 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jun 5 06:26:46 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <3EDF1572.7060900@ghaering.de> References: <5.2.1.1.0.20030604103716.01e91a28@www.thinkware.se> <5.2.1.1.0.20030605110221.01e84318@www.thinkware.se> <3EDF100C.6040903@lemburg.com> <3EDF1572.7060900@ghaering.de> Message-ID: <3EDF1AC4.3020001@lemburg.com> Gerhard H?ring wrote: > M.-A. Lemburg wrote: >>> Certainly. Many drivers implement things beyond the mandatory >>> standard, and if it's possible to get the extras compatible >>> across drivers, that's a big bonus in my opinion. >> >> No objection to adding a note to the spec about this. I still >> think that you have to flesh out a reasonable API for this, >> though, e.g. quoting should implemented on a per data type >> basis rather than on a per SQL statement basis. > > The pyPgSQL, PySQLite and psycopg developers already agreed on an API > for the three projects (the newest versions implement this API): > > If you want to add support for a new datatype for the DB-API module, > this data type should implement a __quote__ method that returns the > appropriate string. Ouch. __xxx__ names are reserved for interpreter internal use. You shouldn't use those in your applications, since it is not clear what will happen to those names in the future, e.g. they could undergo some serious optimizations which could break your code. Other than that, I don't see why a data type should know about 10+ different database backends. The quoting mechanism for a particular database should know about the quoting, not the various data types. > PySQLite and pyPgSQL already have a _quote method like the module-level > quote method Chris was arguing for. But it's currently only an > implementation detail, not part of the public APIs. Before going too much into details, we should vote for the kind of quoting mechanism you would like to have in the spec: 1. quote(sql, parameters) -> quoted SQL statement with embedded SQL literals 2. quote(dataobject, datatype) -> quoted SQL literal Option 2 can be implemented outside the DB API level. Option 1 is possible, but it's not likely to be easily implementable unless the database API already provides a suitable function for it (parsing SQL is hard). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Jun 05 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 19 days left From jacobs at penguin.theopalgroup.com Thu Jun 5 07:46:46 2003 From: jacobs at penguin.theopalgroup.com (Kevin Jacobs) Date: Thu Jun 5 06:47:26 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <3EDF1572.7060900@ghaering.de> Message-ID: On Thu, 5 Jun 2003, Gerhard H?ring wrote: > If you want to add support for a new datatype for the DB-API module, > this data type should implement a __quote__ method that returns the > appropriate string. Again, I think this is a terrible idea. First off, __xxx__ names are reserved in Python. Secondly, there is never a guarantee that there is a single canonical quoted representation for any datum. Different backends, bound backend schema types, encoding parameters, application contexts can all alter these representations. I see __quote__ as a terrible violation of encapsulation. > PySQLite and pyPgSQL already have a _quote method like the module-level > quote method Chris was arguing for. But it's currently only an > implementation detail, not part of the public APIs. And pyPgSQL's _quote is designed only to be used on simple data values in the context of an INSERT or an UPDATE. Used out of context, it is incomplete and in some cases wrong. For one, with ambiguously typed datetime literals). e.g.: SELECT 'foo' = '2001-01-01'; -> FALSE SELECT 'foo' = DATE '2001-01-01'; -> ERROR: Bad date external representation 'foo' PySQLite has a chance of getting _quote right since (last I checked) it only stores untyped string values. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From jacobs at penguin.theopalgroup.com Thu Jun 5 07:50:51 2003 From: jacobs at penguin.theopalgroup.com (Kevin Jacobs) Date: Thu Jun 5 06:51:27 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <3EDF1AC4.3020001@lemburg.com> Message-ID: On Thu, 5 Jun 2003, M.-A. Lemburg wrote: > Gerhard H?ring wrote: > > If you want to add support for a new datatype for the DB-API module, > > this data type should implement a __quote__ method that returns the > > appropriate string. > > Ouch. __xxx__ names are reserved for interpreter internal use. > You shouldn't use those in your applications, since it is not > clear what will happen to those names in the future, e.g. they > could undergo some serious optimizations which could break your > code. I Agree: +10**10*10 > Other than that, I don't see why a data type should know about > 10+ different database backends. The quoting mechanism for > a particular database should know about the quoting, not the > various data types. Ditto. > > PySQLite and pyPgSQL already have a _quote method like the module-level > > quote method Chris was arguing for. But it's currently only an > > implementation detail, not part of the public APIs. > > Before going too much into details, we should vote for the kind > of quoting mechanism you would like to have in the spec: > > 1. quote(sql, parameters) -> quoted SQL statement with embedded SQL > literals +0 if it is option. I certainly would _never_ use it, though. > 2. quote(dataobject, datatype) -> quoted SQL literal -1 for inclusion in DB-API +1 for a higher-level API. I'll even volunteer to manage such a project. The only painful side is that every DB-API driver, even those for the same backend database, uses different backend type codes. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From chris at cogdon.org Thu Jun 5 09:19:37 2003 From: chris at cogdon.org (Chris Cogdon) Date: Thu Jun 5 11:20:18 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <3EDF1572.7060900@ghaering.de> Message-ID: <1E9990FD-9769-11D7-8BD4-000393B658A2@cogdon.org> On Thursday, Jun 5, 2003, at 03:03 US/Pacific, Gerhard H?ring wrote: > M.-A. Lemburg wrote: >> No objection to adding a note to the spec about this. I still >> think that you have to flesh out a reasonable API for this, >> though, e.g. quoting should implemented on a per data type >> basis rather than on a per SQL statement basis. > > The pyPgSQL, PySQLite and psycopg developers already agreed on an API > for the three projects (the newest versions implement this API): > > If you want to add support for a new datatype for the DB-API module, > this data type should implement a __quote__ method that returns the > appropriate string. > > PySQLite and pyPgSQL already have a _quote method like the > module-level quote method Chris was arguing for. But it's currently > only an implementation detail, not part of the public APIs. My proposal was to have a 'quote' function with exactly the same interface as 'execute' which would, just like execute, have the same quoting semantics and be able to handle all the same data types. Perhaps 'stringize' would be a better name? :) -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From mal at lemburg.com Thu Jun 5 18:28:50 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jun 5 11:29:31 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <1E9990FD-9769-11D7-8BD4-000393B658A2@cogdon.org> References: <1E9990FD-9769-11D7-8BD4-000393B658A2@cogdon.org> Message-ID: <3EDF61B2.7050906@lemburg.com> Chris Cogdon wrote: > > On Thursday, Jun 5, 2003, at 03:03 US/Pacific, Gerhard H?ring wrote: > >> M.-A. Lemburg wrote: >> >>> No objection to adding a note to the spec about this. I still >>> think that you have to flesh out a reasonable API for this, >>> though, e.g. quoting should implemented on a per data type >>> basis rather than on a per SQL statement basis. >> >> >> The pyPgSQL, PySQLite and psycopg developers already agreed on an API >> for the three projects (the newest versions implement this API): >> >> If you want to add support for a new datatype for the DB-API module, >> this data type should implement a __quote__ method that returns the >> appropriate string. >> >> PySQLite and pyPgSQL already have a _quote method like the >> module-level quote method Chris was arguing for. But it's currently >> only an implementation detail, not part of the public APIs. > > My proposal was to have a 'quote' function with exactly the same > interface as 'execute' which would, just like execute, have the same > quoting semantics and be able to handle all the same data types. Perhaps > 'stringize' would be a better name? :) I'd like to hear some more feedback from other db-sig subscribers. I personally think that a quote function to generate SQL literals for various backends available as separate Python module would go much further than choosing this highly specific kind of interface. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Jun 05 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 19 days left From chris at cogdon.org Thu Jun 5 09:38:09 2003 From: chris at cogdon.org (Chris Cogdon) Date: Thu Jun 5 11:38:13 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <3EDF186A.3080806@lemburg.com> Message-ID: Was tempted to that this off-list, but... I think it's useful for anyone that's not that familiar with ODBC databases. On Thursday, Jun 5, 2003, at 03:16 US/Pacific, M.-A. Lemburg wrote: > Chris Cogdon wrote: >> I believe the python driver ALREADY has access to this information, >> since it needs it to determine how to correctly pass this information >> to the DBMS, be it in a string or binary format. > > Not necessarily. In ODBC the database can tell the driver to send > the data using a certain data type, but that information is only > known for SQL statements that you actually plan to actually execute > (and not even in all cases). So, in other words, it might work like this: 1. application sends a string and some parameters through the 'execute' method. 2. driver sends the string (only) to the backend through a function that asks 'how should I send you the parameters?' 3. backend responds. 4. driver sends the string (perhaps), plus parameters (encoded according to the previous response) to the backend for execution. Is that right? If so, that seems to be a particularly bad performance hit, almost not worth the savings in sending binary data. (Unless i'm missing someting, of course :) -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From mal at lemburg.com Thu Jun 5 19:18:26 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jun 5 12:19:00 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: References: Message-ID: <3EDF6D52.1010300@lemburg.com> Chris Cogdon wrote: > Was tempted to that this off-list, but... I think it's useful for anyone > that's not that familiar with ODBC databases. I believe it's similar to what other DB interfaces do one way or another... > On Thursday, Jun 5, 2003, at 03:16 US/Pacific, M.-A. Lemburg wrote: > >> Chris Cogdon wrote: >> >>> I believe the python driver ALREADY has access to this information, >>> since it needs it to determine how to correctly pass this information >>> to the DBMS, be it in a string or binary format. >> >> >> Not necessarily. In ODBC the database can tell the driver to send >> the data using a certain data type, but that information is only >> known for SQL statements that you actually plan to actually execute >> (and not even in all cases). > > So, in other words, it might work like this: > > 1. application sends a string and some parameters through the 'execute' > method. > 2. driver sends the string (only) to the backend through a function that > asks 'how should I send you the parameters?' Not quite: the driver/database prepares the statement, ie. it parses the SQL, finds out which data types the bound parameters needs and then passes this information on to the application. Whether this happens in the driver and/or in part in the database is up to the driver/database combo. Many drivers cache data type information locally to avoid the network traffic and do at least part of the parsing themselves. OTOH, the database server can already start setting up the right access paths to the data while the driver on the client side still waits for the application to pass in the data. > 3. backend responds. > 4. driver sends the string (perhaps), plus parameters (encoded according > to the previous response) to the backend for execution. Not "encoded"; converted to whatever the backend requires. > Is that right? > > If so, that seems to be a particularly bad performance hit, almost not > worth the savings in sending binary data. (Unless i'm missing someting, > of course :) Huh ? Where's the performance hit ? The more meta-data you have on the client side the better and faster the whole process can be made. That's also why drivers usually keep a cache of schema information on the client side. Side note: database wire protocols often have limits to the size of the SQL statement you can send, e.g. SAP DB won't accept anything much longer than 16kB worth of SQL. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Jun 05 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 19 days left From chris at cogdon.org Thu Jun 5 11:36:20 2003 From: chris at cogdon.org (Chris Cogdon) Date: Thu Jun 5 13:36:26 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <3EDF6D52.1010300@lemburg.com> Message-ID: <386F86C6-977C-11D7-8BD4-000393B658A2@cogdon.org> On Thursday, Jun 5, 2003, at 09:18 US/Pacific, M.-A. Lemburg wrote: > Not quite: the driver/database prepares the statement, ie. it parses > the SQL, finds out which data types the bound parameters needs and > then passes this information on to the application. I know that the postgresql and mysql drivers gleen the datatype information only from the data being passed as parameters. Do any other drivers get this from other places, such as the sql or from information back from the database ? >> Is that right? >> If so, that seems to be a particularly bad performance hit, almost >> not worth the savings in sending binary data. (Unless i'm missing >> someting, of course :) > > Huh ? Where's the performance hit ? The more meta-data you have > on the client side the better and faster the whole process can > be made. That's also why drivers usually keep a cache of > schema information on the client side. It was only a performance hit if the driver was waiting for a response from the database before sending the data. > Side note: database wire protocols often have limits to the > size of the SQL statement you can send, e.g. SAP DB won't > accept anything much longer than 16kB worth of SQL. Point conceded. I never suggested that the proposed quote function would be useful in all circumstances for all datatypes. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From jekabs.andrusaitis at tietoenator.com Thu Jun 5 21:12:24 2003 From: jekabs.andrusaitis at tietoenator.com (Jekabs Andrushaitis) Date: Thu Jun 5 14:36:33 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method References: <386F86C6-977C-11D7-8BD4-000393B658A2@cogdon.org> Message-ID: <3EDF8808.8090709@tietoenator.com> Usually DB itself does convert the query arguments to the correct types So: 1) the client passes sql statement with arguments in whatever data type it wants; 2) database engine parses the sql statement, decides on query execution path and what types data should be in; 3) database engine converts the arguments to correct data types and executes the statement and returns results. So what normally Python database adapter does is - converting the Python data types to the underlaying db engine data types, while letting the db engine do the conversion from one type to another. Of course there might be exceptions, like db engines which do not use "bind variables" at all, and all the parameters have to be passed through the sql statement itself. -- Jekabs Andrushaitis Senior system analyst TietoEnator Financial Solutions 41 Lacplesa Str. Riga, LV-1011, Latvia Tel: +371 7286660 Fax: +371 7243000 E-mail: jekabs.andrusaitis@tietoenator.com From davidrushby at yahoo.com Thu Jun 5 13:00:29 2003 From: davidrushby at yahoo.com (David Rushby) Date: Thu Jun 5 15:00:53 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <386F86C6-977C-11D7-8BD4-000393B658A2@cogdon.org> Message-ID: <20030605190029.57030.qmail@web11002.mail.yahoo.com> --- Chris Cogdon wrote: > > On Thursday, Jun 5, 2003, at 09:18 US/Pacific, M.-A. Lemburg wrote: > > > Not quite: the driver/database prepares the statement, ie. it parses > > the SQL, finds out which data types the bound parameters needs and > > then passes this information on to the application. > > I know that the postgresql and mysql drivers gleen the datatype > information only from the data being passed as parameters. Yes, but that approach is overly simplistic. There are cases where the same Python object received as a parameter value could require dramatically different internal representations based on the context to which it's being submitted. Database arrays are an example. Witness the following description of the PgArray class from the current version of pyPgSql: """ # Description: A Python wrapper class for PostgreSQL array. # It is used so that the list type can be used as an arg- # ument to Connection.execute() without being treated as # a PostgreSQL array. """ PgArray is an ad hoc, UserList-like wrapper around a built-in list. The sole purpose of the class, as far as I can tell, is to allow the driver to distinguish between: a) an incoming list of distinct parameters bound for several fields b) an incoming list bound for a database array field In a database engine that provides parameter type information via the native client API, such a hack would not be necessary. As another example, what if an incoming string parameter contains unprintable characters, such as the null character? The database might be capable of storing such a value if it were *bound* as a string, but not if it's submitted as a *literal* in a SQL statement. As another example, what about date and time localization? Some database engines have variable modes of localization for dates and times. If the localization mode can vary on a per-field or per-table basis rather than only on a database-wide basis, then in order to generate a locale-correct string representation of an incoming date parameter, the Python driver must parse the SQL statement, determine the fields involved, and look up their metadata. Whereas, *bound* dates and times are typically submitted to the database engine in a locale-independent manner, or the metadata structures provided by the native client API include information about the expected representation, so that client drivers can convert to it. As another example, some databases have data types that cannot be represented as literals--they must be stored and retrieved using specialized API calls. Database arrays and/or blobs are perhaps the most common cases. Arguably, the inability to represent all datatypes literally is a flaw in the engine--one which PostgreSQL and MySQL do not (AFAIK) suffer from. However, there may be good reasons for this limitation, as with partial, stream-based I/O of blobs, or I/O of specific slices of database arrays. > I know that the postgresql and mysql drivers gleen the datatype > information only from the data being passed as parameters. > Do any other drivers get this from other places, such as the sql or > from information back from the database? As far as I know, most other drivers obtain type information for parameters from the database engine via the native client API. The driver submits the parameterized SQL string to the engine; the engine parses it and returns both a prepared binary representation of the SQL statement and a metadata structure containing parameter information. > >> Is that right? > >> If so, that seems to be a particularly bad performance hit, almost > >> not worth the savings in sending binary data. (Unless i'm missing > >> someting, of course :) > > > Huh ? Where's the performance hit ? The more meta-data you have > > on the client side the better and faster the whole process can > > be made. That's also why drivers usually keep a cache of > > schema information on the client side. > > It was only a performance hit if the driver was waiting for a response > from the database before sending the data. It's a one-time, up-front performance hit that pays off dramatically over the course of numerous executions of the same parameterized statement. As long as the SQL string submitted to the Cursor.execute method remains the same as the previously executed SQL string, the prepared statement and the metadata can be reused without additional overhead--only the parameter values vary. For example, here's the relevant section of kinterbasdb's Cursor.execute implementation (from _kicore_execution.c): -------------------------------------- static int _prepare_statement_if_necessary( CursorObject *cursor, PyObject *sql ) { [...] PyObject *prev_sql = cursor->previous_sql; if (prev_sql != NULL) { /* If the PyObject pointers point to the same memory location, the two ** objects are certainly equal. If the pointers refer to different ** memory locations, the two object might still be equal if their ** contents match. */ if (sql == prev_sql || PyObject_Compare(sql, prev_sql) == 0) { cursor->_state = CURSOR_STATE_OPEN; return 0; } } [... else, ask the engine to prepare a new statement] } -------------------------------------- The overhead is very low in repeated executions of the same SQL string--just a pointer comparison, if the current SQL string and the previous one are the same Python string object. __________________________________ Do you Yahoo!? Yahoo! Calendar - Free online calendar with sync to Outlook(TM). http://calendar.yahoo.com From Harald.Meland at usit.uio.no Fri Jun 6 02:28:21 2003 From: Harald.Meland at usit.uio.no (Harald Meland) Date: Thu Jun 5 19:28:27 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: (Kevin Jacobs's message of "Thu, 5 Jun 2003 06:50:51 -0400 (EDT)") References: Message-ID: [Kevin Jacobs] > On Thu, 5 Jun 2003, M.-A. Lemburg wrote: >> 2. quote(dataobject, datatype) -> quoted SQL literal > > -1 for inclusion in DB-API > > +1 for a higher-level API. I'll even volunteer to manage such a project. > The only painful side is that every DB-API driver, even those for the same > backend database, uses different backend type codes. I seem to remember that the question of having all DB-API drivers derive (most of) their types from some common, distributed-with-python non-DBMS-specific DB-API module, has come up here before: However, I'm unable to either a) locate that discussion in the archives or b) recollect what conclusion one (I hope) eventually reached on this question. Of course, this wouldn't work equally well for *all* types; if e.g. the DBMS-independent module doesn't have a specific superclass for e.g. 'distance', while DBMS X does indeed have a type for such values, then the writers of two different X drivers wouldn't necessarily agree on which superclass their driver's class should be based on. A possible workaround is for the DBMS-independent module to include a separate type for "types not covered elsewhere", and require compliant drivers to derive all their "uncommon" driver-specific types from this. -- Harald From Harald.Meland at usit.uio.no Fri Jun 6 02:38:33 2003 From: Harald.Meland at usit.uio.no (Harald Meland) Date: Thu Jun 5 19:38:42 2003 Subject: [DB-SIG] Asymmetry in DB-API 2.0 types In-Reply-To: <3EDE02B6.6060801@halfcooked.com> (Andy Todd's message of "Wed, 04 Jun 2003 15:31:18 +0100") References: <3EDDA210.8020807@lemburg.com> <3EDE02B6.6060801@halfcooked.com> Message-ID: [Andy Todd] > A standard way to access date, time and datetime objects is, in my > opinion, a good idea. Most of the drivers I use work with mxDateTime > but that can cause issues for some people because of the license. There seems to be no license problem with mxDateTime; however, I would still feel more comfortable if the DB-API spec included a specification of what API the various types must support, rather than *requiring* all driver modules to use a specific third-party module for e.g. DateTime objects. Of course, such an type object API specification for DateTime objects should coincide with (a subset of) the currently recommended implementation, i.e. mxDateTime. Now, when all that's said, I also would mention that I think it would be beneficial if similar type object APIs were hammered out for the other standard types (STRING, BINARY, NUMBER and ROWID). As the DB-API spec stands, I'm not sure that it'll always be safe to treat objects for which "type(object) is driver.STRING" as Python strings (even though I would be somewhat surprised if I found a driver where that wasn't the case). -- Harald From greg at cosc.canterbury.ac.nz Fri Jun 6 13:00:25 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Jun 5 20:01:37 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <386F86C6-977C-11D7-8BD4-000393B658A2@cogdon.org> Message-ID: <200306060000.h5600PT08482@oma.cosc.canterbury.ac.nz> Chris Cogdon : > I know that the postgresql and mysql drivers gleen the datatype > information only from the data being passed as parameters. Do any other > drivers get this from other places, such as the sql or from information > back from the database ? Yes. I'm working with the Firebird (formerly Interbase) API at the moment, and it does exactly this. You give it an SQL query containing '?' markers, and it parses and fully groks it, consults the database metadata, and gives you back a structure describing the expected types of all the parameters. It would be utterly impossible to get it to do this with anything less than a fully-formed syntactically and semantically correct SQL statement. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From chris at cogdon.org Thu Jun 5 18:13:41 2003 From: chris at cogdon.org (Chris Cogdon) Date: Thu Jun 5 20:13:45 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: <200306060000.h5600PT08482@oma.cosc.canterbury.ac.nz> Message-ID: On Thursday, Jun 5, 2003, at 17:00 US/Pacific, Greg Ewing wrote: > Chris Cogdon : > >> I know that the postgresql and mysql drivers gleen the datatype >> information only from the data being passed as parameters. Do any >> other >> drivers get this from other places, such as the sql or from >> information >> back from the database ? > > Yes. I'm working with the Firebird (formerly Interbase) API at the > moment, and it does exactly this. You give it an SQL query containing > '?' markers, and it parses and fully groks it, consults the database > metadata, and gives you back a structure describing the expected types > of all the parameters. > > It would be utterly impossible to get it to do this with anything less > than a fully-formed syntactically and semantically correct SQL > statement. Point conceded: if one can only do that with full-formed SQL statements, then any utility of being able to 'piecemeal' the SQL together using strings is defeated. Thank you for giving me an idea on how 'other' databases work. Oh... isn't firebird a browser?? *grins and ducks* -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From greg at cosc.canterbury.ac.nz Fri Jun 6 13:23:10 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Jun 5 20:23:59 2003 Subject: [DB-SIG] API suggestion: expose 'quote' method In-Reply-To: Message-ID: <200306060023.h560NAP08613@oma.cosc.canterbury.ac.nz> > Point conceded: if one can only do that with full-formed SQL > statements, then any utility of being able to 'piecemeal' the SQL > together using strings is defeated. Although it should be feasible to piece together pieces of SQL and their associated parameter values separately in Python before handing them to the DB module, maybe using something like class SQLWithParams: def __init__(self, sql, params): self.sql = sql self.params = params def __add__(self, other): if isinstance(other, SQLWithParams): # SQLWithParams + SQLWithParams return SQLWithParams( self.sql + other.sql, self.params + other.params) else: # SQLWithParams + string return SQLWithParams( self.sql + sql, self.params) def __radd__(self, other): # string + SQLWithParams return SQLWithParams(other + self.sql, self.params) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From mal at lemburg.com Fri Jun 6 19:47:37 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jun 6 12:48:12 2003 Subject: [DB-SIG] Asymmetry in DB-API 2.0 types In-Reply-To: References: <3EDDA210.8020807@lemburg.com> Message-ID: <3EE0C5A9.80005@lemburg.com> Harald Meland wrote: > [M.-A. Lemburg] > > >>The driver output values are driver dependent and there's nothing >>much the API spec could do about this because not all databases >>support all different kinds of data types and it is not even >>clear for which result columns a specific data type is needed >>or desired. > > Wouldn't it be possible to extend the current specification for > e.g. DATETIME: > > DATETIME > This type object is used to describe date/time columns in a > database. Note that DATETIME is a type object, not the object being returned itself. DATETIME (and the others) are only meant to be able to provide a common ground for the cursor.description tuple entries -- nothing more. There are no standard date/time objects defined in the spec. Many DB-API modules use mxDateTime, some return strings, some time.time() integers, etc. In the future some may also use the Python datetime module's objects. Again, DB-API modules are low-level interfaces and you are asking for higher level standardizations which should really go into your abstraction layer rather then the spec. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Jun 06 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2003, Charleroi, Belgium: 18 days left From terekhov at emc.com Fri Jun 6 16:36:22 2003 From: terekhov at emc.com (Mikhail Terekhov) Date: Fri Jun 6 15:33:24 2003 Subject: [DB-SIG] Asymmetry in DB-API 2.0 types References: <3EDDA210.8020807@lemburg.com> <3EE0C5A9.80005@lemburg.com> Message-ID: <3EE0ED36.1080506@emc.com> M.-A. Lemburg wrote: > > Again, DB-API modules are low-level interfaces and you are > asking for higher level standardizations which should really > go into your abstraction layer rather then the spec. > DB-API modules are not so low-level interfaces as it seems. For example they abstract (too much in my view) SQL cursors, simple SQL queries, set commands etc. under "DB-API cursor" thing. Regards, Mikhail From admin at grandcontinental.com.my Sun Jun 8 00:35:41 2003 From: admin at grandcontinental.com.my (admin) Date: Sat Jun 7 11:34:00 2003 Subject: [DB-SIG] WARNING: YOU MAY HAVE A VIRUS Message-ID: <200306072335.AA506200384@grandcontinental.com.my> The Anti-virus software on grandcontinental.com.my has reported an e-mail sent from your computer contains a virus with the subject "Re: Movie" to: kuala_terengganu@grandcontinental.com.my. The virus name is : W32/Sobig.C@mm. This virus has been quarantined by at least one of the recipient's mail receiving server to prevent further damage to the recipient. Please be advised to locate and eliminate this virus from your computer before continuing to send e-mail over the Internet. This mail was auto generated by the recepient's web host provider. From JKANCIAN at arinc.com Wed Jun 11 20:39:23 2003 From: JKANCIAN at arinc.com (Kancianic, Jennifer C. (JKANCIAN)) Date: Wed Jun 11 19:36:41 2003 Subject: [DB-SIG] DCOracle2 excepting after execute Message-ID: <0734A2B8CF21D711B98B00080224869C01BA07F4@exanpmb3.arinc.com> Sometimes, when our code executes an update or insert statement, DCOracle2 will except unexpectedly. Here is the traceback from our logfile: Traceback (most recent call last): File "/home/adc/adc/lib/UserDB.py", line 414, in updateLastAccessed results = self.db.doSQL(sql, (key, )) File "/home/adc/adc/lib/OracleDB.py", line 111, in doSQL self.cursor.execute(sqlStatement, list) File "/usr/local/lib/python2.2/site-packages/DCOracle2/DCOracle2.py", line 1008, in execute self.rowcount = self._cursor.rowcount() AttributeError: 'NoneType' object has no attribute 'rowcount' This appears to be occurring in the DCOracle2 code that tries to get the number of rows affected after the execution. result = self._cursor.execute() if result in RowCountResults: self.rowcount = self._cursor.rowcount() else: self.rowcount = -1 Has anyone encountered this? What can I do anything to NOT get the exception? This is happening regularly and is not good. If this was designed this way, how should I deal with it? Forgive my ignorance. I am using Oracle 1.8, Linux Redhat 7.1, python 2.2, DCOracle 1.3 (I think). Thanks for your help, Jenny From andy47 at halfcooked.com Thu Jun 12 10:50:36 2003 From: andy47 at halfcooked.com (Andy Todd) Date: Thu Jun 12 04:51:39 2003 Subject: [DB-SIG] DCOracle2 excepting after execute In-Reply-To: <0734A2B8CF21D711B98B00080224869C01BA07F4@exanpmb3.arinc.com> References: <0734A2B8CF21D711B98B00080224869C01BA07F4@exanpmb3.arinc.com> Message-ID: <3EE83EDC.10304@halfcooked.com> Kancianic, Jennifer C. (JKANCIAN) wrote: > Sometimes, when our code executes an update or insert statement, DCOracle2 > will except unexpectedly. Here is the traceback from our logfile: > > Traceback (most recent call last): > File "/home/adc/adc/lib/UserDB.py", line 414, in updateLastAccessed > results = self.db.doSQL(sql, (key, )) > File "/home/adc/adc/lib/OracleDB.py", line 111, in doSQL > self.cursor.execute(sqlStatement, list) > File "/usr/local/lib/python2.2/site-packages/DCOracle2/DCOracle2.py", line > 1008, in execute > self.rowcount = self._cursor.rowcount() > AttributeError: 'NoneType' object has no attribute 'rowcount' > > This appears to be occurring in the DCOracle2 code that tries to get the > number of rows affected after the execution. > > result = self._cursor.execute() > if result in RowCountResults: > self.rowcount = self._cursor.rowcount() > else: > self.rowcount = -1 > > Has anyone encountered this? What can I do anything to NOT get the > exception? This is happening regularly and is not good. > > If this was designed this way, how should I deal with it? Forgive my > ignorance. > > I am using Oracle 1.8, Linux Redhat 7.1, python 2.2, DCOracle 1.3 (I think). > > Thanks for your help, > Jenny > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig It possibly is a bug, but it is a little hard to tell without looking at all of the code. Can you produce a nice simple example that causes the same exception to be raised? Or can you strip the SQL from the your example and tell us what happens when you run it in SQL*Plus? I haven't come across a situation where a DCOracle2 cursor object doesn't have a rowcount attribute, but that may just be me. Also, are you sure about the versions of the software you are using? Oracle doesn't currently ship a version 1.8, is it perhaps 8.1? Regards, Andy -- -------------------------------------------------------------------------------- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From JKANCIAN at arinc.com Thu Jun 12 11:04:14 2003 From: JKANCIAN at arinc.com (Kancianic, Jennifer C. (JKANCIAN)) Date: Thu Jun 12 10:01:41 2003 Subject: [DB-SIG] DCOracle2 excepting after execute Message-ID: <0734A2B8CF21D711B98B00080224869C01BA07F6@exanpmb3.arinc.com> Yes - thanks - we use Oracle 8.1.7. Here is some more code. This updates the timestamp on a session key generated by webkit: def updateLastAccessed(self, key): try: sql = '''UPDATE sessions SET last_accessed = SYSDATE WHERE id = :1''' results = self.db.doSQL(sql, (key, )) self.db.commit() except: self.log.exception("updateLastAccessed failed for session key " + key) def doSQL(self, sqlStatement, list=(), noPrefetch=FALSE): """Inserts into the database with the given SQL statement if select flag == 0, othwerwise, performs the given select statement""" sqlStatement = sqlStatement.strip() try: if (not sqlStatement.upper().startswith('SELECT')): self.connect() if (len(list) > 0) and (type(list[0]) == type((1, 2))): return self.cursor.executemany(sqlStatement, list) self.cursor.execute(sqlStatement, list) if sqlStatement.upper().startswith('SELECT'): newconn = self._pool.getConnection() cursor = newconn.cursor() if (noPrefetch): cursor.setPrefetch(None) # DB-SIG - This next statement is where the problem cursor.execute(sqlStatement, list) returnVal = (self.fetchrows(cursor) or []) cursor.close() self._pool.returnConnection(newconn) return returnVal except DCOracle2.Error, e: if sqlStatement.upper().startswith('SELECT'): try: self.disconnect() except: pass self.handleError(sqlStatement, str(e)) When I execute the update statement at the sqlplus command line, it tells me that one row is updated. I have thus far been unable to make the problem happen in our development and test environments. It happens in production regularly (not EVERY time). In particular, there are three different update/insert scenarios where it happens more often. Thanks, Jenny -----Original Message----- From: Andy Todd [mailto:andy47@halfcooked.com] Sent: Thursday, June 12, 2003 4:51 AM To: Kancianic, Jennifer C. (JKANCIAN) Cc: 'db-sig@python.org' Subject: Re: [DB-SIG] DCOracle2 excepting after execute Kancianic, Jennifer C. (JKANCIAN) wrote: > Sometimes, when our code executes an update or insert statement, DCOracle2 > will except unexpectedly. Here is the traceback from our logfile: > > Traceback (most recent call last): > File "/home/adc/adc/lib/UserDB.py", line 414, in updateLastAccessed > results = self.db.doSQL(sql, (key, )) > File "/home/adc/adc/lib/OracleDB.py", line 111, in doSQL > self.cursor.execute(sqlStatement, list) > File "/usr/local/lib/python2.2/site-packages/DCOracle2/DCOracle2.py", line > 1008, in execute > self.rowcount = self._cursor.rowcount() > AttributeError: 'NoneType' object has no attribute 'rowcount' > > This appears to be occurring in the DCOracle2 code that tries to get the > number of rows affected after the execution. > > result = self._cursor.execute() > if result in RowCountResults: > self.rowcount = self._cursor.rowcount() > else: > self.rowcount = -1 > > Has anyone encountered this? What can I do anything to NOT get the > exception? This is happening regularly and is not good. > > If this was designed this way, how should I deal with it? Forgive my > ignorance. > > I am using Oracle 1.8, Linux Redhat 7.1, python 2.2, DCOracle 1.3 (I think). > > Thanks for your help, > Jenny > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig It possibly is a bug, but it is a little hard to tell without looking at all of the code. Can you produce a nice simple example that causes the same exception to be raised? Or can you strip the SQL from the your example and tell us what happens when you run it in SQL*Plus? I haven't come across a situation where a DCOracle2 cursor object doesn't have a rowcount attribute, but that may just be me. Also, are you sure about the versions of the software you are using? Oracle doesn't currently ship a version 1.8, is it perhaps 8.1? Regards, Andy -- ---------------------------------------------------------------------------- ---- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From b.n.lawrence at rl.ac.uk Wed Jun 18 14:25:15 2003 From: b.n.lawrence at rl.ac.uk (Bryan Lawrence) Date: Wed Jun 18 08:25:39 2003 Subject: [DB-SIG] inserting strings into postgres db using pgdb (escaping quotes automagically) Message-ID: <200306181325.15380.b.n.lawrence@rl.ac.uk> Hi Folks I'm just playing at a common db core module for an application. It works ok with gadfly and mysql, and I'm trying to extend it to postgresql. I'm using the pgdb module ... At the moment (mysql) I do inserts with something like data=('a',"b's") cmd="insert into table blah (x,y) values (%s%s)" self.cursor.execute(cmd,data) (replacing the %s with ? for gadfly). This breaks with postgres ... I tried something like cmd='insert into table blah (x,y) values (string.join( map(lambda x: repr(x),data),',')) but that breaks with ERROR: Attribute ... not found Clearly I can muck around parsing strings and escaping things, but that seems remarkably clumsy, and I can't believe I'm the only person with this problem. I've had a google and rummaged in the manuals, but obviously didn't turn over the right leaf ... how does one do this with postgres? I'm not on the mailing list, so if you can help, could you please reply directly? It's not an easy thing to search for in the archive, so apologies if this is a much rehearsed problem. Thanks Bryan From andy47 at halfcooked.com Wed Jun 18 15:06:32 2003 From: andy47 at halfcooked.com (Andy Todd) Date: Wed Jun 18 09:08:07 2003 Subject: [DB-SIG] inserting strings into postgres db using pgdb (escaping quotes automagically) In-Reply-To: <200306181325.15380.b.n.lawrence@rl.ac.uk> References: <200306181325.15380.b.n.lawrence@rl.ac.uk> Message-ID: <3EF063D8.7050008@halfcooked.com> Bryan Lawrence wrote: > Hi Folks > > I'm just playing at a common db core module for an application. It works ok > with gadfly and mysql, and I'm trying to extend it to postgresql. > > I'm using the pgdb module ... > > At the moment (mysql) I do inserts with something like > > data=('a',"b's") > cmd="insert into table blah (x,y) values (%s%s)" > self.cursor.execute(cmd,data) > (replacing the %s with ? for gadfly). > This breaks with postgres ... > > I tried something like > cmd='insert into table blah (x,y) values (string.join( > map(lambda x: repr(x),data),',')) > but that breaks with > ERROR: Attribute ... not found > > Clearly I can muck around parsing strings and escaping things, but > that seems remarkably clumsy, and I can't believe I'm the only person with > this problem. I've had a google and rummaged in the manuals, but obviously > didn't turn over the right leaf ... how does one do this with postgres? > > I'm not on the mailing list, so if you can help, could you please reply > directly? It's not an easy thing to search for in the archive, so apologies > if this is a much rehearsed problem. > > Thanks > Bryan I'm not a PostgreSQL expert so you may want to wait for one of them to supply the answer to your questions. Looking at the PostgreSQL web site there is no explicit documentation of pgdb other than to say that it is a Python interface that conforms to the DB-API version 2.0. In which case I suggest a quick look at; http://www.python.org/topics/database/DatabaseAPI-2.0.html Specifically the module interface called 'paramstyle'. Gadfly uses the 'qmark' style and MySQLdb uses a specialised version of the 'format' style. It is entirely possible that pgdb uses an alternative. To find out, from a python prompt type the following; >>> import pgdb >>> pgdb.paramstyle and a string should be displayed which tells you the paramstyle of your module. If I'm completely wide of the mark and pgdb is not DB-API 2.0 compatible (for instance it doesn't support the paramstyle module interface) there are a number of alternative PostgreSQL interface modules that may be better for you. They are listed on this web page; http://www.python.org/topics/database/modules.html Regards, Andy -- -------------------------------------------------------------------------------- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From b.n.lawrence at rl.ac.uk Wed Jun 18 16:04:21 2003 From: b.n.lawrence at rl.ac.uk (Bryan Lawrence) Date: Wed Jun 18 10:04:44 2003 Subject: [DB-SIG] inserting strings into postgres db using pgdb (escaping quotes automagically) In-Reply-To: <3EF063D8.7050008@halfcooked.com> References: <200306181325.15380.b.n.lawrence@rl.ac.uk> <3EF063D8.7050008@halfcooked.com> Message-ID: <200306181504.21099.b.n.lawrence@rl.ac.uk> Gidday Andy Thanks, problem solved ... well, my code works! However, I couldn't understand pyformat until I found an example which led me to the following mess: def sqlInsert(self,table,values,data): """ Given two lists: a list of keys and a list of values, insert them into a table (yes it's lists for historical reasons""" if self.dbtype=='postgresql': ddata={} for i in range(len(values)):ddata[values[i]]=data[i] cmd='insert into %s (%s) values (%s)'%( table, string.join(values,','), string.join( map( lambda x:'%s%s%s'%('%(',x,')s'),values),',') ) self.cur.execute(cmd,ddata) If anyone can point to a more elegant solution, then great, but otherwise thanks for the rapid help, I can live with this. Cheers, Bryan On Wednesday 18 June 2003 14:06, Andy Todd wrote: > Bryan Lawrence wrote: > > Hi Folks > > > > I'm just playing at a common db core module for an application. It works > > ok with gadfly and mysql, and I'm trying to extend it to postgresql. > > > > I'm using the pgdb module ... > > > > At the moment (mysql) I do inserts with something like > > > > data=('a',"b's") > > cmd="insert into table blah (x,y) values (%s%s)" > > self.cursor.execute(cmd,data) > > (replacing the %s with ? for gadfly). > > This breaks with postgres ... > > > > I tried something like > > cmd='insert into table blah (x,y) values (string.join( > > map(lambda x: repr(x),data),',')) > > but that breaks with > > ERROR: Attribute ... not found > > > > Clearly I can muck around parsing strings and escaping things, but > > that seems remarkably clumsy, and I can't believe I'm the only person > > with this problem. I've had a google and rummaged in the manuals, but > > obviously didn't turn over the right leaf ... how does one do this with > > postgres? > > > > I'm not on the mailing list, so if you can help, could you please reply > > directly? It's not an easy thing to search for in the archive, so > > apologies if this is a much rehearsed problem. > > > > Thanks > > Bryan > > I'm not a PostgreSQL expert so you may want to wait for one of them to > supply the answer to your questions. Looking at the PostgreSQL web site > there is no explicit documentation of pgdb other than to say that it is > a Python interface that conforms to the DB-API version 2.0. In which > case I suggest a quick look at; > > http://www.python.org/topics/database/DatabaseAPI-2.0.html > > Specifically the module interface called 'paramstyle'. Gadfly uses the > 'qmark' style and MySQLdb uses a specialised version of the 'format' > style. It is entirely possible that pgdb uses an alternative. To find > out, from a python prompt type the following; > > >>> import pgdb > >>> pgdb.paramstyle > > and a string should be displayed which tells you the paramstyle of your > module. > > If I'm completely wide of the mark and pgdb is not DB-API 2.0 compatible > (for instance it doesn't support the paramstyle module interface) there > are a number of alternative PostgreSQL interface modules that may be > better for you. They are listed on this web page; > > http://www.python.org/topics/database/modules.html > > Regards, > Andy -- Bryan Lawrence, Head NCAS/British Atmospheric Data Centre web: www.badc.nerc.ac.uk phone: +44 1235 445012 CLRC: Rutherford Appleton Laboratory, Chilton, Didcot, OX110QX, UK From andy47 at halfcooked.com Wed Jun 18 17:14:20 2003 From: andy47 at halfcooked.com (Andy Todd) Date: Wed Jun 18 11:15:52 2003 Subject: [DB-SIG] inserting strings into postgres db using pgdb (escaping quotes automagically) In-Reply-To: <200306181504.21099.b.n.lawrence@rl.ac.uk> References: <200306181325.15380.b.n.lawrence@rl.ac.uk> <3EF063D8.7050008@halfcooked.com> <200306181504.21099.b.n.lawrence@rl.ac.uk> Message-ID: <3EF081CC.2000505@halfcooked.com> Bryan Lawrence wrote: > Gidday Andy > > Thanks, problem solved ... well, my code works! > > However, I couldn't understand pyformat until I found an example which led > me to the following mess: > > def sqlInsert(self,table,values,data): > """ Given two lists: a list of keys and a list of values, insert them > into a table (yes it's lists for historical reasons""" > if self.dbtype=='postgresql': > ddata={} > for i in range(len(values)):ddata[values[i]]=data[i] > cmd='insert into %s (%s) values (%s)'%( > table, > string.join(values,','), > string.join( map( lambda x:'%s%s%s'%('%(',x,')s'),values),',') > ) > self.cur.execute(cmd,ddata) > > If anyone can point to a more elegant solution, then great, but > otherwise thanks for the rapid help, I can live with this. > > Cheers, > Bryan > [snip] > > Yuck. But if it works ... The only quibble I would have is that the 'string' module is deprecated. Given a list called, say, 'myList' you can replace; >>> string.join(myList, ',') With; >>> ','.join(myList) Which you do twice above. Then a couple of minor stylistic quibbles, feel free to ignore these. You can do you string substitution more elegantly (I think) by replacing; '%s%s%s'%'%(',x,')s' with; '%('+'%s'%x+')s' Whilst I was there, I'd also replace the map and lambda with a list comprehension, giving us something like; ','.join(['%('+'%s'%x%')s' for x in values]) Regards, Andy -- -------------------------------------------------------------------------------- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From nathan-pydb at geerbox.com Wed Jun 18 10:05:36 2003 From: nathan-pydb at geerbox.com (Nathan Clegg) Date: Wed Jun 18 12:04:35 2003 Subject: [DB-SIG] inserting strings into postgres db using pgdb (escaping quotes automagically) In-Reply-To: <200306181504.21099.b.n.lawrence@rl.ac.uk> References: <200306181325.15380.b.n.lawrence@rl.ac.uk> <3EF063D8.7050008@halfcooked.com> <200306181504.21099.b.n.lawrence@rl.ac.uk> Message-ID: <16112.36304.357054.949771@jin.int.geerbox.com> > If anyone can point to a more elegant solution, then great, but > otherwise thanks for the rapid help, I can live with this. Well I'm not sure why you are forcing your lists into dictionaries. How about this: def sqlInsert(self,table,values,data): if self.dbtype == 'postgresql': cmd = 'insert into %s (%s) values (%s)' % ( table, ', '.join(values), ', '.join(('%s',) * len(values)), ) self.cur.execute(cmd, data) -- Nathan Clegg GeerBox From andy47 at halfcooked.com Wed Jun 18 18:23:09 2003 From: andy47 at halfcooked.com (Andy Todd) Date: Wed Jun 18 12:24:54 2003 Subject: [DB-SIG] inserting strings into postgres db using pgdb (escaping quotes automagically) In-Reply-To: <16112.36304.357054.949771@jin.int.geerbox.com> References: <200306181325.15380.b.n.lawrence@rl.ac.uk> <3EF063D8.7050008@halfcooked.com> <200306181504.21099.b.n.lawrence@rl.ac.uk> <16112.36304.357054.949771@jin.int.geerbox.com> Message-ID: <3EF091ED.8080400@halfcooked.com> Nathan Clegg wrote: >>If anyone can point to a more elegant solution, then great, but >>otherwise thanks for the rapid help, I can live with this. > > > > Well I'm not sure why you are forcing your lists into dictionaries. > How about this: > > > def sqlInsert(self,table,values,data): > if self.dbtype == 'postgresql': > cmd = 'insert into %s (%s) values (%s)' % ( > table, > ', '.join(values), > ', '.join(('%s',) * len(values)), > ) > self.cur.execute(cmd, data) > > > The column names and values are put into a dictionary because pgdb uses the 'pyformat' paramstyle. The general form of which is; >>> cursor.execute("SELECT column_a FROM table_a WHERE column_b=%(param_a)s", {'param_a':'some value'}) This ensures that values are appropriately quoted and parsed by the database module and not by you. This means that funny (or just unexpected) characters don't cause your query to fail in interesting and spectacular ways. Regards, Andy -- -------------------------------------------------------------------------------- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ From nathan-pydb at geerbox.com Wed Jun 18 10:42:40 2003 From: nathan-pydb at geerbox.com (Nathan Clegg) Date: Wed Jun 18 12:41:38 2003 Subject: [DB-SIG] inserting strings into postgres db using pgdb (escaping quotes automagically) In-Reply-To: <3EF091ED.8080400@halfcooked.com> References: <200306181325.15380.b.n.lawrence@rl.ac.uk> <3EF063D8.7050008@halfcooked.com> <200306181504.21099.b.n.lawrence@rl.ac.uk> <16112.36304.357054.949771@jin.int.geerbox.com> <3EF091ED.8080400@halfcooked.com> Message-ID: <16112.38528.449407.951539@jin.int.geerbox.com> Can't pgdb also use 'format' paramstyle? It is no less safe. The only advantage is being able to ignore order and is convenient for repeating the same param multiple times in the statement, which is unnecessary in this example. My code is no more likely to fail in interesting and spectacular ways than yours, as I am not doing any quoting or parsing myself. Andy Todd wrote: > The column names and values are put into a dictionary because pgdb uses > the 'pyformat' paramstyle. The general form of which is; > > >>> cursor.execute("SELECT column_a FROM table_a WHERE > column_b=%(param_a)s", {'param_a':'some value'}) > > This ensures that values are appropriately quoted and parsed by the > database module and not by you. This means that funny (or just > unexpected) characters don't cause your query to fail in interesting and > spectacular ways. -- Nathan Clegg GeerBox From msanchez at grupoburke.com Wed Jun 18 22:02:57 2003 From: msanchez at grupoburke.com (Marcos =?ISO-8859-1?Q?S=E1nchez?= Provencio) Date: Wed Jun 18 17:02:59 2003 Subject: [DB-SIG] inserting strings into postgres db using pgdb (escaping quotes automagically) In-Reply-To: <16112.38528.449407.951539@jin.int.geerbox.com> References: <200306181325.15380.b.n.lawrence@rl.ac.uk> <3EF063D8.7050008@halfcooked.com> <200306181504.21099.b.n.lawrence@rl.ac.uk> <16112.36304.357054.949771@jin.int.geerbox.com> <3EF091ED.8080400@halfcooked.com> <16112.38528.449407.951539@jin.int.geerbox.com> Message-ID: <1055970122.844.18.camel@renata.macondo.pri> This has been discussed before and recently http://mail.python.org/pipermail/db-sig/2001-April/001719.html http://aspn.activestate.com/ASPN/Mail/Message/1538118 El mi?, 18 de 06 de 2003 a las 18:42, Nathan Clegg escribi?: > Can't pgdb also use 'format' paramstyle? It is no less safe. The > only advantage is being able to ignore order and is convenient for > repeating the same param multiple times in the statement, which is > unnecessary in this example. My code is no more likely to fail in > interesting and spectacular ways than yours, as I am not doing any > quoting or parsing myself. > > > Andy Todd wrote: > > The column names and values are put into a dictionary because pgdb uses > > the 'pyformat' paramstyle. The general form of which is; > > > > >>> cursor.execute("SELECT column_a FROM table_a WHERE > > column_b=%(param_a)s", {'param_a':'some value'}) > > > > This ensures that values are appropriately quoted and parsed by the > > database module and not by you. This means that funny (or just > > unexpected) characters don't cause your query to fail in interesting and > > spectacular ways. -- Marcos S?nchez Provencio www.burke.es From anthony at computronix.com Fri Jun 20 20:31:00 2003 From: anthony at computronix.com (Anthony Tuininga) Date: Fri Jun 20 15:31:01 2003 Subject: [DB-SIG] cx_Oracle 3.0a Message-ID: <1056137434.27168.10.camel@localhost.localdomain> What is cx_Oracle? cx_Oracle is a Python extension module that allows access to Oracle and conforms to the Python database API 2.0 specifications with a few exceptions. Where do I get it? http://starship.python.net/crew/atuining http://www.computronix.com/utilities.shtml (it may be a few days before the second site is updated) What's new? 1) Fixed bug where zero length PL/SQL arrays were being mishandled 2) Fixed support for the data type "float" in Oracle; added one to the display size to allow for the sign of the number, if necessary; changed the display size of unconstrained numbers to 127, which is the largest number that Oracle can handle 3) Added support for retrieving the description of a bound cursor before fetching it 4) Fixed a couple of build issues on Mac OS X, AIX and Solaris (64-bit) 5) Modified documentation slightly based on comments from several people 6) Included files in MANIFEST that are needed to generate the binaries 7) Modified test suite to work within the test environment at Computronix as well as within the packages that are distributed -- Anthony Tuininga anthony@computronix.com Computronix Distinctive Software. Real People. Suite 200, 10216 - 124 Street NW Edmonton, AB, Canada T5N 4A3 Phone: (780) 454-3700 Fax: (780) 454-3838 http://www.computronix.com From postmaster at ns2.admincontrolpanel.com Mon Jun 23 18:05:16 2003 From: postmaster at ns2.admincontrolpanel.com (postmaster@ns2.admincontrolpanel.com) Date: Mon Jun 23 17:05:21 2003 Subject: [DB-SIG] Delivery failure (rc873@hotmail.com.com) Message-ID: Your message has encountered delivery problems to the following recipient(s): rc873@hotmail.com.com Delivery failed Unable to deliver to destination domain Cannot resolve hotmail.com.com -------------- next part -------------- Skipped content of type message/delivery-status-------------- next part -------------- Skipped content of type message/rfc822-headers From marko.mihalec at salomon.at Wed Jun 25 10:42:45 2003 From: marko.mihalec at salomon.at (Marko Mihalec) Date: Wed Jun 25 05:42:47 2003 Subject: [DB-SIG] informixdb Message-ID: <1056534157.2208.16.camel@pc241054.salomon.at> hy, can informixdb 1.3 work fine with python 2.1.3? i have try to make static and become error with some python library: ERROR:: rm -f lib.a ar cr lib.a ranlib lib.a cc -Ae -Wl,-E -Wl,+s -Wl,+b/usr/local/python-2.1.3/lib/python2.1/lib-dynload \ /usr/local/python-2.1.3/lib/python2.1/config/python.o config.o lib.a /usr/local/python-2.1.3/lib/python2.1/config/libpython2.1.a \ -lnsl -ldld -lpthread -lcl -lm \ -o python /usr/ccs/bin/ld: Unexpected end of file in lib.a *** Error exit code 1 /* END OF ERROR*/ thanks From anthony at computronix.com Wed Jun 25 17:23:40 2003 From: anthony at computronix.com (Anthony Tuininga) Date: Wed Jun 25 12:23:41 2003 Subject: [DB-SIG] cx_Oracle mailing list Message-ID: <1056558193.18388.11.camel@localhost.localdomain> A new mailing list has been set up for users of cx_Oracle, a Python extension for accessing Oracle that conforms to the Python database interface API. Please send questions/bug reports and so forth to this mailing list rather than directly to me (although I'll certainly continue to respond to e-mails sent directly to me, others might benefit from the question and the answer). If you wish to subscribe, use the following address http://lists.sourceforge.net/lists/listinfo/cx-oracle-users Since the mailing list is hosted at SourceForge, you can also go to https://sourceforge.net/projects/cx-oracle/ to get information about the project. -- Anthony Tuininga anthony@computronix.com Computronix Distinctive Software. Real People. Suite 200, 10216 - 124 Street NW Edmonton, AB, Canada T5N 4A3 Phone: (780) 454-3700 Fax: (780) 454-3838 http://www.computronix.com From MAILER-DAEMON at mta3.service.uci.edu Fri Jun 27 15:37:35 2003 From: MAILER-DAEMON at mta3.service.uci.edu (MAILER-DAEMON@mta3.service.uci.edu) Date: Fri Jun 27 17:37:41 2003 Subject: [DB-SIG] Returned mail - nameserver error report Message-ID: <200306272137.h5RLbZl05882@mta3.service.uci.edu> --------Message not delivered to the following addresses: shreeve@uci.edu --------Error Detail (phquery V3.0): ---- shreeve@uci.edu Does not exist, and no people with that name or alias(UCInetID) are at UCI. Please contact Postmaster@uci.edu if you have any questions. --------Unsent Message below: Received: (from daemon@localhost) by mta3.service.uci.edu (8.11.4/8.11.2) id h5RLbZK05870 for shreeve@uci.edu.xyzzy; Fri, 27 Jun 2003 14:37:35 -0700 (PDT) Received: from APEXADELL (www.competesmart.net [68.157.84.3]) by mta3.service.uci.edu (8.11.4/8.11.2) with ESMTP id h5RLaVI05075 for ; Fri, 27 Jun 2003 14:36:31 -0700 (PDT) Message-Id: <200306272136.h5RLaVI05075@mta3.service.uci.edu> From: To: Subject: Re: Application Date: Fri, 27 Jun 2003 17:36:31 --0400 Importance: Normal X-Mailer: Microsoft Outlook Express 6.00.2600.0000 X-MSMail-Priority: Normal X-Priority: 3 (Normal) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="CSmtpMsgPart123X456_000_00CC00B4" X-UCIRVINE-MailScanner: Found to be infected This is a multipart message in MIME format --CSmtpMsgPart123X456_000_00CC00B4 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Warning: This message has had one or more attachments removed. Warning: Please read the "VirusWarning.txt" attachment(s) for more information. Please see the attached zip file for details. --CSmtpMsgPart123X456_000_00CC00B4 Content-Type: text/plain; charset="us-ascii"; name="VirusWarning.txt" Content-Disposition: attachment; filename="VirusWarning.txt" Content-Transfer-Encoding: quoted-printable This is a message from the University of California at Irvine, MailScanner E-Mail Virus Protection Service ---------------------------------------------------------------------- The original e-mail attachment "your_details.zip" is on the list of unacceptable attachments for this site and has been replaced by this warning message. At Fri Jun 27 14:36:47 2003 the virus scanner said: The Sobig.d virus in your_details.zip For more information on the Mail Transport Services: Virus Scanning go to http://www.nacs.uci.edu/email/virus-scanning.html. -- Network & Academic Computing Services (949)824-2222 --CSmtpMsgPart123X456_000_00CC00B4-- --------End of Unsent Message