From fumanchu at amor.org Sat Jan 1 03:53:52 2005 From: fumanchu at amor.org (Robert Brewer) Date: Sat Jan 1 03:56:29 2005 Subject: [DB-SIG] BLOB data types Message-ID: <3A81C87DC164034AA4E2DDFE11D258E3024F6A@exchange.hqamor.amorhq.net> Andy Dustman wrote: > What's the current view on the best object class to use for > Binary/BLOB objects? PEP-249 recommends buffer, but buffer > appears deprecated. MySQLdb used to strings, but currently > uses array. mxODBC appears to use strings. psycopg appears > to use buffer. > > Take a look at this for additional comments: > > https://sourceforge.net/tracker/index.php?func=detail > &aid=975831&group_id=22307&atid=374932 > > Anyone got any strong feelings about this? Won't Python strings be encoded, and therefore often choke on values between 128 and 255? To my naive mind, unicode objects would be the preferred choice. Robert Brewer MIS Amor Ministries fumanchu@amor.org From farcepest at gmail.com Sat Jan 1 10:06:33 2005 From: farcepest at gmail.com (Andy Dustman) Date: Sat Jan 1 10:06:36 2005 Subject: [DB-SIG] BLOB data types In-Reply-To: <3A81C87DC164034AA4E2DDFE11D258E3024F6A@exchange.hqamor.amorhq.net> References: <3A81C87DC164034AA4E2DDFE11D258E3024F6A@exchange.hqamor.amorhq.net> Message-ID: <9826f3800501010106108029f1@mail.gmail.com> On Fri, 31 Dec 2004 18:53:52 -0800, Robert Brewer wrote: > Andy Dustman wrote: > > What's the current view on the best object class to use for > > Binary/BLOB objects? PEP-249 recommends buffer, but buffer > > appears deprecated. MySQLdb used to strings, but currently > > uses array. mxODBC appears to use strings. psycopg appears > > to use buffer. > Won't Python strings be encoded, and therefore often choke on values between 128 and 255? To my naive mind, unicode objects would be the preferred choice. No. With MySQL, at least, the only meta-characters it cares about in strings passed to it are single-quote ('), backslash (\), and NUL (zero-byte), and all of these can be escaped with backslash, and there is an API function to do this safely. You don't have to do this with values you pass in; it happens automatically with MySQLdb. Getting 8-bit binary strings out of MySQL and into Python is not a problem either. One reason Unicode objects weren't used because, when PEP-249 was written, they didn't exist (IIRC). -- Computer interfaces should never be made of meat. Using GMail? Setting Reply-to address to <> disables this annoying feature. From mal at egenix.com Sat Jan 1 12:31:21 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Sat Jan 1 12:31:24 2005 Subject: [DB-SIG] BLOB data types In-Reply-To: <9826f3800501010106108029f1@mail.gmail.com> References: <3A81C87DC164034AA4E2DDFE11D258E3024F6A@exchange.hqamor.amorhq.net> <9826f3800501010106108029f1@mail.gmail.com> Message-ID: <41D68A09.1010901@egenix.com> Andy Dustman wrote: > On Fri, 31 Dec 2004 18:53:52 -0800, Robert Brewer wrote: > >>Andy Dustman wrote: >> >>>What's the current view on the best object class to use for >>>Binary/BLOB objects? PEP-249 recommends buffer, but buffer >>>appears deprecated. MySQLdb used to strings, but currently >>>uses array. mxODBC appears to use strings. psycopg appears >>>to use buffer. > > >>Won't Python strings be encoded, and therefore often choke on values between 128 and 255? To my naive mind, unicode objects would be the preferred choice. > > > No. With MySQL, at least, the only meta-characters it cares about in > strings passed to it are single-quote ('), backslash (\), and NUL > (zero-byte), and all of these can be escaped with backslash, and there > is an API function to do this safely. You don't have to do this with > values you pass in; it happens automatically with MySQLdb. Getting > 8-bit binary strings out of MySQL and into Python is not a problem > either. > > One reason Unicode objects weren't used because, when PEP-249 was > written, they didn't exist (IIRC). Unicode is only meant for text data. Until we get a proper binary type in Python, I think strings or buffers are still the best way to go. For input, using a buffer object will make it possible for the DB-API interface to differ between text data (strings) and binary data (buffer) and apply any necessary action when binding paramaters. This is what mxODBC does, BTW, in addition to being careful with non-ASCII string data in strings (not all databases can store binary in CHAR or VARCHAR columns). Happy New Year, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 01 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2004-12-06: Released eGenix mx Extensions for Python 2.4 ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From peter at monicol.co.uk Sat Jan 1 17:15:08 2005 From: peter at monicol.co.uk (Peter Mott) Date: Sat Jan 1 17:15:17 2005 Subject: [DB-SIG] BLOB data types ... API function In-Reply-To: <41D68A09.1010901@egenix.com> Message-ID: > > Andy Dustman wrote: > > On Fri, 31 Dec 2004 18:53:52 -0800, Robert Brewer > > No. With MySQL, at least, the only meta-characters it cares about in > > strings passed to it are single-quote ('), backslash (\), and NUL > > (zero-byte), and all of these can be escaped with backslash, and there > > is an API function to do this safely. That's nice. What is this API function? Peter From farcepest at gmail.com Sat Jan 1 17:29:52 2005 From: farcepest at gmail.com (Andy Dustman) Date: Sat Jan 1 17:29:57 2005 Subject: [DB-SIG] BLOB data types ... API function In-Reply-To: References: <41D68A09.1010901@egenix.com> Message-ID: <9826f380050101082953fa45c@mail.gmail.com> On Sat, 1 Jan 2005 16:15:08 -0000, Peter Mott wrote: > > > > Andy Dustman wrote: > > > On Fri, 31 Dec 2004 18:53:52 -0800, Robert Brewer > > > No. With MySQL, at least, the only meta-characters it cares about in > > > strings passed to it are single-quote ('), backslash (\), and NUL > > > (zero-byte), and all of these can be escaped with backslash, and there > > > is an API function to do this safely. > > That's nice. What is this API function? mysql_real_escape(). It's a MySQL API function., and if you're using MySQLdb, it's used internally; end-users never need to call it. -- Computer interfaces should never be made of meat. Using GMail? Setting Reply-to address to <> disables this annoying feature. From dieter at handshake.de Sat Jan 1 19:04:13 2005 From: dieter at handshake.de (Dieter Maurer) Date: Sat Jan 1 19:15:00 2005 Subject: [DB-SIG] BLOB data types In-Reply-To: <9826f3800501010106108029f1@mail.gmail.com> References: <3A81C87DC164034AA4E2DDFE11D258E3024F6A@exchange.hqamor.amorhq.net> <9826f3800501010106108029f1@mail.gmail.com> Message-ID: <16854.58909.890070.202482@gargle.gargle.HOWL> Andy Dustman wrote at 2005-1-1 04:06 -0500: > ... >One reason Unicode objects weren't used because, when PEP-249 was >written, they didn't exist (IIRC). And they should *NOT* be used for binary data! -- Dieter From linda_ferrari9 at yahoo.com Tue Jan 4 03:02:04 2005 From: linda_ferrari9 at yahoo.com (linda) Date: Tue Jan 4 03:35:16 2005 Subject: [DB-SIG] need help Message-ID: <20050104020204.28043.qmail@web52804.mail.yahoo.com> hello.. my name is linda.. i'm student, ineed your help in how to convert table in oracle into mysql.. how to do it that.. i search all in web how to convert that but their is no solution for me.. i really need your help for my problem.. thanks LINDA linda --------------------------------- Yahoo! Messenger - Communicate instantly..."Ping" your friends today! Download Messenger Now -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/db-sig/attachments/20050104/53220174/attachment.htm From David.Carey at nyscot.ang.af.mil Tue Jan 4 13:47:57 2005 From: David.Carey at nyscot.ang.af.mil (Carey David SSgt 109CF/SCBJ) Date: Tue Jan 4 13:45:02 2005 Subject: [DB-SIG] need help Message-ID: <200501041244.j04CiuDC029928@curly.region1.ang.af.mil> It seems a little vague what your trying to do, but I have a suggestion from what I can gather. go into oracle, at the prompt you need to describe your table. If your table is called 'customers' you would type desc customers it would give you something that looks like the following. Name Null? Type ------------------------------- -------- ---- CUST_ID NOT NULL NUMBER(5) CUST_NAME NOT NULL VARCHAR2(20) CUST_PHONE VARCHAR2(10) Print this out or hand copy it. I'm almost positive that the datatypes from oracle will not match up 100% with the datatypes of MySQL, but I don't have my SQL Bible on me at the moment to look. This is still fine though because a lot of the common data types such a char and varchar should match up no problem. There are tables you can find on the web that will tell you what datatypes MySQL supports. Go into MySQL and create your new table from the information you pulled from the oracle describe. I assume since your using oracle, your familiar with creating tables with SQL, and SQL is SQL pretty much across most databases. That is how I would convert the table. If I had to do more then several, then I would probably write a python program to do it ;-) Now if you have data that you want to move from one table to the next, I'm not sure what exactly would be the best method for you. It depends on your situation. Me personally, I would write a program with 2 database connections. One to the Oracle database and the other to the MySQL database. As the program was iterating through a select statement I would simultaneously have it performing insert statements to the MySQL table I previously created. Another way might be to put it into a csv file, copy it to your other machine, then import it somehow. This is my suggestion. I'm far from a pro at either Oracle or MySQL so there might be some other way that I don't know of, but as far as I know there isn't just one simple program or way to do it. However I'm sure if there is, someone more knowledgeable then myself on the list will comment. David Carey Stratton ANGB Scotia, NY ________________________________ From: linda [mailto:linda_ferrari9@yahoo.com] Sent: Monday, January 03, 2005 9:02 PM To: db-sig@python.org Subject: [DB-SIG] need help hello.. my name is linda.. i'm student, ineed your help in how to convert table in oracle into mysql.. how to do it that.. i search all in web how to convert that but their is no solution for me.. i really need your help for my problem.. thanks LINDA linda ________________________________ Yahoo! Messenger - Communicate instantly..."Ping" your friends today! Download Messenger Now From peter at monicol.co.uk Tue Jan 4 15:04:26 2005 From: peter at monicol.co.uk (Peter Mott) Date: Tue Jan 4 15:04:29 2005 Subject: [DB-SIG] need help In-Reply-To: <200501041244.j04CiuDC029928@curly.region1.ang.af.mil> Message-ID: I don't know Oracle but there should be (surewly must be) an option to dump the contents of a database or table to SQL (in MySQL it is mysqldump) after which you can tweak the SQL and then simply run it from within the MySQL monitor: eg source oracle.sql where your dump file was called oracle.sql You may evenb be ablke toi get away without tweaking the SQL at all. In principle it should not be needed. Peter > -----Original Message----- > From: db-sig-bounces@python.org [mailto:db-sig-bounces@python.org] On > Behalf Of Carey David SSgt 109CF/SCBJ > Sent: 04 January 2005 12:48 > To: 'linda'; 'db-sig@python.org' > Subject: RE: [DB-SIG] need help > > It seems a little vague what your trying to do, but I have a suggestion > from > what I can gather. > > go into oracle, at the prompt you need to describe your table. If your > table > is called 'customers' you would type > > desc customers > > it would give you something that looks like the following. > > Name Null? Type > ------------------------------- -------- ---- > CUST_ID NOT NULL NUMBER(5) > CUST_NAME NOT NULL VARCHAR2(20) > CUST_PHONE VARCHAR2(10) > > Print this out or hand copy it. I'm almost positive that the datatypes > from > oracle will not match up 100% with the datatypes of MySQL, but I don't > have > my SQL Bible on me at the moment to look. This is still fine though > because > a lot of the common data types such a char and varchar should match up no > problem. There are tables you can find on the web that will tell you what > datatypes MySQL supports. Go into MySQL and create your new table from the > information you pulled from the oracle describe. I assume since your using > oracle, your familiar with creating tables with SQL, and SQL is SQL pretty > much across most databases. That is how I would convert the table. If I > had > to do more then several, then I would probably write a python program to > do > it ;-) > > Now if you have data that you want to move from one table to the next, I'm > not sure what exactly would be the best method for you. It depends on your > situation. Me personally, I would write a program with 2 database > connections. One to the Oracle database and the other to the MySQL > database. > As the program was iterating through a select statement I would > simultaneously have it performing insert statements to the MySQL table I > previously created. Another way might be to put it into a csv file, copy > it > to your other machine, then import it somehow. > > This is my suggestion. I'm far from a pro at either Oracle or MySQL so > there > might be some other way that I don't know of, but as far as I know there > isn't just one simple program or way to do it. However I'm sure if there > is, > someone more knowledgeable then myself on the list will comment. > > David Carey > Stratton ANGB > Scotia, NY > > ________________________________ > > From: linda [mailto:linda_ferrari9@yahoo.com] > Sent: Monday, January 03, 2005 9:02 PM > To: db-sig@python.org > Subject: [DB-SIG] need help > > > hello.. > my name is linda.. i'm student, ineed your help in > how to convert table in oracle into mysql.. > how to do it that.. i search all in web how to convert that but > their is no solution for me.. > i really need your help for my problem.. > > thanks > LINDA > > > > linda > > ________________________________ > > Yahoo! Messenger > co > m> - Communicate instantly..."Ping" your friends today! Download > Messenger > Now > co > m/download/index.html> > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig From adijkstra at baandersconsultancy.nl Tue Jan 4 15:27:26 2005 From: adijkstra at baandersconsultancy.nl (Arjen Dijkstra) Date: Tue Jan 4 15:23:30 2005 Subject: [DB-SIG] need help In-Reply-To: References: Message-ID: <41DAA7CE.3020209@baandersconsultancy.nl> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/db-sig/attachments/20050104/0cd499d2/attachment.htm From peter at monicol.co.uk Tue Jan 4 17:35:14 2005 From: peter at monicol.co.uk (Peter Mott) Date: Tue Jan 4 17:35:52 2005 Subject: [DB-SIG] need help In-Reply-To: <41DAA7CE.3020209@baandersconsultancy.nl> Message-ID: I looked at second URL which actually gives a mapping between MySQL and Oracl types. This is a bit of a mysqldump file: -- -- Table structure for table 'categories' -- DROP TABLE IF EXISTS categories; CREATE TABLE categories ( category_name varchar(50) NOT NULL default '', PRIMARY KEY (category_name) ) TYPE=MyISAM; -- -- Dumping data for table 'categories' -- INSERT INTO categories VALUES ('Abortion'),('Accountants'),('Advertising'),('Air'),('Aluminium'),('Ambulan ce'),('Antiques') .... I'm suggesting that Linda generate the Oracle equivalen and the change the CREATE TABLE dump with the equivalent MySQL types. Somrthing like this surely works unless Oracle has no dump process which I find hard to believe (well MSAcess doesn't but .) Cheers Peter _____ From: Arjen Dijkstra [mailto:adijkstra@baandersconsultancy.nl] Sent: 04 January 2005 14:27 To: Peter Mott Cc: 'linda'; db-sig@python.org Subject: Re: [DB-SIG] need help I don't think that will work. Mysql has diverent types: http://www.utexas.edu/its/unix/reference/oracledocs/v92/B10501_01/win.920/a9 7249/toc.htm and especially: http://www.utexas.edu/its/unix/reference/oracledocs/v92/B10501_01/win.920/a9 7249/ch3.htm#1026907 Arjen Peter Mott wrote: I don't know Oracle but there should be (surewly must be) an option to dump the contents of a database or table to SQL (in MySQL it is mysqldump) after which you can tweak the SQL and then simply run it from within the MySQL monitor: eg source oracle.sql where your dump file was called oracle.sql You may evenb be ablke toi get away without tweaking the SQL at all. In principle it should not be needed. Peter -----Original Message----- From: db-sig-bounces@python.org [mailto:db-sig-bounces@python.org] On Behalf Of Carey David SSgt 109CF/SCBJ Sent: 04 January 2005 12:48 To: 'linda'; 'db-sig@python.org' Subject: RE: [DB-SIG] need help It seems a little vague what your trying to do, but I have a suggestion from what I can gather. go into oracle, at the prompt you need to describe your table. If your table is called 'customers' you would type desc customers it would give you something that looks like the following. Name Null? Type ------------------------------- -------- ---- CUST_ID NOT NULL NUMBER(5) CUST_NAME NOT NULL VARCHAR2(20) CUST_PHONE VARCHAR2(10) Print this out or hand copy it. I'm almost positive that the datatypes from oracle will not match up 100% with the datatypes of MySQL, but I don't have my SQL Bible on me at the moment to look. This is still fine though because a lot of the common data types such a char and varchar should match up no problem. There are tables you can find on the web that will tell you what datatypes MySQL supports. Go into MySQL and create your new table from the information you pulled from the oracle describe. I assume since your using oracle, your familiar with creating tables with SQL, and SQL is SQL pretty much across most databases. That is how I would convert the table. If I had to do more then several, then I would probably write a python program to do it ;-) Now if you have data that you want to move from one table to the next, I'm not sure what exactly would be the best method for you. It depends on your situation. Me personally, I would write a program with 2 database connections. One to the Oracle database and the other to the MySQL database. As the program was iterating through a select statement I would simultaneously have it performing insert statements to the MySQL table I previously created. Another way might be to put it into a csv file, copy it to your other machine, then import it somehow. This is my suggestion. I'm far from a pro at either Oracle or MySQL so there might be some other way that I don't know of, but as far as I know there isn't just one simple program or way to do it. However I'm sure if there is, someone more knowledgeable then myself on the list will comment. David Carey Stratton ANGB Scotia, NY ________________________________ From: linda [mailto:linda_ferrari9@yahoo.com] Sent: Monday, January 03, 2005 9:02 PM To: db-sig@python.org Subject: [DB-SIG] need help hello.. my name is linda.. i'm student, ineed your help in how to convert table in oracle into mysql.. how to do it that.. i search all in web how to convert that but their is no solution for me.. i really need your help for my problem.. thanks LINDA linda ________________________________ Yahoo! Messenger m > - Communicate instantly..."Ping" your friends today! Download Messenger Now m/download/index.html > _______________________________________________ DB-SIG maillist - DB-SIG@python.org http://mail.python.org/mailman/listinfo/db-sig _______________________________________________ DB-SIG maillist - DB-SIG@python.org http://mail.python.org/mailman/listinfo/db-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/db-sig/attachments/20050104/773ef7ce/attachment.htm From brunson at brunson.com Tue Jan 4 18:28:42 2005 From: brunson at brunson.com (Eric Brunson) Date: Tue Jan 4 18:16:37 2005 Subject: [DB-SIG] need help In-Reply-To: <20050104020204.28043.qmail@web52804.mail.yahoo.com> References: <20050104020204.28043.qmail@web52804.mail.yahoo.com> Message-ID: <41DAD24A.7060108@brunson.com> linda wrote: > hello.. > my name is linda.. i'm student, ineed your help in > how to convert table in oracle into mysql.. > how to do it that.. i search all in web how to convert that but their is > no solution for me.. > i really need your help for my problem.. > > thanks > LINDA > > > linda > The simplest thing would probably be use use "exp" to export the table, then edit the file to change datatypes into mysql types. From james at pythonweb.org Tue Jan 11 20:32:48 2005 From: james at pythonweb.org (James Gardner) Date: Tue Jan 11 20:33:01 2005 Subject: [DB-SIG] PDBC Draft Specification Message-ID: <41E429E0.9060404@pythonweb.org> Hello All, I've recently been working on a specification for a database abstraction layer for Python. PDBC is designed to address the issue of complexity for end users and enable them to write database portable code. The specification defines a standard SQL subset, field types and an API which all PDBC compliant databases would implement in order to achieve portable database programming for simple operations. The API exposes the existing DB-API 2.0 objects for more complex programming. A reference implementation is being written. An object relational mapper, pure python implementation and interactive prompt are being adapted for PDBC and I'm writing the user documentation at the moment. The URLs for the specification are: http://www.pythonweb.org/projects/pdbc/html/ http://www.pythonweb.org/projects/pdbc/pdbc.txt Any thoughts or comments would be really appreciated, in particular: - Do you think this would be useful to you? - Do you think this would be useful to others such as beginners, those who use languages other than Python or those involved in web programming where database independence is useful for a product to be widely used? - Do you know of any existing packages which do this better already? - Can you think of any critical omissions from API? - Would you be willing to implement the PDBC for the database engine you use? - Can you think of better choices for the emulated field types or their ranges of values? - Is it worth writing an error code system for every error which could conceivably occur within the PDBC SQL specification and would it actually be possible for module implementers to implement the error codes for their database or would this be too much of a challenge? Many thanks, James -- www.pythonweb.org From Michael at Hipp.com Wed Jan 12 23:25:56 2005 From: Michael at Hipp.com (Michael Hipp) Date: Wed Jan 12 23:26:01 2005 Subject: [DB-SIG] Read old Clarion file from Python? Message-ID: <41E5A3F4.2070803@Hipp.com> Does anyone know of a way to read an old Clarion 2.1 DOS file (.DAT) directly from Python? I've found a couple of utilities to dump the file to csv, but nothing to read it directly. Thanks, Michael From anthony at computronix.com Mon Jan 24 20:04:30 2005 From: anthony at computronix.com (Anthony Tuininga) Date: Mon Jan 24 21:16:47 2005 Subject: [DB-SIG] cx_Oracle 4.1 Message-ID: <41F546BE.3020708@computronix.com> What is cx_Oracle? cx_Oracle is a Python extension module that allows access to Oracle and conforms to the Python database API 2.0 specifications with a few exceptions. Where do I get it? http://starship.python.net/crew/atuining What's new? 1) Added support for Python 2.4. In Python 2.4, the datetime module is used for both binding and fetching of date and timestamp data. In Python 2.3, objects from the datetime module can be bound but the internal datetime objects will be returned from queries. 2) Added pickling support for LOB and datetime data. 3) Fully qualified the table name that was missing in an alter table statement in the setup test script as noted by Marc Gehling. 4) Added a section allowing for the setting of the RPATH linker directive in setup.py as requested by Iustin Pop. 5) Added code to raise a programming error exception when an attempt is made to access a LOB locator variable in a subsequent fetch. 6) The username, password and dsn (tnsentry) are stored on the connection object when specified, regardless of whether or not a standard connection takes place. 7) Added additional module level constant called "LOB" as requested by Joseph Canedo. 8) Changed exception type to IntegrityError for constraint violations as requested by Joseph Canedo. 9) If scale and precision are not specified, an attempt is made to return a long integer as requested by Joseph Canedo. 10) Added workaround for Oracle bug which returns an invalid handle when the prepare call fails. Thanks to alantam@hsbc.com for providing the code that demonstrated the problem. 11) The cusor method arravar() will now accept the actual list so that it is not necessary to call cursor.arrayvar() followed immediately by var.setvalue(). 12) Fixed bug where attempts to execute the statement "None" with bind variables would cause a segmentation fault. 13) Added support for binding by position (paramstyle = "numeric"). 14) Removed memory leak created by calls to OCIParamGet() which were not mirrored by calls to OCIDescriptorFree(). Thanks to Mihai Ibanescu for pointing this out and providing a patch. 15) Added support for calling cursor.executemany() with statement None implying that the previously prepared statement ought to be executed. Thanks to Mihai Ibanescu for providing a patch. 16) Added support for rebinding variables when a subsequent call to cursor.executemany() uses a different number of rows. Thanks to Mihai Ibanescu for supplying a patch. 17) The microseconds are now displayed in datetime variables when nonzero similar to method used in the datetime module. 18) Added support for binary_float and binary_double columns in Oracle 10g. Changes since 4.1 beta 1 1) Fixed bug where subclasses of Cursor do not pass the connection in the constructor causing a segfault. 2) DDL statements must be reparsed before execution as noted by Mihai Ibanescu. 3) Add support for setting input sizes by position. 4) Fixed problem with catching an exception during execute and then still attempting to perform a fetch afterwards as noted by Leith Parkin. 5) Rename the types so that they can be pickled and unpickled. Thanks to Harri Pasanen for pointing out the problem. 6) Handle invalid NLS_LANG setting properly (Oracle seems to like to provide a handle back even though it is invalid) and determine the number of bytes per character in order to allow for proper support in the future of multibyte and variable width character sets. 7) Remove date checking from the native case since Python already checks that dates are valid; enhance error message when invalid dates are encountered so that additional processing can be done. 8) Fix bug executing SQL using numeric parameter names with predefined variables (such as what takes place when calling stored procedures with out parameters). 9) Add support for reading CLOB values using multibyte or variable length character sets. -- Anthony Tuininga anthony@computronix.com Computronix Distinctive Software. Real People. Suite 200, 10216 - 124 Street NW Edmonton, AB, Canada T5N 4A3 Phone: (780) 454-3700 Fax: (780) 454-3838 http://www.computronix.com From paul.hide at gmail.com Tue Jan 25 15:49:52 2005 From: paul.hide at gmail.com (Paul Hide) Date: Tue Jan 25 15:49:55 2005 Subject: [DB-SIG] cx_Oracle DLL load failed Message-ID: I recieve the following error: (for version and os info see below) IDLE 1.0.3 >>> import cx_Oracle Traceback (most recent call last): File "", line 1, in -toplevel- import cx_Oracle ImportError: DLL load failed: The specified procedure could not be found. I chased this on google, found some hits , but none helped. This machine has oracle client software set up on it (forms reports, sqlplus) and connects to a remote oracle database on my local intranet. Can anyone offer any suggestions? Paul Hide Python version is: Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] on win32 Os version is: MS Win 2000 sp2 Python 2.4 is also installed on this machine. Path is: C:\Tcl\bin;C:\orant\bin;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;C:\orant\jdk\bin From marcos at burke.ath.cx Tue Jan 25 16:52:16 2005 From: marcos at burke.ath.cx (Marcos =?ISO-8859-1?Q?S=E1nchez?= Provencio) Date: Tue Jan 25 16:52:20 2005 Subject: [DB-SIG] cx_Oracle DLL load failed In-Reply-To: References: Message-ID: <1106668337.4257.11.camel@cynar.proteus> Seems like a version mismatch. Is the cx_Oracle version matching the oracle libs? You may try to use python -v to see more details. El mar, 25-01-2005 a las 14:49 +0000, Paul Hide escribi?: > I recieve the following error: (for version and os info see below) > > IDLE 1.0.3 > >>> import cx_Oracle > > Traceback (most recent call last): > File "", line 1, in -toplevel- > import cx_Oracle > ImportError: DLL load failed: The specified procedure could not be found. > > I chased this on google, found some hits , but none helped. > > This machine has oracle client software set up on it (forms reports, > sqlplus) and connects to a remote oracle database on my local > intranet. > Can anyone offer any suggestions? > > Paul Hide > > > Python version is: > Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] on win32 > > Os version is: > MS Win 2000 sp2 > > Python 2.4 is also installed on this machine. > > Path is: > C:\Tcl\bin;C:\orant\bin;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;C:\orant\jdk\bin > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig -- Marcos S?nchez Provencio From anthony at computronix.com Tue Jan 25 15:58:27 2005 From: anthony at computronix.com (Anthony Tuininga) Date: Tue Jan 25 21:16:02 2005 Subject: [DB-SIG] cx_Oracle DLL load failed In-Reply-To: References: Message-ID: <41F65E93.9080708@computronix.com> You forgot the most important part of the information required to diagnose your problem -- the version of Oracle and the version of cx_Oracle. :-) In addition, when you get that particular error, there is a message box that comes up with some additional information -- what was that information? I would guess that it is complaining about the procedure OCISessionGet() which is only available in Oracle 9.2 and up. If you are running 9.0 or lower, use the Oracle 8i version of cx_Oracle. If that's not it, please reply with the information above. Thanks! Paul Hide wrote: > I recieve the following error: (for version and os info see below) > > IDLE 1.0.3 > >>>>import cx_Oracle > > > Traceback (most recent call last): > File "", line 1, in -toplevel- > import cx_Oracle > ImportError: DLL load failed: The specified procedure could not be found. > > I chased this on google, found some hits , but none helped. > > This machine has oracle client software set up on it (forms reports, > sqlplus) and connects to a remote oracle database on my local > intranet. > Can anyone offer any suggestions? > > Paul Hide > > > Python version is: > Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] on win32 > > Os version is: > MS Win 2000 sp2 > > Python 2.4 is also installed on this machine. > > Path is: > C:\Tcl\bin;C:\orant\bin;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;C:\orant\jdk\bin > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig -- Anthony Tuininga anthony@computronix.com Computronix Distinctive Software. Real People. Suite 200, 10216 - 124 Street NW Edmonton, AB, Canada T5N 4A3 Phone: (780) 454-3700 Fax: (780) 454-3838 http://www.computronix.com From scherrey at proteus-tech.com Tue Jan 25 22:22:39 2005 From: scherrey at proteus-tech.com (Benjamin Scherrey) Date: Tue Jan 25 22:22:43 2005 Subject: [DB-SIG] Python with Rails? Message-ID: <41F6B89F.7030506@proteus-tech.com> Anyone checked out the 'Ruby with Rails' ( http://www.rubyonrails.com ) project? I've not yet played around with it but their mechanism of definging a seemingly seemless object-relational mapping (the Active Record aspect of the project) from within the language is really promising. From what I've seen, their use of reflection to implement this doesn't seem outside of good python programming practice and, I believe, python should be fully capable of supporting a similar construct without modifying the existing language. I post this message here to get feedback on what kind of interest this idea may have and also how reasonable it would be to implement such a system under python. Appreciate any comments or ideas. Ruby does not have anything close to the powerful module support of python, but this thing is so compelling that I could see a significantly increased adaptation of the language, possibly over python. I wonder as to its scalability and flexibility if you want to augment the relational implementation that it creates though. best regards, Ben Scherrey From ianb at colorstudy.com Tue Jan 25 22:49:03 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Tue Jan 25 22:49:59 2005 Subject: [DB-SIG] Python with Rails? In-Reply-To: <41F6B89F.7030506@proteus-tech.com> References: <41F6B89F.7030506@proteus-tech.com> Message-ID: <41F6BECF.8030403@colorstudy.com> Benjamin Scherrey wrote: > Anyone checked out the 'Ruby with Rails' ( http://www.rubyonrails.com > ) project? I've not yet played around with it but their mechanism of > definging a seemingly seemless object-relational mapping (the Active > Record aspect of the project) from within the language is really > promising. From what I've seen, their use of reflection to implement > this doesn't seem outside of good python programming practice and, I > believe, python should be fully capable of supporting a similar > construct without modifying the existing language. Someone's working on it: http://subway.python-hosting.com/ SQLObject, which is what Subway is using, is fairly similar to ActiveRecord. This is what you do to create a class with attributes read from the database: from sqlobject import SQLObject class MyTable(SQLObject): _connection = 'mysql://user:passwd@host/dbname' _fromDatabase = True -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From iiourov at yahoo.com Wed Jan 26 05:16:52 2005 From: iiourov at yahoo.com (Ilia Iourovitski) Date: Wed Jan 26 05:16:54 2005 Subject: [DB-SIG] Python with Rails? In-Reply-To: <41F6BECF.8030403@colorstudy.com> Message-ID: <20050126041652.71877.qmail@web53207.mail.yahoo.com> It is not a problem in Jython land. Use JPublish as web framework. Next any decent web framework such as Tapestry or Webwork allow to create controller in Jython. The only thing which Jython is missing is continuations to compete Javascript in Cocoon. It seems the continuations is the next big thing in Web Development. About persistence. Hibernate is good. Hibernate 3.0 supports dynamic components as Map. So user can access hierarchy of tables as Map of Map. It would be nice to add support to saving Python objects and add ZODB style adapter in order to run Plone on Weblogic. Thx, Ilia --- Ian Bicking wrote: > Benjamin Scherrey wrote: > > Anyone checked out the 'Ruby with Rails' ( > http://www.rubyonrails.com > > ) project? I've not yet played around with it but > their mechanism of > > definging a seemingly seemless object-relational > mapping (the Active > > Record aspect of the project) from within the > language is really > > promising. From what I've seen, their use of > reflection to implement > > this doesn't seem outside of good python > programming practice and, I > > believe, python should be fully capable of > supporting a similar > > construct without modifying the existing language. > > Someone's working on it: > http://subway.python-hosting.com/ > > SQLObject, which is what Subway is using, is fairly > similar to > ActiveRecord. This is what you do to create a class > with attributes > read from the database: > > from sqlobject import SQLObject > class MyTable(SQLObject): > _connection = 'mysql://user:passwd@host/dbname' > _fromDatabase = True > > -- > Ian Bicking / ianb@colorstudy.com / > http://blog.ianbicking.org > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig > From ianb at colorstudy.com Wed Jan 26 06:54:57 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed Jan 26 06:54:59 2005 Subject: [DB-SIG] [ANN] SQLObject 0.6.1 Message-ID: <41F730B1.80601@colorstudy.com> SQLObject 0.6.1 --------------- Version 0.6.1 is a bug fix release for 0.6. Most of the work on this release has been thanks to the contributions of Oleg Broytmann. Thanks Oleg! A brief list of changes: * All class methods take a connection argument. * "DISTINCT" option for select results. * Connection objects have a new module attribute, to get access to the connection's exceptions. * New UnicodeCol() for encoding and decoding values from the database. * Added Indexing (patch from Jeremy Fitzhardinge). * Database connections explicitly closed, instead of just letting them be garbage collected. * selectBy can take be called with instances instead of only IDs for foreign key columns. * DBMConnection was removed (it's been broken a long time). What Is SQLObject? ------------------ SQLObject is an Object-Relational Mapper. Basically it makes your database rows feel like (relatively) normal Python objects: Tables are classes, rows are instances, columns are attributes. SQLObject both allows you to define your classes in Python, and SQLObject will create the tables on your behalf, or SQLObject may use database reflection to dynamically create a class given a table name. SQLObject portably supports a variety of database backends: MySQL, PostgreSQL, SQLite, Firebird, MaxDB/SAPDB, and SyBase. SQLObject is being used in a variety of external projects: ezSQLObject, sqlo for Zope 3, and Subway (a Ruby on Rails clone), among others. Where Is SQLObject? ------------------- Website: http://sqlobject.org News: http://sqlobject.org/docs/News.html#sqlobject-0-6-1 Documentation: http://sqlobject.org/docs/SQLObject.html Subversion repository: http://svn.colorstudy.com/trunk/SQLObject Mailing list: http://lists.sourceforge.net/mailman/listinfo/sqlobject-discuss Archives: http://dir.gmane.org/gmane.comp.python.sqlobject Download: http://prdownloads.sourceforge.net/sqlobject/SQLObject-0.6.1.tar.gz?download -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From anthony at computronix.com Wed Jan 26 23:05:43 2005 From: anthony at computronix.com (Anthony Tuininga) Date: Wed Jan 26 23:36:13 2005 Subject: [DB-SIG] cx_Oracle packaging issue Message-ID: <41F81437.40301@computronix.com> I have received a number of e-mails about the fact that the Python 2.4 builds on Windows were missing the rather vital file cx_Oracle.pyd. :-) Tracking it down resulted in the discovery of a distutils bug in Python 2.4. I got around the problem (and logged a bug on SourceForge so it should get fixed properly later) and new files have been uploaded to SourceForge. As always, you can get at them directly from my web site: http://starship.python.net/crew/atuining My apologies for the packaging mishap. -- Anthony Tuininga anthony@computronix.com Computronix Distinctive Software. Real People. Suite 200, 10216 - 124 Street NW Edmonton, AB, Canada T5N 4A3 Phone: (780) 454-3700 Fax: (780) 454-3838 http://www.computronix.com From Dennis.Wilson at nike.com Wed Jan 26 17:21:15 2005 From: Dennis.Wilson at nike.com (Wilson, Dennis (ETW)) Date: Thu Jan 27 10:02:04 2005 Subject: [DB-SIG] installing cx_Oracle question Message-ID: <200501261621.j0QGLP3A010650@barrierb241.nike.com> I am on a windows machine Windows 2000 with activestate python 2.4. When I install the cx_Oracle package, all I end up with is the documentation. There is nothing in the site-packages directory. I downloaded the package from both starship and sourceforge. Any hints would be appreciated. Thanks :-) From wolfgang.keller.nospam at gmx.de Fri Jan 28 17:31:05 2005 From: wolfgang.keller.nospam at gmx.de (Wolfgang Keller) Date: Fri Jan 28 17:31:11 2005 Subject: [DB-SIG] Python with Rails? In-Reply-To: <41F6B89F.7030506@proteus-tech.com> References: <41F6B89F.7030506@proteus-tech.com> Message-ID: Am Tue, 25 Jan 2005 16:22:39 -0500 schriebst du: > Anyone checked out the 'Ruby with Rails' ( > http://www.rubyonrails.com ) project? I've not yet played around with it > but their mechanism of definging a seemingly seemless object-relational > mapping (the Active Record aspect of the project) from within the > language is really promising. I have no clue of "Active Record", but if I needed an ORM for Python, then I would use Modeling: http://modeling.sourceforge.net/ Best regards Wolfgang Keller From fumanchu at amor.org Fri Jan 28 18:20:23 2005 From: fumanchu at amor.org (Robert Brewer) Date: Fri Jan 28 18:23:38 2005 Subject: [DB-SIG] Python with Rails? Message-ID: <3A81C87DC164034AA4E2DDFE11D258E33982D4@exchange.hqamor.amorhq.net> Wolfgang Keller wrote: > Am Tue, 25 Jan 2005 16:22:39 -0500 schriebst du: > > > Anyone checked out the 'Ruby with Rails' ( > > http://www.rubyonrails.com ) project? I've not yet played > around with it > > but their mechanism of definging a seemingly seemless > object-relational > > mapping (the Active Record aspect of the project) from within the > > language is really promising. > > I have no clue of "Active Record", but if I needed an ORM for > Python, then I would use Modeling: > > http://modeling.sourceforge.net/ "Active Record" refers to an object that maps to a table row (and therefore a system based on such a primitive): http://www.martinfowler.com/eaaCatalog/activeRecord.html FWIW, Dejavu is also an "Active Record" ORM: http://www.aminus.org/rbre/python Robert Brewer MIS Amor Ministries fumanchu@amor.org From wnvcole at peppermillcas.com Fri Jan 28 18:51:14 2005 From: wnvcole at peppermillcas.com (Vernon Cole) Date: Fri Jan 28 18:51:18 2005 Subject: [DB-SIG] Eschew Obfuscation -- down with un-defined TLA's. Message-ID: <1DE30EB3ECE266409FDE6E7032F178C413988E@pcimail1s.peppermillcas.com> This is a general request for everyone to define all TLA's (Three Letter Acronyms) on first use in any text. Our industry is crawling with TLA's and XMLA's (eXtended Multi-Letter Acronyms). It seems in vogue to invent new ones on a weekly basis. Those of us not keeping up with any specific branch of development may not know the meaning of a TLA which others of us use on a regular basis. What is an ORM? ---- Vernon Cole (in the business before WWW, GUI, OOP, WYSIWYG, HTML, i18n, SQL and 4GL were invented.) -----Original Message----- From: db-sig-bounces@python.org [mailto:db-sig-bounces@python.org] On Behalf Of Wolfgang Keller Sent: Friday, January 28, 2005 9:31 AM To: DB-SIG@python.org Subject: Re: [DB-SIG] Python with Rails? Am Tue, 25 Jan 2005 16:22:39 -0500 schriebst du: > Anyone checked out the 'Ruby with Rails' ( > http://www.rubyonrails.com ) project? I've not yet played around with it > but their mechanism of definging a seemingly seemless object-relational > mapping (the Active Record aspect of the project) from within the > language is really promising. I have no clue of "Active Record", but if I needed an ORM for Python, then I would use Modeling: http://modeling.sourceforge.net/ Best regards Wolfgang Keller _______________________________________________ DB-SIG maillist - DB-SIG@python.org http://mail.python.org/mailman/listinfo/db-sig From bkc at murkworks.com Fri Jan 28 19:48:16 2005 From: bkc at murkworks.com (Brad Clements) Date: Fri Jan 28 19:27:26 2005 Subject: [DB-SIG] Table Oriented Programming In-Reply-To: <000a01c3f19b$dc5c7ea0$0a0aa8c0@mikesmonster> Message-ID: <41FA429F.28210.B46A9CD3@coal.murkworks.com> On 12 Feb 2004 at 13:10, Michael Hobbs wrote: > Consider this example code: > departments = DepartmentTable > employees = EmployeeTable > johnsDepartment = (departments * employees) / > (employees.dept == departments.dept) / > (employees.name == 'John') % > departments.name > print johnsDepartment[0].name Has anyone tried expressing sql operations as relational algebra statements in Python, like the example above. I see lots of talk in google, but no code. Is this idea out of fashion already? I can see lots of performance issues.. but it's still interesting to me. -- Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax http://www.wecanstopspam.org/ AOL-IM: BKClements From mal at egenix.com Fri Jan 28 19:45:36 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Jan 28 19:45:39 2005 Subject: [DB-SIG] Table Oriented Programming In-Reply-To: <41FA429F.28210.B46A9CD3@coal.murkworks.com> References: <41FA429F.28210.B46A9CD3@coal.murkworks.com> Message-ID: <41FA8850.3060106@egenix.com> Brad Clements wrote: > On 12 Feb 2004 at 13:10, Michael Hobbs wrote: > > >>Consider this example code: >> departments = DepartmentTable >> employees = EmployeeTable >> johnsDepartment = (departments * employees) / >> (employees.dept == departments.dept) / >> (employees.name == 'John') % >> departments.name >> print johnsDepartment[0].name > > > Has anyone tried expressing sql operations as relational algebra statements in Python, > like the example above. > > I see lots of talk in google, but no code. > > Is this idea out of fashion already? I can see lots of performance issues.. but it's still > interesting to me. FWIW, SQLObject has some logic in place which mimics some of these ideas. In general, I think you could work out a "compiler" which generates SQL from the expression - you'd only need to define a set of special objects that build a logical representation of the expression by using the various hooks (__add__, __sub__, etc.). I don't see any big performance issues, BTW, since you're basically writing SQL in a slightly different way. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 28 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From ianb at colorstudy.com Fri Jan 28 19:53:07 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri Jan 28 19:54:16 2005 Subject: [DB-SIG] Table Oriented Programming In-Reply-To: <41FA8850.3060106@egenix.com> References: <41FA429F.28210.B46A9CD3@coal.murkworks.com> <41FA8850.3060106@egenix.com> Message-ID: <41FA8A13.7020102@colorstudy.com> M.-A. Lemburg wrote: > FWIW, SQLObject has some logic in place which mimics some of these > ideas. > > In general, I think you could work out a "compiler" which generates > SQL from the expression - you'd only need to define a set of special > objects that build a logical representation of the expression by > using the various hooks (__add__, __sub__, etc.). You can find code to do this in sqlobject.sqlbuilder: http://svn.colorstudy.com/trunk/SQLObject/sqlobject/sqlbuilder.py -- it doesn't have any (important) dependencies on the rest of SQLObject. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From scherrey at proteus-tech.com Fri Jan 28 21:20:43 2005 From: scherrey at proteus-tech.com (Benjamin Scherrey) Date: Fri Jan 28 21:20:51 2005 Subject: [DB-SIG] Eschew Obfuscation -- down with un-defined TLA's. In-Reply-To: <1DE30EB3ECE266409FDE6E7032F178C413988E@pcimail1s.peppermillcas.com> References: <1DE30EB3ECE266409FDE6E7032F178C413988E@pcimail1s.peppermillcas.com> Message-ID: <41FA9E9B.8020900@proteus-tech.com> Got your GR re:TLA's. FYI, an ORM == Object / Relational Mapping. HTH & TTYL! Ben Scherrey Vernon Cole wrote: >This is a general request for everyone to define all TLA's (Three Letter >Acronyms) on first use in any text. Our industry is crawling with TLA's >and XMLA's (eXtended Multi-Letter Acronyms). It seems in vogue to invent >new ones on a weekly basis. Those of us not keeping up with any >specific branch of development may not know the meaning of a TLA which >others of us use on a regular basis. > >What is an ORM? >---- >Vernon Cole >(in the business before WWW, GUI, OOP, WYSIWYG, HTML, i18n, SQL and 4GL >were invented.) > > From fumanchu at amor.org Fri Jan 28 21:35:54 2005 From: fumanchu at amor.org (Robert Brewer) Date: Fri Jan 28 21:39:08 2005 Subject: [DB-SIG] Table Oriented Programming Message-ID: <3A81C87DC164034AA4E2DDFE11D258E33982D7@exchange.hqamor.amorhq.net> Brad Clements wrote: > On 12 Feb 2004 at 13:10, Michael Hobbs wrote: > > > Consider this example code: > > departments = DepartmentTable > > employees = EmployeeTable > > johnsDepartment = (departments * employees) / > > (employees.dept == departments.dept) / > > (employees.name == 'John') % > > departments.name > > print johnsDepartment[0].name > > Has anyone tried expressing sql operations as relational > algebra statements in Python, > like the example above. > > I see lots of talk in google, but no code. > > Is this idea out of fashion already? I can see lots of > performance issues.. but it's still > interesting to me. Why is it interesting to you? What benefits do you see over: john = employees.findfirst(name='john') print john.Department.name Granted, the example was simple. Feel free to expand for more meaningful discussion. I'm interested in why you're interested. ;) Robert Brewer MIS Amor Ministries fumanchu@amor.org From bkc at murkworks.com Fri Jan 28 22:12:01 2005 From: bkc at murkworks.com (Brad Clements) Date: Fri Jan 28 21:51:14 2005 Subject: [DB-SIG] Table Oriented Programming In-Reply-To: <3A81C87DC164034AA4E2DDFE11D258E33982D7@exchange.hqamor.amorhq.net> Message-ID: <41FA5F5B.22939.B4EE36F3@coal.murkworks.com> On 28 Jan 2005 at 12:35, Robert Brewer wrote: > Why is it interesting to you? What benefits do you see over: > > john = employees.findfirst(name='john') > print john.Department.name > If only my life was so easy, vs: select first 5000 s.shipmentid, s.ownerid, s.deliverydate, s.shipmentstate, s.shipdate, s.etadatetime, s.billing, p.buyerspurchaseordernumber, p.salesordernumber, p.pieces, p.publictrackingnumber, p.trackingnumber, ro.party, ro.customerrole, ato.company as shiptocompany, ato.postal_code as shiptozip, ato.city as shiptocity, ato.state_province as shiptostate, afrom.company as shipfromcompany, afrom.postal_code as shipfromzip, afrom.city as shipfromcity, afrom.state_province as shipfromstate, ce.documentid, ce.entryid, ce.shipmentid as entryshipmentid, dc.pagecount, cs.carrierid, cs.servicetype from Shipment s inner join Package p on (s.shipmentid = p.shipmentid) inner join PartyShipRelation pss on (s.id = pss.shipment_id) inner join Roles ro on (pss.role_id = ro.id) left join address ato on (s.shipto_id = ato.id) left join address afrom on (s.shipfrom_id = afrom.id) left join CustomsEntryShipLink cesl on (s.shipmentid = cesl.shipmentid) left join CustomsEntries ce on (cesl.entryid = ce.entryid) left join DocumentCatalog dc on (ce.documentid = dc.documentid) inner join CarrierServices cs on (s.carrierserviceid = cs.serviceid) where ( (upper(s.shipmentid) not like 'SFIEXP%') and (s.shipdate >= ?) and (s.shipdate < ?) ) and ( (pss.party_id = ?) ) order by s.shipdate desc, s.shipmentid, p.trackingnumber And that's a simple query. I have 22 variables or so that can be used to control fieldlist, the where portion, and the join. also, the order of joins and where statements greatly impacts the performance of the query. This is why I said in my first post that performance would be an issue, because I'd need a way to specify which clauses are more important. The code that creates the above query (or those like it) is getting pretty long. I'm interested in more expressive mechanisms that still retain the ability to understand, vs this: def __generateDataSpecifier(self,dataSet,criteria=None, opts={}): """return a dataspecifier for use by rowFactory""" if dataSet == 'Default': dataSet = 'ShipmentList' csvmode = False if dataSet in ('ShipmentList','ShipmentListCSV', 'ShipmentCharges'): if dataSet == 'ShipmentCharges': shipcols = self.shipmentChargesColumns packageColumns = () queryShipmentCharges = True else: queryShipmentCharges = False shipcols = (self.dbtype == 'gvib' and self.shipmentListColumnsInterbase) or self.shipmentListColumns packageColumns = ('buyerspurchaseordernumber','salesordernumber', 'pieces', 'publictrackingnumber', 'trackingnumber') if dataSet == 'ShipmentListCSV' or opts.get('includeshipmentdetails'): csvmode = True shipcols += ((computedColumn('totalcharge',self.totalChargeSQL)), (computedColumn('totalpieces',self.totalPiecesSQL)), 'carrierbilltoacctnumber', 'shiptoresidential', 'shipfromtelephone', 'billtotelephone', 'shiptotelephone', 'shiptoattention', 'billtoattention', 'shipfromattention', ) if opts.get('includeactual'): shipcols += (computedColumn('totalactual',self.totalActualSQL),) if opts.get('includepublished'): shipcols += (computedColumn('totalpublished',self.totalPublishedSQL),) if opts.get('includeeta'): shipcols += ('etadatetime', ) dsShipment = dataSpecifier(tablename=self.tableName,prefix="s", columns=shipcols,criteria=criteria) addressToCritera = opts.get('shiptoCriteria') dsAddressTo = dataSpecifier(tablename='address', prefix='ato', columns=csvmode and self.csv_addressToColumns or self.addressToColumns, criteria=addressToCritera) dsAddressFrom = dataSpecifier(tablename='address', prefix='afrom', columns=csvmode and self.csv_addressFromColumns or self.addressFromColumns) if csvmode: dsAddressBillTo = dataSpecifier(tablename='address', prefix='abill', columns=self.csv_addressBillToColumns) else: dsAddressBillTo = None packageCriteria = opts.get('packageCriteria') dsPackage = dataSpecifier(tablename='Package',prefix="p", columns=packageColumns, criteria=packageCriteria) dsCarrierServices = dataSpecifier(tablename='CarrierServices',prefix='cs', columns=('carrierid','servicetype')) partyCriteria = opts.get('partyCriteria') dsPartyShipRelation = dataSpecifier(tablename='PartyShipRelation', prefix='pss', columns=(), criteria=partyCriteria, ) roleCriteria = opts.get('partyRelationCriteria') dsrole = dataSpecifier(tablename="Roles", prefix="ro", columns=('party','customerrole'), criteria=roleCriteria, ) hotlistCriteria = opts.get('hotlistCriteria') if hotlistCriteria: dsShipmentHotlistLink = dataSpecifier(tablename='ShipmentHotlistLink',prefix="shl", columns=(), criteria=hotlistCriteria) else: dsShipmentHotlistLink = None entryCriteria = opts.get('entryCriteria') if entryCriteria: dsCustomsEntryShipLink = dataSpecifier(tablename='CustomsEntryShipLink',prefix="cesl", columns=(), criteria=entryCriteria) else: dsCustomsEntryShipLink = dataSpecifier(tablename='CustomsEntryShipLink',prefix="cesl", columns=() ) dsCustomsEntry = dataSpecifier(tablename='CustomsEntries',prefix="ce", columns=('documentid', 'entryid', computedColumn('entryshipmentid','ce.shipmentid as entryshipmentid'), ) ) dsDocumentCatalog = dataSpecifier(tablename='DocumentCatalog',prefix="dc", columns=('pagecount',) ) if self.dbtype == 'gvib': if dsShipmentHotlistLink: tables = [ RelatedTable(dsShipmentHotlistLink, 'shipment_id', dsShipment, 'id', whichMany=1), RelatedTable(dsShipment,'shipmentid',dsPackage,'shipmentid'), ] elif entryCriteria: if queryShipmentCharges: tables = [ RelatedTable(dsCustomsEntryShipLink, 'shipmentid', dsShipment, 'shipmentid' ,whichMany=1), RelatedTable(dsShipment,'shipmentid',dsPackage,'shipmentid'), ] else: tables = [ RelatedTable(dsCustomsEntryShipLink, 'shipmentid', dsShipment, 'shipmentid' ,whichMany=1), RelatedTable(dsCustomsEntryShipLink, 'entryid', dsCustomsEntry, 'entryid', whichMany=1), RelatedTable(dsCustomsEntry, 'documentid', dsDocumentCatalog, 'documentid', whichMany=1), RelatedTable(dsShipment,'shipmentid',dsPackage,'shipmentid'), ] elif packageCriteria: tables = [ RelatedTable(dsPackage,'shipmentid', dsShipment,'shipmentid', whichMany=1), ] else: tables = [ RelatedTable(dsShipment,'shipmentid',dsPackage,'shipmentid'), ] if queryShipmentCharges: dsCharges = dataSpecifier(tablename='Charges',prefix="c", columns=self.chargeColumns) dsChargeCodes = dataSpecifier(tablename='chargecodes',prefix='cc', columns=('shortdescription', 'chargeclass')) tables.extend( [ RelatedTable(dsShipment, 'shipmentid', dsCharges, 'shipmentid'), RelatedTable(dsCharges,'chargecodeid',dsChargeCodes,'id') ] ) if partyCriteria or roleCriteria: tables.extend( [ RelatedTable(dsShipment, 'id', dsPartyShipRelation, 'shipment_id', whichMany=(not partyCriteria) and 1 or None), ] ) if roleCriteria: tables.append( RelatedTable(dsPartyShipRelation, 'role_id', dsrole, 'id', whichMany=(not partyCriteria) and 1 or None), ) if addressToCritera: tables.append( RelatedTable(dsShipment, 'shipto_id', dsAddressTo, 'id', whichMany=(not addressToCritera) and 1 or None), ) else: tables.extend( [ RelatedTable(dsShipment, 'id', dsPartyShipRelation, 'shipment_id', whichMany=(not partyCriteria) and 1 or None), RelatedTable(dsPartyShipRelation, 'role_id', dsrole, 'id', whichMany=(not partyCriteria) and 1 or None), RelatedTable(dsShipment, 'shipto_id', dsAddressTo, 'id', whichMany=(not addressToCritera) and 1 or None), RelatedTable(dsShipment, 'shipfrom_id', dsAddressFrom, 'id', whichMany=1), ] ) if dsAddressBillTo: tables.append( RelatedTable(dsShipment, 'billto_id', dsAddressBillTo, 'id', whichMany=1) ) if not queryShipmentCharges: if not entryCriteria: tables.extend( [ RelatedTable(dsShipment, 'shipmentid', dsCustomsEntryShipLink, 'shipmentid', whichMany=1), RelatedTable(dsCustomsEntryShipLink, 'entryid', dsCustomsEntry, 'entryid', whichMany=1), RelatedTable(dsCustomsEntry, 'documentid', dsDocumentCatalog, 'documentid', whichMany=1), ] ) tables.append( RelatedTable(dsShipment,'carrierserviceid',dsCarrierServices,'serviceid') ) ds = Join(*tables) else: raise RuntimeError("not supported") tables = [RelatedTable(dsShipment,'shipmentid',dsPackage,'shipmentid'), RelatedTable(dsShipment,'carrierservice_id',dsCarrierServices,'id')] if dsCustomsEntryShipLink: raise RuntimeError("can't handle customs entries link") ds = Join(*tables) else: raise ValueError("Unsupported dataSet (%s)" % dataSet) if queryShipmentCharges: ds.distinct = True else: ds.limit = SHIPMENT_LIST_LIMIT return ds Note that criteria look like this: requirements.append(NOTLIKE('shipmentid', 'SFIEXP')) or if packageCriteria: requirements.append(EQ('shipmentid',const='p.trackingnumber')) opts['packageCriteria'] = AND(*packageCriteria) -- Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax http://www.wecanstopspam.org/ AOL-IM: BKClements From ianb at colorstudy.com Fri Jan 28 22:11:43 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri Jan 28 22:12:48 2005 Subject: [DB-SIG] Table Oriented Programming In-Reply-To: <41FA5F5B.22939.B4EE36F3@coal.murkworks.com> References: <41FA5F5B.22939.B4EE36F3@coal.murkworks.com> Message-ID: <41FAAA8F.4070005@colorstudy.com> Brad Clements wrote: > On 28 Jan 2005 at 12:35, Robert Brewer wrote: > > >>Why is it interesting to you? What benefits do you see over: >> >>john = employees.findfirst(name='john') >>print john.Department.name >> > > > If only my life was so easy, vs: > > select > first 5000 > s.shipmentid, s.ownerid, s.deliverydate, s.shipmentstate, s.shipdate, s.etadatetime, > s.billing, p.buyerspurchaseordernumber, p.salesordernumber, p.pieces, > p.publictrackingnumber, p.trackingnumber, ro.party, ro.customerrole, ato.company as > shiptocompany, ato.postal_code as shiptozip, ato.city as shiptocity, ato.state_province > as shiptostate, afrom.company as shipfromcompany, afrom.postal_code as > shipfromzip, afrom.city as shipfromcity, afrom.state_province as shipfromstate, > ce.documentid, ce.entryid, ce.shipmentid as entryshipmentid, dc.pagecount, > cs.carrierid, cs.servicetype > from > Shipment s inner join Package p on (s.shipmentid = p.shipmentid) > inner join PartyShipRelation pss on (s.id = pss.shipment_id) > inner join Roles ro on (pss.role_id = ro.id) > left join address ato on (s.shipto_id = ato.id) > left join address afrom on (s.shipfrom_id = afrom.id) > left join CustomsEntryShipLink cesl on (s.shipmentid = cesl.shipmentid) > left join CustomsEntries ce on (cesl.entryid = ce.entryid) > left join DocumentCatalog dc on (ce.documentid = dc.documentid) > inner join CarrierServices cs on (s.carrierserviceid = cs.serviceid) Let's say this above part of the query something else entirely. > where > ( (upper(s.shipmentid) not like 'SFIEXP%') and (s.shipdate >= ?) and (s.shipdate > < ?) ) and > ( (pss.party_id = ?) ) > order by s.shipdate desc, s.shipmentid, p.trackingnumber With sqlbuilder you might do something like: s = sqlbuilder.Table('s') clause = ( (~sqlbuilder.func(s.shipmentid).startswith('SFIEXP')) & (s.shipdate >= min_shipment_date) & (s.shipdate <= max_shipment_date) & (pss.party_id = party_id)) # to get the SQL: sqlbuilder.sqlrepr(clause, db_name) It doesn't use bound parameters like ?, instead it does all the SQL generation itself including quotation. So it has to know what database you are using to actually create the SQL, like "mysql", "postgresql", etc. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From mal at egenix.com Fri Jan 28 22:15:45 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Jan 28 22:15:50 2005 Subject: [DB-SIG] Table Oriented Programming In-Reply-To: <41FA5F5B.22939.B4EE36F3@coal.murkworks.com> References: <41FA5F5B.22939.B4EE36F3@coal.murkworks.com> Message-ID: <41FAAB81.3070004@egenix.com> Brad Clements wrote: > On 28 Jan 2005 at 12:35, Robert Brewer wrote: > > >>Why is it interesting to you? What benefits do you see over: >> >>john = employees.findfirst(name='john') >>print john.Department.name >> > > > If only my life was so easy, vs: > > select > first 5000 > s.shipmentid, s.ownerid, s.deliverydate, s.shipmentstate, s.shipdate, s.etadatetime, > s.billing, p.buyerspurchaseordernumber, p.salesordernumber, p.pieces, > p.publictrackingnumber, p.trackingnumber, ro.party, ro.customerrole, ato.company as > shiptocompany, ato.postal_code as shiptozip, ato.city as shiptocity, ato.state_province > as shiptostate, afrom.company as shipfromcompany, afrom.postal_code as > shipfromzip, afrom.city as shipfromcity, afrom.state_province as shipfromstate, > ce.documentid, ce.entryid, ce.shipmentid as entryshipmentid, dc.pagecount, > cs.carrierid, cs.servicetype > from > Shipment s inner join Package p on (s.shipmentid = p.shipmentid) > inner join PartyShipRelation pss on (s.id = pss.shipment_id) > inner join Roles ro on (pss.role_id = ro.id) > left join address ato on (s.shipto_id = ato.id) > left join address afrom on (s.shipfrom_id = afrom.id) > left join CustomsEntryShipLink cesl on (s.shipmentid = cesl.shipmentid) > left join CustomsEntries ce on (cesl.entryid = ce.entryid) > left join DocumentCatalog dc on (ce.documentid = dc.documentid) > inner join CarrierServices cs on (s.carrierserviceid = cs.serviceid) > where > ( (upper(s.shipmentid) not like 'SFIEXP%') and (s.shipdate >= ?) and (s.shipdate > < ?) ) and > ( (pss.party_id = ?) ) > order by s.shipdate desc, s.shipmentid, p.trackingnumber > > And that's a simple query. > > I have 22 variables or so that can be used to control fieldlist, the where portion, and the > join. > > also, the order of joins and where statements greatly impacts the performance of the > query. This is why I said in my first post that performance would be an issue, because > I'd need a way to specify which clauses are more important. I don't think that writing the query using some algebra form will make things easier to comprehend. The problem with your query generator is not the complexity of the generated SQL, it's the many conditions and options that make things hard to read. > The code that creates the above query (or those like it) is getting pretty long. I'm > interested in more expressive mechanisms that still retain the ability to understand, vs > this: I'd try to break down this monster into functional chunks that work on a query object (adding or moving things around as necessary) rather than trying to do everything in one function. Mapping options to methods is also worth a try. > def __generateDataSpecifier(self,dataSet,criteria=None, opts={}): > """return a dataspecifier for use by rowFactory""" > if dataSet == 'Default': > dataSet = 'ShipmentList' > > csvmode = False > if dataSet in ('ShipmentList','ShipmentListCSV', 'ShipmentCharges'): > if dataSet == 'ShipmentCharges': > shipcols = self.shipmentChargesColumns > packageColumns = () > queryShipmentCharges = True > else: > queryShipmentCharges = False > shipcols = (self.dbtype == 'gvib' and self.shipmentListColumnsInterbase) or self.shipmentListColumns > packageColumns = ('buyerspurchaseordernumber','salesordernumber', > 'pieces', 'publictrackingnumber', 'trackingnumber') > > if dataSet == 'ShipmentListCSV' or opts.get('includeshipmentdetails'): > csvmode = True > shipcols += ((computedColumn('totalcharge',self.totalChargeSQL)), > (computedColumn('totalpieces',self.totalPiecesSQL)), > 'carrierbilltoacctnumber', > 'shiptoresidential', > 'shipfromtelephone', > 'billtotelephone', > 'shiptotelephone', > 'shiptoattention', > 'billtoattention', > 'shipfromattention', > ) > > > if opts.get('includeactual'): > shipcols += (computedColumn('totalactual',self.totalActualSQL),) > > > if opts.get('includepublished'): > shipcols += (computedColumn('totalpublished',self.totalPublishedSQL),) > > if opts.get('includeeta'): > shipcols += ('etadatetime', ) > > dsShipment = dataSpecifier(tablename=self.tableName,prefix="s", > columns=shipcols,criteria=criteria) > > addressToCritera = opts.get('shiptoCriteria') > > dsAddressTo = dataSpecifier(tablename='address', prefix='ato', > columns=csvmode and self.csv_addressToColumns or self.addressToColumns, > criteria=addressToCritera) > > dsAddressFrom = dataSpecifier(tablename='address', prefix='afrom', > columns=csvmode and self.csv_addressFromColumns or self.addressFromColumns) > > if csvmode: > dsAddressBillTo = dataSpecifier(tablename='address', prefix='abill', > columns=self.csv_addressBillToColumns) > else: > dsAddressBillTo = None > > packageCriteria = opts.get('packageCriteria') > dsPackage = dataSpecifier(tablename='Package',prefix="p", > columns=packageColumns, criteria=packageCriteria) > > dsCarrierServices = dataSpecifier(tablename='CarrierServices',prefix='cs', > columns=('carrierid','servicetype')) > > partyCriteria = opts.get('partyCriteria') > dsPartyShipRelation = dataSpecifier(tablename='PartyShipRelation', prefix='pss', > columns=(), > criteria=partyCriteria, > ) > > > roleCriteria = opts.get('partyRelationCriteria') > dsrole = dataSpecifier(tablename="Roles", prefix="ro", > columns=('party','customerrole'), > criteria=roleCriteria, > ) > > hotlistCriteria = opts.get('hotlistCriteria') > if hotlistCriteria: > dsShipmentHotlistLink = dataSpecifier(tablename='ShipmentHotlistLink',prefix="shl", > columns=(), criteria=hotlistCriteria) > else: > dsShipmentHotlistLink = None > > entryCriteria = opts.get('entryCriteria') > if entryCriteria: > dsCustomsEntryShipLink = dataSpecifier(tablename='CustomsEntryShipLink',prefix="cesl", > columns=(), criteria=entryCriteria) > else: > dsCustomsEntryShipLink = dataSpecifier(tablename='CustomsEntryShipLink',prefix="cesl", > columns=() ) > > > dsCustomsEntry = dataSpecifier(tablename='CustomsEntries',prefix="ce", > columns=('documentid', 'entryid', > computedColumn('entryshipmentid','ce.shipmentid as entryshipmentid'), > ) > ) > > dsDocumentCatalog = dataSpecifier(tablename='DocumentCatalog',prefix="dc", > columns=('pagecount',) ) > > if self.dbtype == 'gvib': > if dsShipmentHotlistLink: > tables = [ > RelatedTable(dsShipmentHotlistLink, 'shipment_id', dsShipment, 'id', whichMany=1), > RelatedTable(dsShipment,'shipmentid',dsPackage,'shipmentid'), > ] > elif entryCriteria: > if queryShipmentCharges: > tables = [ > RelatedTable(dsCustomsEntryShipLink, 'shipmentid', dsShipment, 'shipmentid' ,whichMany=1), > RelatedTable(dsShipment,'shipmentid',dsPackage,'shipmentid'), > ] > else: > tables = [ > RelatedTable(dsCustomsEntryShipLink, 'shipmentid', dsShipment, 'shipmentid' ,whichMany=1), > RelatedTable(dsCustomsEntryShipLink, 'entryid', dsCustomsEntry, 'entryid', whichMany=1), > RelatedTable(dsCustomsEntry, 'documentid', dsDocumentCatalog, 'documentid', whichMany=1), > RelatedTable(dsShipment,'shipmentid',dsPackage,'shipmentid'), > ] > > elif packageCriteria: > tables = [ > RelatedTable(dsPackage,'shipmentid', dsShipment,'shipmentid', whichMany=1), > ] > else: > tables = [ > RelatedTable(dsShipment,'shipmentid',dsPackage,'shipmentid'), > ] > > if queryShipmentCharges: > dsCharges = dataSpecifier(tablename='Charges',prefix="c", > columns=self.chargeColumns) > > dsChargeCodes = dataSpecifier(tablename='chargecodes',prefix='cc', > columns=('shortdescription', 'chargeclass')) > > tables.extend( > [ > RelatedTable(dsShipment, 'shipmentid', dsCharges, 'shipmentid'), > RelatedTable(dsCharges,'chargecodeid',dsChargeCodes,'id') > ] > ) > if partyCriteria or roleCriteria: > tables.extend( > [ > RelatedTable(dsShipment, 'id', dsPartyShipRelation, 'shipment_id', whichMany=(not partyCriteria) and 1 or None), > ] > ) > if roleCriteria: > tables.append( > RelatedTable(dsPartyShipRelation, 'role_id', dsrole, 'id', whichMany=(not partyCriteria) and 1 or None), > ) > > if addressToCritera: > tables.append( > RelatedTable(dsShipment, 'shipto_id', dsAddressTo, 'id', whichMany=(not addressToCritera) and 1 or None), > ) > > else: > tables.extend( > [ > RelatedTable(dsShipment, 'id', dsPartyShipRelation, 'shipment_id', whichMany=(not partyCriteria) and 1 or None), > RelatedTable(dsPartyShipRelation, 'role_id', dsrole, 'id', whichMany=(not partyCriteria) and 1 or None), > RelatedTable(dsShipment, 'shipto_id', dsAddressTo, 'id', whichMany=(not addressToCritera) and 1 or None), > RelatedTable(dsShipment, 'shipfrom_id', dsAddressFrom, 'id', whichMany=1), > ] > ) > if dsAddressBillTo: > tables.append( > RelatedTable(dsShipment, 'billto_id', dsAddressBillTo, 'id', whichMany=1) > ) > > if not queryShipmentCharges: > if not entryCriteria: > tables.extend( > [ > RelatedTable(dsShipment, 'shipmentid', dsCustomsEntryShipLink, 'shipmentid', whichMany=1), > RelatedTable(dsCustomsEntryShipLink, 'entryid', dsCustomsEntry, 'entryid', whichMany=1), > RelatedTable(dsCustomsEntry, 'documentid', dsDocumentCatalog, 'documentid', whichMany=1), > ] > ) > > tables.append( > RelatedTable(dsShipment,'carrierserviceid',dsCarrierServices,'serviceid') > ) > > ds = Join(*tables) > else: > raise RuntimeError("not supported") > tables = [RelatedTable(dsShipment,'shipmentid',dsPackage,'shipmentid'), > RelatedTable(dsShipment,'carrierservice_id',dsCarrierServices,'id')] > if dsCustomsEntryShipLink: > raise RuntimeError("can't handle customs entries link") > ds = Join(*tables) > else: > raise ValueError("Unsupported dataSet (%s)" % dataSet) > > if queryShipmentCharges: > ds.distinct = True > else: > ds.limit = SHIPMENT_LIST_LIMIT > return ds > > > > Note that criteria look like this: > > requirements.append(NOTLIKE('shipmentid', 'SFIEXP')) > > or > > if packageCriteria: > requirements.append(EQ('shipmentid',const='p.trackingnumber')) > opts['packageCriteria'] = AND(*packageCriteria) > > > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 28 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From fumanchu at amor.org Fri Jan 28 23:19:11 2005 From: fumanchu at amor.org (Robert Brewer) Date: Fri Jan 28 23:22:26 2005 Subject: [DB-SIG] Table Oriented Programming Message-ID: <3A81C87DC164034AA4E2DDFE11D258E33982DB@exchange.hqamor.amorhq.net> Brad Clements wrote: > The code that creates the above query (or those like it) is > getting pretty long. I'm > interested in more expressive mechanisms that still retain > the ability to understand... You have a lot of logic in there that I don't think is reducible; the structure, however, is made more manageable by refactoring. I'd *start* by passing an object instead of a string as the dataSet parameter, and building attributes and methods into each object rather than using the large if/else statements. That is, rather than if dataSet == 'ShipmentCharges': shipcols = self.shipmentChargesColumns packageColumns = () queryShipmentCharges = True you could write: q = Query() dataSet.populate(q, opts) ...and stuff the dataset-specific logic into the ShipmentCharges class. Find the common code to all classes and stuff that into a superclass from which the other classes inherit. And so on ;) Robert Brewer MIS Amor Ministries fumanchu@amor.org From fog at initd.org Sat Jan 29 13:21:14 2005 From: fog at initd.org (Federico Di Gregorio) Date: Sat Jan 29 13:19:04 2005 Subject: [DB-SIG] Table Oriented Programming In-Reply-To: <41FAAA8F.4070005@colorstudy.com> References: <41FA5F5B.22939.B4EE36F3@coal.murkworks.com> <41FAAA8F.4070005@colorstudy.com> Message-ID: <1107001274.7265.4.camel@iris> Il giorno ven, 28-01-2005 alle 15:11 -0600, Ian Bicking ha scritto: > It doesn't use bound parameters like ?, instead it does all the SQL > generation itself including quotation. So it has to know what database > you are using to actually create the SQL, like "mysql", "postgresql", etc. This is.. er.. "horrible". PostgreSQL supports a *lot* of different types and the only way to support them all is to let the driver do the quoting (possibly with the user adding its own quoting rules, see PEP-246 adapt() in psycopg 2). I suppose your driver is not quoting PostGis or or even simple geometrical types mapped to Python instances, right? federico From ods at strana.ru Sat Jan 29 15:04:38 2005 From: ods at strana.ru (Denis S. Otkidach) Date: Sat Jan 29 15:05:03 2005 Subject: [DB-SIG] Table Oriented Programming In-Reply-To: <1107001274.7265.4.camel@iris> References: <41FA5F5B.22939.B4EE36F3@coal.murkworks.com> <41FAAA8F.4070005@colorstudy.com> <1107001274.7265.4.camel@iris> Message-ID: <20050129170438.2b479e5a.ods@strana.ru> On Sat, 29 Jan 2005 13:21:14 +0100 Federico Di Gregorio wrote: > Il giorno ven, 28-01-2005 alle 15:11 -0600, Ian Bicking ha scritto: > > It doesn't use bound parameters like ?, instead it does all the SQL > > generation itself including quotation. So it has to know what > > database you are using to actually create the SQL, like "mysql", > > "postgresql", etc. > > This is.. er.. "horrible". PostgreSQL supports a *lot* of different > types and the only way to support them all is to let the driver do the > quoting (possibly with the user adding its own quoting rules, see > PEP-246 adapt() in psycopg 2). I suppose your driver is not quoting > PostGis or or even simple geometrical types mapped to Python > instances, right? AFAIK, Oleg Broytmann is working on changing SQLObject to use bound parameters. But there is a lot of work and probably compatibility issues. Anyway, builder must know database adapter to use appropriate paramstyle. -- Denis S. Otkidach http://www.python.ru/ [ru] From farcepest at gmail.com Sat Jan 29 16:50:52 2005 From: farcepest at gmail.com (Andy Dustman) Date: Sat Jan 29 16:51:40 2005 Subject: [DB-SIG] Table Oriented Programming In-Reply-To: <20050129170438.2b479e5a.ods@strana.ru> References: <41FA5F5B.22939.B4EE36F3@coal.murkworks.com> <41FAAA8F.4070005@colorstudy.com> <1107001274.7265.4.camel@iris> <20050129170438.2b479e5a.ods@strana.ru> Message-ID: <9826f38005012907504d50d5cb@mail.gmail.com> On Sat, 29 Jan 2005 17:04:38 +0300, Denis S. Otkidach wrote: > On Sat, 29 Jan 2005 13:21:14 +0100 > Federico Di Gregorio wrote: > > > Il giorno ven, 28-01-2005 alle 15:11 -0600, Ian Bicking ha scritto: > > > It doesn't use bound parameters like ?, instead it does all the SQL > > > generation itself including quotation. So it has to know what > > > database you are using to actually create the SQL, like "mysql", > > > "postgresql", etc. > > > > This is.. er.. "horrible". PostgreSQL supports a *lot* of different > > types and the only way to support them all is to let the driver do the > > quoting (possibly with the user adding its own quoting rules, see > > PEP-246 adapt() in psycopg 2). I suppose your driver is not quoting > > PostGis or or even simple geometrical types mapped to Python > > instances, right? > > AFAIK, Oleg Broytmann is working on changing SQLObject to use bound > parameters. But there is a lot of work and probably compatibility > issues. Anyway, builder must know database adapter to use appropriate > paramstyle. All it has to do is look at modules.paramstyle to figure this out. I did a little work along these lines at one point, and if you construct your query using %s as the placeholder (which is likely what happens already), you can put the right placeholders in there fairly easily. With module.paramstyle = "qmark", you can do something like this: query2 = query % ('?',) * len(params) -- Computer interfaces should never be made of meat. Using GMail? Setting Reply-to address to <> disables this annoying feature. From ianb at colorstudy.com Sat Jan 29 21:13:47 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sat Jan 29 21:13:43 2005 Subject: [DB-SIG] Table Oriented Programming In-Reply-To: <1107001274.7265.4.camel@iris> References: <41FA5F5B.22939.B4EE36F3@coal.murkworks.com> <41FAAA8F.4070005@colorstudy.com> <1107001274.7265.4.camel@iris> Message-ID: <41FBEE7B.5080802@colorstudy.com> Federico Di Gregorio wrote: > Il giorno ven, 28-01-2005 alle 15:11 -0600, Ian Bicking ha scritto: > >>It doesn't use bound parameters like ?, instead it does all the SQL >>generation itself including quotation. So it has to know what database >>you are using to actually create the SQL, like "mysql", "postgresql", etc. > > > This is.. er.. "horrible". PostgreSQL supports a *lot* of different > types and the only way to support them all is to let the driver do the > quoting (possibly with the user adding its own quoting rules, see > PEP-246 adapt() in psycopg 2). I suppose your driver is not quoting > PostGis or or even simple geometrical types mapped to Python instances, > right? No, but it could, as it has hooks for new types of data. I'm not saying its the best technique, but then psycopg's technique isn't all that great either (falling back on __repr__ is particular bad, IMHO). At some level sqlbuilder does its own SQL generation, that's the whole point of it. The way it does it is fairly simple, there's a registry of converters, and also a __sqlrepr__ method that things like AND and expressions define which generates compound statements. It would be good to use parameters, but it is a lot harder to keep track of those, which is why it hasn't happened yet. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From fog at initd.org Sat Jan 29 22:51:54 2005 From: fog at initd.org (Federico Di Gregorio) Date: Sat Jan 29 22:49:38 2005 Subject: [DB-SIG] Table Oriented Programming In-Reply-To: <41FBEE7B.5080802@colorstudy.com> References: <41FA5F5B.22939.B4EE36F3@coal.murkworks.com> <41FAAA8F.4070005@colorstudy.com> <1107001274.7265.4.camel@iris> <41FBEE7B.5080802@colorstudy.com> Message-ID: <1107035514.7265.28.camel@iris> Il giorno sab, 29-01-2005 alle 14:13 -0600, Ian Bicking ha scritto: > Federico Di Gregorio wrote: > > Il giorno ven, 28-01-2005 alle 15:11 -0600, Ian Bicking ha scritto: > > > >>It doesn't use bound parameters like ?, instead it does all the SQL > >>generation itself including quotation. So it has to know what database > >>you are using to actually create the SQL, like "mysql", "postgresql", etc. > > > > > > This is.. er.. "horrible". PostgreSQL supports a *lot* of different > > types and the only way to support them all is to let the driver do the > > quoting (possibly with the user adding its own quoting rules, see > > PEP-246 adapt() in psycopg 2). I suppose your driver is not quoting > > PostGis or or even simple geometrical types mapped to Python instances, > > right? > > No, but it could, as it has hooks for new types of data. I'm not saying > its the best technique, but then psycopg's technique isn't all that > great either (falling back on __repr__ is particular bad, IMHO). That was the old psycopg. psycopg 2 will fail on unknown types. Anyway, each backend has its own peculiarities and the right way to cope with them is the driver, not the ORM. Hey, I *love* SQLObject (apart from the primary key support); I am just saying that quoting in the ORM is much more error prone that leaving that to the driver. federico From peter at monicol.co.uk Sun Jan 30 12:50:12 2005 From: peter at monicol.co.uk (Peter Mott) Date: Sun Jan 30 12:50:16 2005 Subject: [DB-SIG] MySQL operational error Message-ID: Hi, I got this error in a Python script X which until yesterday ran without problem: OperationalError: (2005, "Unknown MySQL Server Host 'mysql4db.gradwell.net' (2)") The script, X, calls a script Y. The script X has already successfully connected to the database using the above host. Script Y contains a helper function foo() which does a look up in the database and to do this creates a connection. It was Y that incurred the error. The error was fixed by adding a further optional argument to foo() that holds an open database connection. My problem is that the erorr message is clearly wrong. If there is some way that someone knows whereby Python, MyDQL and MySQLdb could manufacture this erroneous error message I';d be glad to hear of it for I cetainly have no idea where to look. Otherwise it is an error with my ISP's setup (Gradwell.com) which is complex. This is using MySQLdb on 'FreeBSD 4.8-STABLE (RED03072003)' Python is Python 2.3.4 (#2, Nov 14 2004, 18:06:48) [GCC 2.95.4 20020320 [FreeBSD]] on freebsd4' (provided the Python that Apache is using is the samePython as my shell uses which I believe that it is) Thanks for any suggestions, Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/db-sig/attachments/20050130/b31d31bf/attachment.htm From farcepest at gmail.com Sun Jan 30 20:03:16 2005 From: farcepest at gmail.com (Andy Dustman) Date: Sun Jan 30 20:03:20 2005 Subject: [DB-SIG] MySQL operational error In-Reply-To: References: Message-ID: <9826f38005013011037aa0a5f0@mail.gmail.com> On Sun, 30 Jan 2005 11:50:12 -0000, Peter Mott wrote: > OperationalError: (2005, "Unknown MySQL Server Host 'mysql4db.gradwell.net' > (2)") This is not really the right place for this question. It's the sort of thing you should use the mysql-python help forum for. https://sourceforge.net/forum/forum.php?forum_id=70461 However, this is pretty obviously a DNS resolution error, though your host resolves from here. Don't forget to mention what version of MySQLdb you have. -- Computer interfaces should never be made of meat. Using GMail? Setting Reply-to address to <> disables this annoying feature. From bkc at murkworks.com Mon Jan 31 22:56:41 2005 From: bkc at murkworks.com (Brad Clements) Date: Mon Jan 31 22:35:28 2005 Subject: [DB-SIG] Table Oriented Programming In-Reply-To: <33741.162.136.193.1.1107202645.squirrel@162.136.193.1> References: <41FA429F.28210.B46A9CD3@coal.murkworks.com> Message-ID: <41FE5E43.25060.C48A1D54@coal.murkworks.com> On 31 Jan 2005 at 14:17, Michael Hobbs wrote: > I had worked on a prototype for this a long time ago, before I got bored > with the problem. ;-) I believe that it is completely undocumented, in a > very raw state, and problably written in bad newbie Python, but if no one > objects, I can post it as an attachment to this mail-list. > > Let me know, > - Mike I would like to see it! -- Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax http://www.wecanstopspam.org/ AOL-IM: BKClements From michael at hobbshouse.org Mon Jan 31 21:17:25 2005 From: michael at hobbshouse.org (Michael Hobbs) Date: Mon Jan 31 23:12:57 2005 Subject: [DB-SIG] Table Oriented Programming In-Reply-To: <41FA429F.28210.B46A9CD3@coal.murkworks.com> References: <000a01c3f19b$dc5c7ea0$0a0aa8c0@mikesmonster> <41FA429F.28210.B46A9CD3@coal.murkworks.com> Message-ID: <33741.162.136.193.1.1107202645.squirrel@162.136.193.1> Brad Clements said: > On 12 Feb 2004 at 13:10, Michael Hobbs wrote: > >> Consider this example code: >> departments = DepartmentTable >> employees = EmployeeTable >> johnsDepartment = (departments * employees) / >> (employees.dept == departments.dept) / >> (employees.name == 'John') % >> departments.name >> print johnsDepartment[0].name > > Has anyone tried expressing sql operations as relational algebra > statements in Python, > like the example above. I had worked on a prototype for this a long time ago, before I got bored with the problem. ;-) I believe that it is completely undocumented, in a very raw state, and problably written in bad newbie Python, but if no one objects, I can post it as an attachment to this mail-list. I took a look at the sqlbuilder code from SQLObject, mentioned in a previous post, but that is still rather SQL-like and not very Relational Algebra-like. (For example, it uses words such as "select" and "where" instead of "project" and "select".) Let me know, - Mike