From ramrom@earthling.net Sat Feb 1 19:53:13 2003 From: ramrom@earthling.net (Bob Gailer) Date: Sat, 01 Feb 2003 12:53:13 -0700 Subject: [DB-SIG] RDBMS In-Reply-To: <5.1.0.14.0.20030201002642.02ac0bf8@www.thinkware.se> References: <000101c2c94a$bb062460$108b09ca@zohar> Message-ID: <5.2.0.9.0.20030201121415.02f2e278@66.28.54.253> --=======3A6E747B======= Content-Type: text/plain; x-avg-checked=avg-ok-72668EC; charset=us-ascii; format=flowed Content-Transfer-Encoding: 8bit >At 15:53 2003-01-31 +0530, zohar wrote: >>(3) Has a userbase that is connected by mailing list so you cannot expect >>support as of Oracle but you can get a solution. Oracle support is not always the best. I've worked with a client who uses Oracle; I've had to deal with Oracle's MetaLink help service; it is often a long and arduous task to find the right subset of specialists, then to find someone who understands the problem, then to get a solution. Often it turned out that the problem was an error or unclarity in the documentation, or software that did not issue error messages, or error messages that are vague. Instead of fixing things at the source Oracle just writes an endless number of documents trying to explain the problem. Trying to search these many thousand documents is a huge hassle. >>database application in python as by conventional way if I go making >>160+ forms, 250+ reports and graphs would take about 4-5 months. I have a project that's intended to make some of this work easier. Parts of it are in the concept stage, some pieces a partially fleshed out, and some are already established in a simple form in an application already running. One goal of the project is to move the structuring of data and relationships out of the hands of a DBA and into the hands of someone who knows the model but not necessarily the underlying DBMS. Another benefit is to make form design almost obsolete, using certain preconceived forms do the work of many custom forms. An adjunct is a series of reporting and graphing tools that allow one to create report or graph templates, then fill them with a wide variety of data. Unknowns at this point are how performance will be when compared with the traditional RDBMS apporach. If you'd like to evaluate how this might help you, let's talk. Bob Gailer mailto:ramrom@earthling.net 303 442 2625 --=======3A6E747B======= Content-Type: text/plain; charset=us-ascii; x-avg=cert; x-avg-checked=avg-ok-72668EC Content-Disposition: inline --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.445 / Virus Database: 250 - Release Date: 1/21/2003 --=======3A6E747B=======-- From zohr@eth.net Sat Feb 1 09:47:15 2003 From: zohr@eth.net (zohar) Date: Sat, 1 Feb 2003 15:17:15 +0530 Subject: [DB-SIG] ODBMS Message-ID: <000001c2ca13$213f2150$388b09ca@zohar> I am newbie in DBMS concepts so please forgive me if I am wrong. I saw a web page that says "Object Database Management Systems (ODBMSs) are designed to work well with object programming languages" Do anyone know which ODBMS works with python and it offers evaluation version so that I could learn about it before using it. ("`-/")__..-'"``-._ . . `; -._ )-;-,_`) (v_,)' _ ) / (` ( \ ``-' _.-/) . /) (,/ ,(/.' ((,.-' ((,/ From magnus@thinkware.se Sun Feb 2 00:54:05 2003 From: magnus@thinkware.se (Magnus Lycka) Date: Sun, 02 Feb 2003 01:54:05 +0100 Subject: [DB-SIG] RDBMS In-Reply-To: <5.2.0.9.0.20030201121415.02f2e278@66.28.54.253> References: <5.1.0.14.0.20030201002642.02ac0bf8@www.thinkware.se> <000101c2c94a$bb062460$108b09ca@zohar> Message-ID: <5.1.0.14.0.20030202015120.02b10650@www.thinkware.se> At 12:53 2003-02-01 -0700, Bob Gailer wrote: >Oracle support is not always the best. Perhaps not always...but my experience of telephone support from Oracle is very positive. I've rarely used it, but they've been very helpful when I needed them. Oracle training was good as well. This was in Sweden several years ago though. YMMV. -- Magnus Lycka, Thinkware AB Alvans vag 99, SE-907 50 UMEA, SWEDEN phone: int+46 70 582 80 65, fax: int+46 70 612 80 65 http://www.thinkware.se/ mailto:magnus@thinkware.se From pobrien@orbtech.com Sun Feb 2 13:57:01 2003 From: pobrien@orbtech.com (Patrick K. O'Brien) Date: Sun, 2 Feb 2003 07:57:01 -0600 Subject: [DB-SIG] ODBMS In-Reply-To: <000001c2ca13$213f2150$388b09ca@zohar> References: <000001c2ca13$213f2150$388b09ca@zohar> Message-ID: <200302020757.01202.pobrien@orbtech.com> On Saturday 01 February 2003 03:47 am, zohar wrote: > I am newbie in DBMS concepts so please forgive me if I am wrong. > > I saw a web page that says "Object Database Management Systems (ODBMSs) > are designed to work well with object programming languages" > Do anyone know which ODBMS works with python and it offers evaluation > version so that I could learn about it before using it. There are really only two ODBMS-like choices (that I know of) for persisting Python objects without translating into a relational database: ZODB and PyPerSyst. Both are Open Source and free of charge. ZODB: http://www.zope.org/Products/StandaloneZODB PyPerSyst: http://sourceforge.net/projects/pypersyst/ -- Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." ----------------------------------------------- From jacobs@penguin.theopalgroup.com Sun Feb 2 15:51:14 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Sun, 2 Feb 2003 10:51:14 -0500 (EST) Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030202142023.GI912@localhost> Message-ID: On Sun, 2 Feb 2003, Luke Kenneth Casson Leighton wrote: > has anyone considered doing database helper functions based > off of the DB 2.0 api? Yes; though the conversation is best had on the DB-SIG mailing list. > the question is inspired by the amount of work that is > needed to create a useable suite of database queries in > the shortest period of time - something that is important > when both developing databases and also when doing lots > of database applications one after the other. I've found that rapid initial development is far less important than ease of validation, maintenance, and portability. My belief is based on writing and maintaining Python applications that interface to, at times >8 heterogeneous database backends, each with many hundreds of tables and distinct OTP and OLAP queries. > also for some of the background please see the advogato article > http://advogato.org/article/600.html particularly the link to > jmg's code which contains some very simple to use but obscure > coding methods to wrap SQL databases in objects. We (my company) and many others have their own SQL generation and OR mapping implementations that are considerably more advanced. Some (like mine) are in the process of transitioning to an open source development model, while some are already freely available (see http://colorstudy.com/software/SQLObject for one released just last week). So maybe it is time for all of us to get together and put the Open Source "bazaar" to the test and see what comes out. > the reason i ask is because the DB 2.0 api is great in > that it makes things easy-ish to add in different databases > (e.g. MySQL, MS-SQL, PostgreSQL) if you can deal with > the syntactical differences, but it doesn't help you > do things like: > > - make a query that gets you some results > > - modify one of the results > > - write the modified results back. > > i would _expect_ that the cursor class would have a function > called save() which performed the last of these operations. I would not (for many reasons what I will be happy to elaborate on later). Primarily, DB-API is intended to supply low level access to database connections, transactions, simple result sets, SQL pass-through, and some data type mapping capabilities. The functionality you desire is complex enough that it deserves its own layer/tier that sits on top of DB-API modules. > i was therefore wondering about two things: > > - what do people on the python-dev list think of this concept? The idea is great, though there is a good deal of design that needs to be done. I say this from experience, having built such a system. > - is it something appropriate to be included in the > python libs, given that the python db API at present > is more of a convention than something that is > enforced as part of the python library suite. Absolutely not. Even after such a scheme has been battle-tested for a few years and has a large and active user base, it will likely be very large and have a radically different development cycle than the core interpreter. However, if you care that much about formal Python legitimacy, then you can work with DB-SIG to write a new standard for the features you desire. Anyhow, we should move the rest of this discussion to the DB-SIG mailing list (db-sig@python.org). Thanks, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From ianb@colorstudy.com Sun Feb 2 20:30:37 2003 From: ianb@colorstudy.com (Ian Bicking) Date: 02 Feb 2003 14:30:37 -0600 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: References: Message-ID: <1044217837.11685.17.camel@lothlorien> On Sun, 2003-02-02 at 09:51, Kevin Jacobs wrote: > > also for some of the background please see the advogato article > > http://advogato.org/article/600.html particularly the link to > > jmg's code which contains some very simple to use but obscure > > coding methods to wrap SQL databases in objects. > > We (my company) and many others have their own SQL generation and OR mapping > implementations that are considerably more advanced. Some (like mine) are > in the process of transitioning to an open source development model, while > some are already freely available (see http://colorstudy.com/software/SQLObject > for one released just last week). So maybe it is time for all of us to get > together and put the Open Source "bazaar" to the test and see what comes out. There is a list of a number of such interfaces here: http://www.python.org/cgi-bin/moinmoin/HigherLevelDatabaseProgramming If there's any missing I hope someone will extend them. Currently listed are: DbObj db_row ll.sql PgResultSet MiddleKit Object Relational Membrame PyDo SQLBuilder SQLDict SQLObject In various forms these address exactly what Luke was proposing. -- Ian Bicking Colorstudy Web Development ianb@colorstudy.com http://www.colorstudy.com PGP: gpg --keyserver pgp.mit.edu --recv-keys 0x9B9E28B7 4869 N Talman Ave, Chicago, IL 60625 / (773) 275-7241 From dario@ita.chalmers.se Mon Feb 3 08:27:14 2003 From: dario@ita.chalmers.se (=?iso-8859-1?Q?Dario_Lopez-K=E4sten?=) Date: Mon, 3 Feb 2003 09:27:14 +0100 Subject: [DB-SIG] RDBMS References: <000101c2c94a$bb062460$108b09ca@zohar> <5.2.0.9.0.20030201121415.02f2e278@66.28.54.253> Message-ID: <008901c2cb5e$0e4a2620$2235b5d4@WALTER> ----- Original Message ----- From: "Bob Gailer" > One goal of the project is to move the structuring of data and > relationships out of the hands of a DBA and into the hands of someone who > knows the model but not necessarily the underlying DBMS. hm... being a bit of both sides (though mostly on the non-dba side), I have to say that this is not necesarily a good thing. On the contrary, it can be quite disastrous for performance and amount of load on the database. The best thing in my experience is to either make sure that the designers thoroughly (sp?) understand about the implications on the DBA side of their design decisions, or to have co-operation from both sides where the desginer experesses his/her wishes and the DBA suggests ways of implementing that in ways that still will make the database behave properly. evebn better if you could have a combination of both. Just my 2 öre worth. Cheers, /dario From lkcl@samba-tng.org Mon Feb 3 12:21:32 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Mon, 3 Feb 2003 12:21:32 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <1044217837.11685.17.camel@lothlorien> References: <1044217837.11685.17.camel@lothlorien> Message-ID: <20030203122132.GO912@localhost> On Sun, Feb 02, 2003 at 02:30:37PM -0600, Ian Bicking wrote: > On Sun, 2003-02-02 at 09:51, Kevin Jacobs wrote: > > > also for some of the background please see the advogato article > > > http://advogato.org/article/600.html particularly the link to > > > jmg's code which contains some very simple to use but obscure > > > coding methods to wrap SQL databases in objects. > > > > We (my company) and many others have their own SQL generation and OR mapping > > implementations that are considerably more advanced. Some (like mine) are > > in the process of transitioning to an open source development model, while > > some are already freely available (see http://colorstudy.com/software/SQLObject > > for one released just last week). So maybe it is time for all of us to get > > together and put the Open Source "bazaar" to the test and see what comes out. > > There is a list of a number of such interfaces here: > http://www.python.org/cgi-bin/moinmoin/HigherLevelDatabaseProgramming > > If there's any missing I hope someone will extend them. Currently > listed are: > > DbObj > db_row > ll.sql > PgResultSet > MiddleKit > Object Relational Membrame > PyDo > SQLBuilder > SQLDict > SQLObject > > In various forms these address exactly what Luke was proposing. GREAT! example, the next application is a replacement for an extremely poorly written php app. it contains classes (base class a db class that does the db connection) the classes basically take a unique primary key, do a SELECT, record the row of info in the class instance as variables, then obviously it's possible to modify any of the variables, then there's a save() method. _all_ of this code has been hand-crafted! madness! take the SQL file, auto-generate the darn code, for pity's sake. i've been working some way towards this "ideal" goal. see http://sf.net/projects/pysqldb. more later, have a real-world application / customer to keep happy :) l. From lkcl@samba-tng.org Mon Feb 3 12:22:01 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Mon, 3 Feb 2003 12:22:01 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: References: <20030202142023.GI912@localhost> Message-ID: <20030203122201.GP912@localhost> On Sun, Feb 02, 2003 at 10:51:14AM -0500, Kevin Jacobs wrote: > On Sun, 2 Feb 2003, Luke Kenneth Casson Leighton wrote: > > has anyone considered doing database helper functions based > > off of the DB 2.0 api? > > Yes; though the conversation is best had on the DB-SIG mailing list. okay thx! will post there in future. From lkcl@samba-tng.org Mon Feb 3 13:43:27 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Mon, 3 Feb 2003 13:43:27 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: References: <20030202142023.GI912@localhost> Message-ID: <20030203134327.GU912@localhost> On Sun, Feb 02, 2003 at 10:51:14AM -0500, Kevin Jacobs wrote: > > do things like: > > > > - make a query that gets you some results > > > > - modify one of the results > > > > - write the modified results back. > > > > i would _expect_ that the cursor class would have a function > > called save() which performed the last of these operations. > > I would not (for many reasons what I will be happy to elaborate on later). > Primarily, DB-API is intended to supply low level access to database > connections, transactions, simple result sets, SQL pass-through, and some > data type mapping capabilities. The functionality you desire is complex > enough that it deserves its own layer/tier that sits on top of DB-API > modules. i stated the most simple of requirements in order to save time. the more complex requirements that i have are more like a substitute for SQL "VIEWs". proper views, not just read-only VIEWs like some SQL database have. one of the input forms has now got so complex that the update function is 160 lines of code and the add function is 100. and that's simply turning HTML POST cgi form dictionaries / lists into dictionaries of fields that are then passed to one-line function calls (product_create, product_update, product_codes_create, prices_update) that do the update, delete and create SQL statements. the tables are: - products - product_codes (per-product and per-supplier ordering codes) - prices (on a per-supplier and per-product basis) - product_groups (bundling of several products / special offers) [i am also intending to split out product sizes / quantities into a separate table, it will reduce the number of entries in the products table and make product reconciliation from suppliers a lot easier.] so whilst simple get, modify and save functionality would be nice, it's only the start. at present all my HTML forms have value="table_name_row_name" which is a clunky convention by which i can enter data. in this way i can have products_id, products_name, product_codes_supplier_code, product_codes_id as HTML form entries; subsequently in a POST i can identify what rows in the separate tables need to be updated, removed or added. having 260 lines of code to do this job is rather ridiculous, especially as it's not the only part of the application where i need to do this kind of multi-table entry. i'm certain that i'm not the only developer who has these kinds of JOINed table entry forms. l. From jacobs@penguin.theopalgroup.com Mon Feb 3 14:48:54 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Mon, 3 Feb 2003 09:48:54 -0500 (EST) Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030203134327.GU912@localhost> Message-ID: On Mon, 3 Feb 2003, Luke Kenneth Casson Leighton wrote: > the more complex requirements that i have are more like a substitute > for SQL "VIEWs". proper views, not just read-only VIEWs like some > SQL database have. Logical read/write views don't really solve your problem either. The heart of your situation involves (roughly) these processing steps: 1) Extract and unencode variables from the CGI enviornment 2) Decode the variable names and map them to schema elements 3) Validate the variable values against the schema and business logic 4) Group variables into transactional batches and sub-batches based on the physical schema 5) Render and execute SQL to perform updates Most of the above can be done fairly generically in Python, with minimal code needed for each particular form you are processing. However, there is no short-cut magic bullet "save" routine or "View" object that can do all of the above. I recommend focusing on implementing concise and powerful abstractions for each step seperately. > one of the input forms has now got so complex that the update > function is 160 lines of code and the add function is 100. > and that's simply turning HTML POST cgi form dictionaries / lists > into dictionaries Then you need to write smarter code! Post some of that code and some of the smarties here are sure to have suggestions on how to implement better abstractions. In the mean time, I'm working hard to release more of my code as open source. It provides many features that would likely help you and others. Regards, -Kevin Jacobs -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From lkcl@samba-tng.org Mon Feb 3 16:17:32 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Mon, 3 Feb 2003 16:17:32 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: References: <20030203134327.GU912@localhost> Message-ID: <20030203161732.GX912@localhost> On Mon, Feb 03, 2003 at 09:48:54AM -0500, Kevin Jacobs wrote: > On Mon, 3 Feb 2003, Luke Kenneth Casson Leighton wrote: > > the more complex requirements that i have are more like a substitute > > for SQL "VIEWs". proper views, not just read-only VIEWs like some > > SQL database have. > > Logical read/write views don't really solve your problem either. you are right. if you assume that certain steps need to be performed manually, it covers a good proportion of the job, but definitely not enough. my goal ultimately is to be able to write a file, maybe in XML or other suitable text file format, that specifies all the reports i want, that they are just tabular display of records, generation of an input form, or generation of create / delete input form. and for there to be like a _single_ cgi program that can take such a specification file and just... do everything. clearly, the validation of some variable values as you specify in 3) below cannot be entirely done in an automated manner: some things are going to be too complex to specify and you'd just have to put a function call out to do the validation. ... unless you have one of those really neato fancy SQL databases that can do CONSTRAINTS etc and can specify logic tests on the input data ;) > The heart > of your situation involves (roughly) these processing steps: for an update / data input process, yes. the process is slightly different and much simpler for report generation. > 1) Extract and unencode variables from the CGI enviornment yep. > 2) Decode the variable names and map them to schema elements > 3) Validate the variable values against the schema and business logic > 4) Group variables into transactional batches and sub-batches > based on the physical schema > 5) Render and execute SQL to perform updates what do you define as schema elements? i think i can work out what you mean by schema by context. against business logic, yep. i am marginally lost but not entirely lost in terminology on 2, 3 and 4. OKAY, i think i know what you mean in 4). in the code i have written, i have grouped variables into dictionaries, where the field name is the key and the value is the i also have a second input on the "update" functions from which the WHERE clause of an UPDATE statement is generated; the first argument is the dictionary of fields from which the SET clause is generated. assuming that code to do all the above is auto-generated from either a SQL file (with CREATE and ALTER TABLE statements including FOREIGN KEY statements from which JOINs can be auto-generated) or from a separate IDL file, then code can be generated for the following: 1) the types of variable fields from the database definition can auto-generate type-checking and also data entry forms and their corresponding input CGIs (dates, ints, floats, char, all can be got from the SQL CREATE statement) 2) variable names for the viewing, generation of data entry forms and their corresponding input CGIs can all be done too. however, this is where i believe the SQL file alone is insufficient to auto-generate such code: you don't necessarily want _all_ of the fields to be presented in a single input form; you don't necessarily want all of the fields to be viewed in a report. 4) grouping variables into transactional batches very easy to auto-generate, if you have the IDL file specifying things. what i have done at present, which is where i am getting in a muddle, is that the grouping is done by looking for the name of the table as the first part of the CGI POST arguments. only those CGI arguments with the first part of the table and then the field name as the second part of any given CGI argument is used to generate the transactional batch input for stage 5). 5) render and execute SQL statements. here, again, this is very easy: the logic is that you receive a dictionary of field names and their values and the SET part of the SQL statement can be generated from it; you also receive some secondary input from which the WHERE clause of the UPDATE can be generated. easy :) > Most of the above can be done fairly generically in Python, with minimal > code needed for each particular form you are processing. However, there is > no short-cut magic bullet "save" routine or "View" object that can do all of > the above. ? ;) i invite you to take a look at what i have started on - pysqldb contains some of the base classes and functions, following on from that is sqlgen.py and sqlhtmlgen.py which take a SQL file and auto-generate enough python code to do exactly the above. where it breaks down is that the SQL file itself is not sufficient to generate: - complex sub-reports on one subset of fields from one table JOINed against another subset of fields in another - business logic what i HAVEN'T added is logic to auto-generate from CONSTRAINTS because my present SQL knowledge runs out at MySQL which doesn't obey constraints, doesn't have VIEWs, and the MySQL authors are arrogant enough to even spell out in their documentation that they don't believe foreign key constraints are necessary. grr. > I recommend focusing on implementing concise and powerful > abstractions for each step seperately. *vigorous nods* > > one of the input forms has now got so complex that the update > > function is 160 lines of code and the add function is 100. > > and that's simply turning HTML POST cgi form dictionaries / lists > > into dictionaries > > Then you need to write smarter code! :) i realise this :) i have a neat system to generate html forms and tables, a neat system to do database queries (if you can call single function to do a single query neat :) i use the python cgi function SvFormContentDict() to get me the HTML POST, and then it goes haywire after that. basically as i am always refactoring code that i write more than twice or three times at the most, this is the one area where i haven't put in enough thought, and it's getting a little out of hand. regarding posting code: i'll locate the view cvs page for you... http://cvs.sf.net/cgi-bin/viewcvs.cgi/custom/custom/cgi/src/python.py line 714 is where check_update_item() starts. line 877 is where check_add_item() starts. the add and update functions basically parse _any_ of the forms involving POSTs: if the HTTP post has "mode=add" then check_add_item() gets called, if it has mode=update, update gets called. there are several forms that stock.py generates, all of which will call one of these two functions. where things have broken down with this over-simplistic approach is that the check_add_item() function needs to find out why it's been called, from the dictionary of cgi args. ... but in some ways this is all a distraction / just a snapshot of the bigger picture (so i went back and filled in a response to the earlier parts of your email). l. From lkcl@samba-tng.org Mon Feb 3 20:52:57 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Mon, 3 Feb 2003 20:52:57 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: References: <20030202142023.GI912@localhost> Message-ID: <20030203205257.GC7645@localhost> On Sun, Feb 02, 2003 at 10:51:14AM -0500, Kevin Jacobs wrote: > > the question is inspired by the amount of work that is > > needed to create a useable suite of database queries in > > the shortest period of time - something that is important > > when both developing databases and also when doing lots > > of database applications one after the other. > > I've found that rapid initial development is far less important than ease of > validation, maintenance, and portability. My belief is based on writing and > maintaining Python applications that interface to, at times >8 heterogeneous > database backends, each with many hundreds of tables and distinct OTP and > OLAP queries. whoa. not small, then. to be honest the most number of tables i've had to deal with is 25 to 30, across at most two separate databases on different servers. ... but i'm _still_ fed up with how much code has to be written by hand! l. From lkcl@samba-tng.org Mon Feb 3 20:42:14 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Mon, 3 Feb 2003 20:42:14 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <1044217837.11685.17.camel@lothlorien> References: <1044217837.11685.17.camel@lothlorien> Message-ID: <20030203204214.GB7645@localhost> On Sun, Feb 02, 2003 at 02:30:37PM -0600, Ian Bicking wrote: > On Sun, 2003-02-02 at 09:51, Kevin Jacobs wrote: > > > also for some of the background please see the advogato article > > > http://advogato.org/article/600.html particularly the link to > > > jmg's code which contains some very simple to use but obscure > > > coding methods to wrap SQL databases in objects. > > > > We (my company) and many others have their own SQL generation and OR mapping > > implementations that are considerably more advanced. Some (like mine) are > > in the process of transitioning to an open source development model, while > > some are already freely available (see http://colorstudy.com/software/SQLObject oo! exciting code! funnily enough andy dustman (mysqldb) has done a similar bit of coding to SQLbuilder. comments on sqlbuilder 0.2: - i like the try/except on locating database modules. in pysqldb, the name of the database module (and the user/pass) is specified in a per-application config file. therefore the database application is expected to provide certain functionality a la DB 2.0 API. the problem comes when that functionality doesn't exist: the pysqldb classes have to deal with missing or different functionality on a per-module basis. [... perhaps there should be a Python-SQL-DB 2.0 API, or an ODBC-like converter API, to ensure that all SQL databases look the same? ] - i like the per-class handling of database modules. it allows for a much cleaner way of doing "odd" things e.g. the get insert ID which is always weird. in MS-SQL you have to call: "SELECT IDENT_CURRENT('tablename') AS id" in a similar fashion to the sqlbuilder 0.2 PostgreSQL's "SELECT nextval('tablename')" - you may not be aware that MS-SQL has a different "escape" system from MySQL (and probably PostgreSQL). you might want to put in a per-class "string" handler function. the escape sequence for quotes within strings/text is replace ("'", "\\'") in MySQL; but is replace("'", "''") in MS-SQL (7.0 and 2000) and end-of-line characters within strings/text is not necessary (to my knowledge) in MySQL but is replace("\n", "\\012") and replace("\r", "\\015") is needed in MS-SQL. i do have a confession to make: i got the above completely wrong in one revision of some MS-SQL / MySQL compatible application i was developing, and because the other DB developer (visual basic. yukk!) was belgian it caused an absolute riot when i made a change to the above EOL replacements, due to language / miscommunications. what i had done was to substitute text "\n" for "\\\\n" which worked absolutely fine for me because i was correctly converting back again: it failed miserably for the windows VB/DB developer of course... BEWARE THE EOL SQL strings! ROLL ON mime-encoding! :) l. p.s. if you're interested in how i managed to get MS-SQL 2000 access from python on a unix system, the code's at http://sf.net/projects/pyxsqmll. From jacobs@penguin.theopalgroup.com Mon Feb 3 21:01:52 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Mon, 3 Feb 2003 16:01:52 -0500 (EST) Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030203204214.GB7645@localhost> Message-ID: On Mon, 3 Feb 2003, Luke Kenneth Casson Leighton wrote: > On Sun, Feb 02, 2003 at 02:30:37PM -0600, Ian Bicking wrote: > > > On Sun, 2003-02-02 at 09:51, Kevin Jacobs wrote: > > > > also for some of the background please see the advogato article > > > > http://advogato.org/article/600.html particularly the link to > > > > jmg's code which contains some very simple to use but obscure > > > > coding methods to wrap SQL databases in objects. > > > > > > We (my company) and many others have their own SQL generation and OR mapping > > > implementations that are considerably more advanced. Some (like mine) are > > > in the process of transitioning to an open source development model, while > > > some are already freely available (see http://colorstudy.com/software/SQLObject > > oo! exciting code! I thought it was pretty cool too, especially since my library uses some very similar concepts (e.g., tablespaces, pluggable database drivers, etc.). > p.s. if you're interested in how i managed to get MS-SQL 2000 > access from python on a unix system, the code's at > http://sf.net/projects/pyxsqmll. I use David Cole's MS-SQL w/ Sybase ASE and his Sybase module with FreeTDS. Both provide fast access to MS-SQL 2000 from linux, though neither is quite production quality yet. Regards, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From lkcl@samba-tng.org Mon Feb 3 21:27:06 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Mon, 3 Feb 2003 21:27:06 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: References: <20030203204214.GB7645@localhost> Message-ID: <20030203212706.GE7645@localhost> On Mon, Feb 03, 2003 at 04:01:52PM -0500, Kevin Jacobs wrote: > > p.s. if you're interested in how i managed to get MS-SQL 2000 > > access from python on a unix system, the code's at > > http://sf.net/projects/pyxsqmll. > > I use David Cole's MS-SQL w/ Sybase ASE and his Sybase module with FreeTDS. > Both provide fast access to MS-SQL 2000 from linux, though neither is > quite production quality yet. the things that pyxsqmll don't do is binary data and HTTPS (i have a 15 line patch for the HTTPS) but binary data i no longer have access to an MS-SQL database so ain't in a position to add it. freetds wasn't in a useable state at the time i needed MS-SQL access so i plumped for the XML interface option in MS-SQL 2000. took only about three days to write that code (based on andy dustman's mysqldb cursor stuff) and once written i haven't ever needed to do anything drastic to it. one thing that _was_ a pain was i used pyxml but on a query containing 1,000 records because pyxml used Unicode the HTTP response started to take up 40 _megabytes_ of memory and well over a minute to process. talk about overly complex code, pyxml is a heavy overweight hitter. i quickly dumped pyxml and went for a SAX interface using the python distro's stonkingly simple xmllib to rapidly look for specific tag names and nothing else: i couldn't care less if MS's code returned bad XML. works pretty well now (for about 3 days work and only 8 cvs commits :) l. From jacobs@penguin.theopalgroup.com Mon Feb 3 21:35:08 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Mon, 3 Feb 2003 16:35:08 -0500 (EST) Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030203212706.GE7645@localhost> Message-ID: On Mon, 3 Feb 2003, Luke Kenneth Casson Leighton wrote: > On Mon, Feb 03, 2003 at 04:01:52PM -0500, Kevin Jacobs wrote: > > > > p.s. if you're interested in how i managed to get MS-SQL 2000 > > > access from python on a unix system, the code's at > > > http://sf.net/projects/pyxsqmll. > > > > I use David Cole's MS-SQL w/ Sybase ASE and his Sybase module with FreeTDS. > > Both provide fast access to MS-SQL 2000 from linux, though neither is > > quite production quality yet. > > the things that pyxsqmll don't do is binary data and HTTPS > (i have a 15 line patch for the HTTPS) but binary data > i no longer have access to an MS-SQL database so ain't in a > position to add it. > > freetds wasn't in a useable state at the time i needed MS-SQL > access so i plumped for the XML interface option in MS-SQL 2000. It isn't in a great state even now, though it is crawling towards a better state. However it is _much_ faster than pyxsqmll, especially for our workloads. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From jacobs@penguin.theopalgroup.com Mon Feb 3 21:31:28 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Mon, 3 Feb 2003 16:31:28 -0500 (EST) Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030203161732.GX912@localhost> Message-ID: On Mon, 3 Feb 2003, Luke Kenneth Casson Leighton wrote: > my goal ultimately is to be able to write a file, maybe in > XML or other suitable text file format, that specifies all the > reports i want, that they are just tabular display of records, > generation of an input form, or generation of create / delete > input form. We've thought about that, but had much more luck rendering reports via customer code with many helper objects and functions. e.g., we have a class that emits HTML tables with pluggable data formatters so that our report code remains uncluttered. We have very powerful data aggregation functions to slice-and-dice information in our OLAP-type reports. We use CSS style sheets extensively so that our rendering code is abstract from many of the details of the final rendered output. We even handle the difference between 'screen' and 'printed' formatting via style sheets, so that a single report renders very differently depending on where you send it. > > 2) Decode the variable names and map them to schema elements > > 3) Validate the variable values against the schema and business logic > > 4) Group variables into transactional batches and sub-batches > > based on the physical schema > > 5) Render and execute SQL to perform updates > > > what do you define as schema elements? > i think i can work out what you mean by schema by context. Schema elements are the relations and columns of your database backend. Those elements have type, unique key, foreign key, domain, and business logic constraints. > assuming that code to do all the above is auto-generated from > either a SQL file (with CREATE and ALTER TABLE statements > including FOREIGN KEY statements from which JOINs can be > auto-generated) or from a separate IDL file, then code can > be generated for the following: Why not get them from the database at runtime? It is easy to query the types, and only slightly harder to extract the unique and foreign keys. However, your forms and reports will end up being very bland and unfriendly if you use a one-size-fits-all approach to generating them. I've looked at some of your code from "Custom" and it looks like a good start, though very verbose. I'm sure you're developing some more powerful abstraction interfaces in "pysqldb", though I haven't had a chance to look. I'm still working on releasing some of my code so that I can show you how we handle some of these cases. Unfortunately, we have some legally encumbered libraries that I have to remove with a chain-saw and then replace with clean-room code before I can do that. Soon though... Regards, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From magnus@thinkware.se Mon Feb 3 22:35:05 2003 From: magnus@thinkware.se (Magnus Lycka) Date: Mon, 03 Feb 2003 23:35:05 +0100 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: References: <20030203161732.GX912@localhost> Message-ID: <5.1.0.14.0.20030203231847.02b3b968@www.thinkware.se> At 16:31 2003-02-03 -0500, Kevin Jacobs wrote: >On Mon, 3 Feb 2003, Luke Kenneth Casson Leighton wrote: > > my goal ultimately is to be able to write a file, maybe in > > XML or other suitable text file format, that specifies all the > > reports i want, that they are just tabular display of records, > > generation of an input form, or generation of create / delete > > input form. What makes you think XML will be a better very high level programming language than Python? Isn't that really what you are saying? And don't you think that special cases will pop up now and then that will be much easier to handle in Python than in XML? I'm sure you are right about constructing something along what you are talking about, or using one of these available tools, but I still think Python scripts, using these more abstract tools, is the optimal tool do describe each form or report in. What is it you can do in XML that you can't to in a structure of nested lists etc in Python? There are certainly a lot of things you can do in Python that you can't easilly do in XML... >We've thought about that, but had much more luck rendering reports via >customer code with many helper objects and functions. Luck has nothing to do with it! ;) >I'm still working on releasing some of my code so that I can show you how we >handle some of these cases. Unfortunately, we have some legally encumbered >libraries that I have to remove with a chain-saw and then replace with >clean-room code before I can do that. Soon though... Soon... You've said that for years! :( Grumble... ;) Well, maybe you haven't said soon before, but you have certainly teased us with this for some time... ;) Do you mean soon as my four year old son interprets the word, or do you happen to come from a family of geologists or astronomers where millions of years are normal time frames? Maybe you could just hack in there with the chain-saw and publish it as it is? If you just give general directions and let others implement it, it should be clean-room enough, don't you think? I mean, it can't be a Mozilla- sized thingie? Right? Or maybe you should join up with Ian Bicking and add the free things you can offer to SQLObject? After all, there aren't really too few OR Mappers, are there? :) -- Magnus Lycka, Thinkware AB Alvans vag 99, SE-907 50 UMEA, SWEDEN phone: int+46 70 582 80 65, fax: int+46 70 612 80 65 http://www.thinkware.se/ mailto:magnus@thinkware.se From jacobs@penguin.theopalgroup.com Mon Feb 3 22:54:50 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Mon, 3 Feb 2003 17:54:50 -0500 (EST) Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <5.1.0.14.0.20030203231847.02b3b968@www.thinkware.se> Message-ID: On Mon, 3 Feb 2003, Magnus Lycka wrote: > >We've thought about that, but had much more luck rendering reports via > >customer code with many helper objects and functions. > > Luck has nothing to do with it! ;) True -- or maybe I should have said: "Defining a non-Turing complete XML description of how to render a report has so far eluded us." ;) Thus, we use Python as our favorite RAD Turing-complete specification language. > >I'm still working on releasing some of my code so that I can show you how we > >handle some of these cases. Unfortunately, we have some legally encumbered > >libraries that I have to remove with a chain-saw and then replace with > >clean-room code before I can do that. Soon though... > > Soon... You've said that for years! :( Grumble... ;) Actually, its only been just under a year. > Well, maybe you haven't said soon before, but you have > certainly teased us with this for some time... ;) > > Do you mean soon as my four year old son interprets > the word, or do you happen to come from a family of > geologists or astronomers where millions of years are > normal time frames? Well, db_row has already been released, though it is only the tip of the iceburg. I'll have our relational operator library ready in the next few days, and our database abstraction layer could see light of day be the end of the month. Unfortunately, our SQL dialect translator is still mired in legal tangles, so I'm working on a Python-based alternative. > Maybe you could just hack in there with the chain-saw and > publish it as it is? If you just give general directions > and let others implement it, it should be clean-room > enough, don't you think? I mean, it can't be a Mozilla- > sized thingie? Right? Our production version is 80,440 lines of code, ~58,000 of which are not redistributable. Unfortunately some of that is very low level, like our connection management system, so unfactoring is non-trivial. Plus, the changes are of the magnitude that our internal documentation doesn't apply, so I have to patch together some basic documentation to go out with any release. > Or maybe you should join up with Ian Bicking and add the free > things you can offer to SQLObject? While there are some nice conceptual similarities with Ian's work, our infrastructure is still quite different, though I do look forward to working with Ian, Luke, and others to improve and sharpen our models. Regards, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From ianb@colorstudy.com Mon Feb 3 23:14:29 2003 From: ianb@colorstudy.com (Ian Bicking) Date: Mon, 3 Feb 2003 17:14:29 -0600 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030203204214.GB7645@localhost> Message-ID: <3EC26C81-37CD-11D7-ACF1-000393C2D67E@colorstudy.com> On Monday, February 3, 2003, at 02:42 PM, Luke Kenneth Casson Leighton wrote: > comments on sqlbuilder 0.2: > > - i like the try/except on locating database modules. > > in pysqldb, the name of the database module (and the user/pass) > is specified in a per-application config file. > > therefore the database application is expected to provide > certain functionality a la DB 2.0 API. the problem comes > when that functionality doesn't exist: the pysqldb > classes have to deal with missing or different functionality > on a per-module basis. I think the wrappers worked out quite well, and the resulting compatibility code can remain fairly short and isolated. Well, there's only two databases supported, but that's at least 50% of the way to universal coverage :) > [... perhaps there should be a Python-SQL-DB 2.0 API, > or an ODBC-like converter API, to ensure that all SQL > databases look the same? ] I don't know what that would look like. I'm more confident of the utility and feasibility of creating a domain-specific wrapper to deal with database differences. For instance, SQLObject's functionality (at least for now) requires only a small wrapper, mostly to handle the insert ID issue. Layering abstractions can quickly become unwieldy, and I feel strongly that SQLObject (or any other similar system) is best written solely on the DB 2.0 API. > - you may not be aware that MS-SQL has a different "escape" > system from MySQL (and probably PostgreSQL). > > you might want to put in a per-class "string" handler > function. > > the escape sequence for quotes within strings/text > is > replace ("'", "\\'") in MySQL; > but is > replace("'", "''") in MS-SQL (7.0 and 2000) > > and end-of-line characters within strings/text is not > necessary (to my knowledge) in MySQL but > is > replace("\n", "\\012") and replace("\r", "\\015") > is needed in MS-SQL. I've avoided database-specific SQL generation, but perhaps it will be necessary at one point. On these counts MySQL supports '' and \012, so the more complete quoting that MS-SQL demands could be implemented globally. > p.s. if you're interested in how i managed to get MS-SQL 2000 > access from python on a unix system, the code's at > http://sf.net/projects/pyxsqmll. Have you ever tried using Pyro (Python Remote Objects) or similar things for communicating over the wire? I played around with similar stuff myself (though the database I was connecting too was ultimately too buggy), but later realized that I could have more easily exported those interfaces over the wire using a generic distributed-object system. Ian From ramrom@earthling.net Tue Feb 4 03:20:26 2003 From: ramrom@earthling.net (Bob Gailer) Date: Mon, 03 Feb 2003 20:20:26 -0700 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030203122132.GO912@localhost> References: <1044217837.11685.17.camel@lothlorien> <1044217837.11685.17.camel@lothlorien> Message-ID: <5.2.0.9.0.20030203201927.02ea3e30@66.28.54.253> --=======780035B7======= Content-Type: text/plain; x-avg-checked=avg-ok-7695760E; charset=us-ascii; format=flowed Content-Transfer-Encoding: 8bit At 12:21 PM 2/3/2003 +0000, Luke Kenneth Casson Leighton wrote: >see http://sf.net/projects/pysqldb. I feel frustrated when pointed to a package that has no documentation. Is there some? Bob Gailer mailto:ramrom@earthling.net 303 442 2625 --=======780035B7======= Content-Type: text/plain; charset=us-ascii; x-avg=cert; x-avg-checked=avg-ok-7695760E Content-Disposition: inline --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.445 / Virus Database: 250 - Release Date: 1/21/2003 --=======780035B7=======-- From eivindt-db-sig@multinet.no Tue Feb 4 08:32:15 2003 From: eivindt-db-sig@multinet.no (Eivind Tagseth) Date: Tue, 4 Feb 2003 09:32:15 +0100 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030203161732.GX912@localhost> References: <20030203134327.GU912@localhost> <20030203161732.GX912@localhost> Message-ID: <20030204083215.GA2070@tagseth-trd.consultit.no> * Luke Kenneth Casson Leighton [030203 16:16]: > On Mon, Feb 03, 2003 at 09:48:54AM -0500, Kevin Jacobs wrote: > my goal ultimately is to be able to write a file, maybe in > XML or other suitable text file format, that specifies all the > reports i want, that they are just tabular display of records, > generation of an input form, or generation of create / delete > input form. > > and for there to be like a _single_ cgi program that can > take such a specification file and just... do everything. I tried creating such a beast myself a while ago. Most simple web applications are just editing of a database anyway, so why spend so much time writing code that's almost the same? As most of my projects, I got bored, and put it away... ;) Here's what I did though: I started out automatically generating source code for data objects based on db_row and the actual postgresql-instance. This was pretty easy and covered my basic needs. This got me column types, primary keys, auto-increment keys and foreign keys. This was mostly a proof of concept for me, I started out making a DBSchemeFactory, using a Facade/factory-pattern to support fetching info about the database from various databases or config files (e.g. XML). For the web-tier I made a general cgi-input handler class, fetching data and inserting/updating data into the data objects. Simple validation could be performed automatically (i.e. was this a string?), while an XML config file could map extra validator functions to special fields. The cgi would first see if a special cgi-input handler class existed, and would fall back on the general one if not. Finally, for output I used Cheetah, and made a general Cheetah-template for showing a database table or database table row. If the cgi found a template matching the data objects' name, it would use that instead. This would make it easy to put up a simple web application, but you would still have full freedom to taylor it to your own needs. Unfortunately, it was never completed. Does this sound like your ideas? > clearly, the validation of some variable values as you specify > in 3) below cannot be entirely done in an automated manner: > some things are going to be too complex to specify and you'd > just have to put a function call out to do the validation. Jakarta have a nice solution for the web-tier, called Validator. It makes your servlet parse an XML-file, mapping input fields to regexps. Pretty cool for the web-tier. Eivind -- Eivind Tagseth, E-mail jobb: eivind.tagseth@consultit.no, E-mail priv: eivindt@multinet.no Mobil: +47 922 43742 From lkcl@samba-tng.org Tue Feb 4 10:20:34 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Tue, 4 Feb 2003 10:20:34 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <5.1.0.14.0.20030203231847.02b3b968@www.thinkware.se> References: <20030203161732.GX912@localhost> <5.1.0.14.0.20030203231847.02b3b968@www.thinkware.se> Message-ID: <20030204102034.GG7645@localhost> On Mon, Feb 03, 2003 at 11:35:05PM +0100, Magnus Lycka wrote: > At 16:31 2003-02-03 -0500, Kevin Jacobs wrote: > >On Mon, 3 Feb 2003, Luke Kenneth Casson Leighton wrote: > >> my goal ultimately is to be able to write a file, maybe in > >> XML or other suitable text file format, that specifies all the > >> reports i want, that they are just tabular display of records, > >> generation of an input form, or generation of create / delete > >> input form. > > What makes you think XML will be a better very high level programming > language than Python? see http://sf.net/projects/virgule and then tell me if you think that question _really_ needs asking of me!!!!!! > Isn't that really what you are saying? nope. > And don't > you think that special cases will pop up now and then that will be much > easier to handle in Python than in XML? i believe i described, later in the email to which you replied, that it should be possible to over-ride special cases. SQLObject does this by checking for the existence of some functions in the over-riding class, where the name of the function to check for is _{column_name}_{update|del|set}() or something like that. for example in the display of certain fields in a table, in pysqldb i provide a means to pass in a function to perform the HTML rendering. if no such function is given on a per-column basis, then cgi.escape(str(value), 1) is used. the idea that i had was that the XML file should specify the module and the function name to be used to display the column. alternatively, the XML file specifies the class name (or a particular class name is expected by default based on other context e.g. the table name, report name etc) and if that class exists, and a function _in_ that class exists with the name display_in_html_format_{column_name}() then it is called to render the relevant field. the same technique may be used to specify the reporting format: the XML file specifies the type of the report by name, and a rendering module is expected to contain a function by that same report name, which does the actual work. ultimately i really don't care what is used. XML, HTML/templating, SQL databases, flat files or python. i just want simpler applications. > I'm sure you are right about constructing something along what you > are talking about, or using one of these available tools, but I still > think Python scripts, using these more abstract tools, is the optimal > tool do describe each form or report in. > > What is it you can do in XML that you can't to in a structure of > nested lists etc in Python? you can't get python accepted by perl or php programmers or non-technical administrators. you _can_ get the pyxml libraries to read XML documents into structures of nested lists etc. you _can_ get XML accepted by other editing systems. win-win. [you can also get flat-file and SQL databases accepted by non-programmers] l. [@BEGIN IRRELEVANT XML STUFF hit delete, now, if you don't want to hear about XML... > There are certainly a lot of things you > can do in Python that you can't easilly do in XML... tell me about it. xvl/virgule is a non-NP-complete programming language with an XML-based-syntax. your comments here lead me to believe that you misunderstand what XML is, can, and cannot do. XML is nothing without meaning being provided. it's just a pretty data format. XML doesn't run, it doesn't execute, it doesn't sing or dance. for XML to have meaning, there must be three things: - the data. - the interpretation of the data. - a program to make use the interpreted data. if you do not have any one of those three things, XML is xxxxing useless. i recommended XML on the basis that it is becoming widely accepted as a data format. people often mistake it as magic :) @END IRRELEVANT XML STUFF] From lkcl@samba-tng.org Tue Feb 4 11:22:36 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Tue, 4 Feb 2003 11:22:36 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <3EC26C81-37CD-11D7-ACF1-000393C2D67E@colorstudy.com> References: <20030203204214.GB7645@localhost> <3EC26C81-37CD-11D7-ACF1-000393C2D67E@colorstudy.com> Message-ID: <20030204112235.GK7645@localhost> On Mon, Feb 03, 2003 at 05:14:29PM -0600, Ian Bicking wrote: > I don't know what that would look like. I'm more confident of the > utility and feasibility of creating a domain-specific wrapper to deal > with database differences. For instance, SQLObject's functionality (at > least for now) requires only a small wrapper, mostly to handle the > insert ID issue. Layering abstractions can quickly become unwieldy, > and I feel strongly that SQLObject (or any other similar system) is > best written solely on the DB 2.0 API. i was basically proposing that the DBconnection module or something like it be made a standard base class / convention, in the same way that the DB 2.0 API is an accepted convention. only specifically related to SQL. a bit like ODBC is a standard. that sort of thing. l. [running out of steam and doing lots of vague hand-waving] p.s. maybe this is too small an issue to be worth while considering as a standard? From lkcl@samba-tng.org Tue Feb 4 11:36:24 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Tue, 4 Feb 2003 11:36:24 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <3EC26C81-37CD-11D7-ACF1-000393C2D67E@colorstudy.com> References: <20030203204214.GB7645@localhost> <3EC26C81-37CD-11D7-ACF1-000393C2D67E@colorstudy.com> Message-ID: <20030204113623.GL7645@localhost> On Mon, Feb 03, 2003 at 05:14:29PM -0600, Ian Bicking wrote: > > you might want to put in a per-class "string" handler > > function. > > [...] > I've avoided database-specific SQL generation, but perhaps it will be > necessary at one point. On these counts MySQL supports '' and \012, so > the more complete quoting that MS-SQL demands could be implemented > globally. ... can you guarantee that all SQL database systems likely to be used will conform to a global string-handling system? > Have you ever tried using Pyro (Python Remote Objects) or similar > things for communicating over the wire? ... no? i wrote the pyxsqmll code very quickly, cut/paste style from andy dustman's mysqldb and from microsoft's examples on their web site to do HTTP POSTs containing SQL queries. it's only 950 lines of code (and it shows :) requiring a dependency on a 130k download seems somewhat... excessive? > I played around with similar > stuff myself (though the database I was connecting too was ultimately > too buggy), but later realized that I could have more easily exported > those interfaces over the wire using a generic distributed-object > system. ian, what are you proposing that pyro be used for? as a gateway to run a local python ODBC client on the MS-SQL server that then passes the results over, with pyro doing all the communication and stuff? l. From magnus@thinkware.se Tue Feb 4 12:36:15 2003 From: magnus@thinkware.se (Magnus Lycka) Date: Tue, 04 Feb 2003 13:36:15 +0100 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030204102034.GG7645@localhost> References: <5.1.0.14.0.20030203231847.02b3b968@www.thinkware.se> <20030203161732.GX912@localhost> <5.1.0.14.0.20030203231847.02b3b968@www.thinkware.se> Message-ID: <5.1.0.14.0.20030204121838.02b3bab0@www.thinkware.se> At 10:20 2003-02-04 +0000, Luke Kenneth Casson Leighton wrote: > i believe i described, later in the email to which you replied, > that it should be possible to over-ride special cases. ... > for example in the display of certain fields in a table, > in pysqldb i provide a means to pass in a function to > perform the HTML rendering. Are you only intending to use this for web applications? With a GUI, there are so many more things you can do, and so many more requirements that might pop up--and be reasonable-- to implement. You might want to do a lot of things without roundtrips to the database server. > the idea that i had was that the XML file should specify the > module and the function name to be used to display the > column. It sounds to me like you don't intend the columns to be aware of the context they are in. Of course, you can make them aware though the database, but only if you actually store data. What if we want a field to get a red background colour as soon as you enter a value that lies outside the min and max values that are presented in other columns. Imagine that you want to be able to change all columns and get visual feedback before you save data to the database. In my experience, these kinds of demands pop up sooner or later, and it seems to me that it would be much easier to solve this with form-specific code. Unless I misundersttod you, you intended to use a generic engine that will use an XML description to find out what fields to assemble on the screen, and all application specific logic is tied to a specific field. > > What is it you can do in XML that you can't to in a structure of > > nested lists etc in Python? > > you can't get python accepted by perl or php programmers or > non-technical administrators. Ok. Politics can be a tricky issue... You're right about that... Another option is to get involved in Python Business Forum, and try to increase the buzz-factor of Python, but it might be faster to write something in XML I suppose. ;) ReportLab's rml2pdf is an example of what you say I guess, but that's limited to creating documents. I think it works better there than in creating user interfaces. I wrote: > > There are certainly a lot of things you > > can do in Python that you can't easilly do in XML... Luke wrote: > your comments here lead me to believe that you misunderstand > what XML is, can, and cannot do. Well, maybe I was unclear... > XML is nothing without meaning being provided. > > it's just a pretty data format. > > XML doesn't run, it doesn't execute, it doesn't sing or dance. Yes, this is what I meant. (Except possibly the "pretty" part. ;) > for XML to have meaning, there must be three things: > > - the data. > > - the interpretation of the data. > > - a program to make use the interpreted data. The same is true for Python of course. Any source code is just data. But with python you have a much higher signal / noise ratio, and you don't have to invent the interpreter or the syntactic rules. It comes out of the box. I hadn't seen virgule, and I haven't used XML a lot, but I think I have a general understanding of what's on the menu. You didn't make me change my mind... I much prefer the built-in power of python--for the special cases-- even if I try to make as abstract and generic programs as possible. Then I can fix the special cases that are bound to pop up. But I suppose you have a point that XML might be easier to get acceptance for than Python. :( -- Magnus Lycka, Thinkware AB Alvans vag 99, SE-907 50 UMEA, SWEDEN phone: int+46 70 582 80 65, fax: int+46 70 612 80 65 http://www.thinkware.se/ mailto:magnus@thinkware.se From jacobs@penguin.theopalgroup.com Tue Feb 4 13:04:28 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Tue, 4 Feb 2003 08:04:28 -0500 (EST) Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <5.2.0.9.0.20030203201927.02ea3e30@66.28.54.253> Message-ID: On Mon, 3 Feb 2003, Bob Gailer wrote: > At 12:21 PM 2/3/2003 +0000, Luke Kenneth Casson Leighton wrote: > >see http://sf.net/projects/pysqldb. > > I feel frustrated when pointed to a package that has no documentation. Is > there some? [Not speaking for Luke] Please be patient -- that package is still in early development and the pointer was meant primarily for other developers of high-level wrappers. You may be happier if you wait until a formal release annoucement is made before evaluating this or some of the other code that is being discussed. Regards, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From jacobs@penguin.theopalgroup.com Tue Feb 4 13:53:06 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Tue, 4 Feb 2003 08:53:06 -0500 (EST) Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030204083215.GA2070@tagseth-trd.consultit.no> Message-ID: On Tue, 4 Feb 2003, Eivind Tagseth wrote: > I started out automatically generating source code for data objects based > on db_row and the actual postgresql-instance. ;) A happy db_row user? How is it working out for you? -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From magnus@thinkware.se Tue Feb 4 15:15:04 2003 From: magnus@thinkware.se (Magnus Lycka) Date: Tue, 04 Feb 2003 16:15:04 +0100 Subject: [DB-SIG] O-R-Mapper for MySQL that don't require 2.2... Message-ID: <5.1.0.14.0.20030204155810.02bb0d50@www.thinkware.se> I see that a lot of the new O-R-mappers use 2.2 features like new style classes. It seems right now that I'd need to build a small app that interfaces to MySQL on a Linux box where I only have Python 2.1.2. As far as I understand, this rules out SQLObject as well as db_row. Can anyone suggest something simple that will fit here? I don't really need a full fledged O-R-mapper, I just thought I'd make this little app as simple as possible. It's a small app, just two tables to start with, and I don't exepect it to grow a lot, so it's no big deal to wrap this manually, but I thought I could try one of the mappers, like Andy Dustman's SQLDict or Fionn Behrens' pSQL. Any suggestions? -- Magnus Lycka, Thinkware AB Alvans vag 99, SE-907 50 UMEA, SWEDEN phone: int+46 70 582 80 65, fax: int+46 70 612 80 65 http://www.thinkware.se/ mailto:magnus@thinkware.se From dieter@handshake.de Tue Feb 4 18:08:44 2003 From: dieter@handshake.de (Dieter Maurer) Date: Tue, 4 Feb 2003 19:08:44 +0100 Subject: [DB-SIG] O-R-Mapper for MySQL that don't require 2.2... In-Reply-To: <5.1.0.14.0.20030204155810.02bb0d50@www.thinkware.se> References: <5.1.0.14.0.20030204155810.02bb0d50@www.thinkware.se> Message-ID: <15936.428.336262.818390@gargle.gargle.HOWL> Magnus Lycka wrote at 2003-2-4 16:15 +0100: > I see that a lot of the new O-R-mappers use 2.2 features > like new style classes. It seems right now that I'd need > to build a small app that interfaces to MySQL on a Linux > box where I only have Python 2.1.2. This would be easy to change, wouldn't it? Dieter From ianb@colorstudy.com Tue Feb 4 19:06:09 2003 From: ianb@colorstudy.com (Ian Bicking) Date: 04 Feb 2003 13:06:09 -0600 Subject: [DB-SIG] O-R-Mapper for MySQL that don't require 2.2... In-Reply-To: <5.1.0.14.0.20030204155810.02bb0d50@www.thinkware.se> References: <5.1.0.14.0.20030204155810.02bb0d50@www.thinkware.se> Message-ID: <1044385569.1129.2.camel@lothlorien> On Tue, 2003-02-04 at 09:15, Magnus Lycka wrote: > I see that a lot of the new O-R-mappers use 2.2 features > like new style classes. It seems right now that I'd need > to build a small app that interfaces to MySQL on a Linux > box where I only have Python 2.1.2. You could try MiddleKit -- it's part of Webware (webware.sf.net), but you can use it independently. -- Ian Bicking Colorstudy Web Development ianb@colorstudy.com http://www.colorstudy.com PGP: gpg --keyserver pgp.mit.edu --recv-keys 0x9B9E28B7 4869 N Talman Ave, Chicago, IL 60625 / (773) 275-7241 From ianb@colorstudy.com Tue Feb 4 19:19:27 2003 From: ianb@colorstudy.com (Ian Bicking) Date: 04 Feb 2003 13:19:27 -0600 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030204102034.GG7645@localhost> References: <20030203161732.GX912@localhost> <5.1.0.14.0.20030203231847.02b3b968@www.thinkware.se> <20030204102034.GG7645@localhost> Message-ID: <1044386367.1129.44.camel@lothlorien> On Tue, 2003-02-04 at 04:20, Luke Kenneth Casson Leighton wrote: > > I'm sure you are right about constructing something along what you > > are talking about, or using one of these available tools, but I still > > think Python scripts, using these more abstract tools, is the optimal > > tool do describe each form or report in. > > > > What is it you can do in XML that you can't to in a structure of > > nested lists etc in Python? > > you can't get python accepted by perl or php programmers or > non-technical administrators. > > you _can_ get the pyxml libraries to read XML documents into > structures of nested lists etc. > > you _can_ get XML accepted by other editing systems. > > win-win. > > [you can also get flat-file and SQL databases accepted by > non-programmers] I would emphasize SQL as the cross-language layer. What you are creating in XML is either a crippled language, or a crippled description -- the description simply can't be general enough without a full programming language, and you can't achieve that a full language. And certainly not a pleasant language, because XML is not a particularly pleasant syntax. A *big* reason I created SQLObject was because I didn't like XML (or CSV) descriptions. Besides poor tool support and an awkward syntax, XML just isn't a powerful enough medium. Sure, you can *create* a programming language in XML, but I like my programming language. If I was going to attempt language portability, I would simply create a ORM in the other languages. It would be easier, and the interface for the other languages could be comfortable for those languages. That was another goal for SQLObject -- the schema shouldn't have any funny requirements, because if the database is an abstraction layer and communication tool then it shouldn't be biased towards one interface. Ian From magnus@thinkware.se Tue Feb 4 20:44:39 2003 From: magnus@thinkware.se (Magnus Lycka) Date: Tue, 04 Feb 2003 21:44:39 +0100 Subject: [DB-SIG] O-R-Mapper for MySQL that don't require 2.2... In-Reply-To: <15936.428.336262.818390@gargle.gargle.HOWL> References: <5.1.0.14.0.20030204155810.02bb0d50@www.thinkware.se> <5.1.0.14.0.20030204155810.02bb0d50@www.thinkware.se> Message-ID: <5.1.0.14.0.20030204214147.02ccfd20@www.thinkware.se> At 19:08 2003-02-04 +0100, Dieter Maurer wrote: >Magnus Lycka wrote at 2003-2-4 16:15 +0100: > > I see that a lot of the new O-R-mappers use 2.2 features > > like new style classes. It seems right now that I'd need > > to build a small app that interfaces to MySQL on a Linux > > box where I only have Python 2.1.2. >This would be easy to change, wouldn't it? Perhaps not... Then I wouldn't have asked. It's not my box, it belongs to my ISP, and I need to set up the app within a few days. (A trivial little thing really, just insert/update one simple table while verifying constraints to another.) -- Magnus Lycka, Thinkware AB Alvans vag 99, SE-907 50 UMEA, SWEDEN phone: int+46 70 582 80 65, fax: int+46 70 612 80 65 http://www.thinkware.se/ mailto:magnus@thinkware.se From lkcl@samba-tng.org Tue Feb 4 22:23:16 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Tue, 4 Feb 2003 22:23:16 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <5.1.0.14.0.20030204121838.02b3bab0@www.thinkware.se> References: <5.1.0.14.0.20030203231847.02b3b968@www.thinkware.se> <20030203161732.GX912@localhost> <5.1.0.14.0.20030203231847.02b3b968@www.thinkware.se> <5.1.0.14.0.20030204121838.02b3bab0@www.thinkware.se> Message-ID: <20030204222316.GB1229@localhost> On Tue, Feb 04, 2003 at 01:36:15PM +0100, Magnus Lycka wrote: > At 10:20 2003-02-04 +0000, Luke Kenneth Casson Leighton wrote: > > i believe i described, later in the email to which you replied, > > that it should be possible to over-ride special cases. > ... > > for example in the display of certain fields in a table, > > in pysqldb i provide a means to pass in a function to > > perform the HTML rendering. > > Are you only intending to use this for web applications? heck, no, one of my goals is to make it possible for different "presenters" to read the report spec files (whether they be in XML, python, SQL tables or flat text files, doesn't matter) and to present the information using those report specs in whatever format. > > the idea that i had was that the XML file should specify the > > module and the function name to be used to display the > > column. > > It sounds to me like you don't intend the columns to be aware > of the context they are in. Of course, you can make them aware > though the database, but only if you actually store data. > > What if we want a field to get a red background colour as soon > as you enter a value that lies outside the min and max values > that are presented in other columns. as ... darn, let me check my email archives... kevin! described earlier: " Logical read/write views don't really solve your problem either. The heart of your situation involves (roughly) these processing steps: 1) Extract and unencode variables from the CGI enviornment 2) Decode the variable names and map them to schema elements 3) Validate the variable values against the schema and business logic 4) Group variables into transactional batches and sub-batches based on the physical schema 5) Render and execute SQL to perform updates " i believe that what you describe above is an example of step 3). however what you are additionally pointing out is a sub-step of 3), yes? namely, that step 4) and 5) are not performed but instead there is an "error" indicating not to proceed. the system i have in place is that the cgi script does some HTML output saying "error with this field, go back on your browser and fix it", and then calls sys.exit(0) which terminates any possibility of proceeding to step 4) and 5). i realise that this is oversimplistic, not very generic, etc. and what _you_ would like to see is: - a per-data-input entry validation as step 3) above - a means to report on a per-input-field basis what the problem is in an HTML environment, that would tend to indicate: - use javascript, avoid the darn problem :) - redirect the POST back to the same page as before, constructing the page exactly as it was, including the data entry so far EXCEPT that: - the fields specifically marked with an error get an "error" display function called. personaly i would implement this as: - the validation process REPLACE the data in the inputted data field with an Error class - the validation process note that an error occurred and that the steps 4) and 5) not proceed, but redisplay the previous page instead. - the Error class have a display function that gets recognised by the templating / reporting process and called to render both the field and its error message / data. > Imagine that you want to be > able to change all columns and get visual feedback before you save > data to the database. this should be done as a two-stage POST, yes? then the actions in step 4) and 5) need to be optional not mandatory, that is not a problem. > In my experience, these kinds of demands pop up sooner or later, > and it seems to me that it would be much easier to solve this > with form-specific code. uhmm... > Unless I misundersttod you, you intended to use a generic engine > that will use an XML description to find out what fields to > assemble on the screen, ... and some default functions supplied to display such fields, which can be easily over-ridden on a per-field basis... > and all application specific logic is > tied to a specific field. i found that even on doing simple field displaying that this was insufficient, and so would never recommend it. personally i solved this in a manner where a function was passed in, and if that function has three arguments, it is called with: - the entire row of data, as a dictionary - the name of the field presently being displayed - the value converted to a string (a legacy / silly decision issue) l. [RELEVANCE TO PYTHON DB-SIG GROUP ENDS HERE! STOP READING NOW! :) ] > The same is true for Python of course. Any source code is just > data. But with python you have a much higher signal / noise ratio, > and you don't have to invent the interpreter or the syntactic rules. > It comes out of the box. I hadn't seen virgule, and I haven't used > XML a lot, but I think I have a general understanding of what's > on the menu. You didn't make me change my mind... i pointed out virgule because i KNOW how apallingly difficult it becomes to attempt to create an XML-syntax programming language. if you EVER consider it, consider shooting yourself in the head first, then think again. ultimately it will be a lot less painful if you follow this simple but graphic advice. [NOTE TO SUE-HAPPY AND TRIGGER-HAPPY AMERICANS: i live in the UK so be as literal-minded and humourless as you like about my joke above: your laws only apply in the US.] From huy@tramada.com.au Tue Feb 4 23:19:34 2003 From: huy@tramada.com.au (Huy Do) Date: Wed, 5 Feb 2003 10:19:34 +1100 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: Message-ID: Hi Kevin, I'd like to say that I'm a very happy db_row user, it has saved me a lot of time. Thanks Kevin Huy > -----Original Message----- > From: db-sig-admin@python.org [mailto:db-sig-admin@python.org]On Behalf > Of Kevin Jacobs > Sent: Wednesday, 5 February 2003 12:53 AM > To: Eivind Tagseth > Cc: db-sig@python.org > Subject: Re: [DB-SIG] Re: [Python-Dev] database APIs > > > On Tue, 4 Feb 2003, Eivind Tagseth wrote: > > I started out automatically generating source code for data > objects based > > on db_row and the actual postgresql-instance. > > ;) > > A happy db_row user? > > How is it working out for you? > > -Kevin > > -- > -- > Kevin Jacobs > The OPAL Group - Enterprise Systems Architect > Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com > Fax: (216) 986-0714 WWW: http://www.theopalgroup.com > > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig > From ianb@colorstudy.com Wed Feb 5 06:39:13 2003 From: ianb@colorstudy.com (Ian Bicking) Date: 05 Feb 2003 00:39:13 -0600 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030204113623.GL7645@localhost> References: <20030203204214.GB7645@localhost> <3EC26C81-37CD-11D7-ACF1-000393C2D67E@colorstudy.com> <20030204113623.GL7645@localhost> Message-ID: <1044427153.1127.834.camel@lothlorien> On Tue, 2003-02-04 at 05:36, Luke Kenneth Casson Leighton wrote: > On Mon, Feb 03, 2003 at 05:14:29PM -0600, Ian Bicking wrote: > > > > you might want to put in a per-class "string" handler > > > function. > > > > [...] > > > > I've avoided database-specific SQL generation, but perhaps it will be > > necessary at one point. On these counts MySQL supports '' and \012, so > > the more complete quoting that MS-SQL demands could be implemented > > globally. > > ... can you guarantee that all SQL database systems likely to > be used will conform to a global string-handling system? No, I can't guarantee anything. But designing for hypotheticals is a bad idea. And there is a SQL standard, so not everything will be database-specific. > > Have you ever tried using Pyro (Python Remote Objects) or similar > > things for communicating over the wire? > > ... no? > > i wrote the pyxsqmll code very quickly, cut/paste style from > andy dustman's mysqldb and from microsoft's examples on their > web site to do HTTP POSTs containing SQL queries. > > it's only 950 lines of code (and it shows :) > > requiring a dependency on a 130k download seems somewhat... > excessive? I don't know, I haven't actually used Pyro before. I suspect a lot of that is documentation and tools you wouldn't use. Anyway, there's other remote object systems that are much more compact. There's a whole category on Parnassus: http://py.vaults.ca/parnassus/apyllo.py?i=549099265 > > > I played around with similar > > stuff myself (though the database I was connecting too was ultimately > > too buggy), but later realized that I could have more easily exported > > those interfaces over the wire using a generic distributed-object > > system. > > ian, what are you proposing that pyro be used for? > > as a gateway to run a local python ODBC client on the MS-SQL server > that then passes the results over, with pyro doing all the > communication and stuff? Yes, you'd have a small program on the MS-SQL server that would create a connection. That connection would be exported as a remote object, and that would be it. There wouldn't be any layer between the client and the database connection (except the remote object system, but that would hopefully seem fairly transparent). -- Ian Bicking ianb@colorstudy.com http://colorstudy.com 4869 N. Talman Ave., Chicago, IL 60625 / 773-275-7241 "There is no flag large enough to cover the shame of killing innocent people" -- Howard Zinn From eivindt-db-sig@multinet.no Wed Feb 5 08:09:58 2003 From: eivindt-db-sig@multinet.no (Eivind Tagseth) Date: Wed, 5 Feb 2003 09:09:58 +0100 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: References: <20030204083215.GA2070@tagseth-trd.consultit.no> Message-ID: <20030205080958.GB2070@tagseth-trd.consultit.no> * Kevin Jacobs [030204 13:53]: > On Tue, 4 Feb 2003, Eivind Tagseth wrote: > > I started out automatically generating source code for data objects based > > on db_row and the actual postgresql-instance. > > ;) > > A happy db_row user? > > How is it working out for you? Working out just fine, thank you! Eivind From duaneh@connexus.net.au Wed Feb 5 09:15:11 2003 From: duaneh@connexus.net.au (Duane Hennessy) Date: Wed, 5 Feb 2003 20:15:11 +1100 Subject: [DB-SIG] BSDDB What now! Message-ID: <002601c2ccf7$1b878590$ef59decb@DUANE01> Hi all, I've decided to use BSDDB to collect data from a lottery application. Saving the data is the problem though. I downloaded the information from SleepyCat Software which gave me the methods for C programs but not Python. Dir'ing the Python module prints '['__doc__', '__file__', '__name__', 'btopen', 'error', 'hashopen', 'rnopen']' with methods to open, read and navigate an existing database but nothing on creating a database, creating records etc. Does anybody know where I can find the information to use BSDDB with Python? Duane Hennessy. Victoria, Australia. From magnus@thinkware.se Wed Feb 5 10:30:04 2003 From: magnus@thinkware.se (Magnus Lycka) Date: Wed, 05 Feb 2003 11:30:04 +0100 Subject: [DB-SIG] BSDDB What now! In-Reply-To: <002601c2ccf7$1b878590$ef59decb@DUANE01> Message-ID: <5.1.0.14.0.20030205111947.02ce63f0@www.thinkware.se> At 20:15 2003-02-05 +1100, Duane Hennessy wrote: > Does anybody know where I can find the information to use BSDDB with >Python? See http://www.python.org/doc/current/lib/module-bsddb.html and http://www.python.org/doc/current/lib/module-anydbm.html In brief you use it as a python dictionary. You have to use a special constructor where you supply the file name and maybe some other parameters, instead of just doing db = {} as you would with a dict. After that you basically treat it as a dictionary, and your data will magically persist. The database will be created when you open it the first time. There is nothing like a database schema or anything like that. A simple db[key] = value will create/update a record. I have a little warning, although it might not be an issue with recent versions: If you open a database and don't store any data, an empty file will be created. Next time you try to open it, you will get an error, since it's not recognized as a database file. But maybe that's fixed by now. It's been some time since I used it. -- Magnus Lycka, Thinkware AB Alvans vag 99, SE-907 50 UMEA, SWEDEN phone: int+46 70 582 80 65, fax: int+46 70 612 80 65 http://www.thinkware.se/ mailto:magnus@thinkware.se From lkcl@samba-tng.org Wed Feb 5 10:03:13 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Wed, 5 Feb 2003 10:03:13 +0000 Subject: [DB-SIG] SQL bridging In-Reply-To: <1044427153.1127.834.camel@lothlorien> References: <20030203204214.GB7645@localhost> <3EC26C81-37CD-11D7-ACF1-000393C2D67E@colorstudy.com> <20030204113623.GL7645@localhost> <1044427153.1127.834.camel@lothlorien> Message-ID: <20030205100312.GD1229@localhost> On Wed, Feb 05, 2003 at 12:39:13AM -0600, Ian Bicking wrote: > > ian, what are you proposing that pyro be used for? > > > > as a gateway to run a local python ODBC client on the MS-SQL server > > that then passes the results over, with pyro doing all the > > communication and stuff? > > Yes, you'd have a small program on the MS-SQL server that would create a > connection. That connection would be exported as a remote object, and > that would be it. There wouldn't be any layer between the client and > the database connection (except the remote object system, but that would > hopefully seem fairly transparent). for the instance where i didn't have control over the MS-SQL server and for political reasons there was a cat in hell's chance of asking them to install a program, i had to plump for the alternative method - match what they specified, and leave it at that. i was lucky enough to be able to ask them to switch on the XML interface. ODBC bridges _were_ available at the time, you see (commercially?) if speed ever became an issue (it didn't) at production time (which didn't happen) then yes, the bridging technique would have been invaluable. if i remember correctly, there were some such bridging projects already under development or listed on parnassus at the time... (18 months ago). l. From lkcl@samba-tng.org Wed Feb 5 15:34:36 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Wed, 5 Feb 2003 15:34:36 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: References: <20030203161732.GX912@localhost> Message-ID: <20030205153436.GF3609@localhost> On Mon, Feb 03, 2003 at 04:31:28PM -0500, Kevin Jacobs wrote: > On Mon, 3 Feb 2003, Luke Kenneth Casson Leighton wrote: > > my goal ultimately is to be able to write a file, maybe in > > XML or other suitable text file format, that specifies all the > > reports i want, that they are just tabular display of records, > > generation of an input form, or generation of create / delete > > input form. > > We've thought about that, but had much more luck rendering reports via > customer code with many helper objects and functions. e.g., we have a class > that emits HTML tables with pluggable data formatters so that our report > code remains uncluttered. i have started the process of moving away from actually having the HTML _in_ the python code, and using a package to template the HTML (apt-get install python-htmltmpl). what are your thoughts on such a move? personally i am not very impressed with python-htmltmpl it smacks of phpisms which always makes me shudder. is there anything better out there? > We have very powerful data aggregation functions > to slice-and-dice information in our OLAP-type reports. We use CSS style > sheets extensively so that our rendering code is abstract from many of the > details of the final rendered output. ah ha! CSS style sheets. i always get confused by style sheets, and always mean to investigate them properly. netscape, which doesn't print properly it ignores the damn stylesheet (!) put me off. > We even handle the difference between > 'screen' and 'printed' formatting via style sheets, so that a single report > renders very differently depending on where you send it. coool. do you have any advice on how to break up into pages because i need to add a "catalogue" / "brochure" and i'm running into lack of knowledge as to how to produce a mozilla-compatible HTML page that will print on the right A4 page boundaries. > > > 2) Decode the variable names and map them to schema elements > > > 3) Validate the variable values against the schema and business logic > > > 4) Group variables into transactional batches and sub-batches > > > based on the physical schema > > > 5) Render and execute SQL to perform updates > > > > > > what do you define as schema elements? > > i think i can work out what you mean by schema by context. > > Schema elements are the relations and columns of your database backend. > Those elements have type, unique key, foreign key, domain, and business > logic constraints. okay. i get it. terminology. okay. as i understand it, most of that can be specified _in_ the sql file, if you use a good enough sql server. e.g. at present i am relying on the ALTER TABLE FOREIGN KEY syntax, even though MySQL doesn't support that, to auto-generate code for JOINs and LEFT JOINs between tables. this Plan doesn't quite cover everything: - it doesn't cover reports although if MySQL supported VIEWs or even supported them but threw them away i WOULD look at specifying reports as VIEWs. - weird validation logic such as mentioned in one example: what to _do_ on errors in certain fields (red, flashing). i described how to deal with this in more detail yesterday, it can be bolted in, exactly as i believe you are already using. > > assuming that code to do all the above is auto-generated from > > either a SQL file (with CREATE and ALTER TABLE statements > > including FOREIGN KEY statements from which JOINs can be > > auto-generated) or from a separate IDL file, then code can > > be generated for the following: > > Why not get them from the database at runtime? > It is easy to query the > types, and only slightly harder to extract the unique and foreign keys. ... now why didn't i think of that? :) the only thing is that MySQL doesn't support foreign key constraints so those would be lost. > However, your forms and reports will end up being very bland and unfriendly > if you use a one-size-fits-all approach to generating them. i realise this. the Custom db/custom/html.py class (not the best place to have put this code i know) presently supports horizontal and vertical table and form display and entry. also i haven't put it in properly yet (path names hard-coded) but i am starting a move towards HTML-templates so you can specify code fragments, i mean it REALLY couldn't be simpler: from htmltmpl import TemplateManager, TemplateProcessor # Compile or load already precompiled template. template = TemplateManager().prepare("template.tmpl") tproc = TemplateProcessor() # Set the title. tproc.set("title", "Our customers") # Create the 'Customers' loop. customers = [] # First customer. customer = {} customer["name"] = "Joe Sixpack" customer["city"] = "Los Angeles" customer["new"] = 0 customers.append(customer) # Second customer. customer = {} customer["name"] = "Paul Newman" customer["city"] = "New York" customer["new"] = 1 customers.append(customer) tproc.set("Customers", customers) # Print the processed template. print tproc.process(template) your HTML template, named (dur template.tmpl) must contain a tag which will (dur) get substituted with the text. it's also possible to do loops, where the variable of the loop is expected to be None or a list of dictionaries. wow, hey what a coincidence, what does fetchall() return? :) :) > I've looked at some of your code from "Custom" and it looks like a good > start, though very verbose. I'm sure you're developing some more powerful > abstraction interfaces in "pysqldb", though I haven't had a chance to look. pysqldb contains the base classes i created that are equivalent to SQLbuilder and SQLObject it's not as complete as SQLObject although SQLObject i notice doesn't handle LEFT JOINs. pysqldb doesn't make it easy to do LEFT JOINs but it does cope: in any SELECT statement containing a list of tables, if any table starts with the text "LEFT JOIN " then it is handled differently from the other tables. tables=['people', 'addresses', """LEFT JOIN account_attributes ON account_attributes.person_id = people.id"""] will generate: SELECT * FROM people , addresses LEFT JOIN account_attributes ON account_attributes.person_id = people.id rather than: v SELECT * FROM people , addresses , ^ LEFT JOIN account_attributes ON account_attributes.person_id = people.id where the end comma after addresses would obviously cause a SQL syntax error. stripping out any table name with LEFT JOIN in it into a separate list and processing them afterwards seems to do the trick: ljlist = filter(lambda x: x[:9] == 'LEFT JOIN', tablelist) tblist = filter(lambda x: x[:9] != 'LEFT JOIN', tablelist) from_stmt = ", ".join(tblist) + "\n" + "\n".join(ljlist) [i realise there are better ways to do the above 3 lines] also i notice you don't support the INSERT INTO ... SELECT syntax (i just had to add that into pysqldb last night due to some WEIRD requirements...) > I'm still working on releasing some of my code so that I can show you how we > handle some of these cases. great! > Unfortunately, we have some legally encumbered > libraries that I have to remove with a chain-saw and then replace with > clean-room code before I can do that. Soon though... hey, we're not fussy. cvs commit it, fix it later _that's_ my motto :) if it helps with the decision process, usually the first open source project to offer needed functionality is the one that becomes the de-facto standard. kick up a political fuss about how some nerd named luke on the db-sig mailing lists is talking about writing alternate code with _his_ name all over it not your company's name, that should melt a few objections :) open source code is free advertising - there's money to be lost by not getting your code out first! l. From eivindt-db-sig@multinet.no Wed Feb 5 15:48:35 2003 From: eivindt-db-sig@multinet.no (Eivind Tagseth) Date: Wed, 5 Feb 2003 16:48:35 +0100 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030205153436.GF3609@localhost> References: <20030203161732.GX912@localhost> <20030205153436.GF3609@localhost> Message-ID: <20030205154835.GC2070@tagseth-trd.consultit.no> * Luke Kenneth Casson Leighton [030205 15:34]: > i have started the process of moving away from actually having > the HTML _in_ the python code, and using a package to template > the HTML (apt-get install python-htmltmpl). > > what are your thoughts on such a move? IMO, it's the only way to go, anything else is absolutely horrible. > personally i am not very impressed with python-htmltmpl it smacks > of phpisms which always makes me shudder. > is there anything better out there? Yes, Cheetah. I've been using it for a while, and it's got what I want from a template engine and achieves a much better separation between logic and presentation than e.g. php and jsp. And while it's not python, it feels quite pythonish... http://cheetahtemplate.org/ Check it out... Eivind From jacobs@penguin.theopalgroup.com Wed Feb 5 16:31:02 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Wed, 5 Feb 2003 11:31:02 -0500 (EST) Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030205153436.GF3609@localhost> Message-ID: On Wed, 5 Feb 2003, Luke Kenneth Casson Leighton wrote: > i have started the process of moving away from actually having > the HTML _in_ the python code, and using a package to template > the HTML (apt-get install python-htmltmpl). > > what are your thoughts on such a move? Our system already does this. Here is some simplified code: # HC == HeaderCell headerFormat = [ HC('Market'), HC('Customer'), HC('PONum'), HC('Quantity'), HC('Shipper'), HC('Date') ] # TC == TextCell, IC = IntegerCell rowFormat = [ TC('Market'), TC('Customer'), TC('PONum'), IC('Quantity'), TC('Shipper'), TC('Date') ] headerData = [ 'Market', 'Customer', 'Purchase Order #', 'Quantity', 'Shipping Location', 'Date Received' ] table = HTMLTable(outputfile) table.table_begin('report') table.add_header('report-header', headerFormat, headerData) sum = 0 for order in orders: rowData = [ order.market, order.Customer, order.PurchaseOrder, order.Quantity, order.Shipper, order.InvoiceReceived ] table.add_row('report-row', rowFormat, rowData) sum += order.Quantity totalData = [ 'Total:', '', '', sum, '', '' ] table.add_row('report-total', rowFormat, totalData) table.table_end() [Note that the first arguments to table_begin, add_header, and add_row are CSS classes used to style the output] > > We have very powerful data aggregation functions > > to slice-and-dice information in our OLAP-type reports. We use CSS style > > sheets extensively so that our rendering code is abstract from many of the > > details of the final rendered output. > > ah ha! CSS style sheets. > > i always get confused by style sheets, and always mean to > investigate them properly. > > netscape, which doesn't print properly it ignores the damn stylesheet > (!) put me off. Netscape as in Netscape 4? Who cares... MSIE >5, Mozilla and Netscape 6/7 are very good with stylesheets. Also, I just happen to be the author of SelectORacle, a tool to help explain CSS stylesheet selectors. See http://selectoracle.theopalgroup.com/selectoracle/ > do you have any advice on how to break up into pages because > i need to add a "catalogue" / "brochure" and i'm running into > lack of knowledge as to how to produce a mozilla-compatible > HTML page that will print on the right A4 page boundaries. It is _very_ hard to do page boundries in (X)HTML correctly. We've had to resort to browser-specific hacks to do it, and even those don't work well. > as i understand it, most of that can be specified _in_ the > sql file, if you use a good enough sql server. We like to query metadata on the fly -- after all, you can't always control the systems you are interfacing with. > # Create the 'Customers' loop. > customers = [] > > # First customer. > customer = {} > customer["name"] = "Joe Sixpack" > customer["city"] = "Los Angeles" > customer["new"] = 0 > customers.append(customer) > > # Second customer. > customer = {} > customer["name"] = "Paul Newman" > customer["city"] = "New York" > customer["new"] = 1 > customers.append(customer) Yech -- check out db_row. http://opensource.theopalgroup.com/ Very busy, more soon, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From brian@dorseys.org Wed Feb 5 17:12:43 2003 From: brian@dorseys.org (Brian Dorsey) Date: Wed, 5 Feb 2003 09:12:43 -0800 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030205154835.GC2070@tagseth-trd.consultit.no> References: <20030203161732.GX912@localhost> <20030205153436.GF3609@localhost> <20030205154835.GC2070@tagseth-trd.consultit.no> Message-ID: <20030205171243.GA574@dorseys.org> I'll throw in my recommendation for Cheetah as well. I've only used it on a few simple projects, but it was a snap. Their mailing list is friendly and helpful as well. And Cheetah is written completly in Python. Take care, -Brian On Wed, Feb 05, 2003 at 04:48:35PM +0100, Eivind Tagseth wrote: > * Luke Kenneth Casson Leighton [030205 15:34]: > > i have started the process of moving away from actually having > > the HTML _in_ the python code, and using a package to template > > the HTML (apt-get install python-htmltmpl). > > > > what are your thoughts on such a move? > > IMO, it's the only way to go, anything else is absolutely horrible. > > > personally i am not very impressed with python-htmltmpl it smacks > > of phpisms which always makes me shudder. > > > is there anything better out there? > > Yes, Cheetah. I've been using it for a while, and it's got what I want > from a template engine and achieves a much better separation between > logic and presentation than e.g. php and jsp. And while it's not > python, it feels quite pythonish... > > http://cheetahtemplate.org/ > > Check it out... > > > > > Eivind > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig From lkcl@samba-tng.org Wed Feb 5 17:48:19 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Wed, 5 Feb 2003 17:48:19 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: References: <20030205153436.GF3609@localhost> Message-ID: <20030205174819.GD4256@localhost> On Wed, Feb 05, 2003 at 11:31:02AM -0500, Kevin Jacobs wrote: > On Wed, 5 Feb 2003, Luke Kenneth Casson Leighton wrote: > > i have started the process of moving away from actually having > > the HTML _in_ the python code, and using a package to template > > the HTML (apt-get install python-htmltmpl). > > > > what are your thoughts on such a move? > > Our system already does this. Here is some simplified code: > > # HC == HeaderCell > headerFormat = [ HC('Market'), HC('Customer'), HC('PONum'), > HC('Quantity'), HC('Shipper'), HC('Date') ] > > # TC == TextCell, IC = IntegerCell > rowFormat = [ TC('Market'), TC('Customer'), TC('PONum'), > IC('Quantity'), TC('Shipper'), TC('Date') ] okay, so... the classes HC, TC and IC specify formatting, and are applied to each of the data rows? > headerData = [ 'Market', > 'Customer', > 'Purchase Order #', > 'Quantity', > 'Shipping Location', > 'Date Received' ] > > table = HTMLTable(outputfile) > > table.table_begin('report') > > table.add_header('report-header', headerFormat, headerData) > > sum = 0 > for order in orders: > rowData = [ order.market, order.Customer, order.PurchaseOrder, order.Quantity, > order.Shipper, order.InvoiceReceived ] > table.add_row('report-row', rowFormat, rowData) > sum += order.Quantity > > totalData = [ 'Total:', '', '', sum, '', '' ] > table.add_row('report-total', rowFormat, totalData) > > table.table_end() > > [Note that the first arguments to table_begin, add_header, and add_row are > CSS classes used to style the output] ack! step 1, get data: ---------------- i went for a less object-orientated approach (no table class) and i also combined the header and database field names into tuples. the reason for this is simply that i find generating new reports to be much simpler: cut/paste one line from one report to the next and you get the database field, heading name and display function all at once. the approach above that you have taken you would need to cut/paste in three or more separate places, running the risk of not putting the right header name with the field. def display_purchase_details(self): title = self.iso.report_yourselection() # obtain a database cursor with a fetchone() method cur = self.dpydb.purchases_join_products_id_start(cust_id=self.cust_id, transaction_id=self.order_id) # list of fields as follows: # (database_field for a dict from fetchone(), # header information, # optional function to display value) fields = [ ("products_description", self.iso.report_proddetails(), None), ("products_item_size", "Size", None), ("purchases_cost", self.iso.report_cost(), purchase_cost_disp_fn), ("purchases_quantity", self.iso.report_quantity(), None), ("purchases_cost", "Total", purchase_total_disp_fn) ] return self.iso.display_table( cur, html_action, title, fields) the reason for using self.iso.report_xxxx() functions is because i have some dynamic stuff on a per-customer language basis that over-rides the creation of the self.iso variable. i haven't kept this up-to-date entirely in the app i'm doing because it's UK only. the display_table() function is where all the magic occurs to do output. html_action is a function that is used to obtain results from the cur and do an optional argument to display_table is a row-displaying function. it all gets a bit higher-order-function-hair-raising and stuff. step 2, display data: -------------------- this is where i lose some of the flexibility that you have, in creating tables. i have to say that i _did_ consider doing classes (like you have already) but i wanted to strip out ALL evidence of HTML from the application so that it becomeso possible to do GUI and curses stuff. okay. having received a cursor (and processed them with the higher order yukky functions into a list of rows and HTMLling them on the way), display_table (and a couple of other functions which do different formatting styles) then calls display_table_template. display_table also takes the list of tuples (parsing twice) to split out the header field (creates row_header below) which is then ignored in the row list creation (rows below) def display_table_template(self, heading, row_heading, rows, r1, border, cellpadding) : fname = "/var/custom/Example/tmpl/table.html" template = TemplateManager().prepare(fname) tproc = TemplateProcessor() tproc.set("border", str(border)) tproc.set("cellpadding", str(cellpadding)) tproc.set("rows", rows) if heading: tproc.set("heading", heading) if heading is not None: tproc.set("row_header", row_heading) return tproc.process(template) this is where i could benefit from style sheets, i could get rid of the border arg etc. as i've only just been experimenting with htmltmpl it's hard-coded. so what's the point of mentioning this stuff? The Plan is to place in a template (xml, sql, object, whatever): - a list of the field names etc. - a list of the optional function names (or is it reasonable to instead have a class which is expected to have display_fieldname_in_html()? - the descriptive heading. - specifications on what to do with the data, how to display it. clearly if you want to do GTK or other GUI applications you use a different template, it specifies do-other-base-class-to-display, etc. l. From jacobs@penguin.theopalgroup.com Wed Feb 5 18:04:11 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Wed, 5 Feb 2003 13:04:11 -0500 (EST) Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030205174819.GD4256@localhost> Message-ID: On Wed, 5 Feb 2003, Luke Kenneth Casson Leighton wrote: > On Wed, Feb 05, 2003 at 11:31:02AM -0500, Kevin Jacobs wrote: > > Our system already does this. Here is some simplified code: > > > > # HC == HeaderCell > > headerFormat = [ HC('Market'), HC('Customer'), HC('PONum'), > > HC('Quantity'), HC('Shipper'), HC('Date') ] > > > > # TC == TextCell, IC = IntegerCell > > rowFormat = [ TC('Market'), TC('Customer'), TC('PONum'), > > IC('Quantity'), TC('Shipper'), TC('Date') ] > > okay, so... the classes HC, TC and IC specify formatting, > and are applied to each of the data rows? They are applied to each cell, and can perform arbitrary functions like adding CSS classes, converting to localized formats, etc. > step 1, get data: > ---------------- > > i went for a less object-orientated approach (no table class) > and i also combined the header and database field names into > tuples. > > the reason for this is simply that i find generating new reports > to be much simpler: cut/paste one line from one report to the > next and you get the database field, heading name and display > function all at once. Many of our reports require formatting, rollups, data dependent logic, URL embedding, etc. > the approach above that you have taken you would need to > cut/paste in three or more separate places, running the risk > of not putting the right header name with the field. The table renderer makes sure that your formats match your data, so it is easy to detect problems like that. In practice, we have hundreds of reports and never have a problem with keeping track of fields. We also have reports that have very complex multi-line headers with spans and text labels that do not match 1:1 with the data. We also have reports that have many different row formats intermixed within them. > step 2, display data: > -------------------- > > this is where i lose some of the flexibility that you have, > in creating tables. i have to say that i _did_ consider doing > classes (like you have already) but i wanted to strip out > ALL evidence of HTML from the application so that it becomeso > possible to do GUI and curses stuff. You didn't see any HTML in our version, did you? We have a wxWindows and ReportLab renderers that work very well. The only tricky part was imlementing our own engine to apply CSS stylesheets for each backend. > so what's the point of mentioning this stuff? > > The Plan is to place in a template (xml, sql, object, whatever): > > - a list of the field names etc. > - a list of the optional function names > (or is it reasonable to instead have a class which is > expected to have display_fieldname_in_html()? > - the descriptive heading. > - specifications on what to do with the data, how to display it. > > clearly if you want to do GTK or other GUI applications you use a > different template, it specifies do-other-base-class-to-display, > etc. Again, this presupposes that you have: 1) Tables with a single, simple header row 2) All rows are formatted homogeneously 3) No rollups, subtotals, or any other forms of aggregation or delimiters. 4) No non-trivially data-dependent formats 5) etc...etc...etc.. ;) I'm not trying to change your mind, but our clients need all of the above (and more!). -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From lkcl@samba-tng.org Wed Feb 5 21:45:51 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Wed, 5 Feb 2003 21:45:51 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: References: <20030205174819.GD4256@localhost> Message-ID: <20030205214551.GH4256@localhost> On Wed, Feb 05, 2003 at 01:04:11PM -0500, Kevin Jacobs wrote: > > okay, so... the classes HC, TC and IC specify formatting, > > and are applied to each of the data rows? > > They are applied to each cell, and can perform arbitrary functions like > adding CSS classes, converting to localized formats, etc. ack! > > step 1, get data: > > ---------------- > > > > i went for a less object-orientated approach (no table class) > > and i also combined the header and database field names into > > tuples. > > > > the reason for this is simply that i find generating new reports > > to be much simpler: cut/paste one line from one report to the > > next and you get the database field, heading name and display > > function all at once. > > Many of our reports require formatting, rollups, data dependent logic, URL > embedding, etc. url embedding i do as a value-specific formatting function. if the function has three arguments the _entire_ dictionary (of cur.fetchone()) is passed to it as arg1; the field name as arg2; and for good luck str(thevalue) i don't know why i did this arg3! there is _one_ instance where i had to do something weird: transposition of an entire table. that _was_ a pain and i created a class with a fetchone() function on it. it was for VAT pricing / totals. the two lists were Quantity and Cost. > > the approach above that you have taken you would need to > > cut/paste in three or more separate places, running the risk > > of not putting the right header name with the field. > > The table renderer makes sure that your formats match your data, so it is > easy to detect problems like that. hmmmm :) easy to detect problem. hmmm. easier to avoid problem in first place, neh? :) > In practice, we have hundreds of reports > and never have a problem with keeping track of fields. once written, > We also have reports > that have very complex multi-line headers with spans and text labels that do > not match 1:1 with the data. We also have reports that have many different > row formats intermixed within them. yeh, it hadn't occurred to me that anyone would want to use radically different row formats. i _did_ put in some code that would allow me to format the ENTIRE entry thereby making it possible to set the colspan, etc. i guess... *thinks*... if i split display_table() down into its basic components... > > > step 2, display data: > > -------------------- > > > > this is where i lose some of the flexibility that you have, > > in creating tables. i have to say that i _did_ consider doing > > classes (like you have already) but i wanted to strip out > > ALL evidence of HTML from the application so that it becomeso > > possible to do GUI and curses stuff. > > You didn't see any HTML in our version, did you? true .... except in the name :) > We have a wxWindows and > ReportLab renderers that work very well. The only tricky part was > imlementing our own engine to apply CSS stylesheets for each backend. oo, sounds really good! > > The Plan is to place in a template (xml, sql, object, whatever): > > > > - a list of the field names etc. > > - a list of the optional function names > > (or is it reasonable to instead have a class which is > > expected to have display_fieldname_in_html()? > > - the descriptive heading. > > - specifications on what to do with the data, how to display it. > > > > clearly if you want to do GTK or other GUI applications you use a > > different template, it specifies do-other-base-class-to-display, > > etc. > > Again, this presupposes that you have: > > 1) Tables with a single, simple header row ah, not necessarily... double-checking... yep, you're right: the custom/html.py display_table_header() function does a single header row - not necessarily simple, but definitely single. > 2) All rows are formatted homogeneously again, like i said: the formatting function could output a , i do have that possibility covered. > 3) No rollups, subtotals, or any other forms of aggregation or delimiters. > 4) No non-trivially data-dependent formats that _can_ be covered by: - pre-processing the data and returning an alternative object on which fetchone() can be called to return its data row by row. - passing in a different display_row_fn - doing a separate table for each different data set, with row-headers switched off. one thing i haven't mentioned is that quite often i do a row where the value display function does yet another database query, so the entry is actually another table (yes i _do_ know how long netscape / mozilla takes to display tables-in-tables). or an