From daniel.dittmar at sap.com Fri Apr 1 11:17:33 2005 From: daniel.dittmar at sap.com (Dittmar, Daniel) Date: Fri Apr 1 11:17:37 2005 Subject: [DB-SIG] URI syntax for databases Message-ID: >> An alternative would be to always use uri_connect. In >addition, a set of >> modules is provided that implements this uri translation for older >> drivers. The user would then have to choose between the real >module or >> the translation module. > >That seems awkward, but I suppose doable. Importing everything from another module (including _*) is not that difficult. Having a separate module would also be an advantage if the driver author implements uri_connect, but decides to require a different uri syntax than the implementation in the abstraction layer. This would invalidate all URIs in configuration files etc. Not sure if this would be a real world problem. > That translating module would >also have to include the rest of the required functions if it >was to be >a DB-API module. What other module content do you need in addition to the connect (or uri_connect)? I found it always cumbersome that code needs the module in addition to the connection object. I think there exists already the extension to the DB API that exception classes should be accessible through the connection object. Maybe everything else should be accessible as well? Daniel -- Daniel Dittmar SAP Labs Berlin daniel.dittmar@sap.com From ianb at colorstudy.com Fri Apr 1 22:22:39 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri Apr 1 22:24:23 2005 Subject: [DB-SIG] [SQLObject] Re: SQLite connection - relative filename In-Reply-To: References: <1112376400.28418.216.camel@dell.datacomm.albany.edu> Message-ID: <424DAD8F.10005@colorstudy.com> Paul Moore wrote: > Maybe _parseURI should use urllib.url2pathname? That would still be > mildly suboptimal, as url2pathname doesn't accept the common form > '///C:/a/b/...'. But that's a stdlib issue. Incidentally, while we're on the topic of URIs, I'd like to move all this URI parsing stuff into a library that doesn't have SQLObject dependencies, so that other people can use the same syntax for defining DB-API connections in any system. Since there's all sort of annoying aspects to URI parsing (many of which probably would have been avoided if I had used urllib from the start :( ), it seems like such a library would be useful. Such a library would probably produce a connection function and keyword parameters, so that actual connection creation could be held off until after parsing, i.e.: # in dburi.py: def uri_connect(uri): func, args, kwargs = uri_parse('...') return func(*args, **kw) module_aliases = { 'pgsql': 'psycopg', 'mysql': 'MySQLdb', # ... } def uri_parse(uri): module_name = uri.split(':', 1) module_name = module_aliases.get(module_name.lower(), module_name) module = __import__(module_name) if hasattr(module, 'uri_parse'): return module.uri_parse(uri) elif globals().has_key('%s_uri_parse' % module_name): return globals()['%s_uri_parse' % module_name](uri, module) else: return generic_uri_parse(uri, module) Then we'll start implementing parsers for backward compatibility, but perhaps upstream driver authors can start including their own uri_parse functions. Well, the one missing part of this is that it would also be nice to construct a URI from an already-existing connection object, and to store the original URI if that's how that connection object was originally created. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From Chris.Clark at ca.com Sat Apr 2 00:19:55 2005 From: Chris.Clark at ca.com (Clark, Chris M) Date: Sat Apr 2 00:19:59 2005 Subject: [DB-SIG] [SQLObject] Re: SQLite connection - relative filename Message-ID: <5F0EF692A76C0F4D970C00F5D5B3445FA9E4F9@usilms28.ca.com> > -----Original Message----- > From: db-sig-bounces@python.org > [mailto:db-sig-bounces@python.org] On Behalf Of Ian Bicking > Sent: Friday, April 01, 2005 12:23 PM > To: Paul Moore > Cc: sqlobject-discuss@lists.sourceforge.net; Python DB-SIG > Mailing List > Subject: [DB-SIG] [SQLObject] Re: SQLite connection - > relative filename > > Paul Moore wrote: > > Maybe _parseURI should use urllib.url2pathname? That would still be > > mildly suboptimal, as url2pathname doesn't accept the common form > > '///C:/a/b/...'. But that's a stdlib issue. > > Incidentally, while we're on the topic of URIs, I'd like to > move all this URI parsing stuff into a library that doesn't > have SQLObject dependencies, so that other people can use the > same syntax for defining DB-API connections in any system. > Since there's all sort of annoying aspects to URI parsing > (many of which probably would have been avoided if I had used > urllib from the start :( ), it seems like such a library > would be useful. Such a library would probably produce a > connection function and keyword parameters, so that actual > connection creation could be held off until after parsing, i.e.: > > ..... > def uri_parse(uri): > module_name = uri.split(':', 1) ..... My 2 cents: for parsing the URI I like semi-colons ';' as separators, it is a little ODBC like but it has less collisions than colons do. Chris From ianb at colorstudy.com Sat Apr 2 00:37:44 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sat Apr 2 00:39:42 2005 Subject: [DB-SIG] [SQLObject] Re: SQLite connection - relative filename In-Reply-To: <5F0EF692A76C0F4D970C00F5D5B3445FA9E4F9@usilms28.ca.com> References: <5F0EF692A76C0F4D970C00F5D5B3445FA9E4F9@usilms28.ca.com> Message-ID: <424DCD38.1060504@colorstudy.com> Clark, Chris M wrote: >>def uri_parse(uri): >> module_name = uri.split(':', 1) > > ..... > > > My 2 cents: for parsing the URI I like semi-colons ';' as separators, it > is a little ODBC like but it has less collisions than colons do. This particular colon is for indicating the database type, like mysql:..., or pgsql:... -- it's what makes something a URI (instead of a DSN... though I'm even sure what DSN stands for). After that it's potentially entirely up to the database driver, though of course some uniformity is nice. Typical DSNs, from what I can tell, are parsed in an up-front manner into a set of keys and values; though maybe they sometimes include a prefix indicating database type...? Java has database URIs too, don't they? But they are all like jdbc:..., correct? Of course, following that pattern you could get dbapi:jdbc:postgresql://... which feels awfully weird to me. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From daniel.dittmar at sap.com Mon Apr 4 12:11:04 2005 From: daniel.dittmar at sap.com (Dittmar, Daniel) Date: Mon Apr 4 12:11:09 2005 Subject: [DB-SIG] URI syntax for database (was: [SQLObject] Re: SQLite connection - relative filename) Message-ID: >From: db-sig-bounces@python.org >[mailto:db-sig-bounces@python.org] On Behalf Of Ian Bicking > After that it's >potentially entirely up to the database driver, though of course some >uniformity is nice. Typical DSNs, from what I can tell, are parsed in >an up-front manner into a set of keys and values; though maybe they >sometimes include a prefix indicating database type...? DSNs (Data Source Names) are registry keys. The entry for that key contains among other things the id of the ODBC driver. The string for a connect contains the DSN (which is evaluated by the driver manager) and additional connect properties (which are passed to the driver). This at least was the general idea, which has probably been bended since. > Java has >database URIs too, don't they? But they are all like >jdbc:..., correct? Right > Of course, following that pattern you could get >dbapi:jdbc:postgresql://... which feels awfully weird to me. a) I'd prefer pydbapi: as a prefix. That way, someone looking at a configuration file could at least get the notion that any problems might be related to some Python lib. b) Using a module path instead of a DBMS product name is generally preferable (and seemed to be the consent so far). With the JDBC scheme, all relevant drivers have to be loaded into memory first, the diver manager then asks each if the driver would accept a specific URI. c) Should username and password be used similar to other protocols? pydbapi:://[:@][: / Daniel Dittmar -- Daniel Dittmar SAP Labs Berlin daniel.dittmar@sap.com From mal at egenix.com Mon Apr 4 12:59:40 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Mon Apr 4 12:59:42 2005 Subject: [DB-SIG] URI syntax for database (was: [SQLObject] Re: SQLite connection - relative filename) In-Reply-To: References: Message-ID: <42511E1C.2050804@egenix.com> Dittmar, Daniel wrote: >>From: db-sig-bounces@python.org >>[mailto:db-sig-bounces@python.org] On Behalf Of Ian Bicking >>After that it's >>potentially entirely up to the database driver, though of course some >>uniformity is nice. Typical DSNs, from what I can tell, are parsed in >>an up-front manner into a set of keys and values; though maybe they >>sometimes include a prefix indicating database type...? > > > DSNs (Data Source Names) are registry keys. The entry for that key > contains among other things the id of the ODBC driver. The string for a > connect contains the DSN (which is evaluated by the driver manager) and > additional connect properties (which are passed to the driver). This at > least was the general idea, which has probably been bended since. The format of ODBC connection strings (sometimes called DSN string or just DSN) goes like this: connection-string ::= empty-string[;] | attribute[;] | attribute; connection-string empty-string ::= attribute ::= attribute-keyword=attribute-value | DRIVER=[{]attribute-value[}] attribute-keyword ::= DSN | UID | PWD | driver-defined-attribute-keyword attribute-value ::= character-string driver-defined-attribute-keyword ::= identifier (taken from http://msdn.microsoft.com/library/default.asp?url=/library/en-us/odbc/htm/odbcsqldriverconnect.asp; scroll down to the comments section) At the very least you have to provide the DSN= part of the connection string. All other parts are optional. >>Java has >>database URIs too, don't they? But they are all like >>jdbc:..., correct? > > > Right > > >> Of course, following that pattern you could get >>dbapi:jdbc:postgresql://... which feels awfully weird to me. > > > a) I'd prefer pydbapi: as a prefix. That way, someone looking at a > configuration file could at least get the notion that any problems might > be related to some Python lib. > > b) Using a module path instead of a DBMS product name is generally > preferable (and seemed to be the consent so far). With the JDBC scheme, > all relevant drivers have to be loaded into memory first, the diver > manager then asks each if the driver would accept a specific URI. > > c) Should username and password be used similar to other protocols? > pydbapi:://[:@][: > / This is not URI conformant (AFAIK): you cannot have multiple schemes separated by colons. pydbapi-://... would work, e.g. pydbapi-mx.ODBC.Windows://:@sqlserver.example.com/?DATABASE=test Looking at this URI I don't think that the URI-approach is going to make things any easier for the user in setting up things. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 04 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From daniel.dittmar at sap.com Mon Apr 4 14:23:02 2005 From: daniel.dittmar at sap.com (Dittmar, Daniel) Date: Mon Apr 4 14:23:11 2005 Subject: [DB-SIG] URI syntax for database Message-ID: >pydbapi-mx.ODBC.Windows://:@sqlserver.example.com/?DA >TABASE=test > >Looking at this URI I don't think that the URI-approach is >going to make things any easier for the user in setting up >things. If I understood Ian right, he doesn't wants to improve on import dbmodule session = dbmodule.connect () He rather wants to have the possibility to store all the information for a connection in configuration files. Ideally, the controlling program shouldn't need any database driver specific code to create a connection object. The alternative was to call the connect method through keyword args. The controlling program could read the properties out of .ini file, out of XML files, parse a URI or whatever. The problem with that approach was that for some parameters, a string value wouldn't be accepted. So the controlling program would need a way to know which parameters require type conversion. The solution to this one could be that the DB API mandates that connect functions accept strings for all arguments. And until this is implemented, the controlling program has to implement this mapping for each database driver itself. The approach through keyword arguments could have the advantage that it is easier to add some property values at runtime, for example, if the password must be entered interactively. Inserting it into a URI would require some knowledge about the URI format (which is supposed to be database specific). Daniel Dittmar -- Daniel Dittmar SAP Labs Berlin daniel.dittmar@sap.com From wnvcole at peppermillcas.com Mon Apr 4 17:37:42 2005 From: wnvcole at peppermillcas.com (Vernon Cole) Date: Mon Apr 4 17:37:49 2005 Subject: [DB-SIG] URI syntax for database (was: [SQLObject] Re: SQLiteconnection - relative filename) Message-ID: <1DE30EB3ECE266409FDE6E7032F178C41398D1@pcimail1s.peppermillcas.com> Ummm... a little caution here. Don't get comfortable assuming that ODBC is a dependable measure of how to access a data base. As is typical of Micro$oft, and many other companies which have grown too large and self-important, they have changed the rules. ODBC, having become an industry quasi-standard, is now obsoleted by ADO, the latest in a long series of Microsoft "standards" for database access. One of the best things about ADO, I.M.H.O., is that it does NOT use DSNs or registry entries. The Python dbapi module for odbc on Windows(r) has not been maintained for years and is acknowledged as buggy. The alternative (sorry, Mark) is a commercial product and therefore not usable for many of us. Adodbapi is my solution of choice. --- Vernon Cole -----Original Message----- ... > DSNs (Data Source Names) are registry keys. The entry for that key > contains among other things the id of the ODBC driver. The string for a > connect contains the DSN (which is evaluated by the driver manager) and > additional connect properties (which are passed to the driver). This at > least was the general idea, which has probably been bended since. The format of ODBC connection strings (sometimes called DSN string or just DSN) goes like this: ... (taken from http://msdn.microsoft.com/library/default.asp?url=/library/en-us/odbc/ht m/odbcsqldriverconnect.asp; At the very least you have to provide the DSN= part of the connection string. All other parts are optional. From ianb at colorstudy.com Mon Apr 4 18:20:51 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon Apr 4 18:21:57 2005 Subject: [DB-SIG] URI syntax for database (was: [SQLObject] Re: SQLite connection - relative filename) In-Reply-To: References: Message-ID: <42516963.6050407@colorstudy.com> Dittmar, Daniel wrote: >> Of course, following that pattern you could get >>dbapi:jdbc:postgresql://... which feels awfully weird to me. > > > a) I'd prefer pydbapi: as a prefix. That way, someone looking at a > configuration file could at least get the notion that any problems might > be related to some Python lib. I guess... blah, feels like the namespacitus problem of Java, though. > b) Using a module path instead of a DBMS product name is generally > preferable (and seemed to be the consent so far). With the JDBC scheme, > all relevant drivers have to be loaded into memory first, the diver > manager then asks each if the driver would accept a specific URI. Yes, it's certainly a problem. For most the module name isn't that bad, though explaining why you have to use MySQLdb: instead of mysql: will be a little annoying, and certainly unsatisfying to the user. I'd feel compelled to create a mapping of aliases, even if I fell back on the module name. There exists the possibility of centrally registering aliases, that is short of importing modules and having them self-register. E.g. put a "db-api-aliases" directory somewhere on sys.path, with filenames like "mysql.txt" that contains "MySQLdb". Maybe multiple lines in a file if there are multiple implementors (and the connector can either bail with a helpful error message about options for being more explicit, or can prefer one driver over another). > c) Should username and password be used similar to other protocols? > pydbapi:://[:@][: > / That was my expectation. Any parameters could also be interpreted as well, like ?autocommit=t&datestyle=mxDateTime -- and so on, which leaves room for any parameters that would be useful. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon Apr 4 18:20:36 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon Apr 4 18:22:12 2005 Subject: [DB-SIG] URI syntax for database (was: [SQLObject] Re: SQLite connection - relative filename) In-Reply-To: <42511E1C.2050804@egenix.com> References: <42511E1C.2050804@egenix.com> Message-ID: <42516954.8010803@colorstudy.com> M.-A. Lemburg wrote: >>c) Should username and password be used similar to other protocols? >> pydbapi:://[:@][: >>/ > > > This is not URI conformant (AFAIK): you cannot have multiple > schemes separated by colons. > > pydbapi-://... Well, I think it's up to scheme to parse the rest of the URI, so the scheme would be "pydbapi", and it in turn would parse out the module. I think this would be more compliant than: > would work, e.g. > > pydbapi-mx.ODBC.Windows://:@sqlserver.example.com/?DATABASE=test Which really puts something more data-like into the scheme. But then I still prefer using a plain module name, like: mxODBC://user:password@host/database?params Of course, I do believe it's illegal to have punctuation in the scheme as well... well, I guess not, according to this: http://www.w3.org/Addressing/URL/5_URI_BNF.html While it's nice to be able to find modules given a URI (without preloading any modules) I must admit I am rather reluctant to burden users with what I consider to be internal organizations, like mx.ODBC.Windows, or a particular Postgres driver. > Looking at this URI I don't think that the URI-approach is > going to make things any easier for the user in setting up > things. Well, I can say that *I* find them very useful and comfortable when configuring SQLObject, even though I initially didn't think they were that important. I suspect that the effect would actually be magnified for someone with less experience using Python database drivers. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From farcepest at gmail.com Mon Apr 4 21:34:43 2005 From: farcepest at gmail.com (Andy Dustman) Date: Mon Apr 4 21:34:47 2005 Subject: [DB-SIG] URI syntax for database (was: [SQLObject] Re: SQLite connection - relative filename) In-Reply-To: <42516963.6050407@colorstudy.com> References: <42516963.6050407@colorstudy.com> Message-ID: <9826f38005040412347d5b4d0c@mail.gmail.com> I vote for: pydbapi-://[[:]]@[[:]]/[][?param1=...¶m2=...] Where: dbms: Name of the database server implementation (mysql, postgresql, oracle, etc.); map server implementation names to callables (factory functions) that should be passed a mapping of all the above parameters and return a DB API connection object. database: database name paramX: optional implementation-dependent parameters user, password, host, port, database: all obvious, standard meaning as in PEP-249 guidelines for connect() For MySQLdb, the factory function would look like this: def MySQLdb_Connection(kwargs): from MySQLdb import connect kwargs2 = kwargs.copy() try: del kwargs2['database'] kwargs2['db'] = kwargs['database'] except KeyError: pass try: del kwargs2['password'] kwargs2['passwd'] = kwargs['password'] except KeyError: pass return connect(**kwargs2) Of course, if you use db instead of database, and passwd instead of password, this reduces to: def MySQLdb_Connection(kwargs): from MySQLdb import connect return connect(**kwargs) The factory functions should be fairly trivial for most database modules. It might also be possible to (ab)use urllib2 by registering an Opener for each database of interest which returns an DB API connection object instead of a file-like object, and then using urllib2.urlopen() to open connections. But it's probably better to have a seperate module for this, i.e. dburllib.urlopen(url) -> connection. urlparse does a good bit of the work already: >>> import urlparse >>> urlparse.uses_netloc.append("pydbapi-mysql") >>> urlparse.uses_query.append("pydbapi-mysql") >>> urlparse.urlparse('pydbapi-mysql://user:password@host:port/database?compress=1') ('pydbapi-mysql', 'user:password@host:port', '/database', '', 'compress=1', '') >>> -- Computer interfaces should never be made of meat. Using GMail? Setting Reply-to address to <> disables this annoying feature. You are in a maze of twisty little passages all alike. To go north, press 2. To go west, press 4. To go east, press 6. To go south, press 8. If you need assistance, press 0 and a little dwarf will assist you. From ianb at colorstudy.com Mon Apr 4 22:56:02 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon Apr 4 22:57:27 2005 Subject: [DB-SIG] URI syntax for database In-Reply-To: <9826f38005040412347d5b4d0c@mail.gmail.com> References: <42516963.6050407@colorstudy.com> <9826f38005040412347d5b4d0c@mail.gmail.com> Message-ID: <4251A9E2.2070409@colorstudy.com> Andy Dustman wrote: > I vote for: > > pydbapi-://[[:]]@[[:]]/[][?param1=...¶m2=...] > > Where: > > dbms: Name of the database server implementation (mysql, postgresql, > oracle, etc.); map server implementation names to callables (factory > functions) that should be passed a mapping of all the above parameters > and return a DB API connection object. > > database: database name > > paramX: optional implementation-dependent parameters > > user, password, host, port, database: all obvious, standard meaning as > in PEP-249 guidelines for connect() The connection also should have some opportunity to parse the URI. At least for SQLite it won't be quite the same (though it actually would fit the pattern). How the rest gets parsed it kind of up to the driver. I think Firebird actually uses a combination of a full path and a hostname. Which fits fine too. Anyway, there's some diversity, though of course I'd always prefer as much consistency as possible. At the same time, it's important to me that I be able to parse a URI without actually connecting to the database. So while I'd like the database drivers to have the opportunity to do their own parsing, I'd also like to invoke the constructor manually (or at least be able to do so). Incidentally, I'd also like access to any extra parameters that the parser doesn't know about, so I can use them if I wish. I guess I'll try to code up an strawman library with this basic feel, since there has to be a library for backward compatibility anyway. > For MySQLdb, the factory function would look like this: > > def MySQLdb_Connection(kwargs): > from MySQLdb import connect > kwargs2 = kwargs.copy() > try: > del kwargs2['database'] > kwargs2['db'] = kwargs['database'] > except KeyError: > pass > try: > del kwargs2['password'] > kwargs2['passwd'] = kwargs['password'] > except KeyError: > pass > return connect(**kwargs2) > > Of course, if you use db instead of database, and passwd instead of > password, this reduces to: > > def MySQLdb_Connection(kwargs): > from MySQLdb import connect > return connect(**kwargs) > > The factory functions should be fairly trivial for most database modules. > > It might also be possible to (ab)use urllib2 by registering an Opener > for each database of interest which returns an DB API connection > object instead of a file-like object, and then using urllib2.urlopen() > to open connections. But it's probably better to have a seperate > module for this, i.e. dburllib.urlopen(url) -> connection. urlparse > does a good bit of the work already: > > >>>>import urlparse >>>>urlparse.uses_netloc.append("pydbapi-mysql") >>>>urlparse.uses_query.append("pydbapi-mysql") >>>>urlparse.urlparse('pydbapi-mysql://user:password@host:port/database?compress=1') > > ('pydbapi-mysql', 'user:password@host:port', '/database', '', 'compress=1', '') urllib also seems to have a lot of undocumented functions that do the necessary parsing. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From carsten at uniqsys.com Mon Apr 4 23:14:21 2005 From: carsten at uniqsys.com (Carsten Haese) Date: Mon Apr 4 23:14:25 2005 Subject: [DB-SIG] Re: Looking for Stephen Turner, maintainer of informixdb In-Reply-To: <42490c40$0$27184$9b4e6d93@newsread4.arcor-online.net> References: <42490c40$0$27184$9b4e6d93@newsread4.arcor-online.net> Message-ID: <1112649261.15320.13.camel@dot.uniqsys.com> On Tue, 2005-03-29 at 03:05, Michael Husmann wrote: > Carsten Haese wrote: > > > Hello everybody: > > > > I have discovered that the functionality for connecting Python to an > > Informix database is currently in a frustrating state of neglect. The > > link to Kinfxdb is dead, and informixdb doesn't build on Python 2. I > > couldn't find any usable workarounds to the build problem, so I worked > > out successfully how to build informixdb using distutils. > > > > Now I'd like to share the result with the community, but the maintainer > > appears to have gone missing. My emails to sjturner@ix.netcom.com and > > stepturn@mediaone.net have bounced back undeliverable, so now I'm > > posting to the lists trying to locate Stephen. > > > > If anybody has any pointers for locating Stephen Turner, please let me > > know. If Stephen can't be located, I'd be willing to take over the > > project, but I'd prefer the torch be given to me rather than me just > > taking it. > > Carsten, > > haven't heard anything from Stephen either, but there are some extensions to > the informixdb module. I'll send the sources to you next week. So if you > like you can bundle everything in one package. Michael, I'd certainly like to take a look at those extensions. (I tried to respond to you off-list, but my mail bounced back. Is this project cursed or am I? :) I hope you're reading this, Michael.) A week has gone by with no replies as to Stephen's whereabouts, and one reply telling me that I can, in good faith, go ahead and release a fork. Unless somebody tells me I should wait longer, I guess I'll do that. Stay tuned for the announcement on when and where my fork will be available. Best regards, -- Carsten Haese - Software Engineer | Phone: (419) 861-3331 Unique Systems, Inc. | FAX: (419) 861-3340 1446 Reynolds Rd, Suite 313 | Maumee, OH 43537 | mailto:carsten@uniqsys.com From mal at egenix.com Tue Apr 5 01:21:55 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Tue Apr 5 01:22:01 2005 Subject: [DB-SIG] URI syntax for database (was: [SQLObject] Re: SQLiteconnection - relative filename) In-Reply-To: <1DE30EB3ECE266409FDE6E7032F178C41398D1@pcimail1s.peppermillcas.com> References: <1DE30EB3ECE266409FDE6E7032F178C41398D1@pcimail1s.peppermillcas.com> Message-ID: <4251CC13.4020903@egenix.com> Vernon Cole wrote: > Ummm... a little caution here. Don't get comfortable assuming that ODBC > is a dependable measure of how to access a data base. As is typical > of Micro$oft, and many other companies which have grown too large and > self-important, they have changed the rules. ODBC, having become an > industry quasi-standard, is now obsoleted by ADO, the latest in a long > series of Microsoft "standards" for database access. One of the best > things about ADO, I.M.H.O., is that it does NOT use DSNs or registry > entries. > > The Python dbapi module for odbc on Windows(r) has not been maintained > for years and is acknowledged as buggy. The alternative (sorry, Mark) is > a commercial product and therefore not usable for many of us. Adodbapi > is my solution of choice. No need to be sorry - we have many happy customers :-) However, I don't really understand your comment: I was just providing the documentation for how an ODBC connection string is formatted. If you want to compare ODBC and ADO, I'd suggest to have a look at: IIS 5.0 Resource Guide: http://www.microsoft.com/technet/prodtechnol/windows2000serv/technologies/iis/reskit/iischp7.mspx Scroll down to table 7.1: """ Table 7.1 TPS (transactions per second) Per Number of Threads by MDAC Technology Threads 1 2 5 10 20 50 ODBC 66.37 146.28 350.46 626.76 900.24 859.91 OLE DB 67.30 141.92 326.19 590.57 794.91 715.78 OLE DB 2.0 61.73 126.93 297.29 506.75 575.35 526.61 ADO 2.0 51.24 108.12 240.91 377.30 361.26 310.34 """ The reason for these performance figures is that ADO is an interface on top of OLE DB which in return is an interface on top of ODBC (for many database backends such as e.g. SQL Server; see figure 7.1 in the above document). This is MS at its best: stacking levels of APIs on top of each other to correct their API designs ;-) > --- > Vernon Cole > > -----Original Message----- > ... > >>DSNs (Data Source Names) are registry keys. The entry for that key >>contains among other things the id of the ODBC driver. The string for > > a > >>connect contains the DSN (which is evaluated by the driver manager) > > and > >>additional connect properties (which are passed to the driver). This > > at > >>least was the general idea, which has probably been bended since. > > > The format of ODBC connection strings (sometimes called DSN string > or just DSN) goes like this: > ... > (taken from > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/odbc/ht > m/odbcsqldriverconnect.asp; > > At the very least you have to provide the DSN= > part of the connection string. All other parts are > optional. > > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 05 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From David.Rogers at mci.com Wed Apr 6 22:19:44 2005 From: David.Rogers at mci.com (Dave Rogers) Date: Wed Apr 6 22:19:50 2005 Subject: [DB-SIG] MySQLdb and INNODB tables Message-ID: <1112818784.5199.9.camel@pc78539.mcilink.com> I need to use MySQL INNODB tables for transactions, however, I have run into a problem. I can do SELECTS on the INNODB tables just fine but INSERTS and UPDATES do not work. No error is generated. I tried the same code using Python interactively and Python returns "1L" for each line I try to insert but in fact nothing is ever inserted. My Python code works perfectly with MyISAM tables. Am I just not holding my mouth right or what? dave -- Dave Rogers MCI Network Security Operations Center vnet 777-6522 From David.Rogers at mci.com Wed Apr 6 23:34:58 2005 From: David.Rogers at mci.com (Dave Rogers) Date: Wed Apr 6 23:35:11 2005 Subject: [DB-SIG] MySQLdb and INNODB tables In-Reply-To: <16980.18475.140043.996159@jin.int.geerbox.com> References: <1112818784.5199.9.camel@pc78539.mcilink.com> <16980.18475.140043.996159@jin.int.geerbox.com> Message-ID: <1112823298.5199.12.camel@pc78539.mcilink.com> Nathan, Thanks! That was it. I incorrectly assumed that I could operate without transactions on InnoDB tables where I did not explicitly need them. Best, dave On Wed, 2005-04-06 at 13:35 -0700, Nathan Clegg wrote: > InnoDB tables are transactional. Are you committing your transactions > after inserts and deletes? This isn't necessary on MyISAM tables. > > > >>>>> "Dave" == Dave Rogers writes: > > Dave> I need to use MySQL INNODB tables for transactions, however, > Dave> I have run into a problem. I can do SELECTS on the INNODB > Dave> tables just fine but INSERTS and UPDATES do not work. No > Dave> error is generated. I tried the same code using Python > Dave> interactively and Python returns "1L" for each line I try to > Dave> insert but in fact nothing is ever inserted. My Python code > Dave> works perfectly with MyISAM tables. > > Dave> Am I just not holding my mouth right or what? > > -- Dave Rogers Network Security Operations Center 777-6522 From farcepest at gmail.com Thu Apr 7 00:52:23 2005 From: farcepest at gmail.com (Andy Dustman) Date: Thu Apr 7 00:52:29 2005 Subject: [DB-SIG] MySQLdb and INNODB tables In-Reply-To: <1112818784.5199.9.camel@pc78539.mcilink.com> References: <1112818784.5199.9.camel@pc78539.mcilink.com> Message-ID: <9826f38005040615522aad8d92@mail.gmail.com> On Apr 6, 2005 4:19 PM, Dave Rogers wrote: > I need to use MySQL INNODB tables for transactions, however, I have run > into a problem. I can do SELECTS on the INNODB tables just fine but > INSERTS and UPDATES do not work. No error is generated. I tried the same > code using Python interactively and Python returns "1L" for each line I > try to insert but in fact nothing is ever inserted. My Python code works > perfectly with MyISAM tables. > > Am I just not holding my mouth right or what? I'll guess that you aren't executing db.commit() to commit the transaction on your transactional tables. It works for MyISAM because they are not transactional (though that is supposed to change in a future version). -- Computer interfaces should never be made of meat. Using GMail? Setting Reply-to address to <> disables this annoying feature. You are in a maze of twisty little passages all alike. To go north, press 2. To go west, press 4. To go east, press 6. To go south, press 8. If you need assistance, press 0 and a little dwarf will assist you. From djc at object-craft.com.au Thu Apr 7 01:07:48 2005 From: djc at object-craft.com.au (Dave Cole) Date: Thu Apr 7 01:07:59 2005 Subject: [DB-SIG] Sybase module 0.37 released Message-ID: <42546BC4.3070501@object-craft.com.au> WHAT IS IT: The Sybase module provides a Python interface to the Sybase relational database system. It supports all of the Python Database API, version 2.0 with extensions. NOTES: The 0.37 release is identical to 0.37pre3 as no problems were reported with the prerelease. This release contains a number of small bugfixes and patches received from users. I have been unable to find the source of the memory leak reported here: http://www.object-craft.com.au/pipermail/python-sybase/2004-December/000346.html The test program I wrote follows: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - import sys import Sybase db = Sybase.connect(..., auto_commit=True) db.execute(''' if exists (select name from sysobjects where name = "sp_test_leak") begin drop procedure sp_test_leak end ''') db.execute(''' create procedure sp_test_leak @arg int as select @arg ''') for i in range(200): for j in range(1000): c = db.cursor() c.callproc('sp_test_leak', {'@arg': 12345 }) sys.stdout.write('%3d\r' % i) sys.stdout.flush() sys.stdout.write('\n') - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - If someone is able to modify this and come up with a leaking result I am interested in working on the fix. You can build for FreeTDS like this: python setup.py build_ext -D HAVE_FREETDS -U WANT_BULKCOPY python setup.py install The module is available here: http://www.object-craft.com.au/projects/sybase/download/sybase-0.37pre3.tar.gz The module home page is here: http://www.object-craft.com.au/projects/sybase/ CHANGES SINCE 0.36: * Fix FreeTDS compilation and rearrange header includes to remove warnings. * Cursor state initialisation fix from Skip Montanaro. * Callback declaration fix on Windows from Vadim Beloborodov. * Cursor output parameters now work when parameters are passed as a sequence. * Output parameters now work for FreeTDS 0.62.4. 1> create procedure sp_test_output 2> @num int, @result int output 3> as 4> select @result = @num 5> go params = c.callproc('sp_test_output', {'@num': 12345, '@result': Sybase.OUTPUT(1)}) print params['@result'] * The CS_STATUS_RESULT result set is now consumed internally in the Cursor and does not appear in the result sets consumed by the fetch and nextset methods. The return value from the stored procedure is available in the .return_status member of the Cursor. It will only contain a meaningful value once all of the row result sets have been consumed. Note that this does not work with FreeTDS 0.62.4. The return_status seems to always be 0. Research shows that the problem is probably in the CT emulation layer as tsql displays the correct value, but sqsh displays 0. * Output hook patch from Ty Sarna has been applied. * Applied patch from Andre Sedinin to improve error handling. * Improved detection of SYBASE_OCS. -- http://www.object-craft.com.au From nathan-db-sig at geerbox.com Wed Apr 6 22:35:55 2005 From: nathan-db-sig at geerbox.com (Nathan Clegg) Date: Thu Apr 7 12:52:15 2005 Subject: [DB-SIG] MySQLdb and INNODB tables In-Reply-To: <1112818784.5199.9.camel@pc78539.mcilink.com> References: <1112818784.5199.9.camel@pc78539.mcilink.com> Message-ID: <16980.18475.140043.996159@jin.int.geerbox.com> InnoDB tables are transactional. Are you committing your transactions after inserts and deletes? This isn't necessary on MyISAM tables. >>>>> "Dave" == Dave Rogers writes: Dave> I need to use MySQL INNODB tables for transactions, however, Dave> I have run into a problem. I can do SELECTS on the INNODB Dave> tables just fine but INSERTS and UPDATES do not work. No Dave> error is generated. I tried the same code using Python Dave> interactively and Python returns "1L" for each line I try to Dave> insert but in fact nothing is ever inserted. My Python code Dave> works perfectly with MyISAM tables. Dave> Am I just not holding my mouth right or what? From carsten at uniqsys.com Tue Apr 12 16:30:23 2005 From: carsten at uniqsys.com (Carsten Haese) Date: Tue Apr 12 16:30:29 2005 Subject: [DB-SIG] Informixdb: New maintainer, new version Message-ID: <1113316222.16002.51.camel@dot.uniqsys.com> Hi Everybody: Since the current maintainer of the informixdb module appears to have gone missing, I have decided to take over the project. The new home of the informixdb module is http://sourceforge.net/projects/informixdb . Version 1.4 features the following improvements: * Build uses distutils instead of deprecated Makefile.pre.in mechanism. * Connect method takes optional username and password parameters for connecting to a remote database. * Cursor attribute .sqlerrd exposes Informix's sqlca.sqlerrd resulting from the cursor's most recent .execute() call. If you have any questions or comments, please let me know. Best regards, -- Carsten Haese - Software Engineer | Phone: (419) 861-3331 Unique Systems, Inc. | FAX: (419) 861-3340 1446 Reynolds Rd, Suite 313 | Maumee, OH 43537 | mailto:carsten@uniqsys.com From thaddon at equilar.com Fri Apr 15 20:48:14 2005 From: thaddon at equilar.com (Tom Haddon) Date: Fri Apr 15 20:48:17 2005 Subject: [DB-SIG] Database Abstraction in Python Message-ID: <0E52D69E86D25840AAE3611CB657F9F8365BAD@millenium.equilar.com> Hi, I'm new to the list, so please forgive me if this has been covered before. I searched the archives and wasn't able to find anything, so here goes... I'm interested in database abstraction for Python, and am wondering what people use for this. Currently I have thrown together a home-grown class that does database abstraction for MySQL and PostgreSQL for the functions I need, but I realize this is not the way to go long term. I did some google searches and it didn't seem like there was real consensus on what's the best for this in Python. The closest I could see was ADOdb for Python, but since this seems to be a rewrite of a PHP framework, I wasn't sure how good/stable/widely-used it was. Is there a "standard" for this, or is it a matter of personal preference? Thanks, Tom --------------------------------------- Tom Haddon Equilar, Inc. 1710 South Amphlett Boulevard Suite 312 San Mateo, CA 94402 650-286-4528 (phone) 650-286-4513 (fax) thaddon@equilar.com CONFIDENTIALITY NOTICE: This is a transmission from Equilar, Inc. and may contain information which is confidential and proprietary. If you are not the addressee, any disclosure, copying or distribution or use of the contents of this message is expressly prohibited. If you have received this transmission in error, please destroy it and notify us immediately at 650-286-4512. Internet and e-mail communications are Equilar's property and Equilar reserves the right to retrieve and read any message created, sent and received. From mal at egenix.com Fri Apr 15 21:37:59 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Apr 15 21:38:01 2005 Subject: [DB-SIG] Database Abstraction in Python In-Reply-To: <0E52D69E86D25840AAE3611CB657F9F8365BAD@millenium.equilar.com> References: <0E52D69E86D25840AAE3611CB657F9F8365BAD@millenium.equilar.com> Message-ID: <42601817.30903@egenix.com> Tom Haddon wrote: > Hi, > > I'm new to the list, so please forgive me if this has been covered before. I searched the archives and wasn't able to find anything, so here goes... > > I'm interested in database abstraction for Python, and am wondering what people use for this. Currently I have thrown together a home-grown class that does database abstraction for MySQL and PostgreSQL for the functions I need, but I realize this is not the way to go long term. I did some google searches and it didn't seem like there was real consensus on what's the best for this in Python. The closest I could see was ADOdb for Python, but since this seems to be a rewrite of a PHP framework, I wasn't sure how good/stable/widely-used it was. Is there a "standard" for this, or is it a matter of personal preference? The problem with database abstraction is not so much the interface, it's the different database dialects, data types, stored procedure syntax, etc. that make this difficult. To get the most out of the databases that you'd like to support, it's usually best to write an abstraction layer that's adjusted to your particular needs and set of database backends. The generic approach will always have to chose the least common denominator of all database that it wants to supports. That's fine for simple applications, but quickly reaches its limits once you want to do more clever things or benefit from specific database features. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 15 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From thaddon at equilar.com Sat Apr 16 00:11:06 2005 From: thaddon at equilar.com (Tom Haddon) Date: Sat Apr 16 00:11:12 2005 Subject: [DB-SIG] Database Abstraction in Python Message-ID: <0E52D69E86D25840AAE3611CB657F9F8365BAE@millenium.equilar.com> Hi Marc-Anderson, Thanks for the response. It sounds like you're saying that the practicalities of database abstraction make its implementation un-pythonic. I'm not sure I fully agree, but I take the point that if you want get the most out of a database, you probably want to write DB-specific code. At the same time, I think the whole point of database abstraction is more from a code re-use/re-factoring standpoint, and that the implicit trade off with features/performance is in some cases worth it. Appreciate the input though. Tom -----Original Message----- From: M.-A. Lemburg [mailto:mal@egenix.com] Sent: Friday, April 15, 2005 12:38 PM To: Tom Haddon Cc: db-sig@python.org Subject: Re: [DB-SIG] Database Abstraction in Python Tom Haddon wrote: > Hi, > > I'm new to the list, so please forgive me if this has been covered before. I searched the archives and wasn't able to find anything, so here goes... > > I'm interested in database abstraction for Python, and am wondering what people use for this. Currently I have thrown together a home-grown class that does database abstraction for MySQL and PostgreSQL for the functions I need, but I realize this is not the way to go long term. I did some google searches and it didn't seem like there was real consensus on what's the best for this in Python. The closest I could see was ADOdb for Python, but since this seems to be a rewrite of a PHP framework, I wasn't sure how good/stable/widely-used it was. Is there a "standard" for this, or is it a matter of personal preference? The problem with database abstraction is not so much the interface, it's the different database dialects, data types, stored procedure syntax, etc. that make this difficult. To get the most out of the databases that you'd like to support, it's usually best to write an abstraction layer that's adjusted to your particular needs and set of database backends. The generic approach will always have to chose the least common denominator of all database that it wants to supports. That's fine for simple applications, but quickly reaches its limits once you want to do more clever things or benefit from specific database features. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 15 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From chris at cogdon.org Sat Apr 16 00:34:48 2005 From: chris at cogdon.org (Chris Cogdon) Date: Sat Apr 16 00:35:08 2005 Subject: [DB-SIG] Database Abstraction in Python In-Reply-To: <0E52D69E86D25840AAE3611CB657F9F8365BAE@millenium.equilar.com> References: <0E52D69E86D25840AAE3611CB657F9F8365BAE@millenium.equilar.com> Message-ID: <207ad078aa57340c8a8e19c1956480e8@cogdon.org> On Apr 15, 2005, at 15:11, Tom Haddon wrote: > Hi Marc-Anderson, > > Thanks for the response. It sounds like you're saying that the > practicalities of database abstraction make its implementation > un-pythonic. I'm not sure I fully agree, but I take the point that if > you want get the most out of a database, you probably want to write > DB-specific code. At the same time, I think the whole point of > database abstraction is more from a code re-use/re-factoring > standpoint, and that the implicit trade off with features/performance > is in some cases worth it. > > Appreciate the input though. There's one more thing that Marc said I think you missed: Any abstraction mechanism that tries to be TOO general, coping with every application type, and every database type, will end up supporting only such a minimal set of database operations that it becomes very, very uninteresting. Instead, write a database abstractor that is specific for your application, or group of applications, and then you just need to write the underlying interface between the abstraction layer and the database connector for every supportable database. This way, it's 'dead simple' to port your application to a new database, since all the 'required operations' are effectively self-documenting. I've explored the issue enough myself that makes me believe that "any attempt to create a abstraction system that covers all application requirements AND all database implementations is doomed to failure, or mediocrity." Now... what might be interesting is a collection of libraries, functions, classes and mixins that makes the task of creating such abstraction layers easy. For example, when I've written such layers for various applications, I've needed functions A B C for Postgresql, and B C D for Mysql... some overlap, but not much. The 'workhorse' databases are going to be very different, too, perhaps very little overlap. But... there are things that seem to recur over and over. Food for thought, but... I'm not too hungry right now. -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From james at pythonweb.org Sat Apr 16 00:56:13 2005 From: james at pythonweb.org (James Gardner) Date: Sat Apr 16 00:55:44 2005 Subject: [DB-SIG] Database Abstraction in Python In-Reply-To: <42601817.30903@egenix.com> References: <0E52D69E86D25840AAE3611CB657F9F8365BAD@millenium.equilar.com> <42601817.30903@egenix.com> Message-ID: <4260468D.4050807@pythonweb.org> Hi Tom I've been working on a database abstraction layer for Python for a few months which I think you might find useful. I with Marc-Andre agree that it can be tricky to create a database abstraction layer because of the differences in SQL syntax, data typing etc but I published a specification to this list in December for what I considered to be the best compromise between providing database portability but also providing enough functionality to be useful. I've since implemented the specification as the web.database module of the python web modules. Chris Cogden Wrote: > "any attempt to create a abstraction system that covers all application > requirements AND all database implementations is doomed to failure, or > mediocrity." Well, see what you think of this one.. it doesn't cover ALL requirements but I and others using the python web modules have used it in commercial projects and found it to be useful. It supports SQLite, MySQL and has a pure Python implementation as a fallback if you don't have a database engine (easyish to support another database). It supports common data types, most basic SQL operations including joins, foreign and primary keys, converts the data to and from the appropriate format and has an object relational mapper built on top of it to treat an SQL database as a series of dictionaries (web.database.object). You can execute the portable SQL dialect specified directly or use Python methods to build queries. It supports a DB-API 2.0 style interface and parameter substitution. It differs from ADOdb which only provides abstraction in the methods used to execute SQL. The SQL you write is guaranteed to run exactly the same on any supported database since information on the table structures is stored as a table in the database and conversions are done transparently. You can also access the underlying DB-API 2.0 cursor if you want to. I planned to release it as a stand-alone package in the next few weeks but since the topic has come up it makes sense to mention it on this list now. It isn't properly packaged up and is documented for use in the web modules so be prepared for a bit of work. James Links ----- Original Specification: http://www.pythonweb.org/projects/pdbc/html/ Download: http://www.pythonweb.org/projects/webmodules/ Documentation: http://www.pythonweb.org/projects/webmodules/doc/0.5.3/html_multipage/lib/module-web.database.html http://www.pythonweb.org/projects/webmodules/doc/0.5.3/html_multipage/lib/object.html From wnvcole at peppermillcas.com Sat Apr 16 01:08:15 2005 From: wnvcole at peppermillcas.com (Vernon Cole) Date: Sat Apr 16 01:08:21 2005 Subject: [DB-SIG] Database Abstraction in Python Message-ID: <1DE30EB3ECE266409FDE6E7032F178C41398DD@pcimail1s.peppermillcas.com> Tom Haddon wrote: > Hi, > > I'm new to the list, so please forgive me if this has been covered before. I searched the archives and wasn't able to find anything, so here goes... > > I'm interested in database abstraction for Python, and am wondering what people use for this. Currently I have thrown together a home-grown class that does database abstraction for MySQL and PostgreSQL for the functions I need, but I realize this is not the way to go long term. I did some google searches and it didn't seem like there was real consensus on what's the best for this in Python. The closest I could see was ADOdb for Python, but since this seems to be a rewrite of a PHP framework, I wasn't sure how good/stable/widely-used it was. Is there a "standard" for this, or is it a matter of personal preference? Tom: You've heard from the "it can't be done" side so I'll speak up for the other. As you have noticed -- there is not a consensus on how to improve the database abstraction in Python. There is however a large swell of opinion that it SHOULD be done. Guido himself has spoken in favor of a 3.0 version of the Python database API standard. The existing 2.0 standard is a very good start, but it leaves too much undone. It lacks even such rudimentary constructs as column names. It supports four different methods of passing parameters to an SQL statement and is "standard" only in that it requires each module to tell you which one of the four it uses. I am sure that Mark is correct about heads-down dirty hardworking data base procedures which always must and will be pretty specific to a single db and operating system. On the other hand, one should be able to write "text book" examples of how to do simple database operations which will work in pretty much every case. We are not there yet. Welcome aboard, please share your ideas, any help is welcome. --- Vernon Cole From chris at cogdon.org Sat Apr 16 01:15:38 2005 From: chris at cogdon.org (Chris Cogdon) Date: Sat Apr 16 01:15:56 2005 Subject: [DB-SIG] Database Abstraction in Python In-Reply-To: <4260468D.4050807@pythonweb.org> References: <0E52D69E86D25840AAE3611CB657F9F8365BAD@millenium.equilar.com> <42601817.30903@egenix.com> <4260468D.4050807@pythonweb.org> Message-ID: On Apr 15, 2005, at 15:56, James Gardner wrote: > Chris Cogden Wrote: >> "any attempt to create a abstraction system that covers all >> application requirements AND all database implementations is doomed >> to failure, or mediocrity." > > Well, see what you think of this one.. it doesn't cover ALL > requirements but I and others using the python web modules have used > it in commercial projects and found it to be useful. > > It supports SQLite, MySQL and has a pure Python implementation as a > fallback if you don't have a database engine (easyish to support > another database). Well, feel free to saut? my own words and feed them to me :) A good test would be to start porting it to a few more database implementations. Oracle or some ODBC-based connection would be great, but I don't have access to that. I can try for PostgreSQL if you like (via pyPgSQL or psycopg) if you like. Does it come with a bunch of unit tests ? -- ("`-/")_.-'"``-._ Chris Cogdon . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' _.- _..-_/ / ((.' ((,.-' ((,/ fL From james at pythonweb.org Sat Apr 16 14:03:10 2005 From: james at pythonweb.org (James Gardner) Date: Sat Apr 16 14:02:42 2005 Subject: [DB-SIG] Database Abstraction in Python In-Reply-To: References: <0E52D69E86D25840AAE3611CB657F9F8365BAD@millenium.equilar.com> <42601817.30903@egenix.com> <4260468D.4050807@pythonweb.org> Message-ID: <4260FEFE.5030205@pythonweb.org> Hi Chris > Well, feel free to saut? my own words and feed them to me :) A good > test would be to start porting it to a few more database > implementations. Oracle or some ODBC-based connection would be great, > but I don't have access to that. I can try for PostgreSQL if you like > (via pyPgSQL or psycopg) if you like. Does it come with a bunch of > unit tests ? :-) I used to support Gadfly and Marc-Andre's ODBC driver too but Gadfly didn't support NULLs and ODBC wasn't open source so I dropped them both. Since the code worked with them both a few releases ago I'm fairly confident that it should be easy to implement drivers for ODBC and indeed most SQL databases, certainly PostgreSQL. There is also a driver mechanism in the pure Python implementation to allow data to be stored in different formats, I've currently only implemented a DBM driver but at one point also had a CSV driver too so the whole system is fairly extensible. If you are keen to implement a PostgreSQL driver that would be fantastic.. you just derive from database.drivers.base, use database.drivers.mysql as an example. There are five unit tests but I haven't distributed them yet. Here is a zip of the CVS tree with the tests if it is helpful: http://pythonweb.org/projects/pdbc/PDBC.zip I'll package up a proper release in the next few weeks and provide some better documentation. Did you notice the SQL console program which works like the MySQL prompt but for all supported databases in portable mode? More generally, do people feel this is a useful piece of software? Since the same API is used to access all underlying DB-API 2.0 drivers, this could be a useful way of distributing all the DB-API drivers in one package. If people wanted to implement drivers for the databases they use we could implement a feasible database abstraction layer that was genuinely portable when used in portable mode and provided full access to the underlying database in direct mode. Would this not form a useful basis for a DB-API 3.0? James From jfranz at neurokode.com Mon Apr 18 14:05:10 2005 From: jfranz at neurokode.com (Jon Franz) Date: Mon Apr 18 13:59:27 2005 Subject: [DB-SIG] Re: Database Abstraction in Python In-Reply-To: <20050417100002.50B141E4008@bag.python.org> References: <20050417100002.50B141E4008@bag.python.org> Message-ID: <4263A276.5040404@neurokode.com> > better documentation. Did you notice the SQL console program which works > like the MySQL prompt but for all supported databases in portable mode? That is a nice. Very nice. > More generally, do people feel this is a useful piece of software? Unfortunately, no - for philosophical and practical reasons I see many issues with it, at least for myself. >Since > the same API is used to access all underlying DB-API 2.0 drivers, this > could be a useful way of distributing all the DB-API drivers in one > package. If people wanted to implement drivers for the databases they > use we could implement a feasible database abstraction layer that was > genuinely portable when used in portable mode and provided full access > to the underlying database in direct mode. Would this not form a useful > basis for a DB-API 3.0? The concept, yes, would form a useful basis for DBAPI 3, but I don't know about the current implementation. ADOdb, being a port, isn't very pythonic (IMHO), and as such is not suitable either. But lets use this opportunity to talk - to talk about what we think is needed in dbapi3. I know I have a lot of ideas/thoughts - and am writing code for a demo implementation of my ideas as well. I've been writing code because the last 2 (3, 5?) times the next dbapi spec has been discussed here, the conversations were animated but eventually died without any actual progress. ~Jon Franz NeuroKode Labs, LLC From mal at egenix.com Mon Apr 18 14:56:27 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Mon Apr 18 14:56:31 2005 Subject: [DB-SIG] Re: Database Abstraction in Python In-Reply-To: <4263A276.5040404@neurokode.com> References: <20050417100002.50B141E4008@bag.python.org> <4263A276.5040404@neurokode.com> Message-ID: <4263AE7B.5070905@egenix.com> Please don't use the term "DB API 3.0" for anything that has to do with database abstraction. The Python DB API has a different scope. It is a driver interface specification, not a specification for higher level interfaces. The latter can easily be built on top of the DB API and there have been a few successful attempts to do so: PyDAL started by Randall Smith last year: http://pydal.sourceforge.net/ ADOdb which was originally written for PHP: http://adodb.sourceforge.net/ SQLObject which is an ORM layer on top of the DB API: http://sqlobject.org/ PDO, an OO API similar to ADO built on top of the DB API: http://pdo.neurokode.com/ (probably others which I don't remember) If you want to start a new project in that direction, please have a look at the above projects first and see whether one of them fits your needs. The next version of the DB API spec will not include any of these abstraction attempts, but instead focus on clarifying the DB API 2.0 and possibly moving some of the currently optional features in the spec to the mandatory sections. The DB API's aim is to provide a solid and working basis for the many possible abstractions which are commonly used in database programming, from general purpose abstraction layers to per-application implementations. That said, please note that my recommendation to Tom Haddon is based on my own experience in Python database programming. I've done quite a few of these abstraction interfaces on top of the DB API in many projects and found that you get much better portability if you implement the abstraction based on the application rather than going for an abstraction of the abstration which is bound to have to make compromises one way or another. Given that the application writer usually knows best what she expects from the database and which database backends she intends to tagret, the per-application abstraction interface has usually turned out to be the best choice in terms of flexibility, robustness and performance. It also makes it easier to adapt the application to various backends - hiding the SQL, matching the database data types to application data types and tweaking varying database semantics to fit the application's needs. Again, that's my experience. YMMV. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 18 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From jfranz at neurokode.com Mon Apr 18 15:43:17 2005 From: jfranz at neurokode.com (Jonathan Franz) Date: Mon Apr 18 15:43:27 2005 Subject: [DB-SIG] Abstraction layer confusion, ULA (was Re: Database Abstraction in Python) Message-ID: <4263B975.1060101@neurokode.com> I responded to an email about an abstraction layer, whose author asked if such a layer was a good basis for a dbapi 3 - to whit my response was basically 'no, but lets talk about dbapi 3'. Perhaps I wasn't clear enough... As for PDO, I think I might know something about it... As a maintainer of one such abstraction layer, I have a vested interest in making sure dbapi 3 provides the facilities needed for a modern, robust user-level api. Currently, dbapi is lacking many features. Marc wrote: """The next version of the DB API spec will not include any of these abstraction attempts, but instead focus on clarifying the DB API 2.0 and possibly moving some of the currently optional features in the spec to the mandatory sections.""" Every time changes to the spec are discussed on this list, you make that same argument, as if dbapi 3 were something on a shelf waiting for release... it is not. Shouldn't we as a community discuss what is or isn't needed in such a spec? I tend to disagree with the argument on application specific abstraction layers; But only because things like PDO (which provides a developer-friendly API, No ORM or other abstractions) and ADO are not application specific in any way, and yet are tools the average developer would (in many cases) choose to utilize rather than write to the driver-level interface. Let us call such a layer a User-Level-Api (ULA), instead of an abstraction layer. Now, what things should such a layer include? And what do we call the driver layer? And do we still use the dbapi name for the bundle of a common ULA and Driver layer? Perhaps it is time for a PEP. From anthony at interlink.com.au Mon Apr 18 16:04:15 2005 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Apr 18 16:04:30 2005 Subject: [DB-SIG] Abstraction layer confusion, ULA (was Re: Database Abstraction in Python) In-Reply-To: <4263B975.1060101@neurokode.com> References: <4263B975.1060101@neurokode.com> Message-ID: <200504190004.16174.anthony@interlink.com.au> Isn't this the same split that Perl has with it's DBI:DBD? In any case, I too would like to see a standard higher level API on top of the DB-API - note that this should not be included as part of the DB-API, but as an additional API on top of it. This API would be a good candidate for the Python standard library - perhaps starting with something like db_row or the like. And the DB-API needs to be tightened up - in particular things like argument passing (binding input variables) and return values from execute are an utter utter pain in the arse right now, as they vary across database interfaces. As an example, an update that modifies two columns of a single row returns 1 on some implementations, 2 on others. So in the interests of spurring action, what other parts of the DB-API need revision? Let's make a concrete list, then this can actually move forward. Lets put aside all higher-level APIs for the moment - I think that this is better handled in a different PEP. Off the top of my head: connection paramaters execute return values binding input variables -- Anthony Baxter It's never too late to have a happy childhood. From mal at egenix.com Mon Apr 18 16:30:51 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Mon Apr 18 16:30:55 2005 Subject: [DB-SIG] Abstraction layer confusion, ULA (was Re: Database Abstraction in Python) In-Reply-To: <4263B975.1060101@neurokode.com> References: <4263B975.1060101@neurokode.com> Message-ID: <4263C49B.1020208@egenix.com> Jonathan Franz wrote: > Currently, dbapi is lacking many features. Hmm, I don't see how the DB API limits any module writer from adding new features to their implementation. In fact, many authors have added features to their implementation and as a result we have added a special section to the DB API to make sure that these features use standardized semantics. It would really help if you could be more specific about those lacking features. The only feature which I'd really like to see in DB API 3.0 is a standard way to tell the module how to map database types to Python types and vice-versa. > Marc wrote: > """The next version of the DB API spec will not include any > of these abstraction attempts, but instead focus on > clarifying the DB API 2.0 and possibly moving some of > the currently optional features in the spec to the > mandatory sections.""" > > Every time changes to the spec are discussed on this list, you make that > same argument, as if dbapi 3 were something on a shelf waiting for > release... it is not. Shouldn't we as a community discuss what is or > isn't needed in such a spec? Of course ! My main concern is that you are trying to reuse the term "DB API" in a way which historically doesn't make sense: the DB API has always been a very simple API specification focussed on very basic database building blocks. The goal was to make it easy for module writers to build modules which adhere to the spec and to make it easy for users to write code on top of it. Think of it as middleware. If you want to design a new layer on top of the DB API, you are welcome to do so, but please use a different name. You see, I don't think that we need a specification for an abstraction layer. The reason is simple: you will only write this abstraction layer once and not have many implementations which adhere to it (which is what the DB API spec is all about). If you believe that we need to add features to the DB API spec to make the implementation of an abstraction layer easier, we can discuss this aspect. If you would like to see e.g. PDO or a similar package in Python's standard lib, that's a different discussion. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 18 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at egenix.com Mon Apr 18 16:37:16 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Mon Apr 18 16:37:19 2005 Subject: [DB-SIG] Abstraction layer confusion, ULA (was Re: Database Abstraction in Python) In-Reply-To: <200504190004.16174.anthony@interlink.com.au> References: <4263B975.1060101@neurokode.com> <200504190004.16174.anthony@interlink.com.au> Message-ID: <4263C61C.4040605@egenix.com> Anthony Baxter wrote: > Isn't this the same split that Perl has with it's DBI:DBD? > > In any case, I too would like to see a standard higher level API on top > of the DB-API - note that this should not be included as part of the DB-API, > but as an additional API on top of it. This API would be a good candidate > for the Python standard library - perhaps starting with something like db_row > or the like. +1 > And the DB-API needs to be tightened up - in particular things like argument > passing (binding input variables) and return values from execute are an utter > utter pain in the arse right now, as they vary across database interfaces. > As an example, an update that modifies two columns of a single row returns > 1 on some implementations, 2 on others. > > So in the interests of spurring action, what other parts of the DB-API need > revision? Let's make a concrete list, then this can actually move forward. > Lets put aside all higher-level APIs for the moment - I think that this is > better handled in a different PEP. > > Off the top of my head: > > connection paramaters > execute return values Uhm, .executeXXX() methods don't have a return value ?! DB API 2.0 specifies that .rowcount is to be used to access the number of rows touched in an update, insert, etc. .rowcount This read-only attribute specifies the number of rows that the last executeXXX() produced (for DQL statements like 'select') or affected (for DML statements like 'update' or 'insert'). The attribute is -1 in case no executeXXX() has been performed on the cursor or the rowcount of the last operation is not determinable by the interface. [7] Note: Future versions of the DB API specification could redefine the latter case to have the object return None instead of -1. > binding input variables -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 18 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From jfranz at neurokode.com Mon Apr 18 16:40:26 2005 From: jfranz at neurokode.com (Jonathan Franz) Date: Mon Apr 18 16:40:34 2005 Subject: [Fwd: Re: [DB-SIG] Abstraction layer confusion, ULA (was Re: Database Abstraction in Python)] In-Reply-To: <4263C067.1080202@neurokode.com> References: <4263C067.1080202@neurokode.com> Message-ID: <4263C6DA.5070605@neurokode.com> > Isn't this the same split that Perl has with it's DBI:DBD? Yes, and the same sort of split is used by ADO, deplhi, Ruby DBI and others. > So in the interests of spurring action, what other parts of the DB-API need > revision? Let's make a concrete list, then this can actually move forward. > Lets put aside all higher-level APIs for the moment - I think that this is > better handled in a different PEP. A different PEP was the idea - but we will need to keep in mind what, if any, changes to the driver layer would be needed to support functions we might want. Heres my list of dbapi changes (adds to yours): - parameter marks (choose 1 or two, get rid of others) - round-trip typing (or hooks to allow it at the ULA, including type enumeration) - better definition of all return types (undefined shouldn't be allowed, either it is defined, or it returns nothing - leaving it to the module only induces confusion) (extends execute return values in your list) - datetime removal (use python datetime types) - better documentation (descriptor fields are poorly defined, and are based upon odbc fields, yet the docs don't define them, or point at the odbc docs) > > Off the top of my head: > > connection paramaters > execute return values > binding input variables > From jfranz at neurokode.com Mon Apr 18 16:51:20 2005 From: jfranz at neurokode.com (Jonathan Franz) Date: Mon Apr 18 16:51:31 2005 Subject: [DB-SIG] Abstraction layer confusion, ULA (was Re: Database Abstraction in Python) In-Reply-To: <4263C49B.1020208@egenix.com> References: <4263B975.1060101@neurokode.com> <4263C49B.1020208@egenix.com> Message-ID: <4263C968.1070909@neurokode.com> > Hmm, I don't see how the DB API limits any module writer > from adding new features to their implementation. > > In fact, many authors have added features to their implementation > and as a result we have added a special section to the DB API > to make sure that these features use standardized semantics. Ack! I guess my definition of a standard is different than yours - I would see all extensions either made part of the core standard, or removed from the document entirely. Optional features just confuse the things. > It would really help if you could be more specific about those > lacking features. > > The only feature which I'd really like to see in DB API 3.0 > is a standard way to tell the module how to map database types > to Python types and vice-versa. That is one of the main ones I want/need for PDO, see my response to Anthony for others. > My main concern is that you are trying to reuse the term "DB API" > in a way which historically doesn't make sense: the DB API > has always been a very simple API specification focussed on > very basic database building blocks. The goal was to make > it easy for module writers to build modules which adhere to the > spec and to make it easy for users to write code on top of it. > Think of it as middleware. Ah, that makes sense, its a naming-and-scope thing. OK. > If you believe that we need to add features to the DB API spec > to make the implementation of an abstraction layer easier, > we can discuss this aspect. Perfect, that is all I request! Hopefully and/all added features that would make abstraction easier would also be useful for the application writer who writes directly to the spec himself, and for those who write app-specific abstractions. > If you would like to see e.g. PDO or a similar package > in Python's standard lib, that's a different discussion. Whatever the standard ULA is/becomes, that should be suggested for the stlib, but only after it matures. From fog at initd.org Mon Apr 18 18:51:38 2005 From: fog at initd.org (Federico Di Gregorio) Date: Mon Apr 18 18:51:42 2005 Subject: [DB-SIG] Abstraction layer confusion, ULA (was Re: Database Abstraction in Python) In-Reply-To: <4263C49B.1020208@egenix.com> References: <4263B975.1060101@neurokode.com> <4263C49B.1020208@egenix.com> Message-ID: <1113843098.3198.2.camel@lana.initd.org> Il giorno lun, 18/04/2005 alle 16.30 +0200, M.-A. Lemburg ha scritto: > The only feature which I'd really like to see in DB API 3.0 > is a standard way to tell the module how to map database types > to Python types and vice-versa. Hint: currently both psycopg 2 and pysqlite 2 implement some subset of PEP 246 (Object adaptation). Maybe is this the way to go? Another this I really like to see in DBAPI 3 is asynchronous calls, cursors implementing the fileno() method (allowing them to be used in select calls), etc. -- Federico Di Gregorio http://people.initd.org/fog Debian GNU/Linux Developer fog@debian.org INIT.D Developer fog@initd.org Abandon the search for Truth; settle for a good fantasy. -- Anonymous -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/db-sig/attachments/20050418/0b5ed761/attachment.pgp From james at pythonweb.org Mon Apr 18 19:52:37 2005 From: james at pythonweb.org (James Gardner) Date: Mon Apr 18 19:51:58 2005 Subject: [DB-SIG] Abstraction layer confusion, ULA (was Re: Database Abstraction in Python) In-Reply-To: <4263B975.1060101@neurokode.com> References: <4263B975.1060101@neurokode.com> Message-ID: <4263F3E5.8080005@pythonweb.org> Hi All, In developing PDBC (the web.database module) I decided there were 5 levels of database abstraction so perhaps we can agree on naming them like this: Driver DB-API User Level API Portability Layer Object Relational Mapper The driver is whatever libraries are used to connect to the database at the lowest level. The DB-API 2.0 wraps the driver functionality to allow low level database access for full control via SQL (perhaps with some sensible type conversions). The user level API wraps up the DB-API functionality into an easier to use form but doesn't abstract the database functionality, only the API. The portability layer provides database abstraction in both SQL and type conversions so that code works identically on all supported databases. The object relational mapper maps Python objects to SQL relationships so that no SQL is necessarily needed. To the best of my knowledge PDO, PyDAL, adodb are what I would describe as user level APIs that is they simply wrap the DB-API methods in a more convenient and standard form. SQLObject is an Object Relational Mapper. The only portability layer I know of is my own. That is, an API that behaves EXACTLY the same with EXACTLY the same SQL commands on all supported databases by making a compromise between features and portability (regardless of whether we agree this is actually the best approach!) It sounds like what is needed then is a DB-API with slightly less flexibility left to the implementor which takes into account any changes like the adoption of the datetime module and perhaps support for a reduced number of parameter types, tighter use of connection parameters etc. Are we also agreeing we need a user level API, perhaps similar to PDO to be included as part of the standard library?? If so, would we also include the database drivers themselves to provide database functionality out of the box? Is there no general desire to work on a portability layer or object relational mapper to be included in the standard library? Cheers, James From thaddon at equilar.com Mon Apr 18 20:08:19 2005 From: thaddon at equilar.com (Tom Haddon) Date: Mon Apr 18 20:08:23 2005 Subject: [DB-SIG] Abstraction layer confusion, ULA (was Re: Database Abstraction in Python) Message-ID: <0E52D69E86D25840AAE3611CB657F9F8365BB7@millenium.equilar.com> Hi James, I like a lot of the features you discuss, but have two concerns that I'm hoping to understand better: - As I understand it there is "metadata" that is needed to perform the abstraction in web.database and this is stored in the database itself. I'd also read somewhere that if you make some data changes to directly on the database that it might cause issues for web.database. This seems to me like a potential problem if it's true, as you're losing the ability to manipulate the database outside of the application for which it was written. - Just wondering about it's relationship to the web module. Is it really part of/dependant on the rest of the web module, or is it more accurately a separate database module? My preference would be separate as otherwise it seems like the implication is that it's only applicable for/suitable for web applications. Thanks, Tom -----Original Message----- From: db-sig-bounces@python.org [mailto:db-sig-bounces@python.org]On Behalf Of James Gardner Sent: Monday, April 18, 2005 10:53 AM To: db-sig@python.org Subject: Re: [DB-SIG] Abstraction layer confusion,ULA (was Re: Database Abstraction in Python) Hi All, In developing PDBC (the web.database module) I decided there were 5 levels of database abstraction so perhaps we can agree on naming them like this: Driver DB-API User Level API Portability Layer Object Relational Mapper The driver is whatever libraries are used to connect to the database at the lowest level. The DB-API 2.0 wraps the driver functionality to allow low level database access for full control via SQL (perhaps with some sensible type conversions). The user level API wraps up the DB-API functionality into an easier to use form but doesn't abstract the database functionality, only the API. The portability layer provides database abstraction in both SQL and type conversions so that code works identically on all supported databases. The object relational mapper maps Python objects to SQL relationships so that no SQL is necessarily needed. To the best of my knowledge PDO, PyDAL, adodb are what I would describe as user level APIs that is they simply wrap the DB-API methods in a more convenient and standard form. SQLObject is an Object Relational Mapper. The only portability layer I know of is my own. That is, an API that behaves EXACTLY the same with EXACTLY the same SQL commands on all supported databases by making a compromise between features and portability (regardless of whether we agree this is actually the best approach!) It sounds like what is needed then is a DB-API with slightly less flexibility left to the implementor which takes into account any changes like the adoption of the datetime module and perhaps support for a reduced number of parameter types, tighter use of connection parameters etc. Are we also agreeing we need a user level API, perhaps similar to PDO to be included as part of the standard library?? If so, would we also include the database drivers themselves to provide database functionality out of the box? Is there no general desire to work on a portability layer or object relational mapper to be included in the standard library? Cheers, James _______________________________________________ DB-SIG maillist - DB-SIG@python.org http://mail.python.org/mailman/listinfo/db-sig From james at pythonweb.org Mon Apr 18 20:35:58 2005 From: james at pythonweb.org (James Gardner) Date: Mon Apr 18 20:35:21 2005 Subject: [DB-SIG] Database Abstraction in Python In-Reply-To: <0E52D69E86D25840AAE3611CB657F9F8365BB7@millenium.equilar.com> References: <0E52D69E86D25840AAE3611CB657F9F8365BB7@millenium.equilar.com> Message-ID: <4263FE0E.5070500@pythonweb.org> Hi Tom, >- As I understand it there is "metadata" that is needed to perform the abstraction in web.database and this is stored in the database itself. > > That's right, the metadata is simply a table which tells PDBC (web.database) what the field types are and what the table relationships are so that it can perform correct type conversions and enforce foreign key constraints. This isn't actually completely necessary in the architecture because type data could be obtained from the cursor.description object but I found that DB-API implementors tended to change their type codes between versions so storing the information in a table seemed an easier option. > I'd also read somewhere that if you make some data changes to directly on the database that it might cause issues for web.database. > > That's right too, if you altered the table structure externally without updating the meta data, PDBC would get incorrect information about the database structure. Most of the time you wouldn't want to alter a table structure after you have created it and if you did it is easy to also update the meta data table (just not documented yet, hence the warning in the current documentation). The other issue is that it is possible that a field type used by PDBC is supported in the database engine by a field which natively accepts a broader range of values than PDBC allows so an external program could potentially set a value which PDBC wouldn't understand. This isn't an issue in any of the current implementations as far as I know, I was just thinking ahead to possible future implementations. Most of the time it is perfectly alright to update the information from an external program. Again, this isn't documented fully, hence the warning. >- Just wondering about it's relationship to the web module. Is it really part of/dependant on the rest of the web module, or is it more accurately a separate database module? My preference would be separate as otherwise it seems like the implication is that it's only applicable for/suitable for web applications. > > It is a separate module which grew out of my initial attempts to create the sort of abstraction layer for the web modules which Marc-Andre was advocating earlier in this thread. I am currently refactoring it so that I can distribute it as a separate package named PDBC so that it can be used with any program requiring database access, not just the web modules. Hope that helps, James From jfranz at neurokode.com Mon Apr 18 20:52:40 2005 From: jfranz at neurokode.com (Jonathan Franz) Date: Mon Apr 18 20:52:43 2005 Subject: [Fwd: Re: [DB-SIG] Abstraction layer confusion, ULA (was Re: Database Abstraction in Python)] In-Reply-To: <4263FC46.509@neurokode.com> References: <4263FC46.509@neurokode.com> Message-ID: <426401F8.1030807@neurokode.com> > Hi All, > > In developing PDBC (the web.database module) I decided there were 5 > levels of database abstraction so perhaps we can agree on naming them > like this: > > Driver > DB-API > User Level API > Portability Layer > Object Relational Mapper I'd imagine these inverted, just so that the highest-level abstraction/utility is at the top, and lowest at the bottom :) DB-API is a driver specification, so I would think it should be more like this: ORM Portability Layer ULA Driver (DBAPI) > The driver is whatever libraries are used to connect to the database at > the lowest level. The DB-API 2.0 wraps the driver functionality to allow > low level database access for full control via SQL (perhaps with some > sensible type conversions). Almost, dbapi is the interface a driver must provide, not a wrapper. (though some dbapi modules may be wrappers of existing drivers, that is an implementation choice, not a requirement.) > The user level API wraps up the DB-API > functionality into an easier to use form but doesn't abstract the > database functionality, only the API. The portability layer provides > database abstraction in both SQL and type conversions so that code works > identically on all supported databases. The object relational mapper > maps Python objects to SQL relationships so that no SQL is necessarily > needed. > > To the best of my knowledge PDO, PyDAL, adodb are what I would describe > as user level APIs that is they simply wrap the DB-API methods in a more > convenient and standard form. adodb does a little bit more - like standard ways to limit the # of results ("SELECT TOP n" on MSSQL, "SELECT ... LIMIT n" on MySQL and others) - but it is far from a true portability layer. PDO will hopefully, eventually (if we get round trip typing) support editable result sets (would require the results come from a single table, of course), but the SQL generation capabilities needed to do such a thing are very simple compared to a true portability layer. I have experimental versions of this working - but I had to hack together a type-inspection system that is nasty and unreliable: Types that aren't defined in the spec aren't used when determining the type of a column for updates and inserts. > SQLObject is an Object Relational Mapper. > The only portability layer I know of is my own. That is, an API that > behaves EXACTLY the same with EXACTLY the same SQL commands on all > supported databases by making a compromise between features and > portability (regardless of whether we agree this is actually the best > approach!) It is a neat approach, but you'll need JOINs, subselects, LIMIT and other pieces to really make it useful for the majority of people - otherwise the portability layer will fall into the trap of being too limited in functionality to be popular. > It sounds like what is needed then is a DB-API with slightly less > flexibility left to the implementor which takes into account any changes > like the adoption of the datetime module and perhaps support for a > reduced number of parameter types, tighter use of connection parameters > etc. Some sort of limited schema inspection might be useful as well - Right now, for example, the primary key has to be passed to the constructor for an editable resultset in xPDO (The experimental version mentioned above). This may not be an easy feature for driver maintainers to implement though... Easier to implement but still useful: how about dbapi drivers expose the SQL inspection commands for the current connection as properties? (I would just hard code these, but for modules such as mxODBC, the inspection commands change based upon the ODBC driver used). Such as: Connection.listTables Connection.listDBs Connection.inspectTable (names subject to change) > Are we also agreeing we need a user level API, perhaps similar to > PDO to be included as part of the standard library?? +1 > If so, would we > also include the database drivers themselves to provide database > functionality out of the box? I'm of two minds on this one: A) It is a good idea, and works with the batteries-included philosophy. B) Including 3rd party drivers in the stdlib would be hairy from a licensing standpoint, as well as a stdlib-bloat standpoint. Google for the debate on including SQLLite in the stdlib. > Is there no general desire to work on a > portability layer or object relational mapper to be included in the > standard library? I think there may be desire. Is there enough to get something into the stdlib? Unknown. From mcfletch at rogers.com Mon Apr 18 21:36:29 2005 From: mcfletch at rogers.com (Mike C. Fletcher) Date: Mon Apr 18 21:35:44 2005 Subject: [Fwd: Re: [DB-SIG] Abstraction layer confusion, ULA (was Re: Database Abstraction in Python)] In-Reply-To: <426401F8.1030807@neurokode.com> References: <4263FC46.509@neurokode.com> <426401F8.1030807@neurokode.com> Message-ID: <42640C3D.8090209@rogers.com> Jonathan Franz wrote: ... >> The user level API wraps up the DB-API functionality into an easier >> to use form but doesn't abstract the database functionality, only the >> API. The portability layer provides database abstraction in both SQL >> and type conversions so that code works identically on all supported >> databases. The object relational mapper maps Python objects to SQL >> relationships so that no SQL is necessarily needed. >> >> To the best of my knowledge PDO, PyDAL, adodb are what I would >> describe as user level APIs that is they simply wrap the DB-API >> methods in a more convenient and standard form. > > > adodb does a little bit more - like standard ways to limit the # of > results ("SELECT TOP n" on MSSQL, "SELECT ... LIMIT n" on MySQL and > others) - but it is far from a true portability layer. PyTable also does minimal rewriting of queries to support different parameter formats. Again, though, not a true portability layer, it assumes you are writing any given piece of SQL against the current database, it just lets you use the rich pyformat operator everywhere. The idea being that the ORM provides the bulk of the code you want to be cross-database, and the sql queries are for when you want to get down-and-dirty to get past an impedance mismatch (of course, it being a limited ORM, it's quite frequent that you wind up needing to do that :) ). > PDO will hopefully, eventually (if we get round trip typing) support > editable result sets (would require the results come from a single > table, of course), but the SQL generation capabilities needed to do > such a thing are very simple compared to a true portability layer. I > have experimental versions of this working - but I had to hack > together a type-inspection system that is nasty and unreliable: Types > that aren't defined in the spec aren't used when determining the type > of a column for updates and inserts. PyTable allows for this via its schema objects (generally written in Python code to allow extra annotations, but they can also be derived from the internal database structures), but that's part of the ORM from my point of view (you tell a result record to delete itself/update itself/refresh itself). > SQLObject is an Object Relational Mapper. The only portability layer I > know of is my own. That is, an API that behaves EXACTLY the same with > EXACTLY the same SQL commands on all supported databases by making a > compromise between features and portability (regardless of whether we > agree this is actually the best approach!) PyTable has a limited ORM as well. >> It sounds like what is needed then is a DB-API with slightly less >> flexibility left to the implementor which takes into account any >> changes like the adoption of the datetime module and perhaps support >> for a reduced number of parameter types, tighter use of connection >> parameters etc. > > > Some sort of limited schema inspection might be useful as well - Right > now, for example, the primary key has to be passed to the constructor > for an editable resultset in xPDO (The experimental version mentioned > above). This may not be an easy feature for driver maintainers to > implement though... > > Easier to implement but still useful: how about dbapi drivers expose > the SQL inspection commands for the current connection as properties? > (I would just hard code these, but for modules such as mxODBC, the > inspection commands change based upon the ODBC driver used). > > Such as: > Connection.listTables > Connection.listDBs > Connection.inspectTable In PyTable these are spelled: DBDriver.listDatabases( cursor/connection ) DBDriver.listTables( cursor/connection ) DBDriver.listIndices( cursor/connection ) DBDriver.tableStructure( cursor/connection, tableName ) as well as (on PostgreSQL): DBDriver.listNamespaces( cursor/connection ) DBDriver.listNamespaceTables( cursor/connection ) (retrieves all tables in non-root namespaces) I'd suggest making them function calls rather than properties, these are things you don't want to be hiding from the user if they are going to take a significant amount of time. Have fun, Mike -- ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com From gh at ghaering.de Fri Apr 22 01:34:32 2005 From: gh at ghaering.de (Gerhard Haering) Date: Fri Apr 22 10:14:57 2005 Subject: [DB-SIG] [ANN] pysqlite 2.0.alpha3 Message-ID: <20050421233432.GA15825@mylene.ghaering.de> =================== pysqlite 2.0.alpha3 =================== I'm glad to announce pysqlite 2.0.alpha3, which will be the last alpha release. Documentation and more testing will be what I'm concentrating on in the beta test phase, so here's the explanation of the new features within the change log. Download it from Sourceforge: http://sourceforge.net/project/showfiles.php?group_id=54058 Changes since 2.0.alpha2 ======================== - Fixed various memory leaks and refcount problems. - Connection, Cursor and row factories. o Connection factory: Subclass pysqlite2.dbapi2.Connection and provide the parameter to pysqlite2.dbapi2.connect to use that class instead. o Cursor factory: Subclass pysqlite2.dbapi2.Cursor and provide the parameter to the cursor() method of the Connection class so that cursors of your class are created. o Row factory: set the row_factory attribute in your cursor to a callable that accepts a tuple and returns the "real" row object. Combine the three for maximimal power and convenience like in the following example, where we transparently use the db_row module from http://opensource.theopalgroup.com/ to be able to access columns by name instead of by index. A fast alternative to PgResultSet in pysqlite 1.x. from pysqlite2 import dbapi2 as sqlite import db_row class DbRowConnection(sqlite.Connection): def __init__(self, *args, **kwargs): sqlite.Connection.__init__(self, *args, **kwargs) def cursor(self): return DbRowCursor(self) class DbRowCursor(sqlite.Cursor): def execute(self, *args, **kwargs): sqlite.Cursor.execute(self, *args, **kwargs) if self.description: self.row_factory = db_row.IMetaRow(self.description) con = sqlite.connect(":memory:") #, factory=DbRowConnection) cur = con.cursor(factory=DbRowCursor) cur.execute("create table test(i integer)") cur.execute("select 4+5 as foo") for row in cur: print ">", row print row["foo"] print cur.fetchone()["foo"] - The first parameter to .execute() and .executemany() can now also be a Unicode string: cur.execute(u"insert into person(name) values ('H?ring')") - The conversion in SQLite types mode for TEXT is now unicode, not UTF-8 encoded bytestrings. i. e. if the column type is TEXT, then you get back unicode objects now. - Implemented user functions and aggregates. Where a number of parameters is expected, -1 means variable number of arguments. The functions and aggregates must return values of the type NoneType, int, float, str, unicode or buffer (BLOB). Example code: def f(*args): return sum(args) class Sum: def __init__(self): self.sum = 0 def step(self, x): self.sum += int(x) def finalize(self): return self.sum con = sqlite.connect(":memory:") # Syntax: function name, number of parameters, callable con.create_function("myfunc", -1, f) # Syntax: aggregate name, number of parameters, aggregate class con.create_aggregate("myaggr", 1, Sum) - Added MANIFEST.in file, so that bdist_rpm and sdist will work. - Release GIL during SQLite calls. - After a cursor.executemany, cursor.rowcount now delivers the sum of all changes, not only the changes in the last call to the prepared statement. - Implemented checks that the SQLite objects are used from the same thread they were created in. Otherwise raise a ProgrammingError. If you're 100% sure that you want to use the same connection object in multiple threads, because you use proper locking, you can turn the check of by using the parameter ''check_same_thread=0''. - Allow for parameters to be dictionaries, too. Then, the name-based binding in SQLite 3 is used: cur.execute("select name from test where name=:name", {"name": "foo"}) - Improved error handling. - Allow for explicit transaction start: >>> con = sqlite.connect(":memory:", no_implicit_begin=True) ... >>> con.begin() ... >>> con.rollback() ... >>> con.commit() - Implemented cursor.lastrowid - Avoid code duplication: handle execute() and executemany() internally in the same function. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/db-sig/attachments/20050422/8325befd/attachment.pgp From gh at ghaering.de Fri Apr 22 01:36:41 2005 From: gh at ghaering.de (Gerhard Haering) Date: Fri Apr 22 10:15:18 2005 Subject: [DB-SIG] Re: [ANN] pysqlite 2.0.alpha3 In-Reply-To: <20050421233432.GA15825@mylene.ghaering.de> References: <20050421233432.GA15825@mylene.ghaering.de> Message-ID: <20050421233641.GA15865@mylene.ghaering.de> On Fri, Apr 22, 2005 at 01:34:32AM +0200, Gerhard Haering wrote: > =================== > pysqlite 2.0.alpha3 > =================== > > I'm glad to announce pysqlite 2.0.alpha3, which will be the last alpha > release. Documentation and more testing will be what I'm concentrating > on in the beta test phase, so here's the explanation of the new features > within the change log. > > Download it from Sourceforge: > http://sourceforge.net/project/showfiles.php?group_id=54058 > [...] Forgot the pysqlite homepage: http://pysqlite.org/ with Wiki, BugTracker and Subversion repository. -- Gerhard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/db-sig/attachments/20050422/2ef6819f/attachment.pgp From jfranz at neurokode.com Mon Apr 25 16:03:59 2005 From: jfranz at neurokode.com (Jonathan Franz) Date: Mon Apr 25 16:04:12 2005 Subject: [DB-SIG] adodbapi In-Reply-To: <20050422100003.8E2F11E4009@bag.python.org> References: <20050422100003.8E2F11E4009@bag.python.org> Message-ID: <426CF8CF.5090608@neurokode.com> Does anyone know how I can reach the maintainer of adodbapi? I submitted a fix for a bug with transaction handling two weeks ago, and heard nothing back. Looking closer at the project on SF, it looks like there are open bugs dating back almost 2 years... is adodbapi now an orphaned project? ~Jon Franz NeuroKode Labs, LLC From ccurvey at gmail.com Mon Apr 25 18:11:41 2005 From: ccurvey at gmail.com (Chris Curvey) Date: Mon Apr 25 18:11:46 2005 Subject: [DB-SIG] adodbapi In-Reply-To: <426CF8CF.5090608@neurokode.com> References: <20050422100003.8E2F11E4009@bag.python.org> <426CF8CF.5090608@neurokode.com> Message-ID: <25ef693e050425091156c676f0@mail.gmail.com> I think I had posted something several years ago and never heard anything back. Since Python-to-MSSQL has been a pain in the neck for years now, I'd be willing to help in an effort to re-do adodbapi. On 4/25/05, Jonathan Franz wrote: > > Does anyone know how I can reach the maintainer of adodbapi? I > submitted a fix for a bug with transaction handling two weeks ago, and > heard nothing back. > > Looking closer at the project on SF, it looks like there are open bugs > dating back almost 2 years... is adodbapi now an orphaned project? > > ~Jon Franz > NeuroKode Labs, LLC > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/db-sig/attachments/20050425/605339d6/attachment.html From jfranz at neurokode.com Mon Apr 25 18:33:24 2005 From: jfranz at neurokode.com (Jonathan Franz) Date: Mon Apr 25 18:33:32 2005 Subject: [DB-SIG] adodbapi In-Reply-To: <25ef693e050425091156c676f0@mail.gmail.com> References: <20050422100003.8E2F11E4009@bag.python.org> <426CF8CF.5090608@neurokode.com> <25ef693e050425091156c676f0@mail.gmail.com> Message-ID: <426D1BD4.4020304@neurokode.com> I think a re-do may be going a little far, but the source is there, and it is easy to modify. A fork may be called for, and people to help maintain the fork would be needed. What issues have you had with MSSQL and adodbapi? ~Jon Franz NeuroKode Labs, LLC Chris Curvey wrote: > I think I had posted something several years ago and never heard > anything back. Since Python-to-MSSQL has been a pain in the neck for > years now, I'd be willing to help in an effort to re-do adodbapi. > > On 4/25/05, *Jonathan Franz* > wrote: > > Does anyone know how I can reach the maintainer of adodbapi? I > submitted a fix for a bug with transaction handling two weeks ago, and > heard nothing back. > > Looking closer at the project on SF, it looks like there are open bugs > dating back almost 2 years... is adodbapi now an orphaned project? > > ~Jon Franz > NeuroKode Labs, LLC > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://mail.python.org/mailman/listinfo/db-sig > > From jfranz at neurokode.com Mon Apr 25 19:36:31 2005 From: jfranz at neurokode.com (Jonathan Franz) Date: Mon Apr 25 19:36:33 2005 Subject: [DB-SIG] adodbapi In-Reply-To: <25ef693e05042509397c54cea5@mail.gmail.com> References: <20050422100003.8E2F11E4009@bag.python.org> <426CF8CF.5090608@neurokode.com> <25ef693e050425091156c676f0@mail.gmail.com> <426D1BD4.4020304@neurokode.com> <25ef693e05042509397c54cea5@mail.gmail.com> Message-ID: <426D2A9F.3070004@neurokode.com> Ouch, that is nasty, did you end up having to use two connections? The bug I found, and fixed was with transaction handling. The current adodbapi looks at a property on the underlying ADO connection to determine if transactions are supported. Unfortunately, it just looks for the presence of that property, not the value of the property. Thus, when using the DBF driver adodbapi raises errors because it tries to call BeginTrans on the ADO connection. The fix was clean and simple - just use the value of the property if the property exists. Chris Curvey wrote: > Trying to remember, I gave up so long ago.... > > IIRC, I think that I was unable to get two cursors from the same > connection at the same time. Which meant I had to be really careful > when designing tranasctions. I think I looked at the source code to try > to fix it, but it was something architectural about the design of the > library that prevented me from suggesting a fix. > > > From gvwilson at cs.utoronto.ca Tue Apr 26 19:34:53 2005 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Tue Apr 26 20:45:53 2005 Subject: [DB-SIG] re: Data Crunching Message-ID: Readers of this newsgroup might be interested in a new book on data crunching, which is available from Amazon: http://www.amazon.com/exec/obidos/ASIN/0974514071 or directly from the Pragmatic Programmers: http://www.pragmaticprogrammer.com/titles/gwd/index.html The book covers basic text processing, regular expressions, XML manipulation, binary data handling, and the 10% of relational databases that every programmer should know. Most of the examples are in Python (though Unix command line tools, XSL, and SQL are in there as well). The book is aimed at beginner and intermediate programmers; I hope everyone will find it useful. Thanks, Greg Wilson (its grinning author) From rijishvr at rediffmail.com Fri Apr 29 17:31:05 2005 From: rijishvr at rediffmail.com (rijish valoorthodi rajan) Date: Fri Apr 29 17:29:40 2005 Subject: [DB-SIG] bigin with python Message-ID: <20050429153105.28943.qmail@webmail35.rediffmail.com> i have been working on Visual studio, and now i want to try make my application using PYTHON. I have to make a client-server application where a data base application is to be run in the server and a VB like application interface is to be run in the client. how should i approch the project if i am to make the application using PYTHON ? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/db-sig/attachments/20050429/ed2a0d04/attachment.html From nicolas.grilly at garden-paris.com Fri Apr 29 19:00:27 2005 From: nicolas.grilly at garden-paris.com (Nicolas Grilly) Date: Fri Apr 29 19:00:38 2005 Subject: [DB-SIG] bigin with python In-Reply-To: <20050429153105.28943.qmail@webmail35.rediffmail.com> Message-ID: <20050429170034.D52E81E72AA@mrelay1.st2.lyceu.net> Hello, Like you, I use Python to build a client GUI accessing a database server. I build the GUI with wxPython (www.wxpython.org). I use the ODBC driver built-in in Active State's Python distribution (www.activestate.com/Products/ActivePython). But this driver has some serious limitations and it can be hard to find workarounds if you don't know very well SQL databases and ODBC drivers. It's why you can also use mxODBC (www.egenix.com/files/python/mxODBC.html). But you need to pay (a little) if you are a business. wxPython is packaged with an excellent documentation, an excellent web site full of useful information, and a very simple but excellent tool to build GUI (XRCed). I recommend you break your code in three tiers/modules : a module to handle data, a module to handle views (data aware widgets and dialogs - think to use Validators) and a module for your application itself. You can also build a true .exe for Windows if you need that (idem for Mac). Sincerely, Nicolas -- Nicolas Grilly Garden - Conseil et Etudes CRM Conqu?te, fid?lisation, data mining, tableaux de bord, bases de donn?es marketing Tel +33 1 45 72 48 78 - Fax +33 1 56 72 21 32 - Mob +33 6 03 00 25 34 - www.garden-paris.com _____ De : db-sig-bounces@python.org [mailto:db-sig-bounces@python.org] De la part de rijish valoorthodi rajan Envoy? : vendredi 29 avril 2005 17:31 ? : db-sig@python.org Objet : [DB-SIG] bigin with python i have been working on Visual studio, and now i want to try make my application using PYTHON. I have to make a client-server application where a data base application is to be run in the server and a VB like application interface is to be run in the client. how should i approch the project if i am to make the application using PYTHON ? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/db-sig/attachments/20050429/f4f2d0e1/attachment.html