From vernondcole at gmail.com  Mon Sep 22 16:25:38 2008
From: vernondcole at gmail.com (Vernon Cole)
Date: Mon, 22 Sep 2008 08:25:38 -0600
Subject: [DB-SIG] First suggestion for db-api 3.0
Message-ID: <b7eb4a700809220725o68934982xa7f759aa4d49e1ae@mail.gmail.com>

Dear Pythonaholics:

I have not been following the development of Python 2.6 and 3.0, so the
following took me by surprise when I read it this morning. It seems to me on
first glance, that this new feature, "Named Tuple", is exactly what is
needed to make fields in database records (or columns in database rows if
you prefer) more accessible to a python programmer.

From: Discussion of IronPython <users at lists.ironpython.com>...

On Fri, Sep 19, 2008 at 6:26 AM, Michael Foord <fuzzyman at voidspace.org.uk>wrote:
Hello all,

At PyCon UK Raymond Hettinger showed off the Named Tuple; a very useful
recipe for creating tuples with named fields. It is becoming part of the
standard library in Python 2.6.

http://code.activestate.com/recipes/500261/

>>> from namedtuple import namedtuple
>>> thing = namedtuple('thing', ('whizz', 'pop'))
>>> thing
<class '__builtin__.thing'>
>>> i = thing(1, 2)
>>> i.whizz
1
>>> i.pop
2
>>> w, p = i
>>> w, p
(1, 2)
>>> i
thing(whizz=1, pop=2)
>>>

I would like to suggest that we start the process of creating a dbapi 3.0
specification, and that the new spec define the returned data as an iterator
of Named Tuples.
--
Vernon Cole
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/db-sig/attachments/20080922/7f42c06f/attachment.htm>

From james at jamesh.id.au  Tue Sep 23 09:15:02 2008
From: james at jamesh.id.au (James Henstridge)
Date: Tue, 23 Sep 2008 15:15:02 +0800
Subject: [DB-SIG] First suggestion for db-api 3.0
In-Reply-To: <b7eb4a700809220725o68934982xa7f759aa4d49e1ae@mail.gmail.com>
References: <b7eb4a700809220725o68934982xa7f759aa4d49e1ae@mail.gmail.com>
Message-ID: <237378990809230015s36dbc27aw123eec1e82cb1a62@mail.gmail.com>

2008/9/22 Vernon Cole <vernondcole at gmail.com>:
> Dear Pythonaholics:
>
> I have not been following the development of Python 2.6 and 3.0, so the
> following took me by surprise when I read it this morning. It seems to me on
> first glance, that this new feature, "Named Tuple", is exactly what is
> needed to make fields in database records (or columns in database rows if
> you prefer) more accessible to a python programmer.
>
> From: Discussion of IronPython <users at lists.ironpython.com>...
>
> On Fri, Sep 19, 2008 at 6:26 AM, Michael Foord <fuzzyman at voidspace.org.uk>
> wrote:
> Hello all,
>
> At PyCon UK Raymond Hettinger showed off the Named Tuple; a very useful
> recipe for creating tuples with named fields. It is becoming part of the
> standard library in Python 2.6.
>
> http://code.activestate.com/recipes/500261/
>
>>>> from namedtuple import namedtuple
>>>> thing = namedtuple('thing', ('whizz', 'pop'))
>>>> thing
> <class '__builtin__.thing'>
>>>> i = thing(1, 2)
>>>> i.whizz
> 1
>>>> i.pop
> 2
>>>> w, p = i
>>>> w, p
> (1, 2)
>>>> i
> thing(whizz=1, pop=2)
>>>>
>
> I would like to suggest that we start the process of creating a dbapi 3.0
> specification, and that the new spec define the returned data as an iterator
> of Named Tuples.

Note that there has been development on DB-API since the 2.0 release
in the form of extensions listed at the end of the spec.

There is already an optional extension for retrieving a result set
using iterator protocol on the cursor, so that bit is already handled.
 Having the results returned as named tuples could also be handled as
an optional extension.

As for making it part of the core specification, I think the question
in the PEP's FAQ about returning dictionaries applies:

    Question:

       How can I construct a dictionary out of the tuples returned by
       .fetch*():

    Answer:

       There are several existing tools available which provide
       helpers for this task. Most of them use the approach of using
       the column names defined in the cursor attribute .description
       as basis for the keys in the row dictionary.

       Note that the reason for not extending the DB API specification
       to also support dictionary return values for the .fetch*()
       methods is that this approach has several drawbacks:

       * Some databases don't support case-sensitive column names or
         auto-convert them to all lowercase or all uppercase
         characters.

       * Columns in the result set which are generated by the query
         (e.g.  using SQL functions) don't map to table column names
         and databases usually generate names for these columns in a
         very database specific way.

       As a result, accessing the columns through dictionary keys
       varies between databases and makes writing portable code
       impossible.

So such an API may not be implementable on all databases, and may not
give useful results on others.

James.

From vernondcole at gmail.com  Wed Sep 24 20:28:57 2008
From: vernondcole at gmail.com (Vernon Cole)
Date: Wed, 24 Sep 2008 12:28:57 -0600
Subject: [DB-SIG] First suggestion for db-api 3.0
In-Reply-To: <237378990809230015s36dbc27aw123eec1e82cb1a62@mail.gmail.com>
References: <b7eb4a700809220725o68934982xa7f759aa4d49e1ae@mail.gmail.com>
	<237378990809230015s36dbc27aw123eec1e82cb1a62@mail.gmail.com>
Message-ID: <b7eb4a700809241128p7af54693yde2c940b795dd19@mail.gmail.com>

James:
Yes, I read the PEP, and the FAQ. (Since I am a dbapi maintainer, I thought
it would be a good idea.) Lots of the things that have been wished for in
the past were rejected because they would be hard for the api writer to
implement. BDFL has suggested that doing things the easy way for
implementers may not be the most pythonic answer. We (tool writers) are
supposed to make things easy for the user, not for ourselves. For example, I
think that the idea of telling the user which of four different types of
parameter markers your api is using, so that he can code his program four
different ways, is the height of laziness.  On the other hand, if the user
was SETTING .paramstyle to tell ME which style HE is using, then such an
attribute would be a good thing. I would like to implement that. But as long
as my new feature were an "extension" and not part of the standard, who
would use it? That is why we need an updated standard IMHO.

As to the two 'impossible' problems you quote:
 1) "Some databases don't support case-sensitive column names...". True. And
some operating systems don't support case-sensitive file names. So what?  If
a database designer includes two columns which have names differing only in
letter case, he should be shot, not catered to. Make the dbapi standard case
insensitve (like Internet urls) and be done with it.
  2) "...databases usually generate names ... in a very database specific
way." So if you want to control what name your db returns for a function,
you need to use an "AS" clause in your SQL.  My documentation says that "as"
is implemented as a standard in the Entry level of SQL-92. It should be
pretty much universal.

  Doesn't sound 'impossible' to me. If it is 'impossible', then how do those
'several existing tools' do it?

Now let me show you a simple case where you DO need something other than
counting columns. (I am including the code as an attachment, for those who
want to read it.) Snip.py opens Microsoft Active Directory as a table and
returns a row for each user. Note in the result below that the order of the
columns IS NOT the same as the order in the query!

(configuration for the following: Windows XP, Active Directory 2003, Python
2.5.2, adodbapi 2.2.1)

H:\Python>snip.py
Executing the command: "select title, displayName, sAMAccountName,
givenName, adsPath from 'LDAP://dc=wc,dc=peppermillcas,dc=com' where
objectCategory = 'User'"

result data description is:
(NAME TypeCd DispSize IntrnlSz Prec Scale Null?)
(u'adsPath', 202, 108, 256, 255, 255, False)
(u'givenName', 202, 0, 256, 255, 255, False)
(u'sAMAccountName', 202, 10, 256, 255, 255, False)
(u'displayName', 202, 0, 256, 255, 255, False)
(u'title', 202, 0, 256, 255, 255, False)

The column names are exactly what I asked for, but
the order is all wrong. If I asked for row[1] I will get the wrong thing. If
I could ask for row.displayName it would be correct.

After all is said and done, why should it be that I can say
"row.displayName" in COBOL, but not in Python?

(How can ANYTHING be worse than COBOL?)
--
Vernon Cole
former COBOL compiler tester.

On Tue, Sep 23, 2008 at 1:15 AM, James Henstridge <james at jamesh.id.au>wrote:

> 2008/9/22 Vernon Cole <vernondcole at gmail.com>:
> > Dear Pythonaholics:
> >
> > I have not been following the development of Python 2.6 and 3.0, so the
> > following took me by surprise when I read it this morning. It seems to me
> on
> > first glance, that this new feature, "Named Tuple", is exactly what is
> > needed to make fields in database records (or columns in database rows if
> > you prefer) more accessible to a python programmer.
> >
> > From: Discussion of IronPython <users at lists.ironpython.com>...
> >
> > On Fri, Sep 19, 2008 at 6:26 AM, Michael Foord <
> fuzzyman at voidspace.org.uk>
> > wrote:
> > Hello all,
> >
> > At PyCon UK Raymond Hettinger showed off the Named Tuple; a very useful
> > recipe for creating tuples with named fields. It is becoming part of the
> > standard library in Python 2.6.
> >
> > http://code.activestate.com/recipes/500261/
> >
> >>>> from namedtuple import namedtuple
> >>>> thing = namedtuple('thing', ('whizz', 'pop'))
> >>>> thing
> > <class '__builtin__.thing'>
> >>>> i = thing(1, 2)
> >>>> i.whizz
> > 1
> >>>> i.pop
> > 2
> >>>> w, p = i
> >>>> w, p
> > (1, 2)
> >>>> i
> > thing(whizz=1, pop=2)
> >>>>
> >
> > I would like to suggest that we start the process of creating a dbapi 3.0
> > specification, and that the new spec define the returned data as an
> iterator
> > of Named Tuples.
>
> Note that there has been development on DB-API since the 2.0 release
> in the form of extensions listed at the end of the spec.
>
> There is already an optional extension for retrieving a result set
> using iterator protocol on the cursor, so that bit is already handled.
>  Having the results returned as named tuples could also be handled as
> an optional extension.
>
> As for making it part of the core specification, I think the question
> in the PEP's FAQ about returning dictionaries applies:
>
>    Question:
>
>       How can I construct a dictionary out of the tuples returned by
>       .fetch*():
>
>    Answer:
>
>       There are several existing tools available which provide
>       helpers for this task. Most of them use the approach of using
>       the column names defined in the cursor attribute .description
>       as basis for the keys in the row dictionary.
>
>       Note that the reason for not extending the DB API specification
>       to also support dictionary return values for the .fetch*()
>       methods is that this approach has several drawbacks:
>
>       * Some databases don't support case-sensitive column names or
>         auto-convert them to all lowercase or all uppercase
>         characters.
>
>       * Columns in the result set which are generated by the query
>         (e.g.  using SQL functions) don't map to table column names
>         and databases usually generate names for these columns in a
>         very database specific way.
>
>       As a result, accessing the columns through dictionary keys
>       varies between databases and makes writing portable code
>       impossible.
>
> So such an API may not be implementable on all databases, and may not
> give useful results on others.
>
> James.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/db-sig/attachments/20080924/53b99837/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: snip.py
Type: text/x-java
Size: 801 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/db-sig/attachments/20080924/53b99837/attachment.java>

From bkline at rksystems.com  Wed Sep 24 21:06:52 2008
From: bkline at rksystems.com (Bob Kline)
Date: Wed, 24 Sep 2008 15:06:52 -0400
Subject: [DB-SIG] First suggestion for db-api 3.0
In-Reply-To: <b7eb4a700809241128p7af54693yde2c940b795dd19@mail.gmail.com>
References: <b7eb4a700809220725o68934982xa7f759aa4d49e1ae@mail.gmail.com>	<237378990809230015s36dbc27aw123eec1e82cb1a62@mail.gmail.com>
	<b7eb4a700809241128p7af54693yde2c940b795dd19@mail.gmail.com>
Message-ID: <48DA8FCC.5010005@rksystems.com>

Vernon Cole wrote:
> The column names are exactly what I asked for, but
> the order is all wrong. If I asked for row[1] I will get the wrong 
> thing. If I could ask for row.displayName it would be correct.

My first reaction when I read this was "surely the implementation of the 
API is not behaving correctly"; but then I looked at what the 
specification actually says, and indeed, there is no requirement that 
the order of the tuples in the description sequence match the order of 
the columns in the result set.  Regardless of the merits of Vernon's 
proposal (and I think his arguments have some validity), this seems like 
a hole in the specification which should be addressed.

-- 
Bob Kline
http://www.rksystems.com
mailto:bkline at rksystems.com


From mal at egenix.com  Wed Sep 24 22:05:28 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 24 Sep 2008 22:05:28 +0200
Subject: [DB-SIG] First suggestion for db-api 3.0
In-Reply-To: <48DA8FCC.5010005@rksystems.com>
References: <b7eb4a700809220725o68934982xa7f759aa4d49e1ae@mail.gmail.com>	<237378990809230015s36dbc27aw123eec1e82cb1a62@mail.gmail.com>	<b7eb4a700809241128p7af54693yde2c940b795dd19@mail.gmail.com>
	<48DA8FCC.5010005@rksystems.com>
Message-ID: <48DA9D88.1030508@egenix.com>

On 2008-09-24 21:06, Bob Kline wrote:
> Vernon Cole wrote:
>> The column names are exactly what I asked for, but
>> the order is all wrong. If I asked for row[1] I will get the wrong
>> thing. If I could ask for row.displayName it would be correct.
> 
> My first reaction when I read this was "surely the implementation of the
> API is not behaving correctly"; but then I looked at what the
> specification actually says, and indeed, there is no requirement that
> the order of the tuples in the description sequence match the order of
> the columns in the result set.  Regardless of the merits of Vernon's
> proposal (and I think his arguments have some validity), this seems like
> a hole in the specification which should be addressed.

Agreed.

Luckily implementors have done the right thing in the past 10 years
or so :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 24 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From mal at egenix.com  Wed Sep 24 22:36:34 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 24 Sep 2008 22:36:34 +0200
Subject: [DB-SIG] First suggestion for db-api 3.0
In-Reply-To: <b7eb4a700809241128p7af54693yde2c940b795dd19@mail.gmail.com>
References: <b7eb4a700809220725o68934982xa7f759aa4d49e1ae@mail.gmail.com>	<237378990809230015s36dbc27aw123eec1e82cb1a62@mail.gmail.com>
	<b7eb4a700809241128p7af54693yde2c940b795dd19@mail.gmail.com>
Message-ID: <48DAA4D2.4010809@egenix.com>

On 2008-09-24 20:28, Vernon Cole wrote:
> James:
> Yes, I read the PEP, and the FAQ. (Since I am a dbapi maintainer, I thought
> it would be a good idea.) Lots of the things that have been wished for in
> the past were rejected because they would be hard for the api writer to
> implement. BDFL has suggested that doing things the easy way for
> implementers may not be the most pythonic answer. We (tool writers) are
> supposed to make things easy for the user, not for ourselves. For example, I
> think that the idea of telling the user which of four different types of
> parameter markers your api is using, so that he can code his program four
> different ways, is the height of laziness.  On the other hand, if the user
> was SETTING .paramstyle to tell ME which style HE is using, then such an
> attribute would be a good thing. I would like to implement that. 

Feel free to do so. There has been a lot of discussion about these
parameter styles. The last round concluded that we should strip down
the number of possible styles to 1 or 2 ('?' and ':1' IIRC).

> But as long
> as my new feature were an "extension" and not part of the standard, who
> would use it? That is why we need an updated standard IMHO.

The DB-API standard defines a set of features that all compatible
database modules must implement, plus a set of extensions that they
may implement, but don't have to or indeed may not be able to due to
the backends or interfaces not supporting e.g. two phase commit.

If your DB-API module can do more than the standard spec that's
perfectly fine. We created the set of standard extensions so that
those extensions are not implemented in 10 different ways.

Also note that the spec does indeed allow the module to return
namedtuples, but it doesn't require this. The spec deliberately
uses the term "sequence" instead of tuple or list.

Note that the implementation of namedtuples is not a particularly
nice one (they use dynamic programming). There's also a different
implementation available via the C API called structseq (this is used
e.g. by the time module).

Both create subclasses of the standard Python tuple and are
sequences, so both are permitted to be used by the DB-API to
represent rows.

However, using the Python version in an API that potentially
returns a few thousand rows is not exactly what I'd recommend
to do, since it will slow down the interface a lot.

Due to the nature of the namedtuples, you will also see many
different objects as row object (each query will return a
different object type).

> As to the two 'impossible' problems you quote:
>  1) "Some databases don't support case-sensitive column names...". True. And
> some operating systems don't support case-sensitive file names. So what?  If
> a database designer includes two columns which have names differing only in
> letter case, he should be shot, not catered to. Make the dbapi standard case
> insensitve (like Internet urls) and be done with it.
>   2) "...databases usually generate names ... in a very database specific
> way." So if you want to control what name your db returns for a function,
> you need to use an "AS" clause in your SQL.  My documentation says that "as"
> is implemented as a standard in the Entry level of SQL-92. It should be
> pretty much universal.
> 
>   Doesn't sound 'impossible' to me. If it is 'impossible', then how do those
> 'several existing tools' do it?
> 
> Now let me show you a simple case where you DO need something other than
> counting columns. (I am including the code as an attachment, for those who
> want to read it.) Snip.py opens Microsoft Active Directory as a table and
> returns a row for each user. Note in the result below that the order of the
> columns IS NOT the same as the order in the query!
> 
> (configuration for the following: Windows XP, Active Directory 2003, Python
> 2.5.2, adodbapi 2.2.1)
> 
> H:\Python>snip.py
> Executing the command: "select title, displayName, sAMAccountName,
> givenName, adsPath from 'LDAP://dc=wc,dc=peppermillcas,dc=com' where
> objectCategory = 'User'"
> 
> result data description is:
> (NAME TypeCd DispSize IntrnlSz Prec Scale Null?)
> (u'adsPath', 202, 108, 256, 255, 255, False)
> (u'givenName', 202, 0, 256, 255, 255, False)
> (u'sAMAccountName', 202, 10, 256, 255, 255, False)
> (u'displayName', 202, 0, 256, 255, 255, False)
> (u'title', 202, 0, 256, 255, 255, False)
> 
> The column names are exactly what I asked for, but
> the order is all wrong. If I asked for row[1] I will get the wrong thing. If
> I could ask for row.displayName it would be correct.
> 
> After all is said and done, why should it be that I can say
> "row.displayName" in COBOL, but not in Python?
> 
> (How can ANYTHING be worse than COBOL?)
> --
> Vernon Cole
> former COBOL compiler tester.
> 
> On Tue, Sep 23, 2008 at 1:15 AM, James Henstridge <james at jamesh.id.au>wrote:
> 
>> 2008/9/22 Vernon Cole <vernondcole at gmail.com>:
>>> Dear Pythonaholics:
>>>
>>> I have not been following the development of Python 2.6 and 3.0, so the
>>> following took me by surprise when I read it this morning. It seems to me
>> on
>>> first glance, that this new feature, "Named Tuple", is exactly what is
>>> needed to make fields in database records (or columns in database rows if
>>> you prefer) more accessible to a python programmer.
>>>
>>> From: Discussion of IronPython <users at lists.ironpython.com>...
>>>
>>> On Fri, Sep 19, 2008 at 6:26 AM, Michael Foord <
>> fuzzyman at voidspace.org.uk>
>>> wrote:
>>> Hello all,
>>>
>>> At PyCon UK Raymond Hettinger showed off the Named Tuple; a very useful
>>> recipe for creating tuples with named fields. It is becoming part of the
>>> standard library in Python 2.6.
>>>
>>> http://code.activestate.com/recipes/500261/
>>>
>>>>>> from namedtuple import namedtuple
>>>>>> thing = namedtuple('thing', ('whizz', 'pop'))
>>>>>> thing
>>> <class '__builtin__.thing'>
>>>>>> i = thing(1, 2)
>>>>>> i.whizz
>>> 1
>>>>>> i.pop
>>> 2
>>>>>> w, p = i
>>>>>> w, p
>>> (1, 2)
>>>>>> i
>>> thing(whizz=1, pop=2)
>>> I would like to suggest that we start the process of creating a dbapi 3.0
>>> specification, and that the new spec define the returned data as an
>> iterator
>>> of Named Tuples.
>> Note that there has been development on DB-API since the 2.0 release
>> in the form of extensions listed at the end of the spec.
>>
>> There is already an optional extension for retrieving a result set
>> using iterator protocol on the cursor, so that bit is already handled.
>>  Having the results returned as named tuples could also be handled as
>> an optional extension.
>>
>> As for making it part of the core specification, I think the question
>> in the PEP's FAQ about returning dictionaries applies:
>>
>>    Question:
>>
>>       How can I construct a dictionary out of the tuples returned by
>>       .fetch*():
>>
>>    Answer:
>>
>>       There are several existing tools available which provide
>>       helpers for this task. Most of them use the approach of using
>>       the column names defined in the cursor attribute .description
>>       as basis for the keys in the row dictionary.
>>
>>       Note that the reason for not extending the DB API specification
>>       to also support dictionary return values for the .fetch*()
>>       methods is that this approach has several drawbacks:
>>
>>       * Some databases don't support case-sensitive column names or
>>         auto-convert them to all lowercase or all uppercase
>>         characters.
>>
>>       * Columns in the result set which are generated by the query
>>         (e.g.  using SQL functions) don't map to table column names
>>         and databases usually generate names for these columns in a
>>         very database specific way.
>>
>>       As a result, accessing the columns through dictionary keys
>>       varies between databases and makes writing portable code
>>       impossible.
>>
>> So such an API may not be implementable on all databases, and may not
>> give useful results on others.
>>
>> James.
>>
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> DB-SIG maillist  -  DB-SIG at python.org
> http://mail.python.org/mailman/listinfo/db-sig

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 24 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From vernondcole at gmail.com  Thu Sep 25 00:12:06 2008
From: vernondcole at gmail.com (Vernon Cole)
Date: Wed, 24 Sep 2008 16:12:06 -0600
Subject: [DB-SIG] First suggestion for db-api 3.0
In-Reply-To: <b7eb4a700809241504x457f9080j1ea83f3e916262a6@mail.gmail.com>
References: <b7eb4a700809220725o68934982xa7f759aa4d49e1ae@mail.gmail.com>
	<237378990809230015s36dbc27aw123eec1e82cb1a62@mail.gmail.com>
	<b7eb4a700809241128p7af54693yde2c940b795dd19@mail.gmail.com>
	<48DAA4D2.4010809@egenix.com>
	<b7eb4a700809241504x457f9080j1ea83f3e916262a6@mail.gmail.com>
Message-ID: <b7eb4a700809241512h6aaa48dh4cf2586e93650211@mail.gmail.com>

oops!  Forgot to select "Reply to all".  Sorry for the duplicate, Marc.
--
VC

---------- Forwarded message ----------
From: Vernon Cole <vernondcole at gmail.com>
Date: Wed, Sep 24, 2008 at 4:04 PM
Subject: Re: [DB-SIG] First suggestion for db-api 3.0
To: "M.-A. Lemburg" <mal at egenix.com>



On Wed, Sep 24, 2008 at 2:36 PM, M.-A. Lemburg <mal at egenix.com> wrote:

>  [...] There has been a lot of discussion about these
> parameter styles. The last round concluded that we should strip down
> the number of possible styles to 1 or 2 ('?' and ':1' IIRC).
>

I think it was qmark and named. I agree with dropping the others. -- VC

[...] Also note that the spec does indeed allow the module to return
> namedtuples, but it doesn't require this. The spec deliberately
> uses the term "sequence" instead of tuple or list.
>
> Note that the implementation of namedtuples is not a particularly
> nice one (they use dynamic programming). There's also a different
> implementation available via the C API called structseq (this is used
> e.g. by the time module).
>
> Both create subclasses of the standard Python tuple and are
> sequences, so both are permitted to be used by the DB-API to
> represent rows.
>

So a dbapi 3.0 implementation could be a proper superset of dbapi 2.0. That
would be best. -- VC


> However, using the Python version in an API that potentially
> returns a few thousand rows is not exactly what I'd recommend
> to do, since it will slow down the interface a lot.
>
> Due to the nature of the namedtuples, you will also see many
> different objects as row object (each query will return a
> different object type).
>
> My favorite database (the one I help implement once upon a time) had the
data organized as a virtual array of similar (same class) objects with each
column as a named attribute. (Those are not the terms we used, since object
oriented languages had not been invented then.) That's the model I am used
to thinking in. Our implementation was pretty fast.

Adodbapi is already quite slow, so I doubt if the extra overhead of a named
tuple would actually make that much difference to it. Perhaps I should
download a 2.6 python and muck with it. Those of you working in C would be
emulating the action of a named tuple for performance reasons, I expect,
using your own objects. The implementation details would be unimportant,
provided that the syntax is consistent.

  I only know that we are already requiring the user to know two languages:
Python and SQL, in order to use dbapi. Selecting from among the existing ORM
tool kits, and then learning the tool kit, is more of a learning curve than
I could hack. I guess that's why I still don't use an Object Relational
Mapper.

 I just want named access to my columns somehow -- so I don't add the
complexity (and inevitable errors) of having to remember the column number
of each field. The access to each column should feel like native Python
--
VC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/db-sig/attachments/20080924/0bfdeb33/attachment.htm>

From mal at egenix.com  Mon Sep 29 13:00:47 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 29 Sep 2008 13:00:47 +0200
Subject: [DB-SIG] First suggestion for db-api 3.0
In-Reply-To: <b7eb4a700809241512h6aaa48dh4cf2586e93650211@mail.gmail.com>
References: <b7eb4a700809220725o68934982xa7f759aa4d49e1ae@mail.gmail.com>	<237378990809230015s36dbc27aw123eec1e82cb1a62@mail.gmail.com>	<b7eb4a700809241128p7af54693yde2c940b795dd19@mail.gmail.com>	<48DAA4D2.4010809@egenix.com>	<b7eb4a700809241504x457f9080j1ea83f3e916262a6@mail.gmail.com>
	<b7eb4a700809241512h6aaa48dh4cf2586e93650211@mail.gmail.com>
Message-ID: <48E0B55F.4040803@egenix.com>

On 2008-09-25 00:12, Vernon Cole wrote:
> On Wed, Sep 24, 2008 at 2:36 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
>>  [...] There has been a lot of discussion about these
>> parameter styles. The last round concluded that we should strip down
>> the number of possible styles to 1 or 2 ('?' and ':1' IIRC).
>>
> 
> I think it was qmark and named. I agree with dropping the others. -- VC
> 
> [...] Also note that the spec does indeed allow the module to return
>> namedtuples, but it doesn't require this. The spec deliberately
>> uses the term "sequence" instead of tuple or list.
>>
>> Note that the implementation of namedtuples is not a particularly
>> nice one (they use dynamic programming). There's also a different
>> implementation available via the C API called structseq (this is used
>> e.g. by the time module).
>>
>> Both create subclasses of the standard Python tuple and are
>> sequences, so both are permitted to be used by the DB-API to
>> represent rows.
>>
> 
> So a dbapi 3.0 implementation could be a proper superset of dbapi 2.0. That
> would be best. -- VC
> 
> 
>> However, using the Python version in an API that potentially
>> returns a few thousand rows is not exactly what I'd recommend
>> to do, since it will slow down the interface a lot.
>>
>> Due to the nature of the namedtuples, you will also see many
>> different objects as row object (each query will return a
>> different object type).
>>
>> My favorite database (the one I help implement once upon a time) had the
> data organized as a virtual array of similar (same class) objects with each
> column as a named attribute. (Those are not the terms we used, since object
> oriented languages had not been invented then.) That's the model I am used
> to thinking in. Our implementation was pretty fast.
> 
> Adodbapi is already quite slow, so I doubt if the extra overhead of a named
> tuple would actually make that much difference to it. Perhaps I should
> download a 2.6 python and muck with it. Those of you working in C would be
> emulating the action of a named tuple for performance reasons, I expect,
> using your own objects. The implementation details would be unimportant,
> provided that the syntax is consistent.
> 
>   I only know that we are already requiring the user to know two languages:
> Python and SQL, in order to use dbapi. Selecting from among the existing ORM
> tool kits, and then learning the tool kit, is more of a learning curve than
> I could hack. I guess that's why I still don't use an Object Relational
> Mapper.
> 
>  I just want named access to my columns somehow -- so I don't add the
> complexity (and inevitable errors) of having to remember the column number
> of each field. The access to each column should feel like native Python

Please see the FAQ entry on this topic for arguments why this feature
didn't make it into the DB-API as requirement.

For some backends it may very well work out and those could use the
namedtuple approach or any other similar method such as providing
mapping like access to the row entries. For others, it may well
not work, e.g. if the database backend doesn't provide the result
set column name information at all or only using generated names
such as "column1", "column2", etc.

I usually use tuple unpacking to get the rows mapped back into
Python and have so far not had any major problems with it, e.g.

for row in cursor.fetchall():
    id, name, property1, property2, data = row

This solves multiple problems you can run into otherwise:

 * columns with generated names (such as aggregate function calls
   without "AS" renaming) are supported

 * column name casing as made available by the database doesn't
   matter

 * freedom to choose different names in Python than those used
   in the database

 * ability to address columns which do not have Python identifier
   compatible names (e.g. non-ASCII ones or names with embedded
   spaces)

 * local access to variable data (which is fast)

Other advantages:

 * tuple unpacking is a fast operation in Python (it's a native
   byte code)

 * by sticking to tuples, you can use pickle to send the data
   over the wire or store it in a file; if you use simple types,
   you can even use marshal instead of pickle

 * "automatic" detection of a mismatch between the number of
   result set columns sent by the database and the number expected
   by the Python program

The namedtuple access would only provide more flexibility if I were
to only use a subset of the result set columns... but then, I wouldn't
retrieve them from the database in the first place.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 29 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611