From cito at online.de Sat Jul 16 21:20:20 2011 From: cito at online.de (Christoph Zwerschke) Date: Sat, 16 Jul 2011 21:20:20 +0200 Subject: [DB-SIG] Some obscurity with paramstyle Message-ID: <4E21E474.2030706@online.de> At the PyGreSQL mailing list we're currently wondering whether the 'format' and 'pyformat' paramstyles allow specifying parameters with types other than '%s' - e.g. can I specify my parameter as '%.2f' or '%(name).2f' if I want to round floats to 2 digits? PEP 249 has only '%s' in the example, but does not exclude other types, does this mean these are allowed? Also, should we define a new paramstyle for the advanced string formatting syntax available since Py 2.6? -- Christoph From mike_mp at zzzcomputing.com Sat Jul 16 22:27:18 2011 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Sat, 16 Jul 2011 16:27:18 -0400 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E21E474.2030706@online.de> References: <4E21E474.2030706@online.de> Message-ID: <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> On Jul 16, 2011, at 3:20 PM, Christoph Zwerschke wrote: > At the PyGreSQL mailing list we're currently wondering whether the 'format' and 'pyformat' paramstyles allow specifying parameters with types other than '%s' - e.g. can I specify my parameter as '%.2f' or '%(name).2f' if I want to round floats to 2 digits? > > PEP 249 has only '%s' in the example, but does not exclude other types, > does this mean these are allowed? > > Also, should we define a new paramstyle for the advanced string formatting syntax available since Py 2.6? Not all backends that support "format" or "pyformat" would be able to allow such behavior - often the client API of the database in use is passed, from the DBAPI, the SQL statement with placeholders and the parameters separately - no "string formatting" takes place. A DBAPI that advertises "format" or "pyformat" then may or may not be able to handle the extended syntax. IMHO I would prefer to see the DBAPI have exactly two paramstyles, named and qmark, and have all DBAPIs support both consistently. The (py)format styles continuously introduce the mixing of Python's string formatting behavior with the presentation of bound parameters, which are two completely different things. From carl at personnelware.com Sun Jul 17 01:06:02 2011 From: carl at personnelware.com (Carl Karsten) Date: Sat, 16 Jul 2011 18:06:02 -0500 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> Message-ID: On Sat, Jul 16, 2011 at 3:27 PM, Michael Bayer wrote: > IMHO I would prefer to see the DBAPI have exactly two paramstyles, named and qmark, and have all DBAPIs support both consistently. ? ?The (py)format styles continuously introduce the mixing of Python's string formatting behavior with the presentation of bound parameters, which are two completely different things. bold +1 /bold -- Carl K From vernondcole at gmail.com Sun Jul 17 07:52:49 2011 From: vernondcole at gmail.com (Vernon Cole) Date: Sat, 16 Jul 2011 23:52:49 -0600 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> Message-ID: On Sat, Jul 16, 2011 at 2:27 PM, Michael Bayer wrote: > > On Jul 16, 2011, at 3:20 PM, Christoph Zwerschke wrote: > > > At the PyGreSQL mailing list we're currently wondering whether the > 'format' and 'pyformat' paramstyles allow specifying parameters with types > other than '%s' - e.g. can I specify my parameter as '%.2f' or '%(name).2f' > if I want to round floats to 2 digits? > > > > PEP 249 has only '%s' in the example, but does not exclude other types, > > does this mean these are allowed? > > > > Also, should we define a new paramstyle for the advanced string > formatting syntax available since Py 2.6? > > Not all backends that support "format" or "pyformat" would be able to allow > such behavior - often the client API of the database in use is passed, from > the DBAPI, the SQL statement with placeholders and the parameters separately > - no "string formatting" takes place. A DBAPI that advertises "format" or > "pyformat" then may or may not be able to handle the extended syntax. > > IMHO I would prefer to see the DBAPI have exactly two paramstyles, named > and qmark, and have all DBAPIs support both consistently. The (py)format > styles continuously introduce the mixing of Python's string formatting > behavior with the presentation of bound parameters, which are two completely > different things. > > Another advantage to supporting only those two paramstyles is that it would make the paramstyle attribute obsolete. If the bound parameters form a sequence, qmark is implied, if a mapping, then named is implied. Disadvantages would include: 1) some database engines seem to use %s format internally, and efficiency would be lost in translation. 2) some major applications (e.g. django) assume %s format. But I am +1 for the idea anyway. -- Vernon Cole -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike_mp at zzzcomputing.com Sun Jul 17 16:54:07 2011 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Sun, 17 Jul 2011 10:54:07 -0400 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> Message-ID: <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> On Jul 17, 2011, at 1:52 AM, Vernon Cole wrote: > 2) some major applications (e.g. django) assume %s format. > > But I am +1 for the idea anyway. But not all DBAPIs support %s format, do you mean that Django assumes %s format for those DBAPIs that are known to do so ? Shouldn't they be at least using .paramstyle to determine that ? Is there a path to changes being made in the DBAPI? i.e. would there be a DBAPI 3 ? From vernondcole at gmail.com Mon Jul 18 03:10:42 2011 From: vernondcole at gmail.com (Vernon Cole) Date: Sun, 17 Jul 2011 19:10:42 -0600 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> Message-ID: On Sun, Jul 17, 2011 at 8:54 AM, Michael Bayer wrote: > > On Jul 17, 2011, at 1:52 AM, Vernon Cole wrote: > > > 2) some major applications (e.g. django) assume %s format. > > > > But I am +1 for the idea anyway. > > But not all DBAPIs support %s format, do you mean that Django assumes %s > format for those DBAPIs that are known to do so ? Shouldn't they be at > least using .paramstyle to determine that ? > > Well, that's how the 2.0 spec assumes things should work: the DBAPI module tells you which paramstyle it uses, then you write your program five different ways to cover the five possibilities. But django does not do that (does anyone??) -- it blindly assumes that all DBAPI's use %s. That's why someone made a fork of adodbapi in order to support Microsoft SQL server in django. ADO uses qmark, so the fork version had to convert %s to qmark before executing the SQL statements. I pulled the format conversion code (and some other improvements) back into the release version of adodbapi, fixed it so that it does not break % signs in literals, and added a feature so that a programmer can request either qmark, format, or named paramstyle. That's why I favor the suggestion. Having written a version which does format conversion, I know that it is not hard to do. Is there a path to changes being made in the DBAPI? i.e. would there be a > DBAPI 3 ? That possibility has been discussed before, and is particularly timely given that it is impossible to write a PEP 249 compliant module in Python 3. [For example, the PEP states that an Error exception "must be a subclass of the Python StandardError" -- which Python 3 does not support.] Marc-Andr? Lemburg (the author of PEP 249) came out against an update -- mostly due to performance reasons. This is understandable, because his company produces a very efficient db api module. The features which have been suggested for a possible DBAPI 3.0 would add some significant overhead in some situations. The worry is real. Compare the mxDateTime module (also from Mark-Andre's company) with the (newer) standard datetime module, and you will see that datetime is a slow dog. (I use it anyway.) On the other hand, GVR is in favor of an update. When MAL suggested that some ease-of-use improvements (such as named columns) would make implementation of a module more difficult, The BDFL replied that we should make things easier for the user, not the module author. [ I am relying on my memory -- my copy of the exchange got deleted and I am too lazy to look it up in the archive. ;) ] So Yes, there are a TON of new language constructs which could be addressed in an updated API version. The use of a cursor as an iterator is supported by most DB API modules, but not mentioned in PEP 249. A cursor and/or a connection should probably be context managers, so that they will work in a "with" statement. There should be a better definition of how BINARY fields work with byte() data. -- Vernon -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike_mp at zzzcomputing.com Mon Jul 18 04:29:56 2011 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Sun, 17 Jul 2011 22:29:56 -0400 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> Message-ID: <4B211916-4B5C-43B9-8A6C-50569083F035@zzzcomputing.com> On Jul 17, 2011, at 9:10 PM, Vernon Cole wrote: > > > On Sun, Jul 17, 2011 at 8:54 AM, Michael Bayer wrote: > > On Jul 17, 2011, at 1:52 AM, Vernon Cole wrote: > > > 2) some major applications (e.g. django) assume %s format. > > > > But I am +1 for the idea anyway. > > But not all DBAPIs support %s format, do you mean that Django assumes %s format for those DBAPIs that are known to do so ? Shouldn't they be at least using .paramstyle to determine that ? > > Well, that's how the 2.0 spec assumes things should work: the DBAPI module tells you which paramstyle it uses, then you write your program five different ways to cover the five possibilities. But django does not do that (does anyone??) If I understand correctly, SQLAlchemy has done it this way since its first release six years ago. It's quite simple to have the param behavior vary independently of a specific DBAPI implementation. > -- it blindly assumes that all DBAPI's use %s. That's why someone made a fork of adodbapi in order to support Microsoft SQL server in django. > ADO uses qmark, so the fork version had to convert %s to qmark before executing the SQL statements. I pulled the format conversion code (and some other improvements) back into the release version of adodbapi, fixed it so that it does not break % signs in literals, and added a feature so that a programmer can request either qmark, format, or named paramstyle. Pysqlite doesn't support %s, how'd they work around that one ? > > Is there a path to changes being made in the DBAPI? i.e. would there be a DBAPI 3 ? > > That possibility has been discussed before, and is particularly timely given that it is impossible to write a PEP 249 compliant module in Python 3. [For example, the PEP states that an Error exception "must be a subclass of the Python StandardError" -- which Python 3 does not support.] yeah StandardError, had a funny thread about that one some weeks ago... > > Marc-Andr? Lemburg (the author of PEP 249) came out against an update -- mostly due to performance reasons. This is understandable, because his company produces a very efficient db api module. The features which have been suggested for a possible DBAPI 3.0 would add some significant overhead in some situations. The worry is real. Compare the mxDateTime module (also from Mark-Andre's company) with the (newer) standard datetime module, and you will see that datetime is a slow dog. (I use it anyway.) The spec should provide that every DBAPI support the standard datetime module by default (though this is tricky for sqlite...); hooks to use alternates like mxDateTime should be present as well as I would assume mxDateTime has a very similar interface to the datetime builtin. If datetime is slow, that's a Python bug - an efficient C implementation should be provided for such a critical piece. Similarly, the cDecimal library should be part of Python too as Decimal is extremely slow (I would hope that DBAPI 3.0 references support for Decimal explicitly - many DBAPIs like pymssql are broken due to the assumption of float for precision numerics). I would hope DBAPI3 can say something about supporting Decimal for precision numerics specifically without using lossful floats as a middle layer (we have a lot of workarounds for different DBAPIs to get this part to work, on some it's impossible). > > On the other hand, GVR is in favor of an update. Given the very bad state of the Unicode situation in DBAPI 3, as well as for precision numerics, i think it's pretty necessary. I would opt that all strings from a version 3 DBAPI can be returned as u'' literals in Python 2 (including down to the names present in cursor.description - pyodbc + MSSQL for example decides to fix these as latin-1 despite other encoding settings). It's been my observation that an encoding parameter is pretty much present in some form for every database client API, this should be an explicit part of the DBAPI, and the encoding present would be used for all bytestring/unicode coercions. If performance is a concern here, a documented system of enabling "unicode returns" can be provided. But in Python 3 these strings are all unicodes anyway, so this behavior will be needed in any case. Python unicodes should also always be accepted for all arguments, including bind parameter values, bind parameter names, and the SQL statement itself. SQLAlchemy has to encode these for some backends, for others it has to leave it alone else things break, and for pyodbc it's dependent on if you're using FreeTDS or not [!]. While it might seem a little strange to suggest that unicode is explicitly allowed for all strings/identifiers/names, as well as cursor.description names, we have users in europe and asia that use non-ascii identifiers all the time (and they are accepted by most databases), and we've bent over backwards for years to support them all. Pysqlite is the only DBAPI that seems to have "all unicode everywhere" behavior, with unicode support to extremely varying degrees in all the others (with Psycopg2 probably doing the best job at coming close - even though you need to use non-standard extension calls to enable it fully). On the unicode issue, cx_oracle had a totally complicated progression in version 5.0 when they introduced this poorly-considered UNICODE mode, derived from their Python 3 implementation but made available as a static build option for Python 2, where even the connect() call wouldn't accept any incoming string that was *not* unicode ! Then some user insisted that he absolutely *needed* this ridiculous mode, more bending over backwards for SQLAlchemy to support it, then cx_oracle 5.1, realizing they should just be using NLS_LANG (i.e. the "encoding" parameter provided by OCI), they *removed* the whole thing entirely. A long effort that caused wasted time for many. Explicit, non-ambiguous guidance on unicode handling is sorely needed. > When MAL suggested that some ease-of-use improvements (such as named columns) would make implementation of a module more difficult, The BDFL replied that we should make things easier for the user, not the module author. [ I am relying on my memory -- my copy of the exchange got deleted and I am too lazy to look it up in the archive. ;) ] Ease of use things like named columns, connection.execute(), context manager, these are all fairly trivial adds for all DBAPIs. They are of course all fine. But IMHO this is not the critical piece. It's about behaviors that everyone needs to deal with, like can I send u'select * from table' to execute() or must it be 'select * from table', can the "sequence" sent to execute() be a tuple *or* a list (yes this varies, *incompatibly*, between DBAPIs, some accepting only tuple, others accepting only list), that each DBAPI is currently doing in an arbitrary and different fashion with no guidance establishing non-compliant behaviors as bugs. If the spec were just more decisive about things, I could just go to all these DBAPI authors that all have their own opinions about things and say "no, this is it". Right now, DBAPIs are out there with all kinds of behaviors, I can walk you through a large number of them that we've had to normalize. None of them are bugs. Once the spec comes out and takes a stand, they all become bugs, and until fixed the DBAPi is not spec-compliant. > > So Yes, there are a TON of new language constructs which could be addressed in an updated API version. The use of a cursor as an iterator is supported by most DB API modules, but not mentioned in PEP 249. A cursor and/or a connection should probably be context managers, so that they will work in a "with" statement. There should be a better definition of how BINARY fields work with byte() data. > -- > Vernon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Jul 18 09:33:06 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 18 Jul 2011 09:33:06 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E21E474.2030706@online.de> References: <4E21E474.2030706@online.de> Message-ID: <4E23E1B2.1080303@egenix.com> Christoph Zwerschke wrote: > At the PyGreSQL mailing list we're currently wondering whether the > 'format' and 'pyformat' paramstyles allow specifying parameters with > types other than '%s' - e.g. can I specify my parameter as '%.2f' or > '%(name).2f' if I want to round floats to 2 digits? > > PEP 249 has only '%s' in the example, but does not exclude other types, > does this mean these are allowed? Not really. The DB-API 2.0 allows for a certain number of binding parameter formats, but you shouldn't really invent more. For the next version, we're likely going to reduce the number of allowed formats to just two (%s and ?, IIRC), since having lots of different formats has resulted in much confusion. > Also, should we define a new paramstyle for the advanced string > formatting syntax available since Py 2.6? No. See above. The problem with %s or any other format that resembles a Python string formatting character is that people (rightly) expect the full functionality of those to be available, which is not the case. There's a significant difference between string formatting characters and binding parameter characters: the latter just let the database module know where a certain parameter value should be added, but don't literally include the parameters in the SQL string, nor define how the parameters are passed to the database. Binding parameters just define an association between the SQL and the parameter list, nothing more. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 18 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Mon Jul 18 09:33:50 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 18 Jul 2011 09:33:50 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> Message-ID: <4E23E1DE.10005@egenix.com> Michael Bayer wrote: > > On Jul 16, 2011, at 3:20 PM, Christoph Zwerschke wrote: > >> At the PyGreSQL mailing list we're currently wondering whether the 'format' and 'pyformat' paramstyles allow specifying parameters with types other than '%s' - e.g. can I specify my parameter as '%.2f' or '%(name).2f' if I want to round floats to 2 digits? >> >> PEP 249 has only '%s' in the example, but does not exclude other types, >> does this mean these are allowed? >> >> Also, should we define a new paramstyle for the advanced string formatting syntax available since Py 2.6? > > Not all backends that support "format" or "pyformat" would be able to allow such behavior - often the client API of the database in use is passed, from the DBAPI, the SQL statement with placeholders and the parameters separately - no "string formatting" takes place. A DBAPI that advertises "format" or "pyformat" then may or may not be able to handle the extended syntax. > > IMHO I would prefer to see the DBAPI have exactly two paramstyles, named and qmark, and have all DBAPIs support both consistently. The (py)format styles continuously introduce the mixing of Python's string formatting behavior with the presentation of bound parameters, which are two completely different things. +1 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 18 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Mon Jul 18 09:36:28 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 18 Jul 2011 09:36:28 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> Message-ID: <4E23E27C.7060303@egenix.com> Michael Bayer wrote: > > On Jul 17, 2011, at 1:52 AM, Vernon Cole wrote: > >> 2) some major applications (e.g. django) assume %s format. >> >> But I am +1 for the idea anyway. > > But not all DBAPIs support %s format, do you mean that Django assumes %s format for those DBAPIs that are known to do so ? Shouldn't they be at least using .paramstyle to determine that ? > > Is there a path to changes being made in the DBAPI? i.e. would there be a DBAPI 3 ? Yes, such changes would have to go into a DB API 3. See http://ep2011.europython.eu/conference/talks/introduction-to-python-database-programming for some other things I'd like to see in DBAPI 3. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 18 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Mon Jul 18 09:52:02 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 18 Jul 2011 09:52:02 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> Message-ID: <4E23E622.6000706@egenix.com> Vernon Cole wrote: > On Sun, Jul 17, 2011 at 8:54 AM, Michael Bayer wrote: > >Is there a path to changes being made in the DBAPI? i.e. would there be a >> DBAPI 3 ? > > > That possibility has been discussed before, and is particularly timely given > that it is impossible to write a PEP 249 compliant module in Python 3. [For > example, the PEP states that an Error exception "must be a subclass of the > Python StandardError" -- which Python 3 does not support.] > > Marc-Andr? Lemburg (the author of PEP 249) came out against an update -- > mostly due to performance reasons. Not sure, where you read that :-) I'm not opposed to a DB API 3, but since there are only a few things on the table for DB API 3 and the adoption of Python 3.x has just started, so it's not all that urgent. > This is understandable, because his > company produces a very efficient db api module. The features which have > been suggested for a possible DBAPI 3.0 would add some significant overhead > in some situations. The worry is real. Compare the mxDateTime module (also > from Mark-Andre's company) with the (newer) standard datetime module, and > you will see that datetime is a slow dog. (I use it anyway.) > > On the other hand, GVR is in favor of an update. When MAL suggested that > some ease-of-use improvements (such as named columns) would make > implementation of a module more difficult, The BDFL replied that we should > make things easier for the user, not the module author. [ I am relying on my > memory -- my copy of the exchange got deleted and I am too lazy to look it > up in the archive. ;) ] > > So Yes, there are a TON of new language constructs which could be addressed > in an updated API version. The use of a cursor as an iterator is supported > by most DB API modules, but not mentioned in PEP 249. Please have a look again. Note that we have added quite a few (optional) extensions to the DB API 2 in recent years, so even though the version number and the base set of required features hasn't changed, there are many feature already addresses in the DB API. > A cursor and/or a > connection should probably be context managers, so that they will work in a > "with" statement. This is was discussed before and it should probably go into the DB API in some form. I'm no particular fan of hiding transactions in context managers, though. In practice this often causes problems, since you usually want to apply error handling logic in case of problems, other than simply issuing a .rollback(). Another problem is that the connections used in a with statement will usually have already started a transaction before entering the context, so the .rollback() would remove more than just the things added in the with context. There's also the problem of intuitive use: some users will likely expect the connection to also be closed when leaving the context - much like what happens with files. The case for having cursors automatically closed by is not problematic > There should be a better definition of how BINARY fields > work with byte() data. Right. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 18 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From james at jamesh.id.au Mon Jul 18 11:25:43 2011 From: james at jamesh.id.au (James Henstridge) Date: Mon, 18 Jul 2011 17:25:43 +0800 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E23E622.6000706@egenix.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> Message-ID: On Mon, Jul 18, 2011 at 3:52 PM, M.-A. Lemburg wrote: > Vernon Cole wrote: >> A cursor and/or a >> connection should probably be context managers, so that they will work in a >> "with" statement. > > This is was discussed before and it should probably go into > the DB API in some form. > > I'm no particular fan of hiding transactions in context > managers, though. In practice this often causes problems, since > you usually want to apply error handling logic in case of > problems, other than simply issuing a .rollback(). For most of the applications I work with, the transaction handling has been delegated to function decorators, or have it hidden in the framework (e.g. Django's TransactionMiddleware). If there are clean up tasks that need to happen on transaction commit or roll back (e.g. deleting a file on a failed transaction), then using a global transaction manager like Zope's transaction module seems to be a good fit. For transaction retry or error reporting, I haven't seen much benefit in leaving the transaction in an "open by broken" state over cleaning up with a rollback. > Another problem is that the connections used in a with > statement will usually have already started a transaction > before entering the context, so the .rollback() would > remove more than just the things added in the with > context. None of the databases I've worked with fit the Python DB API's "implicit begin" behaviour for transactions, so their adapters have all needed to run a simple state machine to determine when it is necessary to start a transaction. So it would be pretty easy for them to raise an error if an attempt was made to use the context manager when a transaction was already in progress. If all adapters that support transactions function like that, then I think it would be sensible to require them to raise an error in that case. If they don't raise an error, you know that someone somewhere is going to write code like the following: with transaction: # do stuff with transaction: # do more stuff ... and wonder why things break. > There's also the problem of intuitive use: some users will > likely expect the connection to also be closed when leaving > the context - much like what happens with files. If that is a concern, you could require a method call to get the context manager rather than making the connection itself a context manager. For example: with connection.begin_transaction(): ... > The case for having cursors automatically closed by > is not problematic I don't think it makes sense to make cursors behave as context managers for transaction handling. Transactions are a connection level concept affecting all the cursors for the connection, so this would just confuse matters and lead to coding errors. James. From mal at egenix.com Mon Jul 18 11:34:45 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 18 Jul 2011 11:34:45 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> Message-ID: <4E23FE35.6070007@egenix.com> James Henstridge wrote: > On Mon, Jul 18, 2011 at 3:52 PM, M.-A. Lemburg wrote: >> The case for having cursors automatically closed by >> is not problematic > > I don't think it makes sense to make cursors behave as context > managers for transaction handling. Transactions are a connection > level concept affecting all the cursors for the connection, so this > would just confuse matters and lead to coding errors. Just to clarify: The idea is to open a cursor and have the context manager close it automatically. This doesn't have anything to do with the transaction on the connection. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 18 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From cito at online.de Mon Jul 18 12:44:14 2011 From: cito at online.de (Christoph Zwerschke) Date: Mon, 18 Jul 2011 12:44:14 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E23E622.6000706@egenix.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> Message-ID: <4E240E7E.6070005@online.de> Am 18.07.2011 09:33 schrieb M.-A. Lemburg: > Not really. The DB-API 2.0 allows for a certain number of binding > parameter formats, but you shouldn't really invent more. My main problem was that the specs do not even properly explain what these formats are. E.g. they say 'format' means ANSI C printf format codes, and ANSI C means more than just '%s'. So when writing a driver, do I need to support '%f' or not? A new version of the PEP should be more specific here. I agree wth you it makes sense to reduce the number of formats instead of adding more. Am 18.07.2011 09:52 schrieb M.-A. Lemburg: > I'm not opposed to a DB API 3, but since there are only a few > things on the table for DB API 3 and the adoption of Python 3.x > has just started, so it's not all that urgent. But you can also see it the other way round. Adoption of Python 3.x is so slow because important standards like DB-API or WSGI have not been rewritten early enough so nobody knows how to properly implement these. Therefore if we want to promote Py 3 it is urgent to do something. > Note that we have added quite a few (optional) extensions to the > DB API 2 in recent years, so even though the version number and > the base set of required features hasn't changed, there are many > feature already addresses in the DB API. But if these are optional, you have the same problem as with the many parameter formats. People must catch the warning saying the extension is not supported and must write code for both case. At least some of the extensions like iterator support should become mandatory in the next version of the specs. -- Christoph From mal at egenix.com Mon Jul 18 13:20:07 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 18 Jul 2011 13:20:07 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E240E7E.6070005@online.de> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> <4E240E7E.6070005@online.de> Message-ID: <4E2416E7.1050400@egenix.com> Christoph Zwerschke wrote: > Am 18.07.2011 09:33 schrieb M.-A. Lemburg: >> Not really. The DB-API 2.0 allows for a certain number of binding >> parameter formats, but you shouldn't really invent more. > > My main problem was that the specs do not even properly explain what > these formats are. E.g. they say 'format' means ANSI C printf format > codes, and ANSI C means more than just '%s'. True. Some modules are also using %i and %f to bind integers and floats. > So when writing a driver, > do I need to support '%f' or not? The question is not whether you need to support them. You can choose among the available formats. You don't need to support all of them. One will do just fine. However, it's good practice to look around a bit at what other modules are using. > A new version of the PEP should be > more specific here. I agree wth you it makes sense to reduce the number > of formats instead of adding more. > > Am 18.07.2011 09:52 schrieb M.-A. Lemburg: >> I'm not opposed to a DB API 3, but since there are only a few >> things on the table for DB API 3 and the adoption of Python 3.x >> has just started, so it's not all that urgent. > > But you can also see it the other way round. Adoption of Python 3.x is > so slow because important standards like DB-API or WSGI have not been > rewritten early enough so nobody knows how to properly implement these. > Therefore if we want to promote Py 3 it is urgent to do something. There are not a lot of Python 3 changes necessary and most of the ones that are (like the exception base class StandardError which then reads Exception) are obvious fixes. >> Note that we have added quite a few (optional) extensions to the >> DB API 2 in recent years, so even though the version number and >> the base set of required features hasn't changed, there are many >> feature already addresses in the DB API. > > But if these are optional, you have the same problem as with the many > parameter formats. People must catch the warning saying the extension is > not supported and must write code for both case. At least some of the > extensions like iterator support should become mandatory in the next > version of the specs. That will be up for discussion, yes. We will likely make extensions that are easy to implement for a lot of database backends, mandatory in DB API 3. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 18 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From cito at online.de Mon Jul 18 14:25:03 2011 From: cito at online.de (Christoph Zwerschke) Date: Mon, 18 Jul 2011 14:25:03 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E2416E7.1050400@egenix.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> <4E240E7E.6070005@online.de> <4E2416E7.1050400@egenix.com> Message-ID: <4E24261F.3080600@online.de> Am 18.07.2011 13:20 schrieb M.-A. Lemburg: > True. Some modules are also using %i and %f to bind > integers and floats. > >> So when writing a driver, >> do I need to support '%f' or not? > > The question is not whether you need to support them. You can > choose among the available formats. You don't need to support > all of them. One will do just fine. Sorry, I was unclear. I meant *if* my driver advertizes itself with paramstyle='format', does it need to support '%f' or does it suffice to support '%s'? Same for 'pyformat'? From your answer above, it would need to support '%f' as well. And does it also need to support flags, width, precision, length modifiers such as '%.2f'? My concrete problem was that I wanted to support Infinity and NaN where Postgres has a different spelling from Python and needs quotes. This is more difficult to implement if '%f' is allowed. > There are not a lot of Python 3 changes necessary and most of > the ones that are (like the exception base class StandardError > which then reads Exception) are obvious fixes. But as Michael already pointed out, at least the unicode related issues are not so obvious. That was also the reason why WSGI needed a new PEP. -- Christoph From fog at initd.org Mon Jul 18 14:35:37 2011 From: fog at initd.org (Federico Di Gregorio) Date: Mon, 18 Jul 2011 14:35:37 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E24261F.3080600@online.de> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> <4E240E7E.6070005@online.de> <4E2416E7.1050400@egenix.com> <4E24261F.3080600@online.de> Message-ID: <4E242899.5070906@initd.org> On 18/07/11 14:25, Christoph Zwerschke wrote: > Am 18.07.2011 13:20 schrieb M.-A. Lemburg: >> True. Some modules are also using %i and %f to bind >> integers and floats. >> >>> So when writing a driver, >>> do I need to support '%f' or not? >> >> The question is not whether you need to support them. You can >> choose among the available formats. You don't need to support >> all of them. One will do just fine. > > Sorry, I was unclear. I meant *if* my driver advertizes itself with > paramstyle='format', does it need to support '%f' or does it suffice to > support '%s'? Same for 'pyformat'? From your answer above, it would need > to support '%f' as well. And does it also need to support flags, width, > precision, length modifiers such as '%.2f'? > > My concrete problem was that I wanted to support Infinity and NaN where > Postgres has a different spelling from Python and needs quotes. This is > more difficult to implement if '%f' is allowed. IMO, nothing except %s is allowed. Letting the user choose a format it is very much like letting the user do the escaping: Bad Thing (TM). federico -- Federico Di Gregorio fog at initd.org Qu'est ce que la folie? Juste un sentiment de libert? si fort qu'on en oublie ce qui nous rattache au monde... -- J. de Loctra -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From mal at egenix.com Mon Jul 18 14:54:31 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 18 Jul 2011 14:54:31 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E24261F.3080600@online.de> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> <4E240E7E.6070005@online.de> <4E2416E7.1050400@egenix.com> <4E24261F.3080600@online.de> Message-ID: <4E242D07.3090909@egenix.com> Christoph Zwerschke wrote: > Am 18.07.2011 13:20 schrieb M.-A. Lemburg: >> True. Some modules are also using %i and %f to bind >> integers and floats. >> >>> So when writing a driver, >>> do I need to support '%f' or not? >> >> The question is not whether you need to support them. You can >> choose among the available formats. You don't need to support >> all of them. One will do just fine. > > Sorry, I was unclear. I meant *if* my driver advertizes itself with > paramstyle='format', does it need to support '%f' or does it suffice to > support '%s'? Same for 'pyformat'? From your answer above, it would need > to support '%f' as well. And does it also need to support flags, width, > precision, length modifiers such as '%.2f'? > > My concrete problem was that I wanted to support Infinity and NaN where > Postgres has a different spelling from Python and needs quotes. This is > more difficult to implement if '%f' is allowed. The paramstyle only advertises the supported formatting style. With "format" the supported formatting variants are module dependent, so it's up to you what to support. >> There are not a lot of Python 3 changes necessary and most of >> the ones that are (like the exception base class StandardError >> which then reads Exception) are obvious fixes. > > But as Michael already pointed out, at least the unicode related issues > are not so obvious. That was also the reason why WSGI needed a new PEP. True, the DB-API should provide some guidance here as well. Note that the DB-API spec will have to be valid for Python 2 and 3, so special care has to be taken. There are two things to consider: 1. parameters 2. data input and output For parameters, I think the best way to handle this is by allowing both bytes and Unicode for parameters. Perhaps we can add a module global which then signals the supported variants. For data input/output, the situation is difficult, since not all backends support Unicode, some only configuration dependent character sets and some others are basically ASCII/binary data only. We will likely have to introduce a new TEXT() constructor which maps data objects to text data and then takes care of the database specific encoding issues. There are similar issues with float/decimal and date/time values. It would be great if we could resolve all of these using a data type mapping facility that defines input and output mappings in an efficient and flexible way. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 18 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Mon Jul 18 16:04:18 2011 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Mon, 18 Jul 2011 10:04:18 -0400 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E23E1B2.1080303@egenix.com> References: <4E21E474.2030706@online.de> <4E23E1B2.1080303@egenix.com> Message-ID: <02711192-5402-4136-B416-2F7F0BF44572@zzzcomputing.com> On Jul 18, 2011, at 3:33 AM, M.-A. Lemburg wrote: > > For the next version, we're likely going to reduce the number > of allowed formats to just two (%s and ?, IIRC), since having > lots of different formats has resulted in much confusion. Why have both %s and ? (both positional?) If there is to only be positional, what purpose is served by having two choices for the format ? Also is it accurate that named parameters, support for dictionaries passed to execute(), will be dropped ? This leads things back in the "more difficult to use" direction IMHO. From mal at egenix.com Mon Jul 18 16:08:12 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 18 Jul 2011 16:08:12 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <02711192-5402-4136-B416-2F7F0BF44572@zzzcomputing.com> References: <4E21E474.2030706@online.de> <4E23E1B2.1080303@egenix.com> <02711192-5402-4136-B416-2F7F0BF44572@zzzcomputing.com> Message-ID: <4E243E4C.3020800@egenix.com> Michael Bayer wrote: > > On Jul 18, 2011, at 3:33 AM, M.-A. Lemburg wrote: > >> >> For the next version, we're likely going to reduce the number >> of allowed formats to just two (%s and ?, IIRC), since having >> lots of different formats has resulted in much confusion. > > Why have both %s and ? (both positional?) If there is to only be positional, what purpose is served by having two choices for the format ? It's well possible that I've misremembered the ones we picked in a discussion some time ago. Could well have been named and qmark. > Also is it accurate that named parameters, support for dictionaries passed to execute(), will be dropped ? This leads things back in the "more difficult to use" direction IMHO. Depends on which parameter types we choose. Note that I don't think we'll actually drop the support, just require support for two types and leave the others optionally supported as well. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 18 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Mon Jul 18 16:15:17 2011 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Mon, 18 Jul 2011 10:15:17 -0400 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E242D07.3090909@egenix.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> <4E240E7E.6070005@online.de> <4E2416E7.1050400@egenix.com> <4E24261F.3080600@online.de> <4E242D07.3090909@egenix.com> Message-ID: <9D8D1ECE-E0AD-4724-866D-790E7CB1C66D@zzzcomputing.com> On Jul 18, 2011, at 8:54 AM, M.-A. Lemburg wrote: > > For data input/output, the situation is difficult, since not > all backends support Unicode, some only configuration dependent > character sets and some others are basically ASCII/binary data > only. OK, but even those DBAPIs can present Python unicode objects on the user-facing side, at least as an option (the module global you mention speaks to this - I would suggest the parameter is available as a global and on the connection object). If the only encoding available is "ascii", then that's all you have. Of course bytes should be accepted as well, though some DBAPIs have problems with this. I think passing unicode + an encoding is known at the global/connection level gives the widest latitude to the DBAPI itself in dealing with textual data. > > We will likely have to introduce a new TEXT() constructor > which maps data objects to text data and then takes care > of the database specific encoding issues. > > There are similar issues with float/decimal and date/time > values. > > It would be great if we could resolve all of these using > a data type mapping facility that defines input and output > mappings in an efficient and flexible way. Several DBAPIs have these already, but particularly for the Decimal issue, it's critical that numerics aren't presented as Python floats before there is even a chance to extract the information. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Jul 18 2011) >>>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > http://mail.python.org/mailman/listinfo/db-sig From mike_mp at zzzcomputing.com Tue Jul 19 00:52:58 2011 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Mon, 18 Jul 2011 18:52:58 -0400 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E23E622.6000706@egenix.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> Message-ID: <1DA573C9-A15A-412B-A606-D4C185497E02@zzzcomputing.com> On Jul 18, 2011, at 3:52 AM, M.-A. Lemburg wrote: > Vernon Cole wrote: >> On Sun, Jul 17, 2011 at 8:54 AM, Michael Bayer wrote: >>> Is there a path to changes being made in the DBAPI? i.e. would there be a >>> DBAPI 3 ? >> >> >> That possibility has been discussed before, and is particularly timely given >> that it is impossible to write a PEP 249 compliant module in Python 3. [For >> example, the PEP states that an Error exception "must be a subclass of the >> Python StandardError" -- which Python 3 does not support.] >> >> Marc-Andr? Lemburg (the author of PEP 249) came out against an update -- >> mostly due to performance reasons. > > Not sure, where you read that :-) > > I'm not opposed to a DB API 3, but since there are only a few > things on the table for DB API 3 and the adoption of Python 3.x > has just started, so it's not all that urgent. Do you have an idea in mind when effort on the new spec might be happening ? If I can find the time I'd want to contribute a compliance suite, and/or update the old one (since I'd like DBAPI 3 to be a lot more specific about some things). From mal at egenix.com Tue Jul 19 01:00:31 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 19 Jul 2011 01:00:31 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <1DA573C9-A15A-412B-A606-D4C185497E02@zzzcomputing.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> <1DA573C9-A15A-412B-A606-D4C185497E02@zzzcomputing.com> Message-ID: <4E24BB0F.5080202@egenix.com> Michael Bayer wrote: > > On Jul 18, 2011, at 3:52 AM, M.-A. Lemburg wrote: > >> Vernon Cole wrote: >>> On Sun, Jul 17, 2011 at 8:54 AM, Michael Bayer wrote: >>>> Is there a path to changes being made in the DBAPI? i.e. would there be a >>>> DBAPI 3 ? >>> >>> >>> That possibility has been discussed before, and is particularly timely given >>> that it is impossible to write a PEP 249 compliant module in Python 3. [For >>> example, the PEP states that an Error exception "must be a subclass of the >>> Python StandardError" -- which Python 3 does not support.] >>> >>> Marc-Andr? Lemburg (the author of PEP 249) came out against an update -- >>> mostly due to performance reasons. >> >> Not sure, where you read that :-) >> >> I'm not opposed to a DB API 3, but since there are only a few >> things on the table for DB API 3 and the adoption of Python 3.x >> has just started, so it's not all that urgent. > > Do you have an idea in mind when effort on the new spec might be happening ? If I can find the time I'd want to contribute a compliance suite, and/or update the old one (since I'd like DBAPI 3 to be a lot more specific about some things). We might as well start now. I've been wanting to kick off the process for about a year now, but never really got around to it (too many other projects). The talks I gave at EuroPython mentions a few things I think should go into the DB API 3. The list archives will likely have a few more. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 19 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From federico.digregorio at dndg.it Mon Jul 18 11:28:55 2011 From: federico.digregorio at dndg.it (Federico Di Gregorio) Date: Mon, 18 Jul 2011 11:28:55 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> Message-ID: <4E23FCD7.4050304@dndg.it> On 18/07/11 11:25, James Henstridge wrote: > If all adapters that support transactions function like that, then I > think it would be sensible to require them to raise an error in that > case. If they don't raise an error, you know that someone somewhere > is going to write code like the following: > > with transaction: > # do stuff > with transaction: > # do more stuff > > ... and wonder why things break. This can make sense for backends that support check points (subtransactions). PostgreSQL is an example (even if we don't yet support checkpoints in psycopg). federico -- Federico Di Gregorio federico.digregorio at dndg.it Studio Associato Di Nunzio e Di Gregorio http://dndg.it Quis custodiet ipsos custodes? -- Juvenal, Satires, VI, 347 From federico.digregorio at dndg.it Mon Jul 18 15:01:50 2011 From: federico.digregorio at dndg.it (Federico Di Gregorio) Date: Mon, 18 Jul 2011 15:01:50 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E242D07.3090909@egenix.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> <4E240E7E.6070005@online.de> <4E2416E7.1050400@egenix.com> <4E24261F.3080600@online.de> <4E242D07.3090909@egenix.com> Message-ID: <4E242EBE.2010806@dndg.it> On 18/07/11 14:54, M.-A. Lemburg wrote: [snip] > We will likely have to introduce a new TEXT() constructor > which maps data objects to text data and then takes care > of the database specific encoding issues. > > There are similar issues with float/decimal and date/time > values. > > It would be great if we could resolve all of these using > a data type mapping facility that defines input and output > mappings in an efficient and flexible way. At least 2 drivers (psycopg and pysqlite) provide a Python->backend mechanism based on PEP 246, "Object Adaptation". If other implementors are interested I can write a short explanation about how it works and why it was chosen only for the Python->backend path and not for the reverse. federico -- Federico Di Gregorio federico.digregorio at dndg.it Studio Associato Di Nunzio e Di Gregorio http://dndg.it Se sai che hai un ***** di file cos?, lo manovri subito. -- vodka From vernondcole at gmail.com Tue Jul 19 12:05:26 2011 From: vernondcole at gmail.com (Vernon Cole) Date: Tue, 19 Jul 2011 04:05:26 -0600 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E242EBE.2010806@dndg.it> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> <4E240E7E.6070005@online.de> <4E2416E7.1050400@egenix.com> <4E24261F.3080600@online.de> <4E242D07.3090909@egenix.com> <4E242EBE.2010806@dndg.it> Message-ID: Yes, please. I would like to see that. Adodbapi has an output conversion mechanism, but I don't like it particularly well. I would be happy to create a fork of adodbapi for a reference implementation. It will talk to any major database engine, but unfortunately, only on Windows. Vernon Cole (sent from my 'droid phone) On Jul 19, 2011 3:44 AM, "Federico Di Gregorio" wrote: > On 18/07/11 14:54, M.-A. Lemburg wrote: > [snip] >> We will likely have to introduce a new TEXT() constructor >> which maps data objects to text data and then takes care >> of the database specific encoding issues. >> >> There are similar issues with float/decimal and date/time >> values. >> >> It would be great if we could resolve all of these using >> a data type mapping facility that defines input and output >> mappings in an efficient and flexible way. > > At least 2 drivers (psycopg and pysqlite) provide a Python->backend > mechanism based on PEP 246, "Object Adaptation". If other implementors > are interested I can write a short explanation about how it works and > why it was chosen only for the Python->backend path and not for the reverse. > > federico > > -- > Federico Di Gregorio federico.digregorio at dndg.it > Studio Associato Di Nunzio e Di Gregorio http://dndg.it > Se sai che hai un ***** di file cos?, lo manovri subito. -- vodka > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > http://mail.python.org/mailman/listinfo/db-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniele.varrazzo at gmail.com Tue Jul 19 13:49:24 2011 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Tue, 19 Jul 2011 12:49:24 +0100 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> <4E240E7E.6070005@online.de> <4E2416E7.1050400@egenix.com> <4E24261F.3080600@online.de> <4E242D07.3090909@egenix.com> <4E242EBE.2010806@dndg.it> Message-ID: > On Jul 19, 2011 3:44 AM, "Federico Di Gregorio" > wrote: >> At least 2 drivers (psycopg and pysqlite) provide a Python->backend >> mechanism based on PEP 246, "Object Adaptation". If other implementors >> are interested I can write a short explanation about how it works and >> why it was chosen only for the Python->backend path and not for the >> reverse. On Tue, Jul 19, 2011 at 11:05 AM, Vernon Cole wrote: > Yes, please. I would like to see that. Adodbapi has an output conversion > mechanism, but I don't like it particularly well. For a description: http://initd.org/psycopg/docs/advanced.html#adapting-new-python-types-to-sql-syntax http://initd.org/psycopg/docs/advanced.html#type-casting-of-sql-types-into-python-objects -- Daniele From mal at egenix.com Tue Jul 19 14:18:34 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 19 Jul 2011 14:18:34 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> <4E240E7E.6070005@online.de> <4E2416E7.1050400@egenix.com> <4E24261F.3080600@online.de> <4E242D07.3090909@egenix.com> <4E242EBE.2010806@dndg.it> Message-ID: <4E25761A.9070406@egenix.com> Daniele Varrazzo wrote: >> On Jul 19, 2011 3:44 AM, "Federico Di Gregorio" >> wrote: >>> At least 2 drivers (psycopg and pysqlite) provide a Python->backend >>> mechanism based on PEP 246, "Object Adaptation". If other implementors >>> are interested I can write a short explanation about how it works and >>> why it was chosen only for the Python->backend path and not for the >>> reverse. > > On Tue, Jul 19, 2011 at 11:05 AM, Vernon Cole wrote: >> Yes, please. I would like to see that. Adodbapi has an output conversion >> mechanism, but I don't like it particularly well. > > For a description: > > http://initd.org/psycopg/docs/advanced.html#adapting-new-python-types-to-sql-syntax > http://initd.org/psycopg/docs/advanced.html#type-casting-of-sql-types-into-python-objects While this is a nice system, it's also very slow. It uses function calls and string parsing/conversion for adapting each value. This works if you only have to insert/fetch a few rows, but won't be feasible for larger volumes. I think we need something more low-level, which tries to avoid (Python) function calls if possible, e.g. it should be possible to write adapters in C and only point to them using a symbol (sketching here): cursor.setinputconverter(BINARY, module.BINARY_INPUT_CONVERTER) cursor.setoutputconverter(BINARY, module.BINARY_OUTPUT_CONVERTER) For mxODBC we'd then use something like this: # Convert Python unicode object data to SQLWCHAR data cursor.setinputconverter(unicode, module.SQLWCHAR_INPUT_CONVERTER) # Convert SQL_WCHAR type code data to Python unicode cursor.setoutputconverter(SQL.WCHAR, module.SQLCHAR_OUTPUT_CONVERTER_UNICODE) The advantage here is that the database module could work directly on the internal data structure to implement the conversion rather than having to round-trip to Python. In order to provide the psycopg style adaption mechanism, the converter functions would have to be registered with converter codes, e.g. module.registerinputconverter(module.POINT_INPUT_CONVERTER, adapt_point) module.registeroutputconverter(module.POINT_OUTPUT_CONVERTER, cast_point) Lookup would then be a simple dictionary lookup on output (via the column type code) and a dictionary lookup/isinstance() loop for input mappings. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 19 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From Chris.Clark at ingres.com Tue Jul 19 18:37:48 2011 From: Chris.Clark at ingres.com (Chris Clark) Date: Tue, 19 Jul 2011 09:37:48 -0700 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E25761A.9070406@egenix.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> <4E240E7E.6070005@online.de> <4E2416E7.1050400@egenix.com> <4E24261F.3080600@online.de> <4E242D07.3090909@egenix.com> <4E242EBE.2010806@dndg.it> <4E25761A.9070406@egenix.com> Message-ID: <4E25B2DC.8070602@ingres.com> M.-A. Lemburg wrote: > Daniele Varrazzo wrote: > >>> On Jul 19, 2011 3:44 AM, "Federico Di Gregorio" >>> wrote: >>> >>>> At least 2 drivers (psycopg and pysqlite) provide a Python->backend >>>> mechanism based on PEP 246, "Object Adaptation". If other implementors >>>> are interested I can write a short explanation about how it works and >>>> why it was chosen only for the Python->backend path and not for the >>>> reverse. >>>> >> On Tue, Jul 19, 2011 at 11:05 AM, Vernon Cole wrote: >> >>> Yes, please. I would like to see that. Adodbapi has an output conversion >>> mechanism, but I don't like it particularly well. >>> >> For a description: >> >> http://initd.org/psycopg/docs/advanced.html#adapting-new-python-types-to-sql-syntax >> http://initd.org/psycopg/docs/advanced.html#type-casting-of-sql-types-into-python-objects >> > > While this is a nice system, it's also very slow. It uses function > calls and string parsing/conversion for adapting each value. This > works if you only have to insert/fetch a few rows, but won't be > feasible for larger volumes. > > I think we need something more low-level, which tries to > avoid (Python) function calls if possible, ..... I have to confess the only piece of pep-0246 I read was: > > Rejection Notice > > I'm rejecting this PEP. Something much better is about to happen; > it's too early to say exactly what, but it's not going to resemble > the proposal in this PEP too closely so it's better to start a new > PEP. GvR. > Does anyone know what the "better" option was? The date on the pep is a number of years ago. Chris From ethan at stoneleaf.us Wed Jul 20 01:39:45 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 19 Jul 2011 16:39:45 -0700 Subject: [DB-SIG] dBase III and VFP tables Message-ID: <4E2615C1.1060604@stoneleaf.us> Greetings, I am finally adding Null support to my dbf package (http://pypi.python.org/pypi/dbf) and, as usual, the VFP side is being a pain. The issue is that the Logical, Date, and DateTime field types, even when the table does not allow Null values, can be empty -- and when Null values are allowed, those fields can be empty or Null. (Empty pretty much meaning not initialized -- neither True nor False, and no Date nor Time.) At this point, also, my module does not support the DB API (that is a goal though), so right now you pretty much get direct access to the physical table. My question: for the direct access side, is there any advantage to treating an empty L, D, or T field differently from a Null L, D, or T field? Or should they be semantically the same, always returning, for example, None whether it's actually Null in the table or just empty? ~Ethan~ From carl at personnelware.com Wed Jul 20 01:53:57 2011 From: carl at personnelware.com (Carl Karsten) Date: Tue, 19 Jul 2011 18:53:57 -0500 Subject: [DB-SIG] dBase III and VFP tables In-Reply-To: <4E2615C1.1060604@stoneleaf.us> References: <4E2615C1.1060604@stoneleaf.us> Message-ID: On Tue, Jul 19, 2011 at 6:39 PM, Ethan Furman wrote: > Greetings, > > I am finally adding Null support to my dbf package > (http://pypi.python.org/pypi/dbf) and, as usual, the VFP side is being a > pain. > > The issue is that the Logical, Date, and DateTime field types, even when the > table does not allow Null values, can be empty -- and when Null values are > allowed, those fields can be empty or Null. ?(Empty pretty much meaning not > initialized -- neither True nor False, and no Date nor Time.) > > At this point, also, my module does not support the DB API (that is a goal > though), so right now you pretty much get direct access to the physical > table. > > My question: ?for the direct access side, is there any advantage to treating > an empty L, D, or T field differently from a Null L, D, or T field? ?Or > should they be semantically the same, always returning, for example, None > whether it's actually Null in the table or just empty? for those of you that are wondering about this nuttyness: VFP is a dBase derivative. In there early days, there was no support for Null. But a side effect of adding a row but not storing values to it was empty fields. the best example is a date. dates are stored on disk as a string "YYYYMMDD" and in the case of an empty date: " ". Same with numbers: N(6,2) would store 3.14 as " 3.14" and an empty number was " ". Then they added Null support, but instead of using this empty state, they added a flag, so now a field could be Null, Blank or have a value. which was a mistake, but here we are. For D/T I am not sure you have a choice. What value would you return for an empty date other than Null? for Logical I would return False. Mainly because I am pretty sure that because in VFP an empty logical will be treated as .f. anywhere a logical is evaluated. -- Carl K From ethan at stoneleaf.us Wed Jul 20 03:41:26 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 19 Jul 2011 18:41:26 -0700 Subject: [DB-SIG] dBase III and VFP tables In-Reply-To: References: <4E2615C1.1060604@stoneleaf.us> Message-ID: <4E263246.4090807@stoneleaf.us> Carl Karsten wrote: > for those of you that are wondering about this nuttyness: VFP is a > dBase derivative. In there early days, there was no support for Null. > But a side effect of adding a row but not storing values to it was > empty fields. the best example is a date. dates are stored on disk > as a string "YYYYMMDD" and in the case of an empty date: " ". > Same with numbers: N(6,2) would store 3.14 as " 3.14" and an empty > number was " ". Then they added Null support, but instead of > using this empty state, they added a flag, so now a field could be > Null, Blank or have a value. which was a mistake, but here we are. Thanks for the clarification, Carl. > For D/T I am not sure you have a choice. What value would you return > for an empty date other than Null? As it happens, I created custom Date, Time, and DateTime objects so that I could deal with the empty values. I probably just should have returned None from the beginning, but did I mention that this is the project I have learned Python with? Now that I have them, I'm thinking of keeping and continuing to use them for the empty case... but not sure if that's a great idea, hence my question here. > for Logical I would return False. Mainly because I am pretty sure > that because in VFP an empty logical will be treated as .f. anywhere a > logical is evaluated. The test I tried before I left work seems to confirm that (it was just printing). ~Ethan~ From james at jamesh.id.au Wed Jul 20 04:46:38 2011 From: james at jamesh.id.au (James Henstridge) Date: Wed, 20 Jul 2011 10:46:38 +0800 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E23FCD7.4050304@dndg.it> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> <4E23FCD7.4050304@dndg.it> Message-ID: On Mon, Jul 18, 2011 at 5:28 PM, Federico Di Gregorio wrote: > On 18/07/11 11:25, James Henstridge wrote: >> If all adapters that support transactions function like that, then I >> think it would be sensible to require them to raise an error in that >> case. ?If they don't raise an error, you know that someone somewhere >> is going to write code like the following: >> >> ? ? with transaction: >> ? ? ? ? # do stuff >> ? ? ? ? with transaction: >> ? ? ? ? ? ? # do more stuff >> >> ... and wonder why things break. > > This can make sense for backends that support check points > (subtransactions). PostgreSQL is an example (even if we don't yet > support checkpoints in psycopg). While it is definitely possible to simulate some kind of "sub-transaction" through the use of save points, I'm not sure whether it is actually a good idea to magically substitute one for the other. One feature of committing a transaction is that it makes the work has been saved permanently and is available to other connections, which won't be the case with a save point based sub-transaction. If I wrote some code that relied on those semantics (e.g. one that posts a message to another process after completing its work), then it would be an error to call it from within transaction context and I'd want to see an error explaining the fact. There probably are uses where a recursive transaction decorator/context manager might be useful, but I don't think it is necessarily something users would want by default. James. From carl at personnelware.com Wed Jul 20 05:06:42 2011 From: carl at personnelware.com (Carl Karsten) Date: Tue, 19 Jul 2011 22:06:42 -0500 Subject: [DB-SIG] dBase III and VFP tables In-Reply-To: <4E263246.4090807@stoneleaf.us> References: <4E2615C1.1060604@stoneleaf.us> <4E263246.4090807@stoneleaf.us> Message-ID: On Tue, Jul 19, 2011 at 8:41 PM, Ethan Furman wrote: > Carl Karsten wrote: >> >> for those of you that are wondering about this nuttyness: ?VFP is a >> dBase derivative. ?In there early days, there was no support for Null. >> ?But a side effect of adding a row but not storing values to it was >> empty fields. ?the best example is a date. ?dates are stored on disk >> as a string "YYYYMMDD" ?and in the case of an empty date: " ? ? ? ?". >> Same with numbers: N(6,2) would store 3.14 as " ?3.14" and an empty >> number was " ? ? ?". ?Then they added Null support, but instead of >> using this empty state, they added a flag, so now a field could be >> Null, Blank or have a value. ?which was a mistake, but here we are. > > Thanks for the clarification, Carl. > >> For D/T I am not sure you have a choice. ?What value would you return >> for an empty date other than Null? > > As it happens, I created custom Date, Time, and DateTime objects so that I > could deal with the empty values. ?I probably just should have returned None > from the beginning, but did I mention that this is the project I have > learned Python with? Impressive! > > Now that I have them, I'm thinking of keeping and continuing to use them for > the empty case... but not sure if that's a great idea, hence my question > here. I would say supporting Empty is a backwards compatibility thing: If there was code that relied on it, then you should continue to support it. When I was using VFP I never needed both values. My guess is the custom objects will cause problems and solve none. > >> for Logical I would return False. ?Mainly because I am pretty sure >> that because in VFP an empty logical will be treated as .f. anywhere a >> logical is evaluated. > > The test I tried before I left work seems to confirm that (it was just > printing). -- Carl K From mal at egenix.com Wed Jul 20 10:39:59 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 20 Jul 2011 10:39:59 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> Message-ID: <4E26945F.3050701@egenix.com> James Henstridge wrote: > On Mon, Jul 18, 2011 at 3:52 PM, M.-A. Lemburg wrote: >> Vernon Cole wrote: >>> A cursor and/or a >>> connection should probably be context managers, so that they will work in a >>> "with" statement. >> >> This is was discussed before and it should probably go into >> the DB API in some form. >> >> I'm no particular fan of hiding transactions in context >> managers, though. In practice this often causes problems, since >> you usually want to apply error handling logic in case of >> problems, other than simply issuing a .rollback(). > > For most of the applications I work with, the transaction handling has > been delegated to function decorators, or have it hidden in the > framework (e.g. Django's TransactionMiddleware). If there are clean > up tasks that need to happen on transaction commit or roll back (e.g. > deleting a file on a failed transaction), then using a global > transaction manager like Zope's transaction module seems to be a good > fit. > > For transaction retry or error reporting, I haven't seen much benefit > in leaving the transaction in an "open by broken" state over cleaning > up with a rollback. In more detail: When letting a context manager do a .rollback() on a connection in case of an error, the error still propagates up the call chain. However, since the transaction was rolled back, access to cursors on the connection and the connection itself will no longer work, so your error reporting and cleanup possibilities are left with just the error object. Likewise, when having the context manager do an implicit .commit() when leaving the block, you are giving away control and have to add extra code in case you want to prevent the changes from being committed, e.g. in a debug situation. Things can get really interesting in multi-threaded applications, if you're sharing connections between threads - but it's better to avoid that anyway, so I won't go into detail. Most of these issues can be resolved by coding explicit context managers that deal with your particular case, e.g. with TransactionalContext(connection): ... Overall, I think using a transaction manager that explicitly controls the .commit() and .rollback() calls is a lot cleaner and the direct "with connection:" construct causes more problems than it solves in more complex applications. On the plus side, in simple applications it can help the user to not forget the .commit() call (the .rollback() is implicit at garbage collection time anyway). So in conclusion, I think we should add it, but with a warning to the user that it's often better to write your own context manager. >> Another problem is that the connections used in a with >> statement will usually have already started a transaction >> before entering the context, so the .rollback() would >> remove more than just the things added in the with >> context. > > None of the databases I've worked with fit the Python DB API's > "implicit begin" behaviour for transactions, so their adapters have > all needed to run a simple state machine to determine when it is > necessary to start a transaction. So it would be pretty easy for them > to raise an error if an attempt was made to use the context manager > when a transaction was already in progress. The implicit start of transactions originates from the ODBC standard which is used by quite a few database as native API, so the above cannot easily be generalized. There's also a difference between starting a transaction and actually making changes on that transaction. An error should only be raised in case changes were made, but that would require a lot tedious internal state keeping by the database package, unless the backend provides an API to query whether the transaction actually contains any changes. I don't think there's much we can do about this particular aspect of using connections as context managers. > If all adapters that support transactions function like that, then I > think it would be sensible to require them to raise an error in that > case. If they don't raise an error, you know that someone somewhere > is going to write code like the following: > > with transaction: > # do stuff > with transaction: > # do more stuff > > ... and wonder why things break. Indeed. >> There's also the problem of intuitive use: some users will >> likely expect the connection to also be closed when leaving >> the context - much like what happens with files. > > If that is a concern, you could require a method call to get the > context manager rather than making the connection itself a context > manager. For example: > > with connection.begin_transaction(): > ... True, but as discussed above, what would that method do ? It could do an implicit rollback, but this would then likely cause other problems such as clearing out changes the user actually wants to commit later on. The safe way I know is opening a new connection: with db.connect(...) as connection: ... -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 20 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Wed Jul 20 11:05:44 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 20 Jul 2011 11:05:44 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E25B2DC.8070602@ingres.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> <4E240E7E.6070005@online.de> <4E2416E7.1050400@egenix.com> <4E24261F.3080600@online.de> <4E242D07.3090909@egenix.com> <4E242EBE.2010806@dndg.it> <4E25761A.9070406@egenix.com> <4E25B2DC.8070602@ingres.com> Message-ID: <4E269A68.5050401@egenix.com> Chris Clark wrote: > M.-A. Lemburg wrote: >> Daniele Varrazzo wrote: >> >>>> On Jul 19, 2011 3:44 AM, "Federico Di Gregorio" >>>> wrote: >>>> >>>>> At least 2 drivers (psycopg and pysqlite) provide a Python->backend >>>>> mechanism based on PEP 246, "Object Adaptation". If other implementors >>>>> are interested I can write a short explanation about how it works and >>>>> why it was chosen only for the Python->backend path and not for the >>>>> reverse. >>>>> >>> On Tue, Jul 19, 2011 at 11:05 AM, Vernon Cole >>> wrote: >>> >>>> Yes, please. I would like to see that. Adodbapi has an output >>>> conversion >>>> mechanism, but I don't like it particularly well. >>>> >>> For a description: >>> >>> http://initd.org/psycopg/docs/advanced.html#adapting-new-python-types-to-sql-syntax >>> >>> http://initd.org/psycopg/docs/advanced.html#type-casting-of-sql-types-into-python-objects >>> >>> >> >> While this is a nice system, it's also very slow. It uses function >> calls and string parsing/conversion for adapting each value. This >> works if you only have to insert/fetch a few rows, but won't be >> feasible for larger volumes. >> >> I think we need something more low-level, which tries to >> avoid (Python) function calls if possible, ..... > > I have to confess the only piece of pep-0246 I read was: > >> >> Rejection Notice >> >> I'm rejecting this PEP. Something much better is about to happen; >> it's too early to say exactly what, but it's not going to resemble >> the proposal in this PEP too closely so it's better to start a new >> PEP. GvR. >> > > Does anyone know what the "better" option was? The date on the pep is a > number of years ago. Not that I'm aware off. Python has moved on to ABCs, which address part of the intended use in a different way (the outside objects have to implement the protocol and announce this using an ABC). The adapter pattern is still useful, though. I just think that we need to provide a way for database modules to implement such adapter in straight C (as well as in Python) rather than requiring a round-trip to Python in all cases. The sketch I drew up would allow for this. Most database modules use such C adapters in some form anyway, so these would just need to be refactored a little bit to make them available to the Python application, which should help with the adoption of the new techniques. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 20 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ethan at stoneleaf.us Wed Jul 20 17:58:49 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 20 Jul 2011 08:58:49 -0700 Subject: [DB-SIG] dBase III and VFP tables In-Reply-To: References: <4E2615C1.1060604@stoneleaf.us> <4E263246.4090807@stoneleaf.us> Message-ID: <4E26FB39.2090804@stoneleaf.us> Carl Karsten wrote: > I would say supporting Empty is a backwards compatibility thing: If > there was code that relied on it, then you should continue to support > it. > > When I was using VFP I never needed both values. > > My guess is the custom objects will cause problems and solve none. Okay, I think what I'll do is keep it simple by default: None will be returned both for Empty and Null, and if None is assigned to a field it will be written as either Empty, or Null if that field is Nullable. I'll keep the custom objects around and provide a mechanism to specify which objects to return based on field type/null status, so if anyone wants, for example, Decimals instead of floats they can have them, or Empty/Null objects they can have those too. Seem reasonable? ~Ethan~ From carl at personnelware.com Wed Jul 20 17:47:59 2011 From: carl at personnelware.com (Carl Karsten) Date: Wed, 20 Jul 2011 10:47:59 -0500 Subject: [DB-SIG] dBase III and VFP tables In-Reply-To: <4E26FB39.2090804@stoneleaf.us> References: <4E2615C1.1060604@stoneleaf.us> <4E263246.4090807@stoneleaf.us> <4E26FB39.2090804@stoneleaf.us> Message-ID: On Wed, Jul 20, 2011 at 10:58 AM, Ethan Furman wrote: > Carl Karsten wrote: >> >> I would say supporting Empty is a backwards compatibility thing: If >> there was code that relied on it, then you should continue to support >> it. >> >> When I was using VFP I never needed both values. >> >> My guess is the custom objects will cause problems and solve none. > > > Okay, I think what I'll do is keep it simple by default: None will be > returned both for Empty and Null, and if None is assigned to a field it will > be written as either Empty, or Null if that field is Nullable. I'll keep the > custom objects around and provide a mechanism to specify which objects to > return based on field type/null status, so if anyone wants, for example, > Decimals instead of floats they can have them, or Empty/Null objects they > can have those too. > > Seem reasonable? Yep. I was kinda thinking of keeping the empty-able objects around. I hate the idea of trying to document them; I hardly like trying to keep track of how VFP handles them :) -- Carl K From Chris.Clark at ingres.com Wed Jul 20 17:50:58 2011 From: Chris.Clark at ingres.com (Chris Clark) Date: Wed, 20 Jul 2011 08:50:58 -0700 Subject: [DB-SIG] dBase III and VFP tables In-Reply-To: References: <4E2615C1.1060604@stoneleaf.us> Message-ID: <4E26F962.2020204@ingres.com> Carl Karsten wrote: > On Tue, Jul 19, 2011 at 6:39 PM, Ethan Furman wrote: > >> I am finally adding Null support to my dbf package >> (http://pypi.python.org/pypi/dbf) and, as usual, the VFP side is being a >> pain. >> >> The issue is that the Logical, Date, and DateTime field types, even when the >> table does not allow Null values, can be empty -- and when Null values are >> allowed, those fields can be empty or Null. (Empty pretty much meaning not >> initialized -- neither True nor False, and no Date nor Time.) >> >> At this point, also, my module does not support the DB API (that is a goal >> though), so right now you pretty much get direct access to the physical >> table. >> Take a look at engine (formally SnakeSQL) and squawk for ideas on adding both SQL and a dbapi interface. http://pypi.python.org/pypi/engine/ http://pypi.python.org/pypi/SnakeSQL/ http://pypi.python.org/pypi/squawk/ I've not used either so this isn't a recommendation but they look very cool. >> My question: for the direct access side, is there any advantage to treating >> an empty L, D, or T field differently from a Null L, D, or T field? Or >> should they be semantically the same, always returning, for example, None >> whether it's actually Null in the table or just empty? >> > > for those of you that are wondering about this nuttyness: VFP is a > dBase derivative. In there early days, there was no support for Null. > But a side effect of adding a row but not storing values to it was > empty fields. the best example is a date. dates are stored on disk > as a string "YYYYMMDD" and in the case of an empty date: " ". > Same with numbers: N(6,2) would store 3.14 as " 3.14" and an empty > number was " ". Then they added Null support, but instead of > using this empty state, they added a flag, so now a field could be > Null, Blank or have a value. which was a mistake, but here we are. > > For D/T I am not sure you have a choice. What value would you return > for an empty date other than Null? > > for Logical I would return False. Mainly because I am pretty sure > that because in VFP an empty logical will be treated as .f. anywhere a > logical is evaluated. > My 2 cents, it looks like the "type" number "n(6,2)" is really a string with some restrictions on formatting (i.e. only contains digits, a single period, and spaces). You could simple return strings for everything. An optional (and configurable) type conversion layer could then be used on top of that. Ingres has an old (non ANSI-) date type that we still support, this old date can be empty (empty string '' rather than blank padded as described in earlier in the thread). IMHO dbapi drivers can't go returning empty strings for dates (as this is wildly incompatible with other DBMSs), we return a "magic" date for empty dates (the year 9999). It is reasonable to add an extension return type that supports empty, e.g. one could then do checks like: if my_result is empty: .... But I'd suggest that empty support NOT be enabled by default. So lots of options :-S Chris From ethan at stoneleaf.us Wed Jul 20 18:33:39 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 20 Jul 2011 09:33:39 -0700 Subject: [DB-SIG] dBase III and VFP tables In-Reply-To: <4E26F962.2020204@ingres.com> References: <4E2615C1.1060604@stoneleaf.us> <4E26F962.2020204@ingres.com> Message-ID: <4E270363.5080107@stoneleaf.us> Chris Clark wrote: > Take a look at engine (formally SnakeSQL) and squawk for ideas on adding > both SQL and a dbapi interface. > > http://pypi.python.org/pypi/engine/ > http://pypi.python.org/pypi/SnakeSQL/ > > http://pypi.python.org/pypi/squawk/ Thanks, I'll check them out. > My 2 cents, it looks like the "type" number "n(6,2)" is really a string > with some restrictions on formatting (i.e. only contains digits, a > single period, and spaces). You could simple return strings for > everything. An optional (and configurable) type conversion layer could > then be used on top of that. I'm going for the configurable route. VFP and dBase IV and above have binary fields as well as text fields, so just returning what I find in the table doesn't feel very friendly. By default I'll be returning ints/floats, bools, dates, and Char (another custom type that treats whitespace as false, as well as ignoring trailing white space for comparisons) with a mechanism for overriding either on a by type basis (any Number) or a field basis (only Age). > Ingres has an old (non ANSI-) date type that we still support, this old > date can be empty (empty string '' rather than blank padded as described > in earlier in the thread). IMHO dbapi drivers can't go returning empty > strings for dates (as this is wildly incompatible with other DBMSs), we > return a "magic" date for empty dates (the year 9999). It is reasonable > to add an extension return type that supports empty, e.g. one could then > do checks like: > > if my_result is empty: > .... > > But I'd suggest that empty support NOT be enabled by default. So lots of > options :-S Yes, that's the conclusion I have come to. Thanks for the links and your comments! ~Ethan~ From carl at personnelware.com Wed Jul 20 18:44:39 2011 From: carl at personnelware.com (Carl Karsten) Date: Wed, 20 Jul 2011 11:44:39 -0500 Subject: [DB-SIG] dBase III and VFP tables In-Reply-To: <4E270363.5080107@stoneleaf.us> References: <4E2615C1.1060604@stoneleaf.us> <4E26F962.2020204@ingres.com> <4E270363.5080107@stoneleaf.us> Message-ID: On Wed, Jul 20, 2011 at 11:33 AM, Ethan Furman wrote: > as well as ignoring trailing white space for comparisons) I wouldn't bother supporting that in the driver. Another background lesson: dBase started as an interactive environment. It didn't look like a spreadsheet, but think of a spreadsheet: the typical use is to start typing stuff in by hand, select some data with the mouse, hit sort, select some numbers and click Summation, see the results on the screen. At the time things were still in the relm of teletype machines were the only thing you looked at was printed output. Screens were new, and the API was still much like a printer. (not exactly true, there was the EDIT command that would clear the screen, draw field names and input boxes and let you enter data and tab/back tab around the screen.) It wasn't until later versions that they added the concept of stringing a bunch of commands together. conditionals came later. A design goal was syntax that is easy to type interactively, not make sense in a formal programming language. today we have "incremental search" where as you type query parameters the search is executed and you see results. If typing "KAR" is all you need to figure out "CARL KARSTEN" is the person you are looking for, then 3 keys are all you need to press. likewise, in dBase you could do DISPLAY FOR LNAME="KAR" and see just the rows where LNAME _begines with_ "KAR" which makes sense for an interactive environment, but causes grief when you try to write programs. later they added the == operator, but maintained backwards computability with the _only compare until you run out of chars_ behaviour. _backwards computability_ is the only reason to implement this in the driver, and given there is currenly no code relying on it, nothing will break if you don't. So don't :) -- Carl K From ethan at stoneleaf.us Wed Jul 20 19:36:13 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 20 Jul 2011 10:36:13 -0700 Subject: [DB-SIG] dBase III and VFP tables In-Reply-To: References: <4E2615C1.1060604@stoneleaf.us> <4E26F962.2020204@ingres.com> <4E270363.5080107@stoneleaf.us> Message-ID: <4E27120D.6010506@stoneleaf.us> Carl Karsten wrote: > On Wed, Jul 20, 2011 at 11:33 AM, Ethan Furman wrote: >> as well as ignoring trailing white space for comparisons) > > _backwards computability_ is the only reason to implement this in the > driver, and given there is currenly no code relying on it, nothing > will break if you don't. So don't :) Okay, have you written a book? 'Cause I'm ready to buy it. ;) The reason I have that (and I can make it be the option, not the default), is that without it I have to write.... oh -- I could use .startswith(), couldn't I? Well, I was going to say that code such as: # look for so-and-so... for rec in table: if rec.full_name == 'John Doe': do_something_with(rec) and of course this will fail because FULL_NAME is a 40 character field, and I should have said if rec.full_name == 'John Doe ' and I really don't want to do that! Also, .startswith() is not the same as ignoring trailing whitespace. Given this little example, would you still not worry about trailing whitespace? ~Ethan~ From carl at personnelware.com Wed Jul 20 19:46:00 2011 From: carl at personnelware.com (Carl Karsten) Date: Wed, 20 Jul 2011 12:46:00 -0500 Subject: [DB-SIG] dBase III and VFP tables In-Reply-To: <4E27120D.6010506@stoneleaf.us> References: <4E2615C1.1060604@stoneleaf.us> <4E26F962.2020204@ingres.com> <4E270363.5080107@stoneleaf.us> <4E27120D.6010506@stoneleaf.us> Message-ID: On Wed, Jul 20, 2011 at 12:36 PM, Ethan Furman wrote: > Carl Karsten wrote: >> >> On Wed, Jul 20, 2011 at 11:33 AM, Ethan Furman wrote: >>> >>> as well as ignoring trailing white space for comparisons) >> >> _backwards computability_ is the only reason to implement this in the >> driver, and given there is currenly no code relying on it, nothing >> will break if you don't. ?So don't :) > > Okay, have you written a book? ?'Cause I'm ready to buy it. ?;) > > The reason I have that (and I can make it be the option, not the default), > is that without it I have to write.... oh -- I could use .startswith(), > couldn't I? ?Well, I was going to say that code such as: > > # look for so-and-so... > for rec in table: > ? ?if rec.full_name == 'John Doe': > ? ? ? ?do_something_with(rec) > > and of course this will fail because FULL_NAME is a 40 character field, and > I should have said > > ? ?if rec.full_name == 'John Doe ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?' > > and I really don't want to do that! ?Also, .startswith() is not the same as > ignoring trailing whitespace. > > Given this little example, would you still not worry about trailing > whitespace? Oh yeah, that. >>> "abc ".strip() 'abc' if rec.full_name.strip() == 'John Doe' Personally, I would leave it at that. half baked idea: If you want to add some code to hide that, add some sort of auto-strip to the driver so it drops the extra spaces. -- Carl K From ethan at stoneleaf.us Wed Jul 20 20:46:04 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 20 Jul 2011 11:46:04 -0700 Subject: [DB-SIG] dBase III and VFP tables In-Reply-To: References: <4E2615C1.1060604@stoneleaf.us> <4E26F962.2020204@ingres.com> <4E270363.5080107@stoneleaf.us> <4E27120D.6010506@stoneleaf.us> Message-ID: <4E27226C.6010904@stoneleaf.us> Carl Karsten wrote: >> Given this little example, would you still not worry about trailing >> whitespace? > > > Oh yeah, that. > >--> "abc ".strip() > 'abc' > > if rec.full_name.strip() == 'John Doe' > > Personally, I would leave it at that. > > half baked idea: If you want to add some code to hide that, add some > sort of auto-strip to the driver so it drops the extra spaces. Heh, that's what I've been doing. It did two things for me: 1) made equality checking easier; and 2) made boolean checks make sense. It has its own problems though -- mainly that single character empty fields don't work well: if rtyp in 'FMU': # is this Forwardable, Moved, or Undeliverable? ' ' gets stripped down to '', and '' is in every string, this doesn't work. And yes, I could use a .strip() every time I have a field comparison, but I have *lots* of them in my code, and it feels an awful lot like unnecessary boiler plate. Okay, I'll leave the default at str, and make Char, like Date, DateTime, and Logical, be available options. ~Ethan~ From carl at personnelware.com Wed Jul 20 20:47:07 2011 From: carl at personnelware.com (Carl Karsten) Date: Wed, 20 Jul 2011 13:47:07 -0500 Subject: [DB-SIG] dBase III and VFP tables In-Reply-To: <4E27226C.6010904@stoneleaf.us> References: <4E2615C1.1060604@stoneleaf.us> <4E26F962.2020204@ingres.com> <4E270363.5080107@stoneleaf.us> <4E27120D.6010506@stoneleaf.us> <4E27226C.6010904@stoneleaf.us> Message-ID: On Wed, Jul 20, 2011 at 1:46 PM, Ethan Furman wrote: > Carl Karsten wrote: >>> >>> Given this little example, would you still not worry about trailing >>> whitespace? >> >> >> Oh yeah, that. >> >> --> "abc ? ".strip() >> 'abc' >> >> ?if rec.full_name.strip() == 'John Doe' >> >> Personally, I would leave it at that. >> >> half baked idea: If you want to add some code to hide that, add some >> sort of auto-strip to the driver so it drops the extra spaces. > > Heh, that's what I've been doing. ?It did two things for me: 1) made > equality checking easier; and 2) made boolean checks make sense. ?It has its > own problems though -- mainly that single character empty fields don't work > well: > > if rtyp in 'FMU': # is this Forwardable, Moved, or Undeliverable? > > ' ' gets stripped down to '', and '' is in every string, this doesn't work. huh. yeah, its true. i'm kinda surprised. >>> '' in 'abc' True But this works: >>> '' in ['a','b','c'] False >>> 'b' in ['a','b','c'] True > > And yes, I could use a .strip() every time I have a field comparison, but I > have *lots* of them in my code, and it feels an awful lot like unnecessary > boiler plate. I know what you mean. My advice: 2 wads of code: the dbapi module and some wrapper to make it nicer to work with. > > Okay, I'll leave the default at str, and make Char, like Date, DateTime, and > Logical, be available options. > > ~Ethan~ > _______________________________________________ > DB-SIG maillist ?- ?DB-SIG at python.org > http://mail.python.org/mailman/listinfo/db-sig > -- Carl K From federico.digregorio at dndg.it Tue Jul 19 14:42:37 2011 From: federico.digregorio at dndg.it (Federico Di Gregorio) Date: Tue, 19 Jul 2011 14:42:37 +0200 Subject: [DB-SIG] Some obscurity with paramstyle In-Reply-To: <4E25761A.9070406@egenix.com> References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4510C2E0-EB85-48D8-984B-91BE05E3ABCC@zzzcomputing.com> <4E23E622.6000706@egenix.com> <4E240E7E.6070005@online.de> <4E2416E7.1050400@egenix.com> <4E24261F.3080600@online.de> <4E242D07.3090909@egenix.com> <4E242EBE.2010806@dndg.it> <4E25761A.9070406@egenix.com> Message-ID: <4E257BBD.7010205@dndg.it> On 19/07/11 14:18, M.-A. Lemburg wrote: > Daniele Varrazzo wrote: [snip] > While this is a nice system, it's also very slow. It uses function > calls and string parsing/conversion for adapting each value. This > works if you only have to insert/fetch a few rows, but won't be > feasible for larger volumes. It depends on the backend, too. Sincerely when I need to load hudreds of thousands of rows I just avoid INSERT and go directly for mechanisms like PostgreSQL COPY. INSERT is (usually) so slooow that the drivers doesn't add that much overhead. > I think we need something more low-level, which tries to > avoid (Python) function calls if possible, e.g. it should be > possible to write adapters in C and only point to them using > a symbol (sketching here): > > cursor.setinputconverter(BINARY, module.BINARY_INPUT_CONVERTER) > cursor.setoutputconverter(BINARY, module.BINARY_OUTPUT_CONVERTER) > > For mxODBC we'd then use something like this: > > # Convert Python unicode object data to SQLWCHAR data > cursor.setinputconverter(unicode, module.SQLWCHAR_INPUT_CONVERTER) > > # Convert SQL_WCHAR type code data to Python unicode > cursor.setoutputconverter(SQL.WCHAR, module.SQLCHAR_OUTPUT_CONVERTER_UNICODE) > > The advantage here is that the database module could work > directly on the internal data structure to implement > the conversion rather than having to round-trip to Python. But the disavantage is that you can't easily work with composite types where you need to convert the primitive types and pack them into the composite (arrays, geographic data, ...) while adapt() is perfect for nested types. Anyway, happy that the discussion started :) federico -- Federico Di Gregorio federico.digregorio at dndg.it Studio Associato Di Nunzio e Di Gregorio http://dndg.it I porcellini di terra sono davvero Crostacei! Non lo sapevo! Certo che sono crostacei, hanno la crosta! Allora la pizza ? un crostaceo?! -- discorso all'ESC2k07 From wilk at flibuste.net Sun Jul 24 15:29:42 2011 From: wilk at flibuste.net (William) Date: Sun, 24 Jul 2011 13:29:42 +0000 (UTC) Subject: [DB-SIG] Some obscurity with paramstyle References: <4E21E474.2030706@online.de> <1185873A-7F26-491E-AF5C-AFC1BF4D3324@zzzcomputing.com> <4E23E1DE.10005@egenix.com> Message-ID: On 18-07-2011, M.-A. Lemburg wrote: >> IMHO I would prefer to see the DBAPI have exactly two paramstyles, >> named and qmark, and have all DBAPIs support both consistently. >> The (py)format styles continuously introduce the mixing of Python's >> string formatting behavior with the presentation of bound parameters, >> which are two completely different things. > > +1 > +1 also Too many formats are a risk that user will not bother and use string formatting. To help we could provide reference implementation of current formats converters. It's not a bottleneck since the parsing is done only one time. -- William Dod? - http://flibuste.net Informaticien Ind?pendant