From andy47 at halfcooked.com Sun Jan 3 22:26:48 2016 From: andy47 at halfcooked.com (Andy Todd) Date: Mon, 4 Jan 2016 14:26:48 +1100 Subject: [DB-SIG] SQL CONNECTION In-Reply-To: References: Message-ID: <5689E678.8050405@halfcooked.com> On 31/12/2015 1:31 AM, Dale Wheeler wrote: > Hi- > Can you help? > I am trying to connect to SQL Anywhere (v9) with iPython Ananconda. > I am on a Windows 7 PC and both Python and SQL are installed locally. > > Is this at all possible or do i need to upgrade my version of SQL? > > Many thanks > > Dale > > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > Dale, You need to install a Python module that allows you to talk to SQL Anywhere. A quick search shows this - https://pypi.python.org/pypi/sqlanydb/1.0.5 I've not used SQL Anywhere so can't testify if this is the best solution. As you are using Anaconda it *should* be fairly straightforward to install this Python module. Run the "Anaconda Command Prompt" from your start menu and just type c:\> conda install sqlanydb Then after you have installed the package start up a Python prompt and follow the instructions on the package page link I included above. Good luck, Andy -- ----------------------------------------------------------------------------- From the desk of Andrew J Todd esq - http://www.halfcooked.com/ "It's still magic even if you know how it's done." - Terry Pratchett From gtobias at tobcm.com Sun Jan 10 14:25:45 2016 From: gtobias at tobcm.com (greg tobias) Date: Sun, 10 Jan 2016 14:25:45 -0500 Subject: [DB-SIG] PEP 249--connection problems Message-ID: <000c01d14bdc$b407e680$1c17b380$@tobcm.com> Hi. I am a windows user trying to establish a 'connect' using 'pymysql' in py3. I am looking for documentation on what information I need to put into the 'input fields' ie.. pymysql.connect(host='127.0.0.1',unix_socket='/var/run/mysqld/mysqld.sock',u ser = 'root', passwd = '423094424209484',db='mysql') Thanks a lot for your help. Regards, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Sun Jan 10 18:32:47 2016 From: songofacandy at gmail.com (INADA Naoki) Date: Mon, 11 Jan 2016 08:32:47 +0900 Subject: [DB-SIG] PEP 249--connection problems In-Reply-To: <000c01d14bdc$b407e680$1c17b380$@tobcm.com> References: <000c01d14bdc$b407e680$1c17b380$@tobcm.com> Message-ID: Hi, greg. First of all, there is PyMySQL users Google group. I found you luckily. But the group is better place to ask a question. https://groups.google.com/forum/#!forum/pymysql-users On Mon, Jan 11, 2016 at 4:25 AM, greg tobias wrote: > Hi. I am a windows user trying to establish a 'connect' using 'pymysql' in > py3. I am looking for documentation on what information I need to put into > the 'input fields' ie?. > The docstring. > > > pymysql.connect(host='127.0.0.1',unix_socket='/var/run/mysqld/mysqld.sock',user > = 'root', passwd = '423094424209484',db='mysql') > > Thanks a lot for your help. Regards, Greg. > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > > -- INADA Naoki -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Tue Jan 12 07:59:08 2016 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 12 Jan 2016 21:59:08 +0900 Subject: [DB-SIG] Should Binary accept unicode string? Message-ID: Hi, all. I found DB-API 2.0 defines Binary() as Binary(string). https://www.python.org/dev/peps/pep-0249/#binary What the string means? On Python 2, should Binary accept unicode? On Python 3, should Binary accept str? -- INADA Naoki -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike_mp at zzzcomputing.com Fri Jan 15 09:39:24 2016 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Fri, 15 Jan 2016 09:39:24 -0500 Subject: [DB-SIG] Should Binary accept unicode string? In-Reply-To: References: Message-ID: <5699049C.9090807@zzzcomputing.com> On 01/12/2016 07:59 AM, INADA Naoki wrote: > Hi, all. > > I found DB-API 2.0 defines Binary() as Binary(string). > https://www.python.org/dev/peps/pep-0249/#binary > > What the string means? > On Python 2, should Binary accept unicode? > On Python 3, should Binary accept str? IMO, Py2K's "binary" is str, that's all you should accept, and Py3K's "binary" is "bytes", that's all you should accept. A Unicode object is not a "binary" any more than a Python dictionary is. If your users are looking for the "convenience" of assuming unicodes become binaries by using the nearest available character set to encode it, they need to be reminded of "explicit is better than implicit" :). > > -- > INADA Naoki > > > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > From mal at egenix.com Fri Jan 15 09:47:07 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 15 Jan 2016 15:47:07 +0100 Subject: [DB-SIG] Should Binary accept unicode string? In-Reply-To: References: Message-ID: <5699066B.4080004@egenix.com> On 12.01.2016 13:59, INADA Naoki wrote: > Hi, all. > > I found DB-API 2.0 defines Binary() as Binary(string). > https://www.python.org/dev/peps/pep-0249/#binary > > What the string means? > On Python 2, should Binary accept unicode? > On Python 3, should Binary accept str? The Binary() wrapper is intended to provide extra information for the database module and marks the intent of the user to have the input parameter be bound to the binding parameter as binary rather than text (e.g. VARBINARY rather than VARCHAR). For Python 2, you'd probably use something like Binary=buffer. On Python 3, Binary=bytes or Binary=bytearray seem like a natural choices. The choice of possible input parameters for Binary() is really up to the database module author. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Jan 15 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From mike_mp at zzzcomputing.com Fri Jan 15 10:52:19 2016 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Fri, 15 Jan 2016 10:52:19 -0500 Subject: [DB-SIG] Should Binary accept unicode string? In-Reply-To: <5699066B.4080004@egenix.com> References: <5699066B.4080004@egenix.com> Message-ID: <569915B3.2000609@zzzcomputing.com> On 01/15/2016 09:47 AM, M.-A. Lemburg wrote: > On 12.01.2016 13:59, INADA Naoki wrote: >> Hi, all. >> >> I found DB-API 2.0 defines Binary() as Binary(string). >> https://www.python.org/dev/peps/pep-0249/#binary >> >> What the string means? >> On Python 2, should Binary accept unicode? >> On Python 3, should Binary accept str? > > The Binary() wrapper is intended to provide extra information > for the database module and marks the intent of the user to have > the input parameter be bound to the binding parameter as > binary rather than text (e.g. VARBINARY rather than VARCHAR). > > For Python 2, you'd probably use something like Binary=buffer. > On Python 3, Binary=bytes or Binary=bytearray seem like a natural > choices. > > The choice of possible input parameters for Binary() is > really up to the database module author. I still don't understand this philosophy of pep-249. Allowing DBAPIs to arbitrarily decide how strict / loose they want to be for user-defined data passed to even very well known datatypes has a negative impact on portability. It means that code I write for one DBAPI will fail on another. Is it your view that databases and DBAPIs are so fundamentally different, even for basic things like unicodes/bytes, that attempting to provide for portability is hopeless? Why even have a pep-249 if I should expect that I have to rewrite my whole application when switching DBAPIs anyway? Obviosly, full portability between DBAPIs and databases is never going to be possible. But for easy things where a pro-portability decision is clearly very feasible, like, "do / don't accept a unicode object for a bytes type", why can't a decision be made? > From mal at egenix.com Fri Jan 15 11:15:52 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 15 Jan 2016 17:15:52 +0100 Subject: [DB-SIG] Should Binary accept unicode string? In-Reply-To: <569915B3.2000609@zzzcomputing.com> References: <5699066B.4080004@egenix.com> <569915B3.2000609@zzzcomputing.com> Message-ID: <56991B38.1090607@egenix.com> On 15.01.2016 16:52, Mike Bayer wrote: > > > On 01/15/2016 09:47 AM, M.-A. Lemburg wrote: >> On 12.01.2016 13:59, INADA Naoki wrote: >>> Hi, all. >>> >>> I found DB-API 2.0 defines Binary() as Binary(string). >>> https://www.python.org/dev/peps/pep-0249/#binary >>> >>> What the string means? >>> On Python 2, should Binary accept unicode? >>> On Python 3, should Binary accept str? >> >> The Binary() wrapper is intended to provide extra information >> for the database module and marks the intent of the user to have >> the input parameter be bound to the binding parameter as >> binary rather than text (e.g. VARBINARY rather than VARCHAR). >> >> For Python 2, you'd probably use something like Binary=buffer. >> On Python 3, Binary=bytes or Binary=bytearray seem like a natural >> choices. >> >> The choice of possible input parameters for Binary() is >> really up to the database module author. > > I still don't understand this philosophy of pep-249. Allowing DBAPIs > to arbitrarily decide how strict / loose they want to be for > user-defined data passed to even very well known datatypes has a > negative impact on portability. It means that code I write for one > DBAPI will fail on another. Is it your view that databases and DBAPIs > are so fundamentally different, even for basic things like > unicodes/bytes, that attempting to provide for portability is hopeless? > Why even have a pep-249 if I should expect that I have to rewrite my > whole application when switching DBAPIs anyway? > > Obviosly, full portability between DBAPIs and databases is never going > to be possible. But for easy things where a pro-portability decision is > clearly very feasible, like, "do / don't accept a unicode object for a > bytes type", why can't a decision be made? I think you are misunderstanding the purpose of the Binary() helper: This was added as a portable way to tell the database interface to bind data as binary to the parameter, nothing more. Since some database modules rely on the Python type of the input parameters to tell whether to bind as binary, character, numeric, etc., but did have the distinction between binary and text data in Python 2, as we now do in Python 3, the Binary() wrapper was added to make the distinction clear. The types Binary() allows as input are not part of the DB-API, just like we don't make any comments about the allowed input types for any other parameter type the database interface may support. This adds flexibility and makes it possible to create interface modules which support a great deal more than just a few standard Python data types. Back to the choices I mentioned for Binary(): In Python 2, buffer() does allow unicode objects on input, and what you as result corresponds to the binary representation of the unicode object as used by Python. In Python 3, bytes() requires to be more specific and you need to provide an encoding. The result is an encoded binary version of the text input. Both are reasonable choices for a Binary() wrapper and the result is easy to detect as "bind me as binary data" for the database module. It is not uncommon to convert text data to binary data for storage, esp. when dealing with larger blobs you just want to manage and not work on, or when you want to preserve it in exactly the same form you pass it to the database (without any implicit normalizations, surrogate conversions, warnings, etc.). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Jan 15 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From mike_mp at zzzcomputing.com Fri Jan 15 12:21:33 2016 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Fri, 15 Jan 2016 12:21:33 -0500 Subject: [DB-SIG] Should Binary accept unicode string? In-Reply-To: <56991B38.1090607@egenix.com> References: <5699066B.4080004@egenix.com> <569915B3.2000609@zzzcomputing.com> <56991B38.1090607@egenix.com> Message-ID: <56992A9D.6080509@zzzcomputing.com> On 01/15/2016 11:15 AM, M.-A. Lemburg wrote: > On 15.01.2016 16:52, Mike Bayer wrote: >> >> >> On 01/15/2016 09:47 AM, M.-A. Lemburg wrote: >>> On 12.01.2016 13:59, INADA Naoki wrote: >>>> Hi, all. >>>> >>>> I found DB-API 2.0 defines Binary() as Binary(string). >>>> https://www.python.org/dev/peps/pep-0249/#binary >>>> >>>> What the string means? >>>> On Python 2, should Binary accept unicode? >>>> On Python 3, should Binary accept str? >>> >>> The Binary() wrapper is intended to provide extra information >>> for the database module and marks the intent of the user to have >>> the input parameter be bound to the binding parameter as >>> binary rather than text (e.g. VARBINARY rather than VARCHAR). >>> >>> For Python 2, you'd probably use something like Binary=buffer. >>> On Python 3, Binary=bytes or Binary=bytearray seem like a natural >>> choices. >>> >>> The choice of possible input parameters for Binary() is >>> really up to the database module author. >> >> I still don't understand this philosophy of pep-249. Allowing DBAPIs >> to arbitrarily decide how strict / loose they want to be for >> user-defined data passed to even very well known datatypes has a >> negative impact on portability. It means that code I write for one >> DBAPI will fail on another. Is it your view that databases and DBAPIs >> are so fundamentally different, even for basic things like >> unicodes/bytes, that attempting to provide for portability is hopeless? >> Why even have a pep-249 if I should expect that I have to rewrite my >> whole application when switching DBAPIs anyway? >> >> Obviosly, full portability between DBAPIs and databases is never going >> to be possible. But for easy things where a pro-portability decision is >> clearly very feasible, like, "do / don't accept a unicode object for a >> bytes type", why can't a decision be made? > > I think you are misunderstanding the purpose of the Binary() helper: > This was added as a portable way to tell the database interface > to bind data as binary to the parameter, nothing more. I fully understand its purpose. I'm referring to the scope of what we mean by "bind data". > > Since some database modules rely on the Python type of the > input parameters to tell whether to bind as binary, > character, numeric, etc., but did have the distinction between > binary and text data in Python 2, as we now do in Python 3, > the Binary() wrapper was added to make the distinction clear. > > The types Binary() allows as input are not part of the DB-API, > just like we don't make any comments about the allowed input > types for any other parameter type the database interface > may support. Yes, and it is this philosophy I am commenting on - when it is very obvious that the Python data in question does not align to the database type being referred to without a conversion taking place, where that conversion is non-obvious (e.g. a "guess" must be made). This is not the same as when we have say a Python datetime.datetime() object being mapped to a TIMESTAMP field - there's a natural conversion that can take place here, assuming the types line up as far as presence or not of fields like microseconds or timezone. The fields of datetime objects can line up exactly against a backend database without the need for implicit decisionmaking about format. Whereas if we tried to map an integer value to a TIMESTAMP field, this is not a natural conversion - the integer can mean all kinds of things like the epoch in seconds, days, etc. If this data is not accepted by the database directly and the driver makes an arbitrary choice to make this the epoch in days since 1970, that's pretty arbitrary. It would be surprising behavior. > > This adds flexibility and makes it possible to create > interface modules which support a great deal more than > just a few standard Python data types. that is absolutely true, however, I don't see how you can deny that portability is negatively impacted when on one DBAPI I can say: cursor.execute( "insert into table (q) values (?)", [Binary(u'some unicode')] ) and on another, it fails, and I instead have to type: cursor.execute( "insert into table (q) values (?)", [Binary(u'some unicode'.encode('utf-8'))] ) The latter bit of code in fact can work on *both* systems. The first bit of code, cannot. By disallowing the first style, the developer is encouraged to write portable code. By allowing it, developers are led into writing code that is not portable. This is portability in a nutshell, and I'm sure you understand this. The emphasis pep-249 puts on "flexibility" is often at the cost of encouraging "portability", even in very obvious cases like this one, is the philosophy I find questionable, being that this is Python and not Perl. > > Back to the choices I mentioned for Binary(): > > In Python 2, buffer() does allow unicode objects > on input, and what you as result corresponds to the binary > representation of the unicode object as used by Python. The buffer() type produces Python's internal representation of the unicode data; it does not, for example, try to first encode the unicode object based on the platform charset. For the use case people expect when converting unicode objects to bytes, the behavior of buffer() is completely surprising: >>> list(buffer('f')) ['f'] >>> list(buffer(u'f')) ['f', '\x00', '\x00', '\x00'] The latter is the utf-32 representation that my Python interpreter is using. Other Python interpreters, like the one on my mac, return a utf-16 representation: >>> list(buffer(u'f')) ['f', '\x00'] neither the utf-16 or utf-32 formats are commonly used with database applications as a transport format; utf-8 is vastly more common. It should not be controversial that exposing the internal Python Unicode format, which isn't even portable across individual Python builds, is entirely inappropriate for any kind of data exchange between systems. That the buffer() construct was removed in Python 3 and replaced with memoryview() which does not accept strings at all is part of the bigger story that Python 2's unicode story, including the behavior of buffer(), was considered to be a design mistake. > > In Python 3, bytes() requires to be more specific and you > need to provide an encoding. The result is an encoded binary > version of the text input. > > Both are reasonable choices for a Binary() wrapper and > the result is easy to detect as "bind me as binary data" > for the database module. > > It is not uncommon to convert text data to > binary data for storage, esp. when dealing with larger > blobs you just want to manage and not work on, that's not controversial. But having the driver make the following guess: 1. unicode passed, use buffer() to return the Python interpreter's internal representation of it (or utf-32 always, or utf-16 always, or.. ?) 2. unicode passed, try to encode with utf-8 because that's "probably" want someone wants 3. unicode passed, try to encode with the particular charset that's configured with this driver (if the driver supports configured charsets at all) I would argue this scheme works against portability, because it is impossible to avoid choosing an arbitrary conversion scheme. This arbitrary-ness is a design mistake that Python 3 repaired. or when > you want to preserve it in exactly the same form you pass > it to the database (without any implicit normalizations, > surrogate conversions, warnings, etc.). The "form" of Python Unicode data outside of the interpreter is essentially undefined. This would mean one is storing data in their database that not only can't be read portably between major versions of Python, it can't be read portably between *builds* of the same Python version. Developers would vastly prefer their application raise an error rather than implicitly pass the internal Unicode format of data to their database, since this is not a use case anyone has. From vernondcole at gmail.com Fri Jan 15 17:14:59 2016 From: vernondcole at gmail.com (Vernon D. Cole) Date: Fri, 15 Jan 2016 15:14:59 -0700 Subject: [DB-SIG] Should Binary accept unicode string? In-Reply-To: <56992A9D.6080509@zzzcomputing.com> References: <5699066B.4080004@egenix.com> <569915B3.2000609@zzzcomputing.com> <56991B38.1090607@egenix.com> <56992A9D.6080509@zzzcomputing.com> Message-ID: Mike: Thank you for your long explanation. I got lost somewhere in the middle there, though. If you are suggesting that better documentation be added to PEP-249, then perhaps you could include a suggestion as to a (brief) note which could be appended. If you are suggesting that the PEP be expanded to provide a service not now generally available, then perhaps you ought to start the long-talked-about-but-never-tried task of writing a DBAPI level 3 PEP. The present PEP (written long before Python 3) is lacking (and in at least one case, actually wrong) any specific Python 3 implementation details. Several enhancements and improvements to the PEP have been talked about, most of which -- like the one you seem to be making -- would significantly enhance a programmer's experience. There are also several "standard extensions" and frequently added extras which could be included in the body of a new PEP. From mike_mp at zzzcomputing.com Sat Jan 16 10:27:58 2016 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Sat, 16 Jan 2016 10:27:58 -0500 Subject: [DB-SIG] Should Binary accept unicode string? In-Reply-To: References: <5699066B.4080004@egenix.com> <569915B3.2000609@zzzcomputing.com> <56991B38.1090607@egenix.com> <56992A9D.6080509@zzzcomputing.com> Message-ID: <569A617E.2000708@zzzcomputing.com> On 01/15/2016 05:14 PM, Vernon D. Cole wrote: > Mike: > > Thank you for your long explanation. I got lost somewhere in the > middle there, though. > > If you are suggesting that better documentation be added to PEP-249, > then perhaps you could include a suggestion as to a (brief) note which > could be appended. Well, I wasn't even going that far. I'm trying to get a handle on what pep-249's position is as far as portability of datatypes. It has always struck me as very weak. > > If you are suggesting that the PEP be expanded to provide a service > not now generally available, then perhaps you ought to start the > long-talked-about-but-never-tried task of writing a DBAPI level 3 PEP. Well if a DBAPI driver would like to accept a Python unicode object to a Binary() and produce bytes, some conversion is needed, and there are many possible conversions that could take place - there is every possible encoding, and at typically at least four potential candidates among those available. It's my position that the Binary() type should *not* offer to automatically choose such a conversion and should only accept Python types (or 3rd party extension types, sure) that are explicitly 1-1 mappable to a stream of bytes without a "conversion decision" being made. The type of conversion should not be guessed among a choice of several / hundreds within the Binary type. So definitely, not proposing any new service other than "disallow ambiguous input". > The present PEP (written long before Python 3) is lacking (and in at > least one case, actually wrong) any specific Python 3 implementation > details. Several enhancements and improvements to the PEP have been > talked about, most of which -- like the one you seem to be making -- > would significantly enhance a programmer's experience. There are also > several "standard extensions" and frequently added extras which could > be included in the body of a new PEP. I'd love to write the pep for DBAPI3. That task doesn't daunt me, but the task of evaluating reactions and changes to it and shepherding it to acceptance certainly does. I seem to generally be on the other side of most issues on this list and the task would be more surmountable if I had some allies to assist during the long process of defense. that said if we could at least produce / point me to a document that lists out the general things people are looking for, for me I know its fewer bindparam formats and explicitness about how unicode / strings should be handled, also there's the prepared statement thing. Oh, and the tuple/list thing for result rows that I think I'm the only person who cares about that. > From mal at egenix.com Sat Jan 16 14:30:56 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 16 Jan 2016 20:30:56 +0100 Subject: [DB-SIG] Should Binary accept unicode string? In-Reply-To: <569A617E.2000708@zzzcomputing.com> References: <5699066B.4080004@egenix.com> <569915B3.2000609@zzzcomputing.com> <56991B38.1090607@egenix.com> <56992A9D.6080509@zzzcomputing.com> <569A617E.2000708@zzzcomputing.com> Message-ID: <569A9A70.8010608@egenix.com> On 16.01.2016 16:27, Mike Bayer wrote: > > > On 01/15/2016 05:14 PM, Vernon D. Cole wrote: >> Mike: >> >> Thank you for your long explanation. I got lost somewhere in the >> middle there, though. >> >> If you are suggesting that better documentation be added to PEP-249, >> then perhaps you could include a suggestion as to a (brief) note which >> could be appended. > > > Well, I wasn't even going that far. I'm trying to get a handle on what > pep-249's position is as far as portability of datatypes. It has > always struck me as very weak. > > > >> >> If you are suggesting that the PEP be expanded to provide a service >> not now generally available, then perhaps you ought to start the >> long-talked-about-but-never-tried task of writing a DBAPI level 3 PEP. > > Well if a DBAPI driver would like to accept a Python unicode object to a > Binary() and produce bytes, some conversion is needed, and there are > many possible conversions that could take place - there is every > possible encoding, and at typically at least four potential candidates > among those available. > > It's my position that the Binary() type should *not* offer to > automatically choose such a conversion and should only accept Python > types (or 3rd party extension types, sure) that are explicitly 1-1 > mappable to a stream of bytes without a "conversion decision" being > made. The type of conversion should not be guessed among a choice of > several / hundreds within the Binary type. > > So definitely, not proposing any new service other than "disallow > ambiguous input". I still don't understand why you want to restrict Binary() to perform automatic conversions on the input types. Unicode is just one example of where you can implement such conversion, e.g. a database module may want to automatically convert Unicode to UTF-8. For database backends which don't provide Unicode support, this is usually also being done for string parameter types. mxODBC, for example, allows setting a per connection .encoding attribute to define which encoding to use in such cases. But again, Unicode is just one example. Binary() may also apply automatic conversions for other types, such as images, numeric arrays, etc. The DB-API standard cannot define which types to autoconvert and which not. This is a conscious decision left to the database module authors. They have to make similar choices for all other parameter types as well, e.g. whether to convert datetime values to strings, ticks or whether to reject them. In many cases, the database backends don't provide parameter type information, so the database module has to decide what to do. In other cases, the database module may get type information from the database and then has to decide what to do with the input parameters passed to it from Python. Back to Binary(): What we could do is recommend to use e.g. buffer() for Python 2 as default implementation and bytes() for Python 3. The fact that buffer() does accept Unicode objects in Python 2 is due to the way the buffer interface works in Python 2 (in 2000 we thought it would be a good idea to allow access to the UCS-2 data; later on, when we added UCS-4 support, we could not easily remove this feature anymore). For Python 3, bytes() won't accept Unicode because the buffer interface was changed in Python 3 to no longer expose the binary buffer interface. Would that make you happy ? :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Sat Jan 16 22:51:54 2016 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Sat, 16 Jan 2016 22:51:54 -0500 Subject: [DB-SIG] Should Binary accept unicode string? In-Reply-To: <569A9A70.8010608@egenix.com> References: <5699066B.4080004@egenix.com> <569915B3.2000609@zzzcomputing.com> <56991B38.1090607@egenix.com> <56992A9D.6080509@zzzcomputing.com> <569A617E.2000708@zzzcomputing.com> <569A9A70.8010608@egenix.com> Message-ID: <569B0FDA.8090006@zzzcomputing.com> On 01/16/2016 02:30 PM, M.-A. Lemburg wrote: > On 16.01.2016 16:27, Mike Bayer wrote: >> >> >> On 01/15/2016 05:14 PM, Vernon D. Cole wrote: >>> Mike: >>> >>> Thank you for your long explanation. I got lost somewhere in the >>> middle there, though. >>> >>> If you are suggesting that better documentation be added to PEP-249, >>> then perhaps you could include a suggestion as to a (brief) note which >>> could be appended. >> >> >> Well, I wasn't even going that far. I'm trying to get a handle on what >> pep-249's position is as far as portability of datatypes. It has >> always struck me as very weak. >> >> >> >>> >>> If you are suggesting that the PEP be expanded to provide a service >>> not now generally available, then perhaps you ought to start the >>> long-talked-about-but-never-tried task of writing a DBAPI level 3 PEP. >> >> Well if a DBAPI driver would like to accept a Python unicode object to a >> Binary() and produce bytes, some conversion is needed, and there are >> many possible conversions that could take place - there is every >> possible encoding, and at typically at least four potential candidates >> among those available. >> >> It's my position that the Binary() type should *not* offer to >> automatically choose such a conversion and should only accept Python >> types (or 3rd party extension types, sure) that are explicitly 1-1 >> mappable to a stream of bytes without a "conversion decision" being >> made. The type of conversion should not be guessed among a choice of >> several / hundreds within the Binary type. >> >> So definitely, not proposing any new service other than "disallow >> ambiguous input". > > I still don't understand why you want to restrict Binary() > to perform automatic conversions on the input types. it would be consistent with the philosophy of Python 3 itself that unicode and "bytes" are two different things without an implicit conversion. It's also a place that without clear guidance in the pep, some DBAPIs are going to do, and others not, leading to non-portable code. Basically, if pep-249 said, "The Binary() object should accept Python unicode objects and should encode them to bytes using an encoding indicated by the .encoding attribute on Connection", that would be better than the current situation of that it says nothing at all. It's the "it says nothing at all" part here that's more troubling to me, rather than whether or not a Unicode should be encoded when passed to a Binary(). If the spec had a recommendation for where this encoding should come from, then that allows portable code to be written. > > Unicode is just one example of where you can implement such > conversion, e.g. a database module may want to automatically > convert Unicode to UTF-8. For database backends which don't > provide Unicode support, this is usually also being done > for string parameter types. it is true that even in Python 3, DBAPIs are obviously taking on the task of figuring out an appropriate encoding to use for strings, which are unicode objects in Python 3. It strongly suggests that pep-249 or its successor would be served by referring to an "encoding" setting in a standard way. > > mxODBC, for example, allows setting a per connection .encoding > attribute to define which encoding to use in such cases. > > But again, Unicode is just one example. Binary() may also > apply automatic conversions for other types, such as images, > numeric arrays, etc. > > The DB-API standard cannot define which types to autoconvert > and which not. This is a conscious decision left to the database > module authors. > > They have to make similar choices for all other parameter > types as well, e.g. whether to convert datetime values to > strings, ticks or whether to reject them. > In many cases, the database backends don't provide parameter > type information, so the database module has to decide what > to do. In other cases, the database module may get type information > from the database and then has to decide what to do with the > input parameters passed to it from Python. > > Back to Binary(): What we could do is recommend to use e.g. > buffer() for Python 2 as default implementation and bytes() > for Python 3. I think Binary should certainly accept str in Py2K, since that's the normal place we get "bytes" from in Py2K. If it wants to accept a buffer() also, that's fine (because no guess needs to be made), I doubt anyone will use it though as its a deprecated type and they can just use str. > > The fact that buffer() does accept Unicode objects in Python 2 > is due to the way the buffer interface works in Python 2 (in 2000 > we thought it would be a good idea to allow access to > the UCS-2 data; later on, when we added UCS-4 support, we > could not easily remove this feature anymore). yeah UCS-2 / UCS-4, not too useful in the outside world these days :) > > For Python 3, bytes() won't accept Unicode because the buffer > interface was changed in Python 3 to no longer expose the > binary buffer interface. > > Would that make you happy ? :-) it's all great! I only seek to understand the thinking behind DBAPI's philosophy in areas like these. > From madhusudhan.achar at oracle.com Fri Jan 22 03:45:03 2016 From: madhusudhan.achar at oracle.com (madhusudhan achar) Date: Fri, 22 Jan 2016 14:15:03 +0530 Subject: [DB-SIG] Need clarifications/help on preserving revisions of files/permissions using Python Message-ID: <56A1EC0F.2000409@oracle.com> Hello, My current problem is : I have scripts written currently in python, wherein we make lot of changes to files (system), permission changes to files and directories, changes to kernel parameters. We would like to maintain the revisions of these changes and should be able tor rollback the changes at any point in time. The initial thought is to use MySQLdb. My questions are: 1. Provide me sample references to such problems if they exist. 2. How do we preserve changes to files? we need to save the entire files. 3. We need to make permission changes of files/directories (owner/group changes) 4. How do we store the changes/files? Should they stored as tables? 5. How do we restore the stored tables and restore files and file/dir permissions..? I shall look forward to your help/suggestions... thanks Madhu From mal at egenix.com Mon Jan 25 06:16:46 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 25 Jan 2016 12:16:46 +0100 Subject: [DB-SIG] Need clarifications/help on preserving revisions of files/permissions using Python In-Reply-To: <56A1EC0F.2000409@oracle.com> References: <56A1EC0F.2000409@oracle.com> Message-ID: <56A6041E.4080800@egenix.com> Hello Madhu, I am not sure how your posting is relevant to this mailing list. Most of your questions seem to have to do with OS permissions and storing/restoring files. You can store the necessary details in a database, but the actual logic for managing the files themselves will have to be done using e.g. the os module. You might also want to look at a configuration system such as Ansible or SaltStack for managing your systems. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Jan 25 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ On 22.01.2016 09:45, madhusudhan achar wrote: > Hello, > > My current problem is : > > I have scripts written currently in python, wherein we make lot of changes to files (system), > permission changes to > files and directories, changes to kernel parameters. We would like to maintain the revisions of > these changes and should > be able tor rollback the changes at any point in time. > > The initial thought is to use MySQLdb. > > My questions are: > 1. Provide me sample references to such problems if they exist. > 2. How do we preserve changes to files? we need to save the entire files. > 3. We need to make permission changes of files/directories (owner/group changes) > 4. How do we store the changes/files? Should they stored as tables? > 5. How do we restore the stored tables and restore files and file/dir permissions..? > > I shall look forward to your help/suggestions... > > thanks > Madhu > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig