<br><br><div class="gmail_quote">On 28 September 2010 12:29, Michael Foord <span dir="ltr"><<a href="mailto:fuzzyman@voidspace.org.uk">fuzzyman@voidspace.org.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="im"> On 28/09/2010 12:19, Antoine Pitrou wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
On Mon, 27 Sep 2010 23:45:45 -0400<br>
Steve Holden<<a href="mailto:steve@holdenweb.com" target="_blank">steve@holdenweb.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
On 9/27/2010 11:27 PM, Benjamin Peterson wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
2010/9/27 Meador Inge<<a href="mailto:meadori@gmail.com" target="_blank">meadori@gmail.com</a>>:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
which, as seen in the trace, is because the 'detect_encoding' function in<br>
'Lib/tokenize.py' searches for 'BOM_UTF8' (a 'bytes' object) in the string<br>
to tokenize 'first' (a 'str' object). It seems to me that strings should<br>
still be able to be tokenized, but maybe I am missing something.<br>
Is the implementation of 'detect_encoding' correct in how it attempts to<br>
determine an encoding or should I open an issue for this?<br>
</blockquote>
Tokenize only works on bytes. You can open a feature request if you desire.<br>
<br>
</blockquote>
Working only on bytes does seem rather perverse.<br>
</blockquote>
I agree, the morality of bytes objects could have been better :)<br>
<br>
</blockquote></div>
The reason for working with bytes is that source data can only be correctly decoded to text once the encoding is known. The encoding is determined by reading the encoding cookie.<br>
<br>
I certainly wouldn't be opposed to an API that accepts a string as well though.<br>
<br></blockquote><div><br>Ah, and to explain the design decision when tokenize was ported to py3k - the Python 2 APIs take the readline method of a file object (not a string). <br><br><a href="http://docs.python.org/library/tokenize.html">http://docs.python.org/library/tokenize.html</a><br>
<br>For this to work correctly in Python 3 it *has* to be a file object open in binary read mode in order to decode the source code correctly.<br><br>A new API that takes a string would certainly be nice. The Python 2 API for tokenize is 'interesting'...<br>
<br>All the best,<br><br>Michael Foord<br><br> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
All the best,<br>
<br>
Michael<div><div></div><div class="h5"><br>
<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br>
_______________________________________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org" target="_blank">Python-Dev@python.org</a><br>
<a href="http://mail.python.org/mailman/listinfo/python-dev" target="_blank">http://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk" target="_blank">http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk</a><br>
</blockquote>
<br>
<br></div></div><font color="#888888">
-- <br>
<a href="http://www.ironpythoninaction.com/" target="_blank">http://www.ironpythoninaction.com/</a><br>
<a href="http://www.voidspace.org.uk/blog" target="_blank">http://www.voidspace.org.uk/blog</a><br>
<br>
READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.<br>
<br>
<br>
</font></blockquote></div><br><br clear="all"><br>-- <br><a href="http://www.voidspace.org.uk">http://www.voidspace.org.uk</a><br><br><br>