Mailman 3 May 2000 - Python-Dev

Re: [Python-checkins] CVS: python/dist/src/Tools/idle browser.py,NONE,1.1
by Guido van Rossum 10 May '00

10 May '00

Fred, "browser" is a particularly non-descriptive name for this module. Perhaps it's not too late to rename it to e.g. "BrowserControl"? --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

Re: [Patches] PC\config.[hc] changes for Win64
by Trent Mick 10 May '00

10 May '00

On Mon, May 08, 2000 at 10:16:42AM -0400, Guido van Rossum wrote: > > The patch to config.h looks big but it really is not. These are the effective > > changes: > > - MS_WINxx are keyed off _WINxx > > - SIZEOF_VOID_P is set to 8 for Win64 > > - COMPILER string is changed appropriately for Win64 > > One thing worries me: if COMPILER is changed, that changes > sys.platform to "win64", right? I'm sure that will break plenty of > code which currently tests for sys.platform=="win32" but really wants > to test for any form of Windows. Maybe sys.platform should remain > win32? > No, but yes. :( Actually I forgot to mention that my config.h patch changes the PLATFORM #define from win32 to win64. So yes, you are correct. And, yes (Sigh) you are right that this will break tests for sys.platform == "win32". So I guess the simplest thing to do is to leave it as win32 following the same reasoning for defining MS_WIN32 on Win64: > The idea is that the common case is > that code specific to Win32 will also work on Win64 rather than being > specific to Win32 (i.e. there is more the same than different in WIn32 and > Win64). What if someone needs to do something in Python code for either Win32 or Win64 but not both? Or should this never be necessary (not likely). I would like Mark H's opinion on this stuff. Trent -- Trent Mick trentm(a)activestate.com

8 20

Python Query
by Christian Zupancic 09 May '00

09 May '00

====================================================================== Greetings Python Developers, Please participate in a small survey about Python for BeOpen.com that we are conducting with the guidance of our advisor, and the creator of Python, Guido van Rossum. In return for answering just five short questions, I will mail you up to three (3) BeOpen T-shirts-- highly esteemed by select trade-show attendees as "really cool". In addition, three lucky survey participants will receive a Life-Size Inflatable Penguin (as they say, "very cool"). - Why do you prefer Python over other languages, e.g. Perl? - What do you consider to be (a) competitor(s) to Python? - What are Python's strong points and weaknesses? - What other languages do you program in? - If you had one wish about Python, what would it be? - For Monty Python fans only: What is the average airspeed of a swallow (European, non-migratory)? THANKS! That wasn't so bad, was it? Make sure you've attached a business card or address of some sort so I know where to send your prizes. Best Regards, Christian Zupancic Market Analyst, BeOpen.com

1 0

global encoding?!? (was: [Python-checkins] ... unicodeobject.c)
by Greg Stein 09 May '00

09 May '00

On Tue, 9 May 2000, Fred Drake wrote: > Update of /projects/cvsroot/python/dist/src/Objects > In directory seahag.cnri.reston.va.us:/home/fdrake/projects/python/Objects > > Modified Files: > unicodeobject.c > Log Message: > > M.-A. Lemburg <mal(a)lemburg.com>: > Added support for user settable default encodings. The > current implementation uses a per-process global which > defines the value of the encoding parameter in case it > is set to NULL (meaning: use the default encoding). Umm... maybe I missed something, but I thought there was pretty broad feelings *against* having a global like this. This kind of thing is just nasty. 1) Python modules can't change it, nor can they rely on it being a particular value 2) a mutable, global variable is just plain wrong. The InterpreterState and ThreadState structures were created *specifically* to avoid adding crap variables like this. 3) allowing a default other than utf-8 is sure to cause gotchas and surprises. Some code is going to rightly assume that the default is just that, but be horribly broken when an application changes it. Somebody please say this is hugely experimental. And then say why it isn't just a private patch, rather than sitting in CVS. :-( -g -- Greg Stein, http://www.lyra.org/

3 3

Re: [Patches] make 'b','h','i' raise overflow exception
by Trent Mick 09 May '00

09 May '00

On Mon, May 08, 2000 at 10:00:30AM -0400, Guido van Rossum wrote: > > Changes the 'b', 'h', and 'i' formatters in PyArg_ParseTuple to raise an > > Overflow exception if they overflow (previously they just silently > > overflowed). > > Trent, > > There's one issue with this: I believe the 'b' format is mostly used > with unsigned character arguments in practice. >However on systems > with default signed characters, CHAR_MAX is 127 and values 128-255 are > rejected. I'll change the overflow test to: > > else if (ival > CHAR_MAX && ival >= 256) { > > if that's okay with you. > Okay, I guess. Two things: 1. In a way this defeats the main purpose of the checks. Now a silent overflow could happen for a signed byte value over CHAR_MAX. The only way to automatically do the bounds checking is if the exact type is known, i.e. different formatters for signed and unsigned integral values. I don't know if this is desired (is it?). The obvious choice of 'u' prefixes to specify unsigned is obviously not an option. Another option might be to document 'b' as for unsigned chars and 'h', 'i', 'l' as signed integral values and then set the bounds checks ([0, UCHAR_MAX] for 'b') appropriately. Can we clamp these formatters so? I.e. we would be limiting the user to unsigned or signed depending on the formatter. (Which again, means that it would be nice to have different formatters for signed and unsigned.) I think that the bounds checking is false security unless these restrictions are made. 2. The above aside, I would be more inclined to change the line in question to: else if (ival > UCHAR_MAX) { as this is more explicit about what is being done. > Another issue however is that there are probably cases where an 'i' > format is used (which can't overflow on 32-bit architectures) but > where the int value is then copied into a short field without an > additional check... I'm not sure how to fix this except by a complete > inspection of all code... Not clear if it's worth it. Yes, a complete code inspection seems to be the only way. That is some of what I am doing. Again, I have two questions: 1. There are a fairly large number of downcasting cases in the Python code (not necessarily tied to PyArg_ParseTuple results). I was wondering if you think a generalized check on each such downcast would be advisable. This would take the form of some macro that would do a bounds check before doing the cast. For example (a common one is the cast of strlen's size_t return value to int, because Python strings use int for their length, this is a downcast on 64-bit systems): size_t len = strlen(s); obj = PyString_FromStringAndSize(s, len); would become size_t len = strlen(s); obj = PyString_FromStringAndSize(s, CAST_TO_INT(len)); CAST_TO_INT would ensure that 'len'did not overflow and would raise an exception otherwise. Pros: - should never have to worry about overflows again - easy to find (given MSC warnings) and easy to code in (staightforward) Cons: - more code, more time to execute - looks ugly - have to check PyErr_Occurred every time a cast is done I would like other people's opinion on this kind of change. There are three possible answers: +1 this is a bad change idea because...<reason> -1 this is a good idea, go for it +0 (mostly likely) This is probably a good idea for some case where the overflow *could* happen, however the strlen example that you gave is *not* such a situation. As Tim Peters said: 2GB limit on string lengths is a good assumption/limitation. 2. Microsofts compiler gives good warnings for casts where information loss is possible. However, I cannot find a way to get similar warnings from gcc. Does anyone know if that is possible. I.e. int i = 123456; short s = i; // should warn about possible loss of information should give a compiler warning. Thanks, Trent -- Trent Mick trentm(a)activestate.com

3 4

Cannot declare the largest integer literal.
by Trent Mick 08 May '00

08 May '00

>>> i = -2147483648 OverflowError: integer literal too large >>> i = -2147483648L >>> int(i) # it *is* a valid integer literal -2147483648 As far as I traced back: Python/compile.c::com_atom() calls Python/compile.c::parsenumber(s = "2147483648") calls Python/mystrtoul.c::PyOS_strtol() which returns the ERANGE errno because it is given 2147483648 (which *is* out of range) rather than -2147483648. My question: Why is the minus sign not considered part of the "atom", i.e. the integer literal? Should it be? PyOS_strtol() can properly parse this integer literal if it is given the whole number with the minus sign. Otherwise the special case largest negative number will always erroneously be considered out of range. I don't know how the tokenizer works in Python. Was there a design decision to separate the integer literal and the leading sign? And was the effect on functions like PyOS_strtol() down the pipe missed? Trent -- Trent Mick trentm(a)activestate.com

5 8

Re: Python 1.6a2 Unicode bug (was Re: comparing strings and ints)
by Just van Rossum 08 May '00

08 May '00

I wrote: >A utf-8-encoded 8-bit string in Python is *not* a string, but a "ByteArray". Another way of putting this is: - utf-8 in an 8-bit string is to a unicode string what a pickle is to an object. - defaulting to utf-8 upon coercing is like implicitly trying to unpickle an 8-bit string when comparing it to an instance. Bad idea. Defaulting to Latin-1 is the only logical choice, no matter how western-culture-centric this may seem. Just

16 66

Re: [XML-SIG] Re: [I18n-sig] Re: [Python-Dev] Unicode debate
by ht＠cogsci.ed.ac.uk 07 May '00

07 May '00

Guido van Rossum <guido(a)python.org> writes: > Paul, we're both just saying the same thing over and over without > convincing each other. I'll wait till someone who wasn't in this > debate before chimes in. OK, I've never contributed to this discussion, but I have a long history of shipping widely used Python/Tkinter/XML tools (see my homepage). I care _very_ much that heretofore I have been unable to support full XML because of the lack of Unicode support in Python. I've already started playing with 1.6a2 for this reason. I notice one apparent mis-communication between the various contributors: Treating narrow-strings as consisting of UNICODE code points <= 255 is not necessarily the same thing as making Latin-1 the default encoding. I don't think on Paul and Fredrik's account encoding are relevant to narrow-strings at all. I'd rather go right away to the coherent position of byte-arrays, narrow-strings and wide-strings. Encodings are only relevant to conversion between byte-arrays and strings. Decoding a byte-array with a UTF-8 encoding into a narrow string might cause overflow/truncation, just as decoding a byte-array with a UTF-8 encoding into a wide-string might. The fact that decoding a byte-array with a Latin-1 encoding into a narrow-string is a memcopy is just a side-effect of the courtesy of the UNICODE designers wrt the code points between 128 and 255. This is effectively the way our C-based XML toolset (which we embed in Python) works today -- we build an 8-bit version which uses char* strings, and a 16-bit version which uses unsigned short* strings, and convert from/to byte-streams in any supported encoding at the margins. I'd like to keep byte-arrays at the margins in Python as well, for all the reasons advanced by Paul and Fredrik. I think treating existing strings as a sort of pun between narrow-strings and byte-arrays is a recipe for ongoing confusion. ht -- Henry S. Thompson, HCRC Language Technology Group, University of Edinburgh W3C Fellow 1999--2001, part-time member of W3C Team 2 Buccleuch Place, Edinburgh EH8 9LW, SCOTLAND -- (44) 131 650-4440 Fax: (44) 131 650-4587, e-mail: ht(a)cogsci.ed.ac.uk URL: http://www.ltg.ed.ac.uk/~ht/

4 5

should a float overflow or just equal 'inf'
by Trent Mick 06 May '00

06 May '00

Hi all, I submitted a patch a coupld of days ago to have the 'b', 'i', and 'h' formatter for PyArg_ParseTuple raise an Overflow exception if they overflow (currently they just silently overflow). Presuming that this is considered a good idea, should this be carried to floats. Floats don't really overflow, they just equal 'inf'. Would it be more desireable to raise an Overflow exception for this? I am inclined to think that this would *not* be desireable based on the following quote: """ the-754-committee-probably-did-the-best-job-of-fixing-binary-fp- that-can-be-done-ly y'rs - tim """ In any case, the question stands. I don't really have an idea of the potential pains that this could cause to (1) efficiecy, (2) external code that expects to deal with 'inf's itself. The reason I ask is because I am looking at related issues in the Python code these days. Trent -- Trent Mick trentm(a)activestate.com

2 1

Pickle diffs anyone?
by Jim Fulton 05 May '00

05 May '00

Someone recently made a cool proposal for utilizing diffs to save space taken by old versions in the Zope object database: http://www.zope.org/Members/jim/ZODB/ReverseDiffVersioning To make this work, we need a good way of diffing pickles. I thought maybe someone here would have some good suggestions. I do think that the topic is sort of interesting (for some definition of "interesting" ;). The page above is a Wiki page. (Wiki is awesome. If you haven't seen it before, check out http://joyful.com/zwiki/ZWiki.) If you are a member of zope.org, you can edit the page directly, which would be fine with me. :) Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

3 2