Mailman 3 - Python-Dev

RE: [Python-Dev] re: Unicode as argument for 8-bit strings
by Bill Tutt 08 Apr '00

08 Apr '00

> From: Fredrik Lundh [mailto:fredrik@pythonware.com] > > Bill Tutt wrote: > > > There has been a bug report about the treatment of Unicode > > > objects together with 8-bit format strings. The current > > > implementation converts the Unicode object to UTF-8 and then > > > inserts this value in place of the %s.... > > > > > > I'm inclined to change this to have '...%s...' % u'abc' > > > return u'...abc...' since this is just another case of > > > coercing data to the "bigger" type to avoid information loss. > > > > > > Thoughts ? > > > > Suddenly returning a Unicode string from an operation that > was an 8-bit > > string is likely to give some code exterme fits of despondency. > > why is this different from returning floating point values from > operations involving integers and floats? > > > Converting to UTF-8 didn't give you any data loss, however > it certainly > > might be unexpected to now find UTF-8 characters in what > the user originally > > thought was a binary string containing whatever they had > wanted it to contain. > > the more I've played with this, the stronger my opinion that > the "now it's an ordinary string, now it's a UTF-8 string, now > it's an ordinary string again" approach doesn't work. more on > this in a later post. > Well, unicode string/UTF-8 string, but I definately agree with you. Pick one or the other and make the user convert betwixt the two. > (am I the only one here that has actually tried to write code > that handles both unicode strings and ordinary strings? if not, > can anyone tell me what I'm doing wrong?) > In C++, yes. :) Autoconverting into or out of unicode is bound to lead to trouble for someone. Look at the various messes that misused C++ operator overloading can get you into. Whether its the code that wasn't expecting UTF-8 in a normal string type, or a formatting operation that used to return a normal string type now returning a Unicode string. > > Throwing an exception would at the very least force the > user to make a > > decision one way or the other about what they want to do > with the data. > > They might want to do a codepage translation, or something > else. (aka Hey, > > here's a bug I just found for you!) > > > In what other cases are you suddenly returning a Unicode > string object from > > which previouslly returned a string object? > > if unicode is ever to be a real string type in python, and not just a > nifty extension type, it must be okay to return a unicode string from > any operation that involves a unicode argument... Err. I'm not sure what you're getting at here. If your saying that it'd be nice if we could ditch the current string type and just use the Unicode string type, then I agree with you. However, that doesn't mean you should change the semantics of an operation that existed before unicode came into the picture, since it would break backward compatability. +1 for '%s' % u'\u1234' throwing a TypeError exception. Bill

1 0

Unicode as argument for 8-bit format strings
by M.-A. Lemburg 07 Apr '00

07 Apr '00

There has been a bug report about the treatment of Unicode objects together with 8-bit format strings. The current implementation converts the Unicode object to UTF-8 and then inserts this value in place of the %s.... I'm inclined to change this to have '...%s...' % u'abc' return u'...abc...' since this is just another case of coercing data to the "bigger" type to avoid information loss. Thoughts ? -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

3 6

the regression test suite
by Vladimir.Marangozov＠inrialpes.fr 07 Apr '00

07 Apr '00

My kitchen programs show that regrtest.py keeps requesting more and more memory until it finishes all tests. IOW, it doesn't finalize properly each test. It keeps importing modules, without deleting them after each test. I think that before a particular test is run, we need to save the value of sys.modules, then restore it after the test (before running the next one). In a module enabled interpreter, this reduces the memory consumption almost by half... Patch? Think about the number of new tests that will be added in the future. I don't want to tolerate a silently approaching useless disk swapping :-) -- Vladimir MARANGOZOV | Vladimir.Marangozov(a)inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

2 1

Long Multiplication is not commutative.
by Christian Tismer 07 Apr '00

07 Apr '00

Yikes! No, it is computatively commutative, just not in terms of computation time. :-)) The following factorial loops differ by a remarkable factor of 1.8, and we can gain this speed by changing long_mult to always put the lower multiplicand into the left. This was reported to me by Lenny Kneler, who thought he had found a Stackless bug, but he was actually testing long math. :-) This buddy... >>> def ifact3(n) : ... p = 1L ... for i in range(1,n+1) : ... p = i*p ... return p performs better by a factor of 1.8 than this one: >>> def ifact1(n) : ... p = 1L ... for i in range(1,n+1) : ... p = p*i ... return p The analysis of this behavior is quite simple if you look at the implementation of long_mult. If the left operand is big and the right is small, there are much more carry operations performed, together with more loop overhead. Swapping the multiplicands would be a 5 line patch. Should I submit it? ciao - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com

3 3

Re: [Python-Dev] PYTHON_API_VERSION and threading
by Jack Jansen 07 Apr '00

07 Apr '00

> > Something that just struck me: couldn't we use a couple of bits in the > > PYTHON_API_VERSION to check various other things that make dynamic modules > > break? WITH_THREAD is the one I just ran in to, but there's a few others such > > as the object refcounting statistics and platform-dependent things like the > > debug/nodebug compilation on Windows. > > I'm curious what combination didn't work? The thread APIs are > supposed to be designed so that all combinations work -- the APIs are > always present, they just don't do anything in the unthreaded > version. Oops, the problem was mine: not only was the extension module compiled without threading, but also with the previous version of the I/O library used on the mac. Silly me. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

1 0

Why do we need Traceback Objects?
by Christian Tismer 07 Apr '00

07 Apr '00

Hi, while fixing my design flaws after Just's Stackless Mac port, I was dealing with some overflow conditions and tracebacks. When there is a recursion depth overflow condition, we create a lot of new structure for the tracebacks. This usually happens in a situation where memory is quite exhausted. Even worse if we crash because of a memory error: The system will not have enough memory to build the traceback structure, to report the error. Puh :-) When I look into tracebacks, it turns out to be just a chain like the frame chain, but upward down. It holds references to the frames in a 1-to-1 manner, and it keeps copies of f->f_lasti and f->f_lineno. I don't see why this is needed. I'm thinking to replace the tracebacks by a single pointer in the frames for this purpose. It appears further to be possible to do that without any extra memory, since all the frames have extra temporary fields for exception info, and that isn't used in this context. Traceback objects exist each for one and only one frame, and they could be embedded into their frame. Does this make sense? Do I miss something? I'm considering this for Stackless and would like to know if I should prepare it for orthodox Python as well? ciao - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com

5 10

default encoding for 8-bit string literals (was Unicode and comparisons)
by pf＠artcom-gmbh.de 07 Apr '00

07 Apr '00

Hi! [me]: > > From my POV (using ISO Latin-1 all the time) it would be > > "intuitive"(TM) to assume ISO Latin-1 when interpreting u'äöü' in a > > Python source file so that (u'äöü' == 'äöü') == 1. This is what I see > > on *my* screen, whether there is a 'u' in Front of the string or not. M.-A. Lemburg: > u"äöü" is being interpreted as Latin-1. The problem is the > string 'äöü' to the right: during coercion this string is > being interpreted as UTF-8 and this causes the failure. > > You could say: ok, all my strings use Latin-1, but that would > introduce other problems... esp. when you take different > modules with different encoding assumptions and try to > integrate them into an application. Okay. This wouldn't occur here but we have deal with this possibility. > > In dist/src/Misc/unicode.txt you wrote: > > > > > Note that you should provide some hint to the encoding you used to > > > write your programs as pragma line in one the first few comment lines > > > of the source file (e.g. '# source file encoding: latin-1'). [me]: > > The upcoming 1.6 documentation should probably clarify whether > > the interpreter pays attention to "pragma"s or not. > > This is otherwise misleading. > > This "pragma" is nothing more than a hint for the source code > reader to switch his viewing encoding. The interpreter doesn't > treat the file differently. In fact, Python source code is > supposed to tbe 7-bit ASCII ! Sigh. In our company we use 'german' as our master language so we have string literals containing iso-8859-1 umlauts all over the place. Okay as long as we don't mix them with Unicode objects, this doesn't hurt anybody. What I would love to see, would be a well defined way to tell the interpreter to use 'latin-1' as default encoding instead of 'UTF-8' when dealing with string literals from our modules. The tokenizer in Python 1.6 already contains smart logic to get the size of TABs right (pasting from tokenizer.c): /* Skip comment, while looking for tab-setting magic */ if (c == '#') { static char *tabforms[] = { "tab-width:", /* Emacs */ ":tabstop=", /* vim, full form */ ":ts=", /* vim, abbreviated form */ "set tabsize=", /* will vi never die? */ /* more templates can be added here to support other editors */ }; .. It wouldn't be to hard to add something there to recognize other "pragma" comments like for example: #content-transfer-encoding: iso-8859-1 But what to do with it? May be adding a default encoding to every string object? Is this bloat? Just an idea. Regards, Peter

4 9

Pythons (Like Buses) Considered Harmful
by Ka-Ping Yee 07 Apr '00

07 Apr '00

So, has anyone not seen Doctor Fun today yet? http://metalab.unc.edu/Dave/Dr-Fun/latest.jpg :) :) -- ?!ng

2 1

DLL in the system directory on Windows
by Thomas Heller 07 Apr '00

07 Apr '00

> However, as we import data from Python16.dll rather then purely > addresses, we can't use any of these interception solutions. What's wrong with: #define PyClass_Type *(GetProcAddress(hdll, "PyClass_Type")) I have only looked at PythonCOM15.dll, and it seems that there are only references to a handfull of exported data items: some Py*_Type, plus _PyNone_Struct, _PyTrue_Struct, _PyZero_Struct. Thomas Heller

2 3

DLL in the system directory on Windows.
by Mark Hammond 06 Apr '00

06 Apr '00

The 1.6a1 installer on Windows copies Python16.dll into the Python directory, rather than the system32 directory like 1.5.x. We discussed too long ago on this list not why this was probably not going to work. I guess Guido decided to "suck it and see" - which is fine. But guess what - it doesnt work :-( I couldnt get past the installer! The win32all installer executes some Python code at the end of the install (to generate the .pyc files and install the COM objects). This Python code is executed directly to the installation .EXE, by loading and executing a "shim" DLL I wrote for the purpose. Problem is, try as I might, my shim DLL could not load Python16.dll. The shim DLL _was_ in the same directory as Python16.dll. The only way I could have solved it was to insist the WISE installation .EXE be run from the main Python directory - obviously not an option. And the problem is quite obviously going to exist with COM objects. The problem would appear to go away if the universe switched over the LoadLibraryEx() - but we dont have that control in most cases (eg, COM, WISE etc dictate this to us). So, my solution was to copy Python16.dll to the system directory during win32all installation. This results in duplicate copies of this DLL, so to my mind, it is preferable that Python itself go back to using the System32 directory. The problem this will lead to is that Python 1.6.0 and 1.6.1 will not be able to be installed concurrently. Putting entries on the PATH doesnt solve the underlying problem - you will only be able to have one Python 1.6 directory on your path, else you end up with the same coflicts for the DLL. I dont see any better answer than System32 :-( Thoughts? Mark.

8 35