Mailman 3 September 2005 - Python-Dev

Possible bug in urllib.urljoin
by Andrew Edmondson Sept. 25, 2005

Sept. 25, 2005

Dear all, We've found a problem using urllib.urljoin when upgrading from python 2.3 to 2.4. It no longer joins a particular corner case of URLs correctly (we think!). The code appears to follow the algorithm (from http://www.ietf.org/rfc/rfc1808.txt) for resolving urls almost exacty... I believe the problem occurs when reaching "step 5" (approx line 160) which will happen if the embedded url has no scheme, netloc or path (and is nonempty). Following the algorithm the resulting url should … [View More]

2 1

Re: [Python-Dev] Visibility scope for "for/while/if" statements
by Alexander Myodov Sept. 23, 2005

Sept. 23, 2005

Hello Josiah, >> i = 0 >> while i != 1: >> i += 1 >> j = 5 >> print j JC> Maybe you don't realize this, but C's while also 'leaks' internal JC> variables... JC> int i = 0, j; JC> while (i != 1) { JC> i++; JC> j = 5; JC> } JC> printf("%i %i\n", i, j); Yeah, it may *leak* it in your example. But the advantage is that it may *not leak* as well: for (int i = 0; i <= 5; i++) { int j = 5; } printf ("%i\n", j); // Won't be even … [View More]compiled Or in our "while" case: int i = 0; while (i != 1) { i++; int j = 5; } printf("%i %i\n", i, j); JC> If you haven't yet found a good use for such 'leakage', you should spend JC> more time programming and less time talking; you would find (quite JC> readily) that such 'leaking' is quite beneficial. I see such usages and realize the possible gains from it. But for about five years of software development in a bunch of programming languages, I've found good uses of non-leaking variables *as well*. Up to the level of manually "enclosing" the temporary variables used only once: ... int i = input(); int result; { int j = f1(i); int k = f2(i, j); result = f3(i, j, k); } // Now j and k are dead ... >> accustomed from other languages. That's my sorrowful story. JC> So you mistyped something. I'm crying for you, really I am. Yeah, I did. Just forgot removing the usage / renaming of variable neved initialized (I thought) before. And my error was that I relied upon the interpreter to catch it, mistakenly assuming the locality of variables inside loops as a consistency with other common languages. Mea culpa, without any jeers. The only reason why I was wrote my email was to get the understanding of community assessment of such behaviour, and/or probable workarounds. No blame against Python - it's great even with this unusualty. >> But for the "performance-oriented/human-friendliness" factor, Python >> is anyway not a rival to C and similar lowlevellers. C has >> pseudo-namespaces, though. JC> C does not have pseudo-namespaces or variable encapsulation in for loops. Neither for (int i = 0; i <= 5; i++) { int j = 5; } printf ("%i\n", i); nor for (int i = 0; i <= 5; i++) { int j = 5; } printf ("%i\n", j); gets compiled though with "gcc -std=c99". That is, each loop introduces a new scoping level. JC> Ah hah hah! Look ladies and gentlemen, I caught myself a troll! Python JC> does not rival C in the performance/friendliness realm? Who are you JC> trying to kid? There is a reason why high school teachers are teaching JC> kids Python instead of Pascal, Java, etc., it's because it is easier to JC> learn and use. Ohh, my slip made me considered a troll... I meant only that Python does not rival C in such area as "performance by all means, even with decrease of friendliness" (but the overall performance is indeed good, I've seen even the VoIP solution written purely on on Python). And I believe noone of Python developers want it to be rival. JC> What you are proposing both would reduce speed and JC> usability, which suggests that it wasn't a good idea in the first place. Yes, it lacks performance, but I still believe that an opportunity to close the visibility of variables (*opportunity* rather than *behaviour change*!) is better in terms of "friendliness" rather than lack of it. Just because it offers the alternatives. Because C-style closed visibility can emulate the variable lack, but not vice versa. At least one voice here (of Gareth McCaughan) is not-strictly-against it. So I cannot agree that such *option* can reduce usability. >> "for (int i = 0; i < 10; i++)" works fine nowadays. JC> I'm sorry, but you are wrong. The C99 spec states that you must define JC> the type of i before using it in the loop. Maybe you are thinking of JC> C++, which allows such things. "gcc -std=c99" compiles the line "for (int i = 0; i <= 5; i++);" perfectly. Along with another sample mentioned above . To this day, I relied upon gcc in terms of standards compatibility... JC> Tes your ideas on comp.lang.python first, when more than a handful of JC> people agree with you, come back. Ok. Next time I shall be more careful. Sorry again, and thanks for your time. And thanks for more-or-less patient answers. -- With best regards, Alexander mailto:maa_public@sinn.ru [View Less]

6 5

Adding a conditional expression in Py3.0
by Jim Jewett Sept. 23, 2005

Sept. 23, 2005

Ron Adam wrote: > ( e selects (e1 selects a1, b1), > (e2 selects a2, b2), > (e3 selects a3, b3) ) I think you've just reinvented the case statement, which disagrees with "if" over the order of true and false clauses. For a conditional expression, I think the choices are really down to the following, which was already way too much freedom last (http://www.python.org/peps/pep-0308.html) time: (1) Should it be done at all? + It would be useful,… [View More]

3 2

Compressing MSI files: 2.4.2 candidate?
by "Martin v. Löwis" Sept. 23, 2005

Sept. 23, 2005

I just found that I can save somewhat more than 1MiB in the MSI file by using LZX:21 instead of the standard MSZIP when compressing the CAB file. A resulting package can be found at http://www.dcl.hpi.uni-potsdam.de/home/loewis/python-2.4.2c1.msi Can people please test whether this installs just as well? Is it ok to use this modified procedure for 2.4.2 final (2.4.2c1 still uses MSZIP). Regards, Martin

2 2

Variant of removing GIL.
by Sokolov Yura Sept. 23, 2005

Sept. 23, 2005

Excuse my English. I think I know how to remove GIL!!!! Obviously I am an idiot. First about Py_INCREF and Py_DECREF. We should not remove GIL at all. We should change it. It must be "one writer-many reader" in a following semantic: Lock has a "read-counter" and a "write-counter". Initially both are 0. When "reader" tries to acquire lock for "read" it sleeps until "write-counter" is 0. When he "reader" acquires lock, he increase "read-counter". When "reader" releases lock, he decreases "… [View More]read-counter". One reader will not block other, since he not increases "write-counter". Reader will sleep, if there is any waiting writers, since they are increase "write-counter". When "writer" tries to acquire lock for "write", he increase "write-counter" and sleeps until "read-counter" happens 0. For "writers" lock for "write" - simple lock. when "writer" release lock, he decrease "write-counter". When there is no waiting writers, readers arise. Excuse me for telling obviouse things. I am really reinvent wheel in my head, since I was a bad studient. I think this kind of lock is native for linux (i saw it in a kernel source, but do not know is waiting writer locks new readers or not?). Now, every thread keep an queue of objects to decref. It can be implemented as array, cause it will be freed at once. Initially, every object acquires GIL for "read". Py_INCREF works as usually, Py_DECREF places a ref into a queue. When queue has became full or "100" instruction left ( :-) , it usefull), thread releases GIL for "read" and acquires for "write", when he acquire it, he decrefs all objects stored in a queue and clear queue. After all he acquires GIL for "read". But what could we do with changing objects (dicts,lists and another)? There should be a secondary "one-writer-many-reader" "public-write" GIL - PWGIL. SGIL ought to be more complicated, since it should work in RLOCK semantic for "write" lock. Lets call this lock ROWMR(reentreed one writer - many reader) So semantic for ROWMR can be: When a thread acquires ROWMR lock, it acquires it at a "read" level. Lets name it "write-level"=0. While threads "write-level"=0 it is a "reader". Thread can increase "write-level". When he turns "write-level" from 0 to 1, he becomes "writer". while "write-level">0, thread is writer. Thread can decrease "write-level". When "write-level" turns from 1 to 0, thread becomes "reader". With PWGIL : We can mark every _mutable_ object with a creator thread number. If mark match current thread number, object is "private" for the thread. If mark is 0 (or another imposible thread number) object is "public". If mark !=0 and !=current thread number, object is "alien". When we access _mutable_ object, we check is it "private"? If it is, we can do anything without locking. If it is not and we access for read, we check is it "public". If yes ("read" of "public"), then we can read it without locking. If no, we increase "write-level", if object is "alien", make it "public", if we need to change object, change it, decrease "write-level". Of couse, when we append object to "public" collection, we chould make it "public", "write-level" is already increased so we do not make many separate locks, but when we then will access thouse object for read, we will not lock for make it "public". I don't know, how nested scopes are implemented, but i think it should be considered as a mutable object. So there is a small overhead for a single threaded application ( only for comparing 2 numbers) and in a big part of multithreaded, since we are locking only writting on _mutable_ _public_ objects. Most part of "public" objects is not accessed to write often: they are numbers, classes and mostly-read collections. And one can optimize a program by accumulating results in a "private" collection and then flush it to "public" one. Also, there may be a statement for explicit increasing "write-level" around big update of "public" object and decreasing after it. PWGIL also must be released and reacquired with every "100" instructions left,but only if "write-level=0", it conforms to current GIL semantic. I think, it must be not released with flushing decref queue, since it can happen while we are in C code. And there must be strong think about blocking IO. Mostly awful situation (at my point of view): object O is "private" for a thread A. thread B accesses O and try to mark it "public", so it locks in attempt of increasing "write-level" thread A starts to change O (it is in "write-level 0"), and in a C code it releases PWGIL (around blocking IO, for example). thread B becomes "writer", changes object to "public", becomes "reader" and starts to read O, returning thread A continue to change O , remaining in a "write-level=0". But, I think, well written C code should not attemt to make blocking IO inside of changing non-local objects (and it does not attempt at the moment, as I guess. Am I mistaken?). Or/and, when it returns and continues to change O, it must check, is it "private" or it isn't? I think, big part of checks and manipulation with GIL&PWGIL could be hidden inside of current C API, so we should not change a tons of libraries written in C. Only libraries, which create mutable objects which use notstandart containers for storing. Maybe there should be only one united SGIL for incref-decref and "write public". Summary overhead: each Py_DECREF place reference in a thread local queue (it could be small enough - about 1000, and not dinamic - just an array with counter). every object (mutable?) store thread mark (onle 4 byte, i think) every access to an object whould check - mutable is it? only if yes, 'private' is it? and only for 'mutable public/alien' object we are locking. There would no more than 20% of perfomance overhead, i think. And +50% advantage in ordinary multithreated programm on dual processor box. (Maybe +90% on 3 processor, +110 % on 4 processor, since write block will lock all readers). [View Less]

24 39

Re: [Python-Dev] Visibility scope for "for/while/if" statements
by Alexander Myodov Sept. 23, 2005

Sept. 23, 2005

Hello Josiah, >> Why the variables defined inside "for"/"while"/"if" statements >> (including loop variables for "for") are visible outside this scope? JC> if and while statements don't define variables, so they can't expose JC> them later in the scope. They don't. They just leak internal ones: i = 0 while i != 1: i += 1 j = 5 print j JC> In regards to 'for' loops, they have always done that, there is code JC> that relies on that behavior, and changing … [View More]behavior would unnecessarily JC> break code. I mentioned it below: while unconditional changing the behaviour will definitely break a lot, something like "import strict" would help those ones who rate such behaviour enough error-prone to reconsider proceeding using Python at all. JC> Further, you should clarify what you would want this mythical non-leaky JC> for loop to do in various cases. What would happen in the following JC> case? JC> i = None JC> for i in ...: JC> ... JC> print i JC> ... assuming that the loop executed more than once? Would you always JC> get 'None' printed, or would you get the content of the last variable? As for me (if my humble opinion is useful for you), the best would be to 1. assume that a loop variable is always newly-declared for this loop, 2. all variables first declared inside the loop are local to this loop. Thus, your example falls to case 1: "i" variable is newly declared for this loop. Well, we don't reuse old value of i to start the iteration from a particular place, like below? i = 5 for i in [3,4,5,6,7]: print i, More general, the variables could be assumed local only to the *same or higher* indentation level. JC> What about: JC> x = None JC> for i in ...: JC> x = f(i) JC> ... JC> print x JC> Would 'x' be kept after the loop finished executing? case 2: "x" variable is declared already. And if there is no "x = None" line, it should tell "name 'x' is not defined". Unless we are prone to funny "uncertainly defined" cases, if some variable is defined inside an "in" in the loop. JC> I would imagine in your current code you are running something like JC> this: JC> i = #some important value JC> ... #many lines of code JC> for i in ...: JC> ... JC> #you access the 'i' you bound a long time ago JC> In this case, you are making a common new programmer mistake by using JC> the same variable name for two disparate concepts. Nah, here is my error-case: I made several loops, one by one, using the "i" variable for looping. Then in the latest loop I changed the "i" name to more meaningful "imsi" name in the "for" declaration and whenever I found inside the loop. As I use "i" name *for loops exclusively*, I didn't wittingly reuse the same name for different purposes. The problem was that I missed one occurance of "i" variable inside the loop code, so it gained the same value (from the completion of previous loop) throughout all the "imsi" loop. And the interpreter didn't notice me that "I am using the undefined variable" (since it is considered defined in Python), as accustomed from other languages. That's my sorrowful story. JC> If the 'i' that was JC> initially bound was important through the lifetime of the scope, you JC> should have named it differently than the 'i' that was merely a loop JC> variable. I'm pretty happy that I didn't ever made a list comprehension with "i" variable inside the loop over "i" variable. While I wasn't aware of these side-effects, it would be even harder spottable. Now I'll look more carefully at copy-pasting list comprehensions, in every case examining all of the lower-indent variables for not to clash. JC> I will also point out that in general, not leaking/exposing/etc. such JC> variables would necessarily slow down Python. Why? I agree, this is one of obvious advantages of absence of pseudo-namespaces (wasting the resources to allocation and removal of variables, etc). Though in case of presence of pseudo-namespaces, it an easily be emulated by initializing the variable first-time on the highest-level of indentation. But for the "performance-oriented/human-friendliness" factor, Python is anyway not a rival to C and similar lowlevellers. C has pseudo-namespaces, though. JC> Python semantics seem to have been following the rule of "we are all JC> adults here". I always believed that the programming language (as any computer program) should slave to the human, rather than a human should slave to the program. JC> Generally though, Python follows a common C semantic of variable leakage. JC> C Code: JC> int i; // required in some versions of C JC> for (i=0;i<10;i++) { JC> ... JC> } "for (int i = 0; i < 10; i++)" works fine nowadays. JC> Again: test your assumptions. If your untested assumptions are wrong, JC> don't complain that the language is broken. In my case, as I explained already, it was not a assumption but an uncatched mistype, which is usually catched in other languages due to the presence of "pseudo-namespaces" (and in other languages I even accustomed to enforce the locality of temporary variables by explicitly setting the logically independent blocks of code within the curly braces). I always test the doubtful pieces before use. But *I don't dare to tell that the language is broken*. The existing behaviour has its own pros, and I also don't want a lot of code to be broken by easy behaviour change. What upsets me is the absence of alternatives (as the lack of flexibility) for the new code, especially with the fact that current behaviour mismatches with many of currently popular languages and habits learned from them. JC> Also: python-dev is a mailing list for the development /of/ Python. JC> Being that your questions as of late have been in the realm of "why does JC> or doesn't Python do this?", you should go to python-list (or the JC> equivalent comp.lang.python newsgroup) for answers to questions JC> regarding current Python behavior, and why Python did or didn't do JC> something in its past. I'm sorry for wasting the time of developers. For "for/while/if" statements, I just had an idea which (I believed) could be useful for many peoples, and could be very useful for such a human-orienter language as Python is (actually, this idea is borrowed from another language, but only partly - only in the area of ability to have "for" and "while" inside the single loop expression), so I though that the best people to estimate it should be the ones who potentially could implement this (provided they like it). For this conversation, the explained just looked to me as language-level problem (especially for list comprehensions), so I had to beg a workaroung/explanation/fix to it also from the language creators. If they don't consider it a problem and don't have a workaroung - sorry again for wasting your time, then that's indeed only my problem and I'll try to cope with it myself. I really didn't want to "claim and blame". What I asked and proposed was intended to be useful not only for myself, but - I believed - for many people, especially newcomers after the other languages. -- С уважением, Alexander mailto:maa_public@sinn.ru [View Less]

3 3

IMPORTANT: release24-maint branch is FROZEN from 2005-09-21 00:00 UTC for 2.4.2
by Anthony Baxter Sept. 23, 2005

Sept. 23, 2005

Starting in about 11 hours time, the release24-maint branch is FROZEN for the 2.4.2c1 release. The freeze will last for around a day, and then we're in a state of mostly-frozen for another week, until 2.4.2 (final). During that week, please don't check things into the branch unless you check with me first. Let's make this a nice painless release. I'll send another message once 2.4.2 is done. Thanks, Anthony -- Anthony Baxter <anthony(a)python.org> It's never too late to have a happy childhood.

6 13

Repository for python developers
by Mikhail Kashkin Sept. 23, 2005

Sept. 23, 2005

Moscow, Russia, September 23th, 2005 We are happy to spread good news about `Key Solutions' <http://keysolutions.ru/>`_ new initiative. We have created a new public repository for python developers. The ulitimate goal of this project is to unite companies and people under the umbrella of Russian-speaking-python (Zope/Plone/Zope3/Twisted/etc) open source programmers community, so that we could jointly push forward new programs and technologies in Russia. The repository powered by `… [View More]

1 0

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser Sept. 22, 2005

Sept. 22, 2005

Patch / Bug Summary ___________________ Patches : 337 open ( -6) / 2941 closed (+14) / 3278 total ( +8) Bugs : 908 open ( +0) / 5262 closed (+17) / 6170 total (+17) RFE : 194 open ( +5) / 187 closed ( +2) / 381 total ( +7) New / Reopened Patches ______________________ use LIST_APPEND opcode for X.append() method calls (2005-09-16) http://python.org/sf/1292625 opened by Neal Norwitz python.sty correction - verbatim environment (2005-09-17) http://python.… [View More]

1 0

Visibility scope for "for/while/if" statements
by Alexander Myodov Sept. 22, 2005

Sept. 22, 2005

Hello, Don't want to be importunate annoyingly asking the things probably trivial for experienced community, but need to ask it anyway, after spending about two hours trying to find well-camouflaged error caused by it. Why the variables defined inside "for"/"while"/"if" statements (including loop variables for "for") are visible outside this scope? This litters the namespace, makes things more complex and requires even more attention from the programmer than even the uneasy C. Or … [View More]

5 4