Mailman 3 June 2010 - Python-Dev

Re: [Python-Dev] email package status in 3.X
by P.J. Eby June 25, 2010

June 25, 2010

At 10:57 AM 6/20/2010 -0700, Guido van Rossum wrote: >The problem comes exactly where you find it: when *porting* existing >code that uses aforementioned ways to alleviate the pain, you find >that the hacks no longer work and a properly layered design is needed >that clearly distinguishes between which variables contain bytes and >which text. Actually, I would say that it's more that (in the network protocol case) we *have* bytes, some of which we would like to *treat* as text,… [View More] yet do not wish to constantly convert back and forth to full-blown unicode -- especially since the protocols themselves designate ASCII or latin-1 at the transport layer (sometimes with odder encodings above, but these already have to be explicitly dealt with by existing code). While reading over this thread, I'm wondering whether at least my (WSGI-related) problems in this area would be solved by the availability of a type (say "bstr") that was simply a wrapper providing string-like behavior over an underlying bytes, byte array, or memoryview, that would produce objects of compatible type when combined with strings (by encoding them to match). Then, I could wrap bytes with it to pass them to string operations, and then feed them back into everything else. The bstr type ideally would be directly compatible with bytes I/O, or at least have a .bytes attribute that would be. It seems like that would reduce WSGI porting issues quite a bit, since it would mostly consist of throwing extra bstr() calls in where things are breaking, and maybe grabbing the .bytes attribute for I/O. This approach would still be explicit as to what types you're working with, but would not require O(n) *conversions* at every interaction boundary. It would be limited, of course, to single-byte encodings with all characters (0-255) valid. OTOH, maybe there should just be a bytestrings module with bytestrings.ascii and bytestrings.latin1, and between the two that should cover the network protocol needs quite well. Actually, if the Python 3 str() constructor could do O(1) conversion for the latin-1 case (i.e., just wrapped the underlying bytes), I would just put, "bstr = lambda x: str(x,'latin-1')" at the top of my programs and have roughly the same effect. This idea is still a bit half-baked, but a more baked version might be just the ticket for porting stuff that used str to work with bytes in 2.x, if only because writing, e.g.: newurl = bstr(urljoin(bstr(base), 'subdir')) seems so much saner than writing *this* everywhere: newurl = str(urljoin(str(base, 'latin-1'), 'subdir'), 'latin-1') It is perhaps a bit late to propose this idea, since ideally we would also want to use it in 2.x to aid porting. But I'm curious if any other people here experiencing byte/unicode woes in relation to network protocols would find this a solution to their chief frustration. (i.e., that the stdlib often insists now on strings, where effectively bytes were usable before, and thus one must do conversions both coming and going.) [View Less]

23 85

Summary of Python tracker Issues
by Python tracker June 25, 2010

June 25, 2010

ACTIVITY SUMMARY (2010-06-18 - 2010-06-25) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2795 open (+38) / 18104 closed (+14) / 20899 total (+52) Open issues with patches: 1130 Average duration of open issues: 712 days. Median duration of open issues: 503 days. Open Issues Breakdown open 2765 (+38) languishing 13 ( +0) pending 16 ( +0) Issues Created Or … [View More]

1 0

Re: [Python-Dev] bytes / unicode
by P.J. Eby June 24, 2010

June 24, 2010

At 10:29 AM 6/21/2010 -0700, Guido van Rossum wrote: >Perhaps there are more situations where a polymorphic API would be >helpful. Such APIs are not always so easy to implement, because they >have to be careful with literals or other constants (and even more so >mutable state) used internally -- but it can be done, and there are >plenty of examples in the stdlib. What if we could use the time machine to make the APIs that *were* polymorphic, regain their previously-polymorphic … [View More]status, without needing to actually *change* any of the code of those functions? That's what Barry's ebytes proposal would do, with appropriate coercion rules. Passing ebytes into such a function would yield back ebytes, even if the function used strings internally, as long as those strings could be encoded back to bytes using the ebytes' encoding. (Which would normally be the case, since stdlib constants are almost always ASCII, and the main use cases for ebytes would involve ascii-extended encodings.) >I'm stil unclear on exactly what bstr is supposed to be, but it sounds >a bit like one of the rejected proposals for having a single >(Unicode-capable) str type that is implemented using different width >encodings (Latin-1, UCS-2, UCS-4) underneath. Not quite - as modified by Barry's proposal (which I like better than mine) it'd be an object that just combines bytes with an attribute indicating the underlying encoding. When it interacts with strings, the strings are *encoded* to bytes, rather than upgrading the bytes to text. This is actually a big advantage for error-detection in any application where you're working with data that *must* be encodable in a specific encoding for output, as it allows you to catch errors much *earlier* than you would if you only did the encoding at your output boundary. Anyway, this would not be the normal bytes type or string type; it's "bytes with an encoding". It's also more general than Unicode, in the sense that it allows you to work with character sets that don't really *have* a proper Unicode mapping. One issue I remember from my "enterprise" days is some of the Asian-language developers at NTT/Verio explaining to me that unicode doesn't actually solve certain issues -- that there are use cases where you really *do* need "bytes plus encoding" in order to properly express something. Unfortunately, I never quite wrapped my head around the idea, I just remember it had something to do with the fact that Unicode has single character codes that mean different things in different languages, such that you were actually losing information by converting to unicode, or something like that. (Or maybe the characters were expressed differently in certain encodings according to what language they came from, so you couldn't roundtrip them through unicode without losing information. I think that's probably was what it was; maybe somebody here can chime in more on that point.) Anyway, a type like this would need to have at least a bit of support from the core language, because the str type would need to be able to handle at least the __contains__ and %/.format() coercion cases, since these functions don't have __r*__ equivalents that a user-implemented type could provide... and strings don't have anything like a '__coerce__' either. If sufficient hooks existed, then an ebytes could be implemented outside the stdlib, and still used within it. [View Less]

4 3

os.getgroups() on MacOS X Was: red buildbots on 2.7
by Alexander Belopolsky June 24, 2010

June 24, 2010

On Wed, Jun 23, 2010 at 2:08 AM, Ronald Oussoren <ronaldoussoren(a)mac.com> wrote: .. >> >>> * [Ronald's proposal] results in posix.getgroups not reflecting results of posix.setgroups >>> >> >> This effectively substitutes getgrouplist called on the current user >> for getgroups. In 3.x, I believe the correct action will be to >> provide direct access to getgrouplist which is while not POSIX (yet?), >> is widely available. > &… [View More]gt; I don't mind adding getgrouplist, but that issue is seperator from this one. BTW. Appearently getgrouplist is posix > (<http://refspecs.freestandards.org/LSB_3.1.1/LSB-Core-generic/LSB-Core-gener…>), although this isn't a > requirement for being added to the posix module. > (The link you provided leads to "Linux Standard Base Core Specification," which is different from POSIX, but the distinction is not relevant for our discussion.) > > It is still my opinion that the second option is preferable for better compatibility with system tools, even if the patch > is more complicated and the library function we use can be considered to be broken. Let me try to formulate what the disagreement is. There are two different group lists that can be associated with a running process: 1) The list of current supplementary group IDs maintained by the system for each process and stored in per-process system tables; and 2) The list of the groups that include the uid under which the process is running as a member. The first list is returned by a system call getgroups and the second can be obtained using system database access functions as follows: pw = getpwuid(getuid()) getgrouplist(pw->pw_name, ..) The first list can be modified by privileged processes using setgroups system call, while the second changes when system databases change. The problem that _DARWIN_C_SOURCE introduces is that it replaces system getgroups with a database query effectively making the true process' list of supplementary group IDs inaccessible to programs. See source code at <http://www.opensource.apple.com/source/Libc/Libc-594.1.4/sys/getgroups.c>. The problem is complicated by the fact that OSX true getgroups call appears to truncate the list of groups to NGROUPS_MAX=16. Note, however that it is not clear whether the system call truncates the list or the underlying process tables are limited to 16 entries and additional groups are ignored when the process is created. In my view, getgroups and getgrouplist are two fundamentally different operations and both should be provided by the os module. Redefining os.getgroups to invoke getgrouplist instead of system getgroups on one particular platform to work around that platform's system call limitation is not right. [View Less]

6 7

Use of cgi.escape can lead to XSS vulnerabilities
by Craig Younkins June 24, 2010

June 24, 2010

Hello, The method in question: http://docs.python.org/library/cgi.html#cgi.escape http://svn.python.org/view/python/tags/r265/Lib/cgi.py?view=markup # at the bottom "Convert the characters '&', '<' and '>' in string s to HTML-safe sequences. Use this if you need to display text that might contain such characters in HTML. If the optional flag quote is true, the quotation mark character ('"') is also translated; this helps for inclusion in an HTML attribute value, as in <A HREF=".… [View More]

3 3

swig/python and intel's threadedbuildginblocks
by tullarisc June 23, 2010

June 23, 2010

Hi, I've compiled intel's OSS threadedbuidlingblocks library on OpenBSD and put everything in some swig interfaces. Here you go: http://tullarisc.xtreemhost.com/swig.ttb.tgz Love, tullarisc. -- View this message in context: http://old.nabble.com/swig-python-and-intel%27s-threadedbuildginblocks-tp28… Sent from the Python - python-dev mailing list archive at Nabble.com.

1 0

Re: [Python-Dev] email package status in 3.X
by lutz＠rmi.net June 23, 2010

June 23, 2010

> Python 3.0 was *declared* to be an experimental release, and by most > standards 3.1 (in terms of the core language and functionality) was a > solid release. > > Any reasonable expectation about Python 3 adoption predicted that it > would take years, and would include going through a phase of difficulty > and disappointment... Declaring something to be a turd doesn't change the fact that it's a turd. I have a feeling that most people outside this list would have … [View More]

18 35

red buildbots on 2.7
by Bill Janssen June 23, 2010

June 23, 2010

Considering that we've just released 2.7rc2, there are an awful lot of red buildbots for 2.7. In fact, I don't remember having seen a green buildbot for OS X and 2.7. Shouldn't these be fixed? On OS X Leopard, I'm seeing failures in test_py3kwarn, test_urllib2_localnet, test_uuid. On OS X Tiger, I'm seeing failures in test_pep277, test_py3kwarn, test_ttk_guionly, and test_urllib2_localnet. We don't have a buildbot running Snow Leopard, apparently. Bill

14 42

UserDict in 2.7
by Raymond Hettinger June 23, 2010

June 23, 2010

There's an entry in whatsnew for 2.7 to the effect of "The UserDict class is now a new-style class". I had thought there was a conscious decision to not change any existing classes from old-style to new-style. IIRC, Martin had championed this idea and had rejected all of proposals to make existing classes inherit from object. Raymond

6 10

Re: [Python-Dev] email package status in 3.X
by P.J. Eby June 23, 2010

June 23, 2010

At 10:20 PM 6/21/2010 +1000, Nick Coghlan wrote: >For the idea of avoiding excess copying of bytes through multiple >encoding/decoding calls... isn't that meant to be handled at an >architectural level (i.e. decode once on the way in, encode once on >the way out)? Optimising the single-byte codec case by minimising data >copying (possibly through creative use of PEP 3118) may be something >that we want to look at eventually, but it strikes me as something of >a premature … [View More]optimisation at this point in time (i.e. the old adage >"first get it working, then get it working fast"). The issue is, I'd like to have an idempotent incantation that I can use to make the inputs and outputs to stdlib functions behave in a type-safe manner with respect to bytes, in cases where bytes are really what I want operated on. Note too that this is an argument for symmetry in wrapping the inputs and outputs, so that the code doesn't have to "know" what it's dealing with! After all, right now, if a stdlib function might return bytes or unicode depending on runtime conditions, I can't even hardcode an .encode() call -- it would fail if the return type is a bytes. This basically goes against the "tell, don't ask" pattern, and the Pythonically idempotent approach. That is, Python builtins normally return you back the same thing if it's already what you want - int(someInt)-> someInt, iter(someIter)->someIter, etc. Since this incantation may need to be used often, and in places that are not known to me in advance, I would like it to not impose new overhead in unexpected places. (i.e., the usual argument brought against making changes to the 'list' type that would change certain operations from O(1) to O(log something)). It's more about predictability, and having One *Obvious* Way To Do It, as opposed to "several ways, which you need to think carefully about and restructure your entire architecture around if necessary". One obvious way means I can focus on the mechanical effort of porting *first*, without having to think. So, the performance issue isn't really about performance *per se*, so much as about the "mental UI" of the language. You could just as easily lie and tell me that your bstr implementation is O(1), and I would probably be happy and never notice, because the issue was never really about performance as such, but about having to *think* about it. (i.e., breaking flow.) Really, the entire issue can presumably be dealt with by some series of incantations - it's just code after all. But having to sit and think about *every* situation where I'm dealing with bytes/unicode distinctions seems like a torture compared to being able to say, "okay, so when dealing with this sort of API and this sort of data, this is the One Obvious Way to do the conversions." It's One Obvious Way that I want, but some people seem to be arguing that the One Obvious Way is to Think Carefully About It Every Time -- and that seems to violate the "Obvious" part, IMO. ;-) [View Less]

9 17