Mailman 3 January 2007 - Python-Dev

Re: [Python-Dev] The bytes type
by glyph＠divmod.com 17 Jan '07

17 Jan '07

On 06:49 pm, python(a)rcn.com wrote: >I think we should draw a line in the sand and resolve not to garbage-up Py2.6. >The whole Py3.0 project is about eliminating cruft and being free of the >bonds of backwards compatibility. Adding non-essential cruft to Py2.6 >goes against that philosophy. Emotionally charged like "cruft" and "garbage" are obscuring the issue. Let's replace them with equivalents charged in the opposite direction: "I think we should draw a line in the sand and resolve not to compatibility-up Py2.6. The whole Py3.0 project is about eliminating useful libraries and being free of the bonds of working software. Adding non-essential forward-compatibility to Py2.6 goes against that philosophy." The benefit (to me, and to many others) of 3.x over 2.x is the promise of more future maintenance, not the lack of cruft. In fact, if I made a list of my current top ten problems with Python, "cruft" wouldn't even make it in. There is lots of useful software that will not work in the 3.0 series, and without forward compatibility there is no way to get there from here. As Guido said, if 3.0 is going to break compatibility, that burdens the 2.x series with the need to provide transitional functionality. The upgrade path needs to be available in one version or the other, or 2.x needs to be maintained forever. You can't have it both ways.

7 10

Re: [Python-Dev] Proposed 3.0 compatiblity module
by Phillip J. Eby 17 Jan '07

17 Jan '07

At 03:47 PM 1/16/2007 -0800, Guido van Rossum wrote: >Oh, as long as we're talking adding features to 2.6 I'm fine. I >thought you were proposing changes to the plans for 3.0, as in your >proposal" regarding the dict view API. Nah, that was a side tangent, combined with a partial lack of understanding of how the view thing was supposed to work. Having to wrap 'em with list() calls is annoying but doable. I was just throwing out an idea I didn't realize had already been shot down.

1 0

Proposed 3.0 compatiblity module
by Raymond Hettinger 17 Jan '07

17 Jan '07

ISTM, we can decide to limit 3.0's impact to a single compatibility module, essentially as an alternate set of builtins. import __new_builtins__ for name in 'bytes dict coerce'.split(): # list any new features you want to support setattr(__builtins__, name, getattr(__new_builtins__, name)) The bytes type would just be the new type. The dict would have keys=iterkeys, no has_key, etc. The coerce function would work but have a DeprecationWarning or somesuch. If you want future division, we already have a mechanism for that. If you want to eliminate backticks, just grep for them. If you don't care about 3.0 compatibilty, just don't import the module. Outside of a compatibility module, we can decide to leave Py2.6 unmolested by Py3.0, so I can stop being cranky. Raymond

4 6

Re: [Python-Dev] Proposed 3.0 compatiblity module
by Phillip J. Eby 17 Jan '07

17 Jan '07

At 03:18 PM 1/16/2007 -0800, Guido van Rossum wrote: >On 1/16/07, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > The idea here being that, once 2.6 is widely-enough deployed that it can be > > assumed as a base for one's users, you can simply run the translator once, > > do any cleanup, and then have 3.0-clean code that also still runs for your > > installed base. > > > > That way, there's no chasm to leap; just a code cleanup. > >I understand; I would rather have that too, everything else being the >same. But everything else wouldn't be the same -- it would place many >more restrictions on 3.0, and the common subset would still be much >smaller. For me personally, the weight of the added restrictions to >3.0 is the killer. I don't understand; how would adding features to 2.6 restrict what you could add to 3.0?

2 1

Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater
by "Martin v. Löwis" 17 Jan '07

17 Jan '07

glyph(a)divmod.com schrieb: > Also, the exact strategy I suggested could be implemented in various > ways that might be efficient. Here are a few ways it might be made more > efficient than the straw many of one extra dict lookup per call to > keys() et. al.: I'm not saying that an efficient implementation is impossible, but I still don't see it. The ones you suggest don't work that well: > * Don't make it a dictionary key; make it an attribute of the module at > the C level. Unfortunately, in a function, there is no reference to the module it came from. There is __module__ (on the function, not on the code object), but that is a string. > * Annotate the code object in some way that makes code compiled in that > scope set a global variable to enable the compatibility; protect it with > the GIL. This I don't quite understand: what has the GIL to do with that? Are you suggesting to hold the GIL while the entire function is running? That would restrict concurrency even more than the GIL already restricts it. Also, what about nested functions spanning multiple modules (with different settings)? > * These methods are already function pointers in a data structure in C; > actually have a py->py3k mode switch which swaps out the C function > pointers to the appropriate implementation for the calling code. That doesn't work: The items method is per dictionary (or actually per type, i.e. for the dict type). The same dictionary, when accessed from different modules, should have .items() behave differently. Wrt. that property: # a.py from __future__ import items_is_iter def f(d): return d.items # b.oy import a d = {} print a.f(d)() What should that print? I think I take back my earlier suggestion that I can imagine an inefficient implementation. I cannot imagine any reasonable behaviour of a per-module change of methods on dictionary objects. > Do I just suffer from having an overactive imagination? Are all of > these implementation strategies impossible for some reason, and there > are no others? Yes, and yes. Regards, Martin

3 3

Re: [Python-Dev] The bytes type
by Phillip J. Eby 17 Jan '07

17 Jan '07

At 09:50 AM 1/16/2007 -0800, Guido van Rossum wrote: >Actually it's very easy to write code using keys(), items() and >values() that works as well in 2.2 as it works in 3.0: never use the >iter* variants, and don't sweat the performance costs of creating a >list so much. If you can't afford to create a list, iterate over the >dict itself, which will give you the keys in both versions. Actually, the main reason I use items() or keys() is because I want to mutate the dictionary, or because something else is going to mutate it while I'm looping over it. But yeah, that discussion should be on Py3K. >>Without having at least *some* 2.x version that can run 3.x code, I think >>there is little chance of 3.0 becoming viable. You've been comparing this >>to Zope 2/Zope 3, but in that world there is something called "Five" that >>lets you do Zope 3 things inside of Zope 2, so you have some chance of >>porting your code in an incremental fashion, without having to leap >>everything over in one go. > >I'm not aware of compromises to 3.0's architecture that were made in >order to ease the transition though -- all of the burden was placed on >Zope 2.x (though perhaps some optional compatibility packages were >also added to Zope 3.0?). Right, which is why I don't think we should worry about 2.5 as much as 2.6. >>The analogy isn't perfect, because we are not so much trying to provide >>backward compatibility in "Excel" as to add *forward* compatibility to >>"123", but you get the idea. > >Such analogies are hard to get right; in this case he's talking about >file interchange formats. Perhaps the closest analogy is pickling. Actually, I meant the analogy for *code*. If you can't write code that runs on some 2.x + 3.x combination, you have no way to cross the chasm. If you're writing "spreadsheets for Excel" (code that runs on 3.x), you need to be able to "share them with 1-2-3 users" (run that code on 2.x).

1 0

Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater
by glyph＠divmod.com 16 Jan '07

16 Jan '07

On 07:42 pm, thomas(a)python.org wrote: >On 1/10/07, Raymond Hettinger <raymond.hettinger(a)verizon.net> wrote: >> >><"Anthony Baxter"> >> > Comments? What else should get warnings? >> >>It is my strong preference that we not go down this path. >>Instead, the 2.6 vs 3.0 difference analysis should go in an >>external lint utility. >Having Python 2.6 optionally warn for >3.0-compatibility is a lot easier for the average developer than having a >separate tool or a separately compiled Python. I could not possibly agree more. Given the highly dynamic nature of Python, such a tool will *at best* catch only the most egregious uses of deprecated features. Backticks are easy enough to find, but the *only* way that I can reasonably imagine migrating a body of code like Twisted (or any non-trivial Python library) would be having a way to examine the warning output of its test suite. I am suuuuper +1 on the warnings for 2.6, as well as forward-compatibility at some point in the 2.x series for new syntax. Without the ability to bridge 2.x->3.0 during some interim period, I can say for sure that Twisted _will not_ migrate to 3.0, ever. We are really a small project and just don't have the manpower to maintain two overlapping but mutually incompatible codebases. I've been assuming for some time that the only hope for Py3k compatibility within Twisted would be using PyPy as a translation layer. With the addition of runtime compatibility warnings, it might be feasible that we could run on the "bare metal" (ha ha) of Python3's VM.

22 34

Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater
by glyph＠divmod.com 16 Jan '07

16 Jan '07

On 08:19 am, martin(a)v.loewis.de wrote: >Georg Brandl schrieb: >>> If Python 3.0 was simply a release which removed deprecated features, >>> there would clearly be no issue. I would update my code in advance of >>> the 3.0 release to not use any of those features being removed, and >>> I'm all set. But that's not what I'm hearing. Python 3 is both adding >>> new ways to do things, and removing the older way, in the same >>> version, with no overlap. This makes me very anxious. >> >> It has always been planned that in those cases that allow it, the new way to do >> it will be introduced in a 2.x release too, and the old way removed only in 3.x. > >What does that mean for the example James gave: if dict.items is going >to be an iterator in 3.0, what 2.x version can make it return an >iterator, when it currently returns a list? > >There simply can't be a 2.x version that *introduces* the new way, as it >is not merely a new API, but a changed API. The API isn't changed. It's just dependent on its execution context in a confusing way. That difference right now is being reflected as "2.x VM" vs. "3.0 VM" but there are other ways to reflect it more explicitly. It would certainly be possible to have: from __future__ import items_is_iter be the same as: __py3k_compat_items_is_iter__ = True and have the 2.x series' items() method check the globals() of the calling scope to identify the return value of items() in that particular context. If the actual data structures of module dictionaries and stack objects are too expensive there are other, similar things that could be done at the C level. This implementation strategy is just the obvious thing that occurred to me after maybe 2 minutes of consideration. I'm sure someone more familiar with the internals of Python could come up with something better.

3 3

buglet in long("123\0", 10)
by Guido van Rossum 15 Jan '07

15 Jan '07

What's wrong with this session? :-) Python 2.6a0 (trunk:53416, Jan 13 2007, 15:24:17) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> int('123\0') Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: null byte in argument for int() >>> int('123\0', 10) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: invalid literal for int() with base 10: '123\x00' >>> long('123\0') Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: null byte in argument for long() >>> long('123\0', 10) 123L >>> -- --Guido van Rossum (home page: http://www.python.org/~guido/)

5 5

Rewrite of import in Python source (sans docs) is complete
by Brett Cannon 15 Jan '07

15 Jan '07

Finally, after a few months worth of work, I have finally gotten far enough in my import rewrite that I am willing to stick my neck out and say it is semantically complete! You can find it in the sandbox under import_in_py. So, details of this implementation. I implemented PEP 302 importers/loaders for built-in, frozen, extension, .py, and .pyc files along with rewriting the steps __import__ goes through to do an import. I also developed an API for .py/.pyc file handling so that there is a generic filesystem importer/loader and a separate handler for .py/.pyc files. This should allow for (relatively) easy selective overriding of just how .py/.pyc files are stored (e.g., introducing a database backend) or how variants on .py/.pyc files are handled (e.g., Quixote's .ptl format). This code has extensive tests and so I am fairly confident that it does what is expected of an import rewrite. There are actually more lines in the test file than the implementation. There is also a mock implementation used for testing. Was interesting doing this in such a test-driven, XP style of only coding what I needed. I have run this code through the entire regression test suite and that is where you find out subtle differences between this implementation and the built-in import (you can see for yourself with the regrtest.sh shell script). First test_pkg will fail because currently the new import adds a __loader__ attribute on all modules (that might change for security reasons) and test_pkg is an old, stdout comparing test. Second, test_runpy fails because I have not implemented get_code on the filesystem loader which is required by runpy. Both are shallow issues that can be dealt with. Third, and the hardest difference to deal with, is that you will get some warnings that print out that you normally don't see. This is because warnings.warn and its stack_level argument don't have the effect people are used to when importing a deprecated module. Before you could set stack_level to 2 and it would look like it came from the import statement. But now, with import written in Python and thus on the call stack compared to being in C and thus not showing up, two levels back is still in the import code. I really don't know how this should be dealt with short of saying that the rule of thumb with 2 stack levels back for a warning does not work when done at the import level. It is not blazing fast at the moment. Some things, like the built-in and frozen importers/loaders could be rewritten in C without huge issue. I am also sure I have made some stupid design decisions at various points in the code. But there is benchmarking code in the sandbox called importbench and it showed up a 10x speed slowdown on a Linux box I was using in mid to late December when doing a fresh import of certain types (I don't remember exactly which kind off the top of my head). Because of this current slowness I don't know if people want to rush into trying to make this the default import implementation quite yet or if this is not too big of a thing since the common case of pulling out of sys.modules is not that much slower. I know I am currently not planning on devoting the time to bootstrap it in as I have my security work to finish first along with other Python stuff that seems more pressing to me. And since (I think) I don't need to bootstrap it in order to finish my security work I can't justify spending work time on it. But I can rearrange priorities if people really want to pursue this (especially if I can get some help with it). As with the module's name, it is currently named 'importer', but that is bad since it conflicts with the idea of importers from PEP 302. I was thinking importlib, but I wanted to wait and see what other people thought. Don't know if you guys are okay with me checking this in without having it vetted by the community first like we prefer all new modules to do. I have not done the LaTeX docs yet. I think that is all of the details that I can think of. I am still working towards implementing the security needed so that an application that embeds Python can execute arbitrary code securely. Giving a talk at PyCon on the topic for anyone interested. Special thanks needs to go to Paul Moore who I talked to through most of the design of the code. Nick Coghlan also provided some handy feedback. And Neal Norwitz for bugging about wanting something like this done. Plus thanks to everyone who has shown support. -Brett

2 2