Mailman 3 March 2008 - Python-Dev

2.6 and 3.0 tasks
by Guido van Rossum March 22, 2008

March 22, 2008

Moving this to a new subject to keep the discussion of tasks and the discussion of task tracking tools separate. On Sun, Mar 16, 2008 at 9:42 AM, Christian Heimes <lists(a)cheimes.de> wrote: > I did a quick brainstorming with me, myself and I. I came up with a list > of (IMHO) important tasks. > > * Stabilize the C API of Python 3.0. I like to rename several prefixes > to reduce the confusing: PyBytes -> PyByteArray, +1 (also +1 to backporting this to 2.6) > PyString -> PyBytes ... -1. This will make merging code from 2.6 harder, and causes more work for porting C extensions. > * Backport the new buffer protocol to 2.6. I spoke to Travis yesterday > and he said he is trying to get it done during the PyCon sprint. Maybe > somebody can assist him? Does he need assistance? > When the new buffer protocol is available in 2.6 we can start back > porting bytesarray and the new IO framework. Are these really so closely tied that they have to wait before they can be backported? > * Replace Windows API calls with wide versions to support unicode for > file names, environment etc. +1. This should be broken into separate tasks for each API. > * Get the stdlib reorg done +1. But again, I think this is many small tasks. > and add the fixers to 2to3 +1. I think quite a few changes have not had a fixer added. Again, I think we should maintain a specific list of needed fixers; fixers can easily be developed independently. > * Speed up 2to3. I suggested a GSoC task to improve and speed up 2to3 > but it may be too late when we plan to ship out 3.0 in August. While I know that some people are expecting to use a development model that invokes 2to3 very frequently, I think this is at best a nice-to-have. (I also don't see how it could be done, but maybe I'm blind for the obvious, as the original author.) I have some other tasks to add: - getargs.c (Georg posted about this; his post actually inspired my post.) - Tweak structseq to be more like namedtuple. Maybe they could have a common ABC. I also think we should get rid of the concept of "hidden" fields (that are accessible by name but not through the tuple API). - Devise a way to handle str instances pickled in 2.6 and unpickled in 3.0, and also bytes instances pickled in 3.0 and unpickled in 2.6. - Someone should go over https://spreadsheets.google.com/ccc?key=pCKY4oaXnT81FrGo3ShGHGg and extract more tasks. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

9 22

2.6 and 3.0 project management
by Guido van Rossum March 22, 2008

March 22, 2008

Python 3.0 and 2.6 are coming along really nice. I am optimistic that we can make the projected August date for the final releases of 2.6 and 3.0. As you may remember, Barry (the new release manager for both) suggested that we synchronize releases of 2.6 and 3.0. Not only could this potentially save the release manager and his assistants some time, doing the final releases together sends a clear signal to the community that both versions will receive equal support. However, looking at the calendar, I think we need to do a little more planning and management than we've typically done for Python releases. A final release in August means that we should plan to release a first beta in June and a second one in July. That means we have time for only two more alpha releases (April and May). I'm thinking of 1-2 release candidates 1-2 weeks ahead of the final release. Barry can make up a more detailed timetable. I'm fine with some slippage (especially if planned ahead) of individual releases due to availability of release personnel; but I don't want to be held up by features or bugs unless they are of absolutely dramatic show-stopping nature. In order to make such a tight release schedule we should try to come up with a list of tasks that need to be done, and prioritize them. This should include documentation, and supporting tools like 2to3. It should include features, backports of features, cleanup, bugs, and whatever else needs to be done (e.g. bugbot maintenance). In the past we've used shared spreadsheets in Google for this purpose, but seeing that these haven't been updated in ages, I'm skeptical that they are a sufficient tool. In my day job at Google we've started to do all task management for our project in the bug tracker (but that tracker has some features that make it particularly easy). Does anyone have a suggestion for an online open shared task management system that we cold adopt? Or should we bite the bullet and put everything in the bug tracker? Other suggestions? Anything's better than just email... PS. I realize that I've been rather absent from the day to day activities in the Python 3000 world lately. This is a temporary condition due to an important impending launch in my day job; I expect to have much more time for Python again starting in April. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

12 26

Proposal: Slightly Changed Semantics for from-Import
by Armin Ronacher March 22, 2008

March 22, 2008

Hi all, I propose a small change on the "from"-import behavior. Currently there are two ways to import a module name foo from bar. Namely "import foo.bar as bar" and "from foo import bar". The main problem with the second is that it will not work in every situation. Modules are registered with their names in sys.modules before the import but become and attribute of their parent after. This leads to the surprising behavior that "from foo.bar import baz" works but "from foo import bar" not until the foo.bar module finished setting up. This behavior is especially surprising if you have circular dependencies and you notice that your imports work properly if you import a specific module before another one and will fail if you import it afterwards although it doesn't seem like that should make any difference. My proposal is that "from foo import bar" returns sys.modules["foo.bar"] if this one exists and the foo.bar attribute lookup failed. If a test case is needed I can attach one later on. Regards, Armin

2 1

Re: [Python-Dev] [Distutils] How we can get rid of eggs for 2.6 and beyond
by Phillip J. Eby March 22, 2008

March 22, 2008

At 02:14 PM 3/22/2008 +0000, Paul Moore wrote: >For the system Python, I need: >- a single way to list what's installed (including version) >- a single way to uninstall items as needed >- a way (or more than one) to install 3rd party software *which ties >into the above* Right, and the PEP effort is devoted to having one way to store the information required, to tie these things together. If there is a standard way to store that info on your system, then it doesn't matter how many tools there are, you still have your "one way" to list what's installed or to uninstall things, because you just pick the one lister and/or uninstaller whose UI you prefer.

1 0

Testing documentation snippets
by Georg Brandl March 22, 2008

March 22, 2008

Hi, the newest version of Sphinx supports testing doctest (and other) snippets in the documentation. Since we have many examples in the docs that may get out of date, I think this is a valuable thing to have. I've started making the doctests runnable with Sphinx in three documents; the functional howto, the RE and the Decimal docs. If you want to know how to run the doctests, it should be as easy as $ cd Doc $ make doctest PYTHON=../python on UNIX. The PYTHON=../python is needed to really test against the tree version of Python, not the system-wide installed one. If you want to run only tests in one source file, run $ make doctest PYTHON=../python SOURCES=library/decimal.rst If there is further interest in this, I'll explain the details how to "activate" examples to work with "make doctest". Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

1 0

Python source code on Bazaar vcs
by Matthieu Brucher March 22, 2008

March 22, 2008

Sorry for the double post, Jeroen :| ---------- Forwarded message ---------- From: Matthieu Brucher <matthieu.brucher(a)gmail.com> Date: 21 mars 2008 10:03 Subject: Re: [Python-Dev] Python source code on Bazaar vcs To: Jeroen Ruigrok van der Werven <asmodai(a)in-nomine.org> 2008/3/21, Jeroen Ruigrok van der Werven <asmodai(a)in-nomine.org>: > > (To provide counterweight.) > > > -On [20080320 20:44], Barry Warsaw (barry(a)python.org) wrote: > >We have not made a decision to move to Bazaar officially, nor have we > made > >a decision to even move off of Subversion. > > > Good, because between this now and pytz the other 63 projects I follow use > Subversion or Mercurial. > Bazaar seems to be mostly limited to Ubuntu users and stuff Canonical > does, > so the choice for a Bazaar setup next to Subversion strikes me a bit as > odd. > Mercurial is also Python, distributed and with a, as far as I (can) track > things, bigger market share than Bazaar. > Hi, This is not quite true. One of the main bzr developers is a Windows guy, so the user base is/will be large, far larger than only Ubuntu. A lot of projects switched to bzr because of this. Honestly, I think that Hg and bzr fill the same need, but bzr developement at the moment is fantastic. For a still young product it is a good think (Hg is older). One additional good think is that there is a Sourceforge-like site that uses bzr ; launchpad.net. I don't know of something equivalent for Hg. Just my two (euro) cents ;) Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher

2 1

Primer on distributed revision control?
by skip＠pobox.com March 22, 2008

March 22, 2008

With all these distributed revision control systems now available (bzr, hg, darcs, svk, many more), I find I need an introduction to the concepts and advantages of repository distribution. It seems to me that it has the potential for leading to anarchy, though I can see how some things would be improved (working offline, maintaining local patches). It's not obvious how I push changes back upstream. Can someone point me to some useful content (web pages or books) which will help me wrap my brain around the ideas? Maybe a compare/contrast of the major players? Thanks, Skip

6 5

httplib &c. timeouts and global state
by John J Lee March 22, 2008

March 22, 2008

http://python.org/sf/2451 """ The new timeout support in 2.6 makes use of new function socket.create_connection(). socket.create_connection() provides no way to disable timeouts, other than by relying on socket.getdefaulttimeout() returning None. This is unfortunate, because it was the purpose of the new timeout support to allow control of timeouts without reliance on global state. setdefaultsocket.create_connection() should always call sock.settimeout() with the timeout passed to create_connection(), unless a special non-None value is passed indicating that the global default is to be used. Specific modules may then use that special non-None value where required, to preserve backwards-compatibility. """ Facundo doesn't seem to agree. If I understand him correctly, he objects to the use of a special value other than None as an argument value. I think avoiding this minor complication is secondary to avoiding gratuitously imposing global state on users where it's not needed. Much existing code uses .setdefaulttimeout(), because legacy code does not expose the socket timeout. Of course, not all of that code can be changed immediately (otherwise, why did we wait so long before Facundo did his valuable work on this?). Of course, very often, code that calls socket.setdefaulttimeout(some_non_none_value) wants a default timeout for everything. But that isn't *always* the case, and standard library modules that expose the timeout feature should not make that assumption. For example, an application might only want to set a timeout for some non-essential network operation. As another example, useful but poorly-written library code might call .setdefaulttimeout(). Also, ISTM there's a big difference between nobody having contributed the time to implement the feature on the one hand, and deliberately imposing the global socket timeout state on users on the other! Facundo indicates in the tracker discussion for 2451 that a decision was made here about this, but doesn't recall exactly which post, and directed me here. What I found in the archive is this thread (sorry for the non-python.org URL): http://www.gossamer-threads.com/lists/python/dev/555039?do=post_view_thread… In that discussion, this issue is raised, but I don't see any resolution that answers the objection made in issue 2451. Anyway, apologies in advance if there was a decision here that takes account of the above objection. I do want to thank Facundo for adding the timeout support to modules such as httplib. But I also want to loudly complain about this detail :-) John

1 0

Adventures with x64, VS7 and VS8 on Windows
by Mark Hammond March 22, 2008

March 22, 2008

Hi all, I hope the cross-post is appropriate. I've started playing with getting the pywin32 extensions building under the AMD64 architecture. I started building with Visual Studio 8 (it was what I had handy) and I struck a few issues relating to the compiler version that I thought worth sharing. * In trying to build x64 from a 32bit VS7 (ie, cross-compiling via the PCBuild directory), the python.exe project fails with: pythoncore fatal error LNK1112: module machine type 'X86' conflicts with target machine type 'AMD64' is this a known issue, or am I doing something wrong? * The PCBuild8 project files appear to work without modification (I only tried native compilation here though, not a cross-compile) - however, unlike the PCBuild directory, they place all binaries in a 'PCBuild8/x64' directory. While this means that its possible to build for multiple architectures from the same source tree, it makes life harder for tools like 'distutils' - eg, distutils already knows about the 'PCBuild' directory, but it knows nothing about either PCBuild8 or PCBuild8/x64. A number of other build processes also know to look inside a PCBuild directory (eg, Mozilla), so instead of formalizing PCBuild8, I think we should merge PCBuild8 into PCBuild. This could mean PCBuild/vs7 and PCBuild/vs8 directories with the "project" files, but binaries would still be generated in the 'PCBuild' (or PCBuild/x64) directory. This would mean the same tree isn't capable of hosting 2 builds from different VS compilers, but I think that is reasonable (if it's a problem, just have multiple source directories). I understand that PCBuild8 is not "official", but in the assumption that future versions of Python will use a compiler later than VS7, it makes sense to me to clean this up now - what are others opinions on this? * Re the x64 directory used by the PCBuild8 process. IMO, it makes sense to generate x64 binaries to their own directory - my expectation is that cross-compiling between platforms is a reasonable use-case, and we should support multiple achitectures for the same compiler version. This would mean formalizing the x64 directory in both 'PCBuild' and distutils, and leaving other external build processes to update as they support x64 builds. Does this make sense? Would this fatally break other scripts used for packaging (eg, the MSI framework)? * Wide characters in VS8: PC/pyconfig.h defines PY_UNICODE_TYPE as 'unsigned short', which corresponds with both 'WCHAR' and 'wchar' in previous compiler versions. VS8 defines this as wchar_t, which I'm struggling to find a formal definition for beyond being 2 bytes. My problem is that code which assumes a 'Py_UNICODE *' could be used in place of a 'WCHAR *' now fails. I believe the intent on Windows has always been "PyUNICODE == 'native unicode'" - should PC/pyconfig.h reflect this (ie, should pyconfig.h grow a version specific definition of PyUNICODE as wchar_t)? * Finally, as something from left-field which may well take 12 months or more to pull off - but would there be any interest to moving the Windows build process to a cygwin environment based on the existing autoconf scripts? I know a couple of projects are doing this successfully, including Mozilla, so it has precendent. It does impose a greater burden on people trying to build on Windows, but I'd suggest that in recent times, many people who are likely to want to build Python on Windows are already likely to have a cygwin environment. Simpler mingw builds and nuking MSVC specific build stuff are among the advantages this would bring. It is not worth adding this as "yet another windows build option" - so IMO it is only worth progressing with if it became the "blessed" build process for windows - if there is support for this, I'll work on it as the opportunity presents itself... I'm (obviously) only suggesting we do this on the trunk and am happy to make all agreed changes - but I welcome all suggestions or critisisms of this endeavour... Cheers, Mark

12 57

Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby March 22, 2008

March 22, 2008

At 03:43 PM 3/18/2008 -0500, Guido van Rossum wrote: >Only very few people would care about writing a setup >script that works with this bootstrap module; basically only package >manager implementers. That's true today, sure, but as soon as it is widely available, others are sure to want to use it too. I just want a bright-line distinction between what is and isn't bootstrappable, rather than a murky region of "maybe, if you're not doing anything too complicated". >There seems to be a misunderstanding about what I am proposing we do >instead. The boostrap installer should only be powerful enough to >allow it to be used to install a real package manager like setuptools. Which is why PEP 365 proposed only downloading an archive to a cache directory, and optionally running something from it. It explicitly disavows "installation" of anything, since the downloaded archive wouldn't have been added to sys.path except for the duration of the bootstrap process, and no scripts were to be installed. (Indeed, apart from the methods it would have used to locate the archive on PyPI, and to determine what to run from inside it, there was nothing particularly egg-specific about the proposed bootstrapping process.) So, to fully egg-neutralize the bootstrapping approach, we need only know how to locate an appropriate archive, and how to determine what to run from it. For the latter, we could use the already-in-2.6 convention of running __main__ from a zipfile or directory. (Too bad distutils source distributions have an extra directory name embedded in them, so one can't just execute them directly. Otherwise, we could've just let people drop in a __main__.py next to setup.py. OTOH, maybe it would be enough to use setuptools' algorithm for finding setup.py to locate __main__.py, and I'm fairly sure *that* can be briefly expressed in the PEP.) The other open question is a naming convention and version detection, so that the bootstrap tool can identify which of the files listed on PyPI is suitable for its use. (Both with regard to the version selection, and file type.) However, if PyPI were to grow support for designating the appropriate files and/or versions in some other way, we wouldn't need a naming convention as such. Without one or the other, the bootstrap tool would have to grow a version parsing scheme of some type, and play guessing games with file extensions. (Which is one reason I limited PEP 365's scope to downloading eggs actually *uploaded* to PyPI, rather than arbitrary packages *linked* from PyPI.) So, if I had to propose something right now, I would be inclined to propose: * using setuptools' version parsing semantics for interpretation of alpha/beta/dev/etc. releases * having a bdist_bootstrap format that's essentially a bdist_dumb .zip file with the internal path prefixes stripped off, making it an importable .zip with a different file extension. (Or maybe just .pyboot.zip?) The filename convention would use setuptools' canonicalization and escaping of names and version numbers, to allow unambiguous machine parsing of the filename. A __main__ module would have to be present for the archive to be run, as opposed to just being downloaded to a temporary directory. * calling the bootstrap module 'bootstrap', as in 'python -m bootstrap projectname optionalversion'. The module would expose an API to allow it to be used programmatically as well as the command line, so that bootstrapped packages can use the bootstrap process to locate dependencies if they so desire. (Today's package management tools, at least, are all based on setuptools, so if it's not present they'll need to download that before beginning their own bootstrapping process.) Apart from keeping the PEP self-contained and short, is there anything in this that you think you would object to? (You may reserve the right, of course, to later not like something in the details of setuptools' version/filename rules, after I've put them into the PEP, or really, anything else. I'm just asking if there's anything that's obviously offensive at this point, before I spend time on a new PEP.)

7 9