On Sun, Mar 16, 2008 at 10:23 AM, Guido van Rossum wrote:

Moving this to a new subject to keep the discussion of tasks and the discussion of task tracking tools separate.

...

I did a quick brainstorming with me, myself and I. I came up with a

On Sun, Mar 16, 2008 at 9:42 AM, Christian Heimes wrote: list

...

of (IMHO) important tasks.

* Stabilize the C API of Python 3.0. I like to rename several prefixes to reduce the confusing: PyBytes -> PyByteArray,

+1 (also +1 to backporting this to 2.6)

...

PyString -> PyBytes ...

-1. This will make merging code from 2.6 harder, and causes more work for porting C extensions.

There was a thread about this a few weeks ago: http://mail.python.org/pipermail/python-dev/2008-March/077339.html We can still do the renaming, but alias PyString to PyBytes.

...

* Backport the new buffer protocol to 2.6. I spoke to Travis yesterday and he said he is trying to get it done during the PyCon sprint. Maybe somebody can assist him?

Does he need assistance?

...

When the new buffer protocol is available in 2.6 we can start back porting bytesarray and the new IO framework.

Are these really so closely tied that they have to wait before they can be backported?

...

* Replace Windows API calls with wide versions to support unicode for file names, environment etc.

+1. This should be broken into separate tasks for each API.

...

* Get the stdlib reorg done

+1. But again, I think this is many small tasks.

...

and add the fixers to 2to3

+1. I think quite a few changes have not had a fixer added. Again, I think we should maintain a specific list of needed fixers; fixers can easily be developed independently.

...

* Speed up 2to3. I suggested a GSoC task to improve and speed up 2to3 but it may be too late when we plan to ship out 3.0 in August.

While I know that some people are expecting to use a development model that invokes 2to3 very frequently, I think this is at best a nice-to-have. (I also don't see how it could be done, but maybe I'm blind for the obvious, as the original author.)

I have some other tasks to add:

- getargs.c (Georg posted about this; his post actually inspired my post.)

- Tweak structseq to be more like namedtuple. Maybe they could have a common ABC. I also think we should get rid of the concept of "hidden" fields (that are accessible by name but not through the tuple API).

- Devise a way to handle str instances pickled in 2.6 and unpickled in 3.0, and also bytes instances pickled in 3.0 and unpickled in 2.6.

- Someone should go over https://spreadsheets.google.com/ccc?key=pCKY4oaXnT81FrGo3ShGHGg and extract more tasks.

-- --Guido van Rossum (home page: http://www.python.org/~guido/http://www.python.org/%7Eguido/ ) _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.c...

On Sun, Mar 16, 2008 at 8:23 AM, Guido van Rossum wrote:

On Sun, Mar 16, 2008 at 9:42 AM, Christian Heimes wrote: ......

...

and add the fixers to 2to3

+1. I think quite a few changes have not had a fixer added. Again, I think we should maintain a specific list of needed fixers; fixers can easily be developed independently.

Neal and I are coming up with a list to feed tasks to interested PyCon sprinters.

...

* Speed up 2to3. I suggested a GSoC task to improve and speed up 2to3 but it may be too late when we plan to ship out 3.0 in August.

While I know that some people are expecting to use a development model that invokes 2to3 very frequently, I think this is at best a nice-to-have. (I also don't see how it could be done, but maybe I'm blind for the obvious, as the original author.)

The biggest win in terms of performance would be to reimplement the pattern matching engine used by the fixers.: it by far dominates the running time, taking 99+% of the runtime when I ran 2to3 over Twisted, for example. The current design is a heavily-recursive system, and as such bombs out when it encounters, e.g., files with a thousand assignment statements in a row. I'd also like something more expressive: the current DSL can't express recursive patterns. Collin Winter

...

...

* Replace Windows API calls with wide versions to support unicode for file names, environment etc. +1. This should be broken into separate tasks for each API.

What are we referring to here? Calling the W versions explicitly and using wchar_t for everything, or using the TCHAR/TEXT() approach and keeping the API calls the same, letting the #define UNICODE do the work behind the scenes?

Not sure whose being attributed here, but I think "breaking down" should be done by "information source": command line, environment, registry, file names, sys.path, module names, etc. I have a patch on SF to resolve the command line issue. I don't think we should use Microsoft's Unicode programming model around TCHAR/TEXT. It would allow for two different builds, and given that we don't need to support W9X anymore, an "ANSI" build is pointless, IMO. Regards, Martin

Collin Winter wrote:

The biggest win in terms of performance would be to reimplement the pattern matching engine used by the fixers.: it by far dominates the running time, taking 99+% of the runtime when I ran 2to3 over Twisted, for example. The current design is a heavily-recursive system, and as such bombs out when it encounters, e.g., files with a thousand assignment statements in a row. I'd also like something more expressive: the current DSL can't express recursive patterns.

Do you have time to mentor a GSoC project? Or can you mentor a mentor ...? :) Christian

On Sun, Mar 16, 2008 at 12:32 PM, Guido van Rossum wrote:

On Sun, Mar 16, 2008 at 11:57 AM, Benjamin Peterson wrote: [Guido]

...

...

That's a rather long thread. Was any conclusion reached? I'm not sure how introducing a set of aliases will help merging 2.6 code to 3.0. Can you or Christian describe the proposed approach in more detail?

...

As far as I can see, no objections were raised in that thread.

Hm. I had wanted to register a complaint, but I guess I was too busy.

...

Christian explained the probable approach: http://mail.python.org/pipermail/python-dev/2008-March/077362.html

That's not much of a plan. It doesn't discuss any of the effects of the proposed change on merging code from the 2.6 trunk to the py3k branch.

On Sun, Mar 16, 2008 at 12:09 PM, Christian Heimes wrote:

...

Guido van Rossum wrote:

...

-1. This will make merging code from 2.6 harder, and causes more work for porting C extensions.

I'm happy to pay the price for the sake of a clean and easy-to-recall C API.

The for the C extension problem I already proposed a solution in the thread Benjamin mentioned before.

#include "Python.h" #if PY_VERSION_HEX > 0x03000000 #include "python2_compat.h" #endif

Where python2_compat provides aliases for PyInt and PyString:

#define PyInt_Spam PyLong_Spam ... #define PyString_Egg PyBytes_Egg

So this doesn't address merges at all. Suppose we have some C code that's shared between 2.6 and 3.0 and manipulates binary data (e.g. the gzip codec). It currently uses PyString on both branches, so any changes to the trunk merge smoothly into the py3k branch. But if you change PyString -> PyBytes in the py3k branch, every change you make in the 2.6 code will cause a merge conflict. Is that really what you want? I worry that it will effectively separate the trunk and the py3k branch, losing the advantage of doing development that effects both at once.

We could backport the python2_compact header.

-- --Guido van Rossum (home page: http://www.python.org/~guido/http://www.python.org/%7Eguido/ )

Guido van Rossum wrote:

So this doesn't address merges at all. Suppose we have some C code that's shared between 2.6 and 3.0 and manipulates binary data (e.g. the gzip codec). It currently uses PyString on both branches, so any changes to the trunk merge smoothly into the py3k branch. But if you change PyString -> PyBytes in the py3k branch, every change you make in the 2.6 code will cause a merge conflict. Is that really what you want? I worry that it will effectively separate the trunk and the py3k branch, losing the advantage of doing development that effects both at once.

I'm fully aware of the extra burden. The removal of the PyInt_* functions is already causing merge conflicts and compile errors. Nearly every C code merge contains at least one place that requires manual intervention. The PyInt merge conflicts are trivial to fix - so would errors caused PyString -> PyBytes rename. I'm not worried about the extra work since it's usually trivial and fast to fix. I'm more worried about the API names. Do you *really* want to drag down dead bodies along the road for the next ten years? The dead bodies are going to rot and stink sooner than later. By Python 3.2 everybody surely regrets the confusing names. ;) Christian

On Sun, Mar 16, 2008 at 1:04 PM, Christian Heimes wrote:

Guido van Rossum wrote:

...

So this doesn't address merges at all. Suppose we have some C code that's shared between 2.6 and 3.0 and manipulates binary data (e.g. the gzip codec). It currently uses PyString on both branches, so any changes to the trunk merge smoothly into the py3k branch. But if you change PyString -> PyBytes in the py3k branch, every change you make in the 2.6 code will cause a merge conflict. Is that really what you want? I worry that it will effectively separate the trunk and the py3k branch, losing the advantage of doing development that effects both at once.

I'm fully aware of the extra burden. The removal of the PyInt_* functions is already causing merge conflicts and compile errors.

Even though the more popular PyInt_ APIs are still available (even if only as macros).

Nearly every C code merge contains at least one place that requires manual intervention. The PyInt merge conflicts are trivial to fix - so would errors caused PyString -> PyBytes rename.

I disagree. Bad merges are already a frequent cause of instability in 3.0. This could easily double the problems. And while I want to reduce the instability (I really wish you would no commit until all unittests pass), I also don't want the merges to cost more of your time!

I'm not worried about the extra work since it's usually trivial and fast to fix. I'm more worried about the API names. Do you *really* want to drag down dead bodies along the road for the next ten years? The dead bodies are going to rot and stink sooner than later. By Python 3.2 everybody surely regrets the confusing names. ;)

It doesn't have to be so black and white. Using the macro approach you propose we can fix the situation at any time later. We could also make the changes in 2.6, so that the 2.6 code looks the same as 3.0. (However for binary compatibility I think it would be better if in 2.6 the linker sees PyString where in 3.0 it sees PyBytes.) -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On 3/16/08, Travis Oliphant wrote:

...

Moving this to a new subject to keep the discussion of tasks and the discussion of task tracking tools separate.

On Sun, Mar 16, 2008 at 9:42 AM, Christian Heimes wrote:

...

I did a quick brainstorming with me, myself and I. I came up with a

Guido van Rossum wrote: list

...

...

of (IMHO) important tasks.

* Stabilize the C API of Python 3.0. I like to rename several prefixes to reduce the confusing: PyBytes -> PyByteArray,

+1 (also +1 to backporting this to 2.6)

...

PyString -> PyBytes ...

-1. This will make merging code from 2.6 harder, and causes more work for porting C extensions.

...

* Backport the new buffer protocol to 2.6. I spoke to Travis yesterday and he said he is trying to get it done during the PyCon sprint. Maybe somebody can assist him?

Does he need assistance?

I don't really need help with back-porting the protocol. However, I do need help with the struct module changes. This is a standard-library that I'm hoping to get help with.

-Travis

I'm happy to help with this stuff during the sprint.

On Fri, Mar 21, 2008 at 4:41 PM, Christian Heimes wrote:

Guido van Rossum schrieb:

...

Even though the more popular PyInt_ APIs are still available (even if only as macros).

THe PyInt_* macros are only available when Include/intobject.h is included explicitly. It's not in Python.h any more.

...

I disagree. Bad merges are already a frequent cause of instability in 3.0. This could easily double the problems. And while I want to reduce the instability (I really wish you would no commit until all unittests pass), I also don't want the merges to cost more of your time!

I'm trying my best but sometimes I don't spot the cause of a failing unit test. I got a slightly faster laptop so I'm now able to run the full unit test suite every time I do a svnmerge.py.

:-)

...

It doesn't have to be so black and white. Using the macro approach you propose we can fix the situation at any time later.

We could also make the changes in 2.6, so that the 2.6 code looks the same as 3.0. (However for binary compatibility I think it would be better if in 2.6 the linker sees PyString where in 3.0 it sees PyBytes.)

Let me get this straight. You propose that we replace PyString_ with PyBytes_ in both Python 2.6 and 3.0 core code. In Python 2.6 some macros replace the PyBytes_* functions with PyString_ so the linker sees PyString_*? Mmh, it sounds like an interesting idea :]

Right. We've done this 2-stage rename before, during the great (sometimes known as grand) renaming, in the 1.x days. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

I haven't been around during the 1.x -> 2.x days. I was still in the dark ages (aka PHP user).

:-)

How do you want me to tackle down the PyString / PyBytes problem?

I think it can actually be simplified. I think maintaining binary compatibility between 2.6 and earlier versions is hopeless anyway, so we might as well just rename PyString to PyBytes in 2.6 and 3.0, and have an extra set of macros so that code using PyString needs to be recompiled but not otherwise touched. E.g. typedef { ... } PyBytesObject; #define PyStringObject PyBytesObject ... PyString_Type; #define PyBytes_Type PyString_Type <etc> -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum schrieb:

...

When the new buffer protocol is available in 2.6 we can start back porting bytesarray and the new IO framework.

Are these really so closely tied that they have to wait before they can be backported?

I've started with the backport of the bytearray type in a new branch svn+ssh://pythondev@svn.python.org/python/branches/trunk-bytearray Any help is appreciated. I already solved most issues. Open tasks: * fix bytearray.extend() * add PyString support to bytearray.fromhex * Add old style char buffer interface to bytearray (for codecs) * Add old style read write buffer interface to bytearray (for file.readinto) * Fix pickle and copy support for bytearray Christian