C. Titus Brown wrote:

On Sun, Mar 22, 2009 at 07:30:01PM -0300, Daniel (ajax) Diniz wrote: I do think you should be prepared for pushback from python-dev on any such ideas ;). Don't get too ambitious about writing up *your* way of fixing things, but rather make sure you and the students are prepared to adapt to what people on python-dev think. Mind you, ultimately the people doing the work should make the decisions, but input from python-dev is usually pretty useful even when it's contradictory!

Everything I've seen from Daniel so far seems to be about either making things we already do more efficient, or else providing additional features in ways that don't make the tools any more confusing for others already used to a particular way of doing things. So he seems to be navigating this particular minefield pretty well so far.

Then again, I may be a little biased - some of the recent bug tracker changes are things I had wished for in the past, but never chipped in any code to help make them happen :)

Cheers, Nick.

Oh, I heartily endorse his suggestions! I just want to make sure that we make maximum use of students (and their code doesn't get tossed at the end of the summer, which has serious morale consequences ;)

This is my concern as well.

One of my past students pitched a core project and ended up spending most of his SoC time in the PEP process (http://www.python.org/dev/peps/pep-0368/). Given how frustrating the experience was for him, I'd rather have future SoC students be able to focus on code.

Let's at least have rough consensus on brainstormed ideas before they hit the ideas page. Existing PEPs that have had a time to air and evolve are much better for this reason.

"Martin v. Löwis" wrote:

Yes, I'm also quite grateful for the contributions I have received from Daniel.

Thank you Martin. I'm sure I'd still be going around in circles if it weren't for your guidance, and I'd be MIA after the first time I broke the tracker too. So thanks a lot for the support.

I hope he'll stay around for some time without exhausting.

Me too. I'm trying to leave a big audit trail so other people can join efforts or take over parts of roadmap easily, but that's also a backup for long fieldwork trips or the eventual burn out.

Best regards, Daniel

C. Titus Brown wrote:

Given the relative paucity of core Python GSoC ideas, I think you should go for both of these, *especially* if you have a mentor up front.  So, write 'em up, add 'em to the GSoC page, and let's see who we get... If we get good applications for both, then I think we can "fund" both of them.

Great, thanks a lot for the feedback :)

I do think you should be prepared for pushback from python-dev on any such ideas ;).  Don't get too ambitious about writing up *your* way of fixing things, but rather make sure you and the students are prepared to adapt to what people on python-dev think.  Mind you, ultimately the people doing the work should make the decisions, but input from python-dev is usually pretty useful even when it's contradictory!

Pushback? Luxury! :) Thanks for the great advice, but I'd actually love negative input on this. Better to find out early that there's no way out of the bikeshedding tar pit than to waste a slot (and the student's time).

I think many people don't speak up for or against GSoC proposals because they don't have time to mentor / discuss details. This particular proposal is doomed if nobody voices the itches to be scratched.

So I'm kinda taunting python-dev with a huge proposal that goes around generalities and tries to make the case for joining efforts with PIDA ;)

Skipping to "Suggestions" and the "Food for thought example" might help in deciding whether the wall of text is worth a look...

Best regards, Daniel

== Helper Python core development tools.

There's some amount of repetitive, required steps to work on Python development. This proposal is about improving the workflow of core developers, probably with small glue scripts. As each developer has his own personal workflow, these helper scripts should be optional, easy to extend and aimed at the most common tasks. Of course, there may be no real demand for optimizing the workflow. Avoiding the use of a GSoC slot for unnecessary projects is very important, so if you think it's a wast of resources, please speak up :)

Justification Sometimes, non-obvious bits like the right sequence of svnmerge commands, the right way to link a Rietveld code review to a given issue or checking for correct autoconf version might get in the way of real development. Other obvious steps, like handling issue properties (e.g. Resolution), posting a message that tells in which revs the issue was fixed of or even checking for changes in tabs versus spaces, also require a bit of time. Regardless of obviousness, forgetting one item (or getting it wrong) in the development checklist happens from time to time. If there are ways to factor these repetitive, required tasks out from a core dev's concerns, it can only help the development process. Non-code developers could also benefit from this kind of tool, and could contribute in a more efficient way to Python development, with higher code/ticket quality. Quoting the tracker cleanup proposal: Optimizing the application of main/core developers' time to Python issues is a no-brainer. Being able to add volunteers to the productive time pool is also very desirable.

Pre-Requisites Identifying which tasks and steps should be optimized cannot be left to the student nor to the mentor, it must be a collective effort. Clear goals must be set before someone try to implement them. Sure, there are many descriptions of workflows in past python-dev (and python-list) threads, but the GSoC is about code.

Methods It's up to the mentors/student team. I suggest offering early releases for the scripts and maybe test repositories, trackers, Rietveld instance, etc., as ways of making sure the resulting code will be useful and used by the target public.

I believe most tools should be developed in a generic way, so that they could fit in other projects and workflows. IMHO, this would make the resulting scripts much more valuable.

Student Besides some experience with Python and tools of the trade (VCSs, bug trackers, etc.), the most important requisite is being able to listen to the community and taking criticism well.

Mentors Ali Afshar from PIDA is willing to mentor if the 'generic tools' way is accepted. Since it's about reusing development tools inside an IDE, any of these helper scripts could be integrated into PIDA, the only pre-requisite being that they'd not be too deeply linked to the Python infrastructure. With a small bit of concern about this, it'd be easy to extend/fork the resulting tools to use with other trackers, VCSs, test runners and code review tools. I am also available to help with the Python workflow part, and there's an early effort to integrate Rietveld and our bug tracker under way. At least one core Python developer should mentor, preferably one that knows how to interact well with python-dev. Any project focused on how people work is prone to flamefests, so diverting most heat away from the student might be a necessary skill. Bikeshedding is also very likely to occur, but the proposal's goals is to provide a couple of brushes and some paint cans of different colo(u)rs :)

Deliverables Should include a couple of helper scripts, maybe some patches to currently used tools. Other ideas?

Suggestions Wrappers for working with tracker issues, managing patches, running tests, code reviews, committing (including code checking hooks), merging, etc. E.g., I'm working on a script that allows one to submit a Roundup issue and a Rietveld issue in a way that links them together.

Food for thought example (sorry, not quite how a core dev works): ajaksu@Belkar:~/py3k$ pyfix 5434 --export ../month_delta --no-assign PyFixing... Issue 5434: datetime.MonthDelta Type: feature request Stage: not selected Versions: 3.1 Nosy list: jess.austin,lemburg,tim_one,+ajaksu2 Assigned to: nobody (--no-assign)

SVN exporting current working copy to: ../month_delta Export complete

Downloading files: monthdelta.py -> ../month_delta/patch/monthdelta.py OK monthdelta.diff -> ../month_delta/patch/monthdelta.diff OK Downloaded 2 files: monthdelta.py jess.austin, 2009-03-11 18:39 prototype implementation in python monthdelta.diff jess.austin, 2009-03-12 17:54 updated patch, unified diff against py3k

Downloaded 4 messages: ../month_delta/msgs/msg83272.txt Jess Austin: datetime is a wonderful (...) ../month_delta/msgs/msg83273.txt Jess Austin: This is my first try at (...) ../month_delta/msgs/msg83275.txt Jess Austin: Rietveld link: (...) ../month_delta/msgs/msg83480.txt Jess Austin: This prototype python (...)

Rietveld link found: http://codereview.appspot.com/25079

ajaksu@Belkar:~/py3k$ cd ../month_delta ajaksu@Belkar:~/month_delta$ pyfix patch PyFixing issue 5434... One patch found: patch/monthdelta.diff jess.austin, 2009-03-12 17:54 updated patch, unified diff against py3k Applying... OK ajaksu@Belkar:~/month_delta$ pyfix set stage "patch review" ajaksu@Belkar:~/month_delta$ pyfix status PyFixing issue 5434... Stage: not selected -> "patch review" D M Doc/c-api/datetime.rst 2 chunks 34 lines D M Doc/library/datetime.rst 10 chunks 354 lines B M Include/datetime.h 7 chunks 82 lines T M Lib/test/test_datetime.py 2 chunks 231 lines D M Misc/NEWS 1 chunk 14 lines B M Modules/datetimemodule.c 20 chunks 781 lines B: build required - T: tests - D: docs - G: backtrace

ajaksu@Belkar:~/month_delta$ pyfix test Error: build required, try to test anyway? [n] Test aborted ajaksu@Belkar:~/month_delta$ pyfix build ./configure --quiet make -s Python build finished, but the necessary bits to build these modules were not found: bsddb185 sunaudiodev ajaksu@Belkar:~/month_delta$ pyfix test ./python -E -tt ./Lib/test/regrtest.py -l -uall -rw 344 tests OK. 25 tests skipped. ajaksu@Belkar:~/month_delta$ pyfix up -m"LGTM, needs testing on other platforms." PyFixing issue 5434... No differences found. Setting stage: not selected -> patch review Adding message: Author: Daniel Diniz LGTM, needs testing on other platforms. ajaksu@Belkar:~/month_delta$

Thanks for the feedback, Antoine!

Antoine Pitrou wrote:

Daniel (ajax) Diniz <ajaksu <at> gmail.com> writes:

...

Sometimes, non-obvious bits like the right sequence of svnmerge commands, the right way to link a Rietveld code review to a given issue or checking for correct autoconf version might get in the way of real development.

Well, really, rather than trying to improve svnmerge, we should try to find a way forward to switch to Merc... oops, I mean what will turn out to be the best DVCS for our needs :-)

That was a little bait for input ;)

But the real point is that, regardless of underlying VCS, there is a choice between "having all core developers know by heart the right switches and order of steps to correctly checkout/update ->( branch locally) -> fix -> diff -> commit -> merge -> solve conflicts" and "offering a little, well-documented script that takes care of the switches and sequence of steps".

Maybe a 'untangle svnmerge-created history' tool would help too? :)

...

I am also available to help with the Python workflow part, and there's an early effort to integrate Rietveld and our bug tracker under way.

...

Food for thought example (sorry, not quite how a core dev works):

[...]

...

SVN exporting current working copy to: ../month_delta

Does it work when using an hg/bzr/git mirror? There's already hg and git support in Rietveld's upload.py, so this should be possible.

Given that the pyfix script is completely imaginary ATM, yes, it works as well with hg/bzr/git/perforce/CVS/darcs as it does with SVN :)

In a more serious note, the PIDA offer puts anyvc[1] on the table. So we could support different VCSs as upload.py does it, or aim for a more pluggable way, probably based on anyvc. Either way, them scripts should be simple and follow the Unix way, so e.g. pyfix --export would actually call anyvc --export or pyvcs --export.

Cheers, Daniel

[1] http://pypi.python.org/pypi/anyvc

On Mon, Mar 23, 2009 at 5:03 PM, Antoine Pitrou solipsis@pitrou.net wrote:

Hi,

Daniel (ajax) Diniz <ajaksu <at> gmail.com> writes:

...

But the real point is that, regardless of underlying VCS, there is a choice between "having all core developers know by heart the right switches and order of steps to correctly checkout/update ->( branch locally) -> fix -> diff -> commit -> merge -> solve conflicts" and "offering a little, well-documented script that takes care of the switches and sequence of steps".

Well, it seems to me that most of these steps are separated by manual intervention (e.g. compile and run the test suite to check that everything works smoothly), so there's no real point in making a script out of them.

The real issues with svnmerge are its occasional bugs or failures (it forgot some changesets when merging in the io-c branch!),

Any chance you were not using the latest svnmerge when you did that merge ? I've had problems like this when using older versions.

its slowness, and its limitations (which are really inherent to the SVN model: e.g., if someone commits to the branch you have just started doing an svnmerge to, you have to revert everything and start over with the latest updates).

Regards

Antoine.

---

Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ggpolo%40gmail.com

2009/3/23 Antoine Pitrou solipsis@pitrou.net:

Guilherme Polo <ggpolo <at> gmail.com> writes:

...

Any chance you were not using the latest svnmerge when you did that merge ? I've had problems like this when using older versions.

Well, actually, Benjamin did the merge, so perhaps he can give more info.

I was using the latest svnmerge.py from the SVN trunk. (maybe it's broken?)

Martin v. Löwis wrote:

P.P.S. Are you sure you have to re-merge when somebody commits something unrelated to the branch? Or just when somebody else merges as well?

The latter is the only one I've ever had problems with (and that was due to me forgetting to update before merging rather than someone else actually doing a concurrent merge).

Cheers, Nick.

...

P.S. I don't believe your claim that it forgot changesets. Could it be that it simply forgot adding files, and that it did so because you already had the files in the sandbox, so that the merging failed?

It's more weird actually, it actively forgot some changes in some files but some other changes in the /same/ files did get merged.

I see. Provided there is somebody willing to work on this, it might be interesting to reproduce it.

...

P.P.S. Are you sure you have to re-merge when somebody commits something unrelated to the branch? Or just when somebody else merges as well?

Perhaps only the latter, I'm not sure actually. (but it can also happen that the "unrelated" commit modifies some files you were merging changes in...)

That should not be a problem, unless the to-be-merged changes directly conflict. Just svn-update, then try committing again.

It's actually also possible to recover from overlapping merges also: just re-run svnmerge with --record-only (-M); this assumes nobody else has merged the very same changes concurrently.

Regards, Martin

Glenn Linderman wrote:

One can set CMD into Unicode mode (chcp 65001)... not sure how Python reacts to that either. But even then...

I tried that and others have reported doing so on python-list but no one has gotten that to work.

CMD will only use fixed-width fonts, and none of the standard XP ones seem to contain all of Unicode. Not sure if that has improved on Vista or 7, as they don't run here.

It _would_ be nice to get this resolved, somehow.

Definitely.

tjr

Hi. I'm Japanese and non-ascii charactor user. (cp932)

We have to use "IME" to input non-ascii charactor in Windows. When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe.

So setting codepage to 65001 make output universal but make input ascii-only. Sit!!!

I hope PyQtShell http://code.google.com/p/pyqtshell/ become good IDLE alternative.

Hi.

...

We have to use "IME" to input non-ascii charactor in Windows. When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe.

So setting codepage to 65001 make output universal but make input ascii-only. Sit!!!

Is there a code page that still allows IME input, but supports all of Unicode? I know there is GB18030 - is it any good?

I found WriteConsoleW() API recently. This API can write utf16 string to console directly, without change OutputCodepage.

example: http://bitbucket.org/methane/hg-fixutf8-jp/src/tip/win32helper.py#cl-42

I think this API is good for py3k. When stdout is console and not redirected to [pipe|file], sys.stdout.write(u"foo") can avoid encoding and use WriteConsoleW(L"foo")

When stdout is console and not redirected to [pipe|file], sys.stdout.write(u"foo") can avoid encoding and use WriteConsoleW(L"foo")

I think this would be fairly difficult to implement given the way the output libraries work. If you think it can be done, please provide a patch to bugs.python.org.

Regards, Martin

Terry Reedy tjreedy@udel.edu writes:

If Windows (or other OSes) (to be investigated) does not reliably come with a full unicode font (at least current BMP), is there a public domain or open license font that we can include?

The GNU Unifont at Unifoundry URL:http://unifoundry.com/ is designed for this purpose.