GSoC: Core Python development tools
Hi, I'd like to bring up the general idea of using a PSF slot in GSoC2009 to improve the Python development infrastructure. I also happen to have two concrete proposals for that (such a coincidence!). But I assure you the general idea is more important than my proposals :) General: Solving issues that get in core devs' way when they work on Python development could be a nice PSF GSoC project. I believe there are enough code related tasks that would greatly improve Python development workflow but lack manpower to complete. E.g., ISTM that a student working on svnmerge in past SoC editions would've been able to mitigate some painful shortcomings. If the core developers could come up with a list of peeves that would be solvable in code (in existing tools), that would allow for a very useful GSoC proposal. Proposals: These might fit the description above, and they're both tracker related (yep, one-trick-pony here). The upside is that at least one potential mentor is available for each, and I'd be able to offer support to the student. First, the PSF could invest a slot on integrating different tools of the development workflow. Variations of 'file issue at bug tracker, submit code for review' or 'branch locally to virtualenv, upload failing testcase to tracker, upload patch to tracker' command line utils. If these could be developed as general tools (i.e., not deeply tied to Python dev infrastructure), Ali Afshar from PIDA is willing to mentor it. I'd be available to help with Roundup and Rietveld (integration in progress), but would like to see it work with Launchpad, Bugzilla, Google Code and Review Board :) The other proposal is to use a slot in Roundup proper and the Python tracker instance. This could have a core Roundup developer as mentor, under the condition it's focused on Roundup itself. IMO, are some missing/broken core features in Roundup that would have a positive impact on our tracker, mostly those relating to searches, security and UI. I'd have a lot to contribute to the Python tracker part, based on ongoing work. Even if neither is considered worthy, I'll keep them on my to-do list and hope to slowly and hackishly work towards both proposals' goals. Barring feedback saying that they're out of scope, stupid and downright offensive, I'll post details on each to this thread very soon. So, if the PSF was to use a slot on improving the way you work on Python development, what would you like to see fixed or implemented? Best regards, Daniel
On Sun, Mar 22, 2009 at 07:30:01PM -0300, Daniel (ajax) Diniz wrote: -> Even if neither is considered worthy, I'll keep them on my to-do list -> and hope to slowly and hackishly work towards both proposals' goals. -> Barring feedback saying that they're out of scope, stupid and -> downright offensive, I'll post details on each to this thread very -> soon. Given the relative paucity of core Python GSoC ideas, I think you should go for both of these, *especially* if you have a mentor up front. So, write 'em up, add 'em to the GSoC page, and let's see who we get... If we get good applications for both, then I think we can "fund" both of them. I do think you should be prepared for pushback from python-dev on any such ideas ;). Don't get too ambitious about writing up *your* way of fixing things, but rather make sure you and the students are prepared to adapt to what people on python-dev think. Mind you, ultimately the people doing the work should make the decisions, but input from python-dev is usually pretty useful even when it's contradictory! cheers, --titus -- C. Titus Brown, ctb@msu.edu
C. Titus Brown wrote:
On Sun, Mar 22, 2009 at 07:30:01PM -0300, Daniel (ajax) Diniz wrote: I do think you should be prepared for pushback from python-dev on any such ideas ;). Don't get too ambitious about writing up *your* way of fixing things, but rather make sure you and the students are prepared to adapt to what people on python-dev think. Mind you, ultimately the people doing the work should make the decisions, but input from python-dev is usually pretty useful even when it's contradictory!
Everything I've seen from Daniel so far seems to be about either making things we already do more efficient, or else providing additional features in ways that don't make the tools any more confusing for others already used to a particular way of doing things. So he seems to be navigating this particular minefield pretty well so far. Then again, I may be a little biased - some of the recent bug tracker changes are things I had wished for in the past, but never chipped in any code to help make them happen :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Mon, Mar 23, 2009 at 10:26:54PM +1000, Nick Coghlan wrote: -> C. Titus Brown wrote: -> > On Sun, Mar 22, 2009 at 07:30:01PM -0300, Daniel (ajax) Diniz wrote: -> > I do think you should be prepared for pushback from python-dev on any -> > such ideas ;). Don't get too ambitious about writing up *your* way of -> > fixing things, but rather make sure you and the students are prepared to -> > adapt to what people on python-dev think. Mind you, ultimately the -> > people doing the work should make the decisions, but input from -> > python-dev is usually pretty useful even when it's contradictory! -> -> Everything I've seen from Daniel so far seems to be about either making -> things we already do more efficient, or else providing additional -> features in ways that don't make the tools any more confusing for others -> already used to a particular way of doing things. So he seems to be -> navigating this particular minefield pretty well so far. -> -> Then again, I may be a little biased - some of the recent bug tracker -> changes are things I had wished for in the past, but never chipped in -> any code to help make them happen :) Oh, I heartily endorse his suggestions! I just want to make sure that we make maximum use of students (and their code doesn't get tossed at the end of the summer, which has serious morale consequences ;) --t -- C. Titus Brown, ctb@msu.edu
Oh, I heartily endorse his suggestions! I just want to make sure that we make maximum use of students (and their code doesn't get tossed at the end of the summer, which has serious morale consequences ;)
This is my concern as well. One of my past students pitched a core project and ended up spending most of his SoC time in the PEP process (http://www.python.org/dev/peps/pep-0368/). Given how frustrating the experience was for him, I'd rather have future SoC students be able to focus on code. Let's at least have rough consensus on brainstormed ideas before they hit the ideas page. Existing PEPs that have had a time to air and evolve are much better for this reason.
Everything I've seen from Daniel so far seems to be about either making things we already do more efficient, or else providing additional features in ways that don't make the tools any more confusing for others already used to a particular way of doing things. So he seems to be navigating this particular minefield pretty well so far.
Yes, I'm also quite grateful for the contributions I have received from Daniel. I hope he'll stay around for some time without exhausting. Regards, Martin
"Martin v. Löwis" wrote:
Yes, I'm also quite grateful for the contributions I have received from Daniel.
Thank you Martin. I'm sure I'd still be going around in circles if it weren't for your guidance, and I'd be MIA after the first time I broke the tracker too. So thanks a lot for the support.
I hope he'll stay around for some time without exhausting. Me too. I'm trying to leave a big audit trail so other people can join efforts or take over parts of roadmap easily, but that's also a backup for long fieldwork trips or the eventual burn out.
Best regards, Daniel
Nick Coghlan wrote:
Everything I've seen from Daniel so far seems to be about either making things we already do more efficient, or else providing additional features in ways that don't make the tools any more confusing for others already used to a particular way of doing things. So he seems to be navigating this particular minefield pretty well so far.
Thanks a lot Nick! :) BTW, it seems there's a procedure to follow if my navigation fails[1].
Then again, I may be a little biased - some of the recent bug tracker changes are things I had wished for in the past, but never chipped in any code to help make them happen :)
Not a problem, sir, we accept RFEs devoid of any code bits in this store :) Cheers, Daniel [1] George: If we do happen to step on a mine, Sir, what do we do? Capt. Blackadder: Normal procedure, Lieutenant, is to jump two hundred feet in the air and scatter oneself over a wide area.
C. Titus Brown wrote:
Given the relative paucity of core Python GSoC ideas, I think you should go for both of these, *especially* if you have a mentor up front. So, write 'em up, add 'em to the GSoC page, and let's see who we get... If we get good applications for both, then I think we can "fund" both of them.
Great, thanks a lot for the feedback :)
I do think you should be prepared for pushback from python-dev on any such ideas ;). Don't get too ambitious about writing up *your* way of fixing things, but rather make sure you and the students are prepared to adapt to what people on python-dev think. Mind you, ultimately the people doing the work should make the decisions, but input from python-dev is usually pretty useful even when it's contradictory!
Pushback? Luxury! :) Thanks for the great advice, but I'd actually love negative input on this. Better to find out early that there's no way out of the bikeshedding tar pit than to waste a slot (and the student's time). I think many people don't speak up for or against GSoC proposals because they don't have time to mentor / discuss details. This particular proposal is doomed if nobody voices the itches to be scratched. So I'm kinda taunting python-dev with a huge proposal that goes around generalities and tries to make the case for joining efforts with PIDA ;) Skipping to "Suggestions" and the "Food for thought example" might help in deciding whether the wall of text is worth a look... Best regards, Daniel == Helper Python core development tools. There's some amount of repetitive, required steps to work on Python development. This proposal is about improving the workflow of core developers, probably with small glue scripts. As each developer has his own personal workflow, these helper scripts should be optional, easy to extend and aimed at the most common tasks. Of course, there may be no real demand for optimizing the workflow. Avoiding the use of a GSoC slot for unnecessary projects is very important, so if you think it's a wast of resources, please speak up :) Justification Sometimes, non-obvious bits like the right sequence of svnmerge commands, the right way to link a Rietveld code review to a given issue or checking for correct autoconf version might get in the way of real development. Other obvious steps, like handling issue properties (e.g. Resolution), posting a message that tells in which revs the issue was fixed of or even checking for changes in tabs versus spaces, also require a bit of time. Regardless of obviousness, forgetting one item (or getting it wrong) in the development checklist happens from time to time. If there are ways to factor these repetitive, required tasks out from a core dev's concerns, it can only help the development process. Non-code developers could also benefit from this kind of tool, and could contribute in a more efficient way to Python development, with higher code/ticket quality. Quoting the tracker cleanup proposal: Optimizing the application of main/core developers' time to Python issues is a no-brainer. Being able to add volunteers to the productive time pool is also very desirable. Pre-Requisites Identifying which tasks and steps should be optimized cannot be left to the student nor to the mentor, it must be a collective effort. Clear goals must be set before someone try to implement them. Sure, there are many descriptions of workflows in past python-dev (and python-list) threads, but the GSoC is about code. Methods It's up to the mentors/student team. I suggest offering early releases for the scripts and maybe test repositories, trackers, Rietveld instance, etc., as ways of making sure the resulting code will be useful and used by the target public. I believe most tools should be developed in a generic way, so that they could fit in other projects and workflows. IMHO, this would make the resulting scripts much more valuable. Student Besides some experience with Python and tools of the trade (VCSs, bug trackers, etc.), the most important requisite is being able to listen to the community and taking criticism well. Mentors Ali Afshar from PIDA is willing to mentor if the 'generic tools' way is accepted. Since it's about reusing development tools inside an IDE, any of these helper scripts could be integrated into PIDA, the only pre-requisite being that they'd not be too deeply linked to the Python infrastructure. With a small bit of concern about this, it'd be easy to extend/fork the resulting tools to use with other trackers, VCSs, test runners and code review tools. I am also available to help with the Python workflow part, and there's an early effort to integrate Rietveld and our bug tracker under way. At least one core Python developer should mentor, preferably one that knows how to interact well with python-dev. Any project focused on how people work is prone to flamefests, so diverting most heat away from the student might be a necessary skill. Bikeshedding is also very likely to occur, but the proposal's goals is to provide a couple of brushes and some paint cans of different colo(u)rs :) Deliverables Should include a couple of helper scripts, maybe some patches to currently used tools. Other ideas? Suggestions Wrappers for working with tracker issues, managing patches, running tests, code reviews, committing (including code checking hooks), merging, etc. E.g., I'm working on a script that allows one to submit a Roundup issue and a Rietveld issue in a way that links them together. Food for thought example (sorry, not quite how a core dev works): ajaksu@Belkar:~/py3k$ pyfix 5434 --export ../month_delta --no-assign PyFixing... Issue 5434: datetime.MonthDelta Type: feature request Stage: not selected Versions: 3.1 Nosy list: jess.austin,lemburg,tim_one,+ajaksu2 Assigned to: nobody (--no-assign) SVN exporting current working copy to: ../month_delta Export complete Downloading files: monthdelta.py -> ../month_delta/patch/monthdelta.py OK monthdelta.diff -> ../month_delta/patch/monthdelta.diff OK Downloaded 2 files: monthdelta.py jess.austin, 2009-03-11 18:39 prototype implementation in python monthdelta.diff jess.austin, 2009-03-12 17:54 updated patch, unified diff against py3k Downloaded 4 messages: ../month_delta/msgs/msg83272.txt Jess Austin: datetime is a wonderful (...) ../month_delta/msgs/msg83273.txt Jess Austin: This is my first try at (...) ../month_delta/msgs/msg83275.txt Jess Austin: Rietveld link: (...) ../month_delta/msgs/msg83480.txt Jess Austin: This prototype python (...) Rietveld link found: http://codereview.appspot.com/25079 ajaksu@Belkar:~/py3k$ cd ../month_delta ajaksu@Belkar:~/month_delta$ pyfix patch PyFixing issue 5434... One patch found: patch/monthdelta.diff jess.austin, 2009-03-12 17:54 updated patch, unified diff against py3k Applying... OK ajaksu@Belkar:~/month_delta$ pyfix set stage "patch review" ajaksu@Belkar:~/month_delta$ pyfix status PyFixing issue 5434... Stage: not selected -> "patch review" D M Doc/c-api/datetime.rst 2 chunks 34 lines D M Doc/library/datetime.rst 10 chunks 354 lines B M Include/datetime.h 7 chunks 82 lines T M Lib/test/test_datetime.py 2 chunks 231 lines D M Misc/NEWS 1 chunk 14 lines B M Modules/datetimemodule.c 20 chunks 781 lines B: build required - T: tests - D: docs - G: backtrace ajaksu@Belkar:~/month_delta$ pyfix test Error: build required, try to test anyway? [n] Test aborted ajaksu@Belkar:~/month_delta$ pyfix build ./configure --quiet make -s Python build finished, but the necessary bits to build these modules were not found: bsddb185 sunaudiodev ajaksu@Belkar:~/month_delta$ pyfix test ./python -E -tt ./Lib/test/regrtest.py -l -uall -rw 344 tests OK. 25 tests skipped. ajaksu@Belkar:~/month_delta$ pyfix up -m"LGTM, needs testing on other platforms." PyFixing issue 5434... No differences found. Setting stage: not selected -> patch review Adding message: Author: Daniel Diniz LGTM, needs testing on other platforms. ajaksu@Belkar:~/month_delta$
Daniel (ajax) Diniz <ajaksu <at> gmail.com> writes:
Sometimes, non-obvious bits like the right sequence of svnmerge commands, the right way to link a Rietveld code review to a given issue or checking for correct autoconf version might get in the way of real development.
Well, really, rather than trying to improve svnmerge, we should try to find a way forward to switch to Merc... oops, I mean what will turn out to be the best DVCS for our needs :-)
I am also available to help with the Python workflow part, and there's an early effort to integrate Rietveld and our bug tracker under way.
Food for thought example (sorry, not quite how a core dev works): [...]
SVN exporting current working copy to: ../month_delta
Does it work when using an hg/bzr/git mirror? There's already hg and git support in Rietveld's upload.py, so this should be possible.
Thanks for the feedback, Antoine! Antoine Pitrou wrote:
Daniel (ajax) Diniz <ajaksu <at> gmail.com> writes:
Sometimes, non-obvious bits like the right sequence of svnmerge commands, the right way to link a Rietveld code review to a given issue or checking for correct autoconf version might get in the way of real development.
Well, really, rather than trying to improve svnmerge, we should try to find a way forward to switch to Merc... oops, I mean what will turn out to be the best DVCS for our needs :-)
That was a little bait for input ;) But the real point is that, regardless of underlying VCS, there is a choice between "having all core developers know by heart the right switches and order of steps to correctly checkout/update ->( branch locally) -> fix -> diff -> commit -> merge -> solve conflicts" and "offering a little, well-documented script that takes care of the switches and sequence of steps". Maybe a 'untangle svnmerge-created history' tool would help too? :)
I am also available to help with the Python workflow part, and there's an early effort to integrate Rietveld and our bug tracker under way.
Food for thought example (sorry, not quite how a core dev works): [...]
SVN exporting current working copy to: ../month_delta
Does it work when using an hg/bzr/git mirror? There's already hg and git support in Rietveld's upload.py, so this should be possible.
Given that the pyfix script is completely imaginary ATM, yes, it works as well with hg/bzr/git/perforce/CVS/darcs as it does with SVN :) In a more serious note, the PIDA offer puts anyvc[1] on the table. So we could support different VCSs as upload.py does it, or aim for a more pluggable way, probably based on anyvc. Either way, them scripts should be simple and follow the Unix way, so e.g. pyfix --export would actually call anyvc --export or pyvcs --export. Cheers, Daniel [1] http://pypi.python.org/pypi/anyvc
Hi, Daniel (ajax) Diniz <ajaksu <at> gmail.com> writes:
But the real point is that, regardless of underlying VCS, there is a choice between "having all core developers know by heart the right switches and order of steps to correctly checkout/update ->( branch locally) -> fix -> diff -> commit -> merge -> solve conflicts" and "offering a little, well-documented script that takes care of the switches and sequence of steps".
Well, it seems to me that most of these steps are separated by manual intervention (e.g. compile and run the test suite to check that everything works smoothly), so there's no real point in making a script out of them. The real issues with svnmerge are its occasional bugs or failures (it forgot some changesets when merging in the io-c branch!), its slowness, and its limitations (which are really inherent to the SVN model: e.g., if someone commits to the branch you have just started doing an svnmerge to, you have to revert everything and start over with the latest updates). Regards Antoine.
On Mon, Mar 23, 2009 at 5:03 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Hi,
Daniel (ajax) Diniz <ajaksu <at> gmail.com> writes:
But the real point is that, regardless of underlying VCS, there is a choice between "having all core developers know by heart the right switches and order of steps to correctly checkout/update ->( branch locally) -> fix -> diff -> commit -> merge -> solve conflicts" and "offering a little, well-documented script that takes care of the switches and sequence of steps".
Well, it seems to me that most of these steps are separated by manual intervention (e.g. compile and run the test suite to check that everything works smoothly), so there's no real point in making a script out of them.
The real issues with svnmerge are its occasional bugs or failures (it forgot some changesets when merging in the io-c branch!),
Any chance you were not using the latest svnmerge when you did that merge ? I've had problems like this when using older versions.
its slowness, and its limitations (which are really inherent to the SVN model: e.g., if someone commits to the branch you have just started doing an svnmerge to, you have to revert everything and start over with the latest updates).
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ggpolo%40gmail.com
-- -- Guilherme H. Polo Goncalves
2009/3/23 Antoine Pitrou <solipsis@pitrou.net>:
Guilherme Polo <ggpolo <at> gmail.com> writes:
Any chance you were not using the latest svnmerge when you did that merge ? I've had problems like this when using older versions.
Well, actually, Benjamin did the merge, so perhaps he can give more info.
I was using the latest svnmerge.py from the SVN trunk. (maybe it's broken?) -- Regards, Benjamin
The real issues with svnmerge are its occasional bugs or failures (it forgot some changesets when merging in the io-c branch!), its slowness, and its limitations (which are really inherent to the SVN model: e.g., if someone commits to the branch you have just started doing an svnmerge to, you have to revert everything and start over with the latest updates).
I think addressing the slowness should surely be in scope for SoC: I would hope that rewriting it to only use a single session should provide some speedup (i.e. through the Python bindings, rather than the command line). Of course, such a project might better be mentored at subversion than Python. Regards, Martin P.S. I don't believe your claim that it forgot changesets. Could it be that it simply forgot adding files, and that it did so because you already had the files in the sandbox, so that the merging failed? P.P.S. Are you sure you have to re-merge when somebody commits something unrelated to the branch? Or just when somebody else merges as well?
Martin v. Löwis wrote:
P.P.S. Are you sure you have to re-merge when somebody commits something unrelated to the branch? Or just when somebody else merges as well?
The latter is the only one I've ever had problems with (and that was due to me forgetting to update before merging rather than someone else actually doing a concurrent merge). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Martin v. Löwis <martin <at> v.loewis.de> writes:
P.S. I don't believe your claim that it forgot changesets. Could it be that it simply forgot adding files, and that it did so because you already had the files in the sandbox, so that the merging failed?
It's more weird actually, it actively forgot some changes in some files but some other changes in the /same/ files did get merged.
P.P.S. Are you sure you have to re-merge when somebody commits something unrelated to the branch? Or just when somebody else merges as well?
Perhaps only the latter, I'm not sure actually. (but it can also happen that the "unrelated" commit modifies some files you were merging changes in...)
P.S. I don't believe your claim that it forgot changesets. Could it be that it simply forgot adding files, and that it did so because you already had the files in the sandbox, so that the merging failed?
It's more weird actually, it actively forgot some changes in some files but some other changes in the /same/ files did get merged.
I see. Provided there is somebody willing to work on this, it might be interesting to reproduce it.
P.P.S. Are you sure you have to re-merge when somebody commits something unrelated to the branch? Or just when somebody else merges as well?
Perhaps only the latter, I'm not sure actually. (but it can also happen that the "unrelated" commit modifies some files you were merging changes in...)
That should not be a problem, unless the to-be-merged changes directly conflict. Just svn-update, then try committing again. It's actually also possible to recover from overlapping merges also: just re-run svnmerge with --record-only (-M); this assumes nobody else has merged the very same changes concurrently. Regards, Martin
Antoine Pitrou wrote:
Well, it seems to me that most of these steps are separated by manual intervention (e.g. compile and run the test suite to check that everything works smoothly)
Agreed.
so there's no real point in making a script out of them.
The idea would be to provide scripts for each step that people think should be easier/automated. Like running the provided testcase before and after applying the corresponding patch. Or making sure to build before running tests for a patch that touches C code.
The real issues with svnmerge are its occasional bugs or failures (it forgot some changesets when merging in the io-c branch!)
Who's handling those bugs? Is any of them avoidable/easier to catch/easier to workaround with a script that checks for, e.g., svnmerge version, svnmerge correct usage and completeness of the merge? I think this proposal could help reproducing those issues and fixing/mitigating them until a better solution is available.
its slowness,
Maybe something could be improved in svnmerge, as Martin suggests. Or merging could be made a bit easier, so people would backport their commits themselves more often and the laundry list merges could be a little smaller.
and its limitations (which are really inherent to the SVN model: e.g., if someone commits to the branch you have just started doing an svnmerge to, you have to revert everything and start over with the latest updates).
We have discussed this off-list and I have no answer regarding SVN limitations. If a svnmerge-on-top-of-hg or svnmerge-on-top-of-bzr script can be used to merge SVN branches with the exact same results of the real svnmerge, only faster and prettier, maybe it should be considered for GSoC too. If/when the main Python repository moves to a DVCS, scripts that allow people to perform the same steps they currently perform to achieve the same results might help the transition. I won't mention the bzr-on-top-of-git part :) Cheers, Daniel
One of the disappointments of CPython 3.0 on Windows is that the switch to unicode for text (str), coupled with the continued use of a unicode-oblivious (obtuse) user interface (MS 'Command Prompt'), means that print can no longer print all str strings, or all legal Python code (as in a traceback). Different people have tried and failed fix this bug by fiddling with 'Command Prompt' configeration. This would make a useful summer project, though I don't know if it would involve enough coding. Call the project something like 3.0 Unicode UI Improvement. I see two possibilities. 1) Find an C-coded open-source C-P replacement whose author will license to PSF and, as needed, modify or integrate it with CPython. 2) IDLE does much better but its support seems to still be imcomplete. Upgrade tk/tkinter/IDLE (wherever the problems lie) and make IDLE's shell an alternate UI. If Windows (or other OSes) (to be investigated) does not reliably come with a full unicode font (at least current BMP), is there a public domain or open license font that we can include? Terry Jan Reedy
On approximately 3/22/2009 8:48 PM, came the following characters from the keyboard of Terry Reedy:
One of the disappointments of CPython 3.0 on Windows is that the switch to unicode for text (str), coupled with the continued use of a unicode-oblivious (obtuse) user interface (MS 'Command Prompt'), means that print can no longer print all str strings, or all legal Python code (as in a traceback).
Different people have tried and failed fix this bug by fiddling with 'Command Prompt' configeration. This would make a useful summer project, though I don't know if it would involve enough coding. Call the project something like 3.0 Unicode UI Improvement.
I see two possibilities.
1) Find an C-coded open-source C-P replacement whose author will license to PSF and, as needed, modify or integrate it with CPython.
2) IDLE does much better but its support seems to still be imcomplete. Upgrade tk/tkinter/IDLE (wherever the problems lie) and make IDLE's shell an alternate UI.
If Windows (or other OSes) (to be investigated) does not reliably come with a full unicode font (at least current BMP), is there a public domain or open license font that we can include?
One can, of course, set CMD into Latin-1 mode (chcp 1252)... not sure how Python reacts to that, as I've only used it with Perl, until I gave up on Perl's Unicode support (which someone finally seems to be fixing, but then there is CPAN to improve). But that doesn't solve the font problem, nor characters above 255. One can set CMD into Unicode mode (chcp 65001)... not sure how Python reacts to that either. But even then... CMD will only use fixed-width fonts, and none of the standard XP ones seem to contain all of Unicode. Not sure if that has improved on Vista or 7, as they don't run here. It _would_ be nice to get this resolved, somehow. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
Glenn Linderman wrote:
One can set CMD into Unicode mode (chcp 65001)... not sure how Python reacts to that either. But even then...
I tried that and others have reported doing so on python-list but no one has gotten that to work.
CMD will only use fixed-width fonts, and none of the standard XP ones seem to contain all of Unicode. Not sure if that has improved on Vista or 7, as they don't run here.
It _would_ be nice to get this resolved, somehow.
Definitely. tjr
On approximately 3/23/2009 12:12 PM, came the following characters from the keyboard of Terry Reedy:
Glenn Linderman wrote:
One can set CMD into Unicode mode (chcp 65001)... not sure how Python reacts to that either. But even then...
I tried that and others have reported doing so on python-list but no one has gotten that to work.
http://support.microsoft.com/kb/247815 http://www.microsoft.com/downloads/details.aspx?familyid=22e69ae4-7e40-4807-8a86-b3d36fab68d3&displaylang=en (python 3) import ctypes k=ctypes.WinDLL('kernel32') x = k.SetConsoleOutputCP(65001) if x!= 1: print("x was ", x ) exit( 1 ) print (''.join(chr(i) for i in range(0x410, 0x430)).encode('utf-8')) produces a nice b'\xd0\x90\d0\x91....' stream of hex representations of UTF-8 encoded Unicode characters... The only thing that seems to be missing is that Python won't emit them to the screen that way. So surely some python-dev that is smarter than me, could provide that magic incantation. Will go search, but that isn't in my current knowledge banks. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
Hi. I'm Japanese and non-ascii charactor user. (cp932) We have to use "IME" to input non-ascii charactor in Windows. When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe. So setting codepage to 65001 make output universal but make input ascii-only. Sit!!! I hope PyQtShell <http://code.google.com/p/pyqtshell/> become good IDLE alternative.
Hi. I'm Japanese and non-ascii charactor user. (cp932)
We have to use "IME" to input non-ascii charactor in Windows. When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe.
So setting codepage to 65001 make output universal but make input ascii-only. Sit!!!
Is there a code page that still allows IME input, but supports all of Unicode? I know there is GB18030 - is it any good? In any case, for *that* problem, I think you need to look for a different terminal program.
I hope PyQtShell <http://code.google.com/p/pyqtshell/> become good IDLE alternative.
Could well be. To see it integrated into Python (and perhaps replace IDLE), a lot of steps would need to happen, though. But perhaps you weren't asking for it to be included in Python - it can also be a good alternative if it's not included (like several other IDEs) Regards, Martin
Hi.
We have to use "IME" to input non-ascii charactor in Windows. When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe.
So setting codepage to 65001 make output universal but make input ascii-only. Sit!!!
Is there a code page that still allows IME input, but supports all of Unicode? I know there is GB18030 - is it any good?
I found WriteConsoleW() API recently. This API can write utf16 string to console directly, without change OutputCodepage. example: http://bitbucket.org/methane/hg-fixutf8-jp/src/tip/win32helper.py#cl-42 I think this API is good for py3k. When stdout is console and not redirected to [pipe|file], sys.stdout.write(u"foo") can avoid encoding and use WriteConsoleW(L"foo") -- Naoki INADA <songofacandy@gmail.com>
On Thu, Jul 23, 2009, INADA Naoki wrote:
I found WriteConsoleW() API recently. This API can write utf16 string to console directly, without change OutputCodepage.
example: http://bitbucket.org/methane/hg-fixutf8-jp/src/tip/win32helper.py#cl-42
I think this API is good for py3k. When stdout is console and not redirected to [pipe|file], sys.stdout.write(u"foo") can avoid encoding and use WriteConsoleW(L"foo")
Please submit a feature request to bugs.python.org -- with a patch would be even nicer, of course. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "The volume of a pizza of thickness 'a' and radius 'z' is given by pi*z*z*a"
When stdout is console and not redirected to [pipe|file], sys.stdout.write(u"foo") can avoid encoding and use WriteConsoleW(L"foo")
I think this would be fairly difficult to implement given the way the output libraries work. If you think it can be done, please provide a patch to bugs.python.org. Regards, Martin
On approximately 3/24/2009 10:16 AM, came the following characters from the keyboard of INADA Naoki:
Hi. I'm Japanese and non-ascii charactor user. (cp932)
We have to use "IME" to input non-ascii charactor in Windows. When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe.
So setting codepage to 65001 make output universal but make input ascii-only. Sit!!!
I hope PyQtShell <http://code.google.com/p/pyqtshell/> become good IDLE alternative.
Thanks for the feedback. So at least one version of the code I posted shows that programmatically, the code page can be set differently for input and output, although the last version brought both to 65001. It seems that the chcp 65001 always does both. If the IME only works for cp932, then leave input at cp932, and set output to 65001? I have no idea if that could be a solution for you, but I would be interested in your results if you find that it is, or isn't, as that would add to the collective knowledge base about the subject. This is idea 2, below, where I tried to cover the solution space more broadly. Looking briefly at the definition of cp932, it seems that it covers most of the Unicode characters... so perhaps any or several of the following could happen: 1) the IME could be converted to produce UTF-8 instead of cp932, allowing use of 65001 for input and output 2) the split code page could be used to avoid the conversion of Unicode to cp932 for output. 3) Unicode could be converted to cp932 for output, allowing use of cp932 for both input and output. These are listed in the order of increased overhead for character handling. Perhaps you could enlighten us all as to the issues with each of these ideas. I realize the IME exists today, and is likely coded to use cp932, and that it would take some work to convert it to produce Unicode. However, there seems to be a straightforward conversion chart between cp932 and Unicode at Wikipedia, so perhaps that isn't a huge effort. It seems that the long term goal of having all software speak Unicode would increase the efficiency of all software when dealing with multi-lingual issues, as a common solution can be applied universally, rather than re-inventing solutions that only work for particular code pages. But I'm not fully aware of whether or not the design or implementation of Unicode precludes universal solutions: I have heard rumors that certain characters must be interpreted differently in different locale contexts, which seems to be counter to the "one solution fits all" possibility. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
Terry Reedy <tjreedy@udel.edu> writes:
If Windows (or other OSes) (to be investigated) does not reliably come with a full unicode font (at least current BMP), is there a public domain or open license font that we can include?
The GNU Unifont at Unifoundry <URL:http://unifoundry.com/> is designed for this purpose. -- \ “For my birthday I got a humidifier and a de-humidifier. I put | `\ them in the same room and let them fight it out.” —Steven Wright | _o__) | Ben Finney
2) IDLE does much better but its support seems to still be imcomplete. Upgrade tk/tkinter/IDLE (wherever the problems lie) and make IDLE's shell an alternate UI.
That is certainly *no* good SoC project. Instead, just report it as a *specific* bug report (rather than saying "it seems incomplete"). Regards, Martin
participants (13)
-
"Martin v. Löwis" -
Aahz -
Antoine Pitrou -
Arc Riley -
Ben Finney -
Benjamin Peterson -
C. Titus Brown -
Daniel (ajax) Diniz -
Glenn Linderman -
Guilherme Polo -
INADA Naoki -
Nick Coghlan -
Terry Reedy