Mailman 3 July 2012 - Python-Dev

Re: [Python-Dev] cpython (2.7): Make it easier to search for the grouper() recipe.
by Georg Brandl 06 Jul '12

06 Jul '12

On 03.07.2012 06:09, raymond.hettinger wrote: > http://hg.python.org/cpython/rev/d32f21d87363 > changeset: 77921:d32f21d87363 > branch: 2.7 > parent: 77914:366df7ba1616 > user: Raymond Hettinger <python(a)rcn.com> > date: Mon Jul 02 21:08:45 2012 -0700 > summary: > Make it easier to search for the grouper() recipe. > > files: > Doc/library/itertools.rst | 3 ++- > 1 files changed, 2 insertions(+), 1 deletions(-) > > > diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst > --- a/Doc/library/itertools.rst > +++ b/Doc/library/itertools.rst > @@ -733,7 +733,8 @@ > return izip(a, b) > > def grouper(n, iterable, fillvalue=None): > - "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx" > + "Collect data into fixed-length chunks or blocks" > + # grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx" > args = [iter(iterable)] * n > return izip_longest(fillvalue=fillvalue, *args) Will you please apply this also to 3.x? Georg

1 0

GitHub mirror (Was: Bitbucket mirror?)
by anatoly techtonik 06 Jul '12

06 Jul '12

On Fri, Jun 29, 2012 at 6:58 AM, Eli Bendersky <eliben(a)gmail.com> wrote: > > The devguide (http://docs.python.org/devguide/committing.html) says: > > Bitbucket also maintain an up to date clone of the main cpython repository > that can be used as the basis for a new clone or patch queue. > > [the link goes to https://bitbucket.org/mirror/cpython/overview] This one is better than Atlassian's (looks nice, shorter URL) and worked long before Atlassian promotion as "makers of Bitbucket", which is dead wrong BTW, because everybody knows they bought BB. I don't remember if I commented on the Python Insider or if it was censored. Whatever.. it hurts Atlassian, not me. On the subject. Is there a mirror of CPython on GitHub? I abandoned my efforts to create online editor for docs, which should have been both web fronted independent, maintainable and clean. It appeared much easier to just use GitHub - so far it is the only one from open source code hosting services that directly commits online changes to repository (and allows anonymous to do this). I've made more than a dozen proposal for fixing docs, because as a matter of fact - filling a bug AND explaining why docs are wrong, why they need to be fixed, what should be added - all of this is a way *much easier* (and less time consuming!) than just fixing them. Unfortunately.

6 5

Summary of Python tracker Issues
by Python tracker 06 Jul '12

06 Jul '12

ACTIVITY SUMMARY (2012-06-29 - 2012-07-06) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3495 (+10) closed 23546 (+31) total 27041 (+41) Open issues with patches: 1462 Issues opened (32) ================== #10571: "setup.py upload --sign" broken: TypeError: 'str' does not sup http://bugs.python.org/issue10571 reopened by eric.araujo #15223: datetime instances lack __module__ attribute http://bugs.python.org/issue15223 opened by cjerdonek #15227: Fatal Python error: PyEval_RestoreThread: NULL tstate on examp http://bugs.python.org/issue15227 opened by tmaslach #15228: os.utime() docs not clear on behavior on nonexistant files http://bugs.python.org/issue15228 opened by dewin #15230: runpy.run_path broken: Breaks scoping; pollutes sys.modules; d http://bugs.python.org/issue15230 opened by Benjamin.S.Wolf #15231: update PyPI upload doc to say --no-raw passed to rst2html.py http://bugs.python.org/issue15231 opened by cjerdonek #15232: email.generator.Generator doesn't mangle "From " lines in MIME http://bugs.python.org/issue15232 opened by Chris.Pickett #15233: atexit: guarantee order of execution of registered functions? http://bugs.python.org/issue15233 opened by georg.brandl #15234: avoid runtime library path for extensions found in system dire http://bugs.python.org/issue15234 opened by doko #15235: allow newer berkley db versions http://bugs.python.org/issue15235 opened by doko #15237: Add capsule API to _decimal http://bugs.python.org/issue15237 opened by skrah #15238: shutil.copystat should copy Linux extended attributes http://bugs.python.org/issue15238 opened by larry #15239: Abandoned Tools/unicode/mkstringprep.py http://bugs.python.org/issue15239 opened by storchaka #15242: PyImport_GetMagicTag() should use the same const char * as sys http://bugs.python.org/issue15242 opened by eric.snow #15243: Misleading documentation for __prepare__ http://bugs.python.org/issue15243 opened by William.Schwartz #15244: Support for opening files with FILE_SHARE_DELETE on Windows http://bugs.python.org/issue15244 opened by sbt #15245: ast.literal_eval fails on some literals http://bugs.python.org/issue15245 opened by JBernardo #15246: Line coverage for collectionts.abc.Set http://bugs.python.org/issue15246 opened by agentultra #15247: io.open() is inconsistent re os.open() http://bugs.python.org/issue15247 opened by apalala #15248: In "TypeError: 'tuple' object is not callable", explain that a http://bugs.python.org/issue15248 opened by o11c #15249: email.generator.BytesGenerator doesn't mangle "From " lines wh http://bugs.python.org/issue15249 opened by petri.lehtinen #15250: document that filecmp.dircmp comparisons are "shallow" http://bugs.python.org/issue15250 opened by cjerdonek #15251: new.code and new.function crashes Python iterpretter http://bugs.python.org/issue15251 opened by jeffdharper #15255: spam / garbage report http://bugs.python.org/issue15255 opened by andisthermal555 #15256: Typo in error message http://bugs.python.org/issue15256 opened by Marc.Abramowitz #15257: Misc/.gdbinit:pystack is too brittle http://bugs.python.org/issue15257 opened by rian #15258: argparse documentation: Improve optparse section regarding all http://bugs.python.org/issue15258 opened by nejucomo #15259: "Helping with Documentation" references missing dailybuild.py http://bugs.python.org/issue15259 opened by cjerdonek #15260: Mention how to order Misc/NEWS entries http://bugs.python.org/issue15260 opened by cjerdonek #15262: Idle does not show traceback in other threads http://bugs.python.org/issue15262 opened by Mark #15263: Guard against invalid file handles in os functions http://bugs.python.org/issue15263 opened by larry #1207613: Bottom Scroll Bar http://bugs.python.org/issue1207613 reopened by terry.reedy Most recent 15 issues with no replies (15) ========================================== #15262: Idle does not show traceback in other threads http://bugs.python.org/issue15262 #15260: Mention how to order Misc/NEWS entries http://bugs.python.org/issue15260 #15259: "Helping with Documentation" references missing dailybuild.py http://bugs.python.org/issue15259 #15258: argparse documentation: Improve optparse section regarding all http://bugs.python.org/issue15258 #15257: Misc/.gdbinit:pystack is too brittle http://bugs.python.org/issue15257 #15256: Typo in error message http://bugs.python.org/issue15256 #15250: document that filecmp.dircmp comparisons are "shallow" http://bugs.python.org/issue15250 #15244: Support for opening files with FILE_SHARE_DELETE on Windows http://bugs.python.org/issue15244 #15234: avoid runtime library path for extensions found in system dire http://bugs.python.org/issue15234 #15230: runpy.run_path broken: Breaks scoping; pollutes sys.modules; d http://bugs.python.org/issue15230 #15228: os.utime() docs not clear on behavior on nonexistant files http://bugs.python.org/issue15228 #15227: Fatal Python error: PyEval_RestoreThread: NULL tstate on examp http://bugs.python.org/issue15227 #15201: C argument errors and Python arguments error are different http://bugs.python.org/issue15201 #15199: Default mimetype for javascript should be application/javascri http://bugs.python.org/issue15199 #15195: test_distutils fails when ARCHFLAGS is set on a Mac http://bugs.python.org/issue15195 Most recent 15 issues waiting for review (15) ============================================= #15257: Misc/.gdbinit:pystack is too brittle http://bugs.python.org/issue15257 #15250: document that filecmp.dircmp comparisons are "shallow" http://bugs.python.org/issue15250 #15249: email.generator.BytesGenerator doesn't mangle "From " lines wh http://bugs.python.org/issue15249 #15246: Line coverage for collectionts.abc.Set http://bugs.python.org/issue15246 #15245: ast.literal_eval fails on some literals http://bugs.python.org/issue15245 #15244: Support for opening files with FILE_SHARE_DELETE on Windows http://bugs.python.org/issue15244 #15242: PyImport_GetMagicTag() should use the same const char * as sys http://bugs.python.org/issue15242 #15239: Abandoned Tools/unicode/mkstringprep.py http://bugs.python.org/issue15239 #15238: shutil.copystat should copy Linux extended attributes http://bugs.python.org/issue15238 #15234: avoid runtime library path for extensions found in system dire http://bugs.python.org/issue15234 #15233: atexit: guarantee order of execution of registered functions? http://bugs.python.org/issue15233 #15231: update PyPI upload doc to say --no-raw passed to rst2html.py http://bugs.python.org/issue15231 #15220: Reduce parsing overhead in email.feedparser.BufferedSubFile http://bugs.python.org/issue15220 #15209: Re-raising exceptions from an expression http://bugs.python.org/issue15209 #15207: mimetypes.read_windows_registry() uses the wrong regkey, creat http://bugs.python.org/issue15207 Top 10 most discussed issues (10) ================================= #14243: tempfile.NamedTemporaryFile not particularly useful on Windows http://bugs.python.org/issue14243 15 msgs #3754: cross-compilation support for python build http://bugs.python.org/issue3754 12 msgs #3871: cross and native build of python for mingw32 with packaging http://bugs.python.org/issue3871 11 msgs #15202: followlinks/follow_symlinks/symlinks flags unification http://bugs.python.org/issue15202 11 msgs #15233: atexit: guarantee order of execution of registered functions? http://bugs.python.org/issue15233 11 msgs #15245: ast.literal_eval fails on some literals http://bugs.python.org/issue15245 11 msgs #15180: Cryptic traceback from os.path.join when mixing str & bytes http://bugs.python.org/issue15180 6 msgs #15198: multiprocessing Pipe send of non-picklable objects doesn't rai http://bugs.python.org/issue15198 6 msgs #1602: windows console doesn't print or input Unicode http://bugs.python.org/issue1602 5 msgs #3561: Windows installer should add Python and Scripts directories to http://bugs.python.org/issue3561 5 msgs Issues closed (28) ================== #1677: Ctrl-C will exit out of Python interpreter in Windows http://bugs.python.org/issue1677 closed by tim.golden #2345: Using an exception variable outside an 'except' clause should http://bugs.python.org/issue2345 closed by brett.cannon #4485: fast swap of "default" Windows python versions http://bugs.python.org/issue4485 closed by tim.golden #9239: zipfile: truncating comment can corrupt the zipfile http://bugs.python.org/issue9239 closed by pitrou #14330: don't use host python, use host search paths for host compiler http://bugs.python.org/issue14330 closed by doko #14591: Value returned by random.random() out of valid range on 64-bit http://bugs.python.org/issue14591 closed by python-dev #14902: test_logging failed http://bugs.python.org/issue14902 closed by python-dev #14967: distutils2.utils.resolve_name cannot be implemented to give co http://bugs.python.org/issue14967 closed by eric.araujo #15020: default value for progname in pythonrun.c should be python3 fo http://bugs.python.org/issue15020 closed by pitrou #15030: PyPycLoader can't read cached .pyc files http://bugs.python.org/issue15030 closed by brett.cannon #15033: Different exit status when using -m http://bugs.python.org/issue15033 closed by orsenthil #15141: IDLE horizontal scroll bar missing (Win-XPsp3) http://bugs.python.org/issue15141 closed by terry.reedy #15166: Implement imp.get_tag() using sys.implementation http://bugs.python.org/issue15166 closed by brett.cannon #15174: amd64\python_d.exe -m test fails http://bugs.python.org/issue15174 closed by skrah #15210: importlib.__init__ checks for the wrong exception when looking http://bugs.python.org/issue15210 closed by brett.cannon #15212: Rename SC_GLOBAL_EXPLICT to SC_GLOBAL_EXPLICIT in compiler mod http://bugs.python.org/issue15212 closed by pitrou #15224: Range: Additional Methods (min/max/__and__) http://bugs.python.org/issue15224 closed by mark.dickinson #15225: Add negative tests for passing str to hmac.HMAC and hmac.new http://bugs.python.org/issue15225 closed by pitrou #15226: max( str ) should be fast with PEP 393 http://bugs.python.org/issue15226 closed by loewis #15229: stringification of subclasses of OSError can cause crash http://bugs.python.org/issue15229 closed by pitrou #15236: SEGFAULT in visit_decref http://bugs.python.org/issue15236 closed by amaury.forgeotdarc #15240: ctype Structure keeps reference to function pointers http://bugs.python.org/issue15240 closed by vpelletier #15241: venv module and pyvenv script documented, but not python behav http://bugs.python.org/issue15241 closed by python-dev #15252: Delivery reports about your e-mail http://bugs.python.org/issue15252 closed by benjamin.peterson #15253: xmlrpclib.ServerProxy does not support 2-tuple value for uri p http://bugs.python.org/issue15253 closed by loewis #15254: 08 is invalid token in lists. http://bugs.python.org/issue15254 closed by tim.golden #15261: os.stat(fd) crashes on Windows if fd does not exist http://bugs.python.org/issue15261 closed by sbt #798876: windows sys.path contains nonexistant directory http://bugs.python.org/issue798876 closed by tim.golden

1 0

Re: [Python-Dev] [Python-checkins] peps: specify Provides-Extra
by Éric Araujo 06 Jul '12

06 Jul '12

Le 06/07/2012 10:36, benjamin.peterson a écrit : > http://hg.python.org/peps/rev/4dc72bbf9ed2 > changeset: 4483:4dc72bbf9ed2 > user: Daniel Holth<dholth(a)fastmail.fm> > date: Wed Jul 04 07:30:08 2012 -0400 > summary: > specify Provides-Extra Sorry, but this was premature. The mail on distutils-sig only describes the syntax for a new field without any explanation of the semantics and use case, not the relation to the Setup-Requires and Test-Requires fields that we’ve been discussing for some time (and which would make part of the Extra proposal obsolete). I know Alexis and I are not as active as other devs in mailing lists and PEPs, but these things are on our radar and we want to push them forward. Please allow the discussion to take place before editing the packaging PEPs. Regards

1 0

EuroPython 2012 Language Summit is Canceled.
by Larry Hastings 06 Jul '12

06 Jul '12

I only got one more RSVP and zero topics for the docket. So let's sprint instead. See you at the PyCon 2013 Language Summit, //arry/

1 0

Re: [Python-Dev] Bloody FAQ (Was: [Python-ideas] itertools.chunks(iterable, size, fill=None))
by Paul Boddie 05 Jul '12

05 Jul '12

Stefan Behnel wrote: > anatoly techtonik, 05.07.2012 21:41: > > > > Make the bloody FAQ and summarize this stuff? Why waste each others > > time? > > Yes, that is exactly the question. > > It takes time to write things up nicely. I mean, once someone has pointed > out to you that this has been discussed before, you could just go, look it > up (or search for it), and then put it into a Wiki or blog post yourself, > or sum it up and send it to the mailing list as a reply. Why rely on others > to do it for you? To be fair, Anatoly has done quite a bit of maintenance on some of the Wiki content around various aspects of the project, so it's not as if he's demanding anything out of the ordinary or asking for others to do things that he isn't already doing in some sense. My experience is that there usually needs to be some willingness on the other end of the transaction, and if it takes repetition to encourage it amongst those who don't see the current situation as a problem for them, then so be it. Of course, this kind of documentation activity, where one gathers together historical decisions and the consensus from long-forgotten discussions, is pretty thankless work. I occasionally regard it as worthwhile if only to bring up something someone said as an inconvenient interruption in any current discussion, but that's a pretty minimal reward for all the effort unless one has such work as part of one's daily routine. Paul

1 0

Bloody FAQ (Was: [Python-ideas] itertools.chunks(iterable, size, fill=None))
by anatoly techtonik 05 Jul '12

05 Jul '12

On Thu, Jul 5, 2012 at 7:50 PM, Stefan Behnel <stefan_ml(a)behnel.de> wrote: > anatoly techtonik, 05.07.2012 15:36: >> On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy wrote: >>> From Raymond's first message on http://bugs.python.org/issue6021 , add >>> grouper: >>> >>> "This has been rejected before. >> >> I quite often see such arguments and I can't stand to repeat that >> these are not arguments. It is good to know, but when people use that >> as a reason to close tickets - that's just disgusting. > > The *real* problem is that people keep bringing up topics (and even spell > them out in the bug tracker) without searching for existing discussions > and/or tickets first. That's why those who do such a search (or who know > what they are talking about anyway) close these tickets with the remark > "this has been rejected before", instead of repeating an entire heap of > arguments all over again to feed a discussion that would only lead to the > same result as it did before, often several times before. Make the bloody FAQ and summarize this stuff? Why waste each others time? If people don't enjoy repeating themselves over and over - there is a bloody wiki. What should happen to people to start extracting gems of knowledge from piles of dusty sheets called list "archives" for others to admire. No, it is easier to say "it was already discussed many times", "why don't you Google yourself", "so far you're only complaining", etc. If people can't find anything - why everybody thinks they are ignorant and lazy. Even if it so, why nobody thinks that maybe that bloody Xapian index is dead again for a bloody amount of moons nobody knows why and how many exactly? Why nobody thinks that lazy coders can also help with development? Maybe that laziness is the primary reason some major groups actually prefer Python to Java, C++ and other more interesting languages (such as PHP) when it comes to typing? Make it easy and the patches will follow. Answers like "this was discussed before" don't make it easy to understand, and leaving users rereading old 19xx archives that people don't reread themselves will likely make users bounce and never (NEVER!) come up with some proposal again. An "organic" way to keep traffic low. Miscommunication is a bad experience for users, bad experience for developers, everybody is annoyed and as a result such nice language as Python loses points on TIOBE (and convenient chunk() functions to munch-munch on the sequence data). Wheew. :-F

3 2

Re: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size, fill=None)
by anatoly techtonik 05 Jul '12

05 Jul '12

Before anything else I must apologize for significant lags in my replies. I can not read all of them to hold in my head, so I reply one by one as it goes trying not to miss a single point out there. It would be much easier to do this in unified interface for threaded discussions, but for now there is no capabilities for that neither in Mailman nor in GMail. And when it turns out that the amount of text is too big, and I spend a lot of time trying to squeeze it down and then it becomes pointless to send at all. Now back on the topic: On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy <tjreedy(a)udel.edu> wrote: > On 6/29/2012 4:32 PM, Georg Brandl wrote: >> >> On 26.06.2012 10:03, anatoly techtonik wrote: >>> >>> Now that Python 3 is all about iterators (which is a user killer >>> feature for Python according to StackOverflow - >>> http://stackoverflow.com/questions/tagged/python) would it be nice to >>> introduce more first class functions to work with them? One function >>> to be exact to split string into chunks. > > Nothing special about strings. It seemed so, but it just appeared that grouper recipe didn't work for me. >>> itertools.chunks(iterable, size, fill=None) > > This is a renaming of itertools.grouper in 9.1.2. Itertools Recipes. You > should have mentioned this. I think of 'blocks' rather than 'chunks', but I > notice several SO questions with 'chunk(s)' in the title. I guess `block` gives too low signal/noize ration in search results. That's why it probably also called chunks in other languages, where `block` stand for something else (I speak of Ruby blocks). >>> Which is the 33th most voted Python question on SO - >>> >>> http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-even… > > I am curious how you get that number. I do note that there are about 15 > other Python SO questions that seem to be variations on the theme. There > might be more if 'blocks' and 'groups' were searched for. It's easy: 1. Go http://stackoverflow.com/ 2. Search [python] 3. Click `votes` tab 4. Choose `30 per page` at the bottom 5. Jump to the second page, there it is 4th from the top: http://stackoverflow.com/questions/tagged/python?page=2&sort=votes&pagesize… As for duplicates - feel free to mark them as such. SO allows everybody to do this (unlike Roundup). >> Anatoly, so far there were no negative votes -- would you care to go >> another step and propose a patch? > > That is because Raymond H. is not reading either list right now ;-) > Hence the Cc:. Also because I did not yet respond to a vague, very > incomplete idea. > > From Raymond's first message on http://bugs.python.org/issue6021 , add > grouper: > > "This has been rejected before. I quite often see such arguments and I can't stand to repeat that these are not arguments. It is good to know, but when people use that as a reason to close tickets - that's just disgusting. To the Raymond's honor he cares to explain. > * It is not a fundamental itertool primitive. The recipes section in > the docs shows a clean, fast implementation derived from zip_longest(). What is the definition of 'fundamental primitive'? To me the fact that top answer for chunking strings on SO has 2+ times more votes than itertools versions is a clear 5 sigma indicator that something is wrong with this Standard model without chunks boson. > * There is some debate on a correct API for odd lengths. Some people > want an exception, some want fill-in values, some want truncation, and > some want a partially filled-in tuple. The alone is reason enough not > to set one behavior in stone. use case 3.1: odd lengths exception (CHOOSE ONE) 1. I see that no itertools function throws exceptions, check manually: len(iterable) / float(size) == len(iterable) // float(size) 2. Explicitly - itertools.chunks(iterable, size, fill=None) + itertools.chunks(iterable, size, fill=None, exception=False) use case 3.2. fill in value. it is here (SOLVED) use case 3.3: truncation no itertools support truncation, do manually chunks(iter, size)[:len(iter)//size) use case 4: partially filled-in tuple What should be there? >>> chunks('ABCDEFG', 3, 'x') >>> | More replies and workarounds to some of the raised points are below. > * There is an issue with having too many itertools. The module taken as > a whole becomes more difficult to use as new tools are added." There can be only two reasons to that: * chosen basis is bad (many functions that are rarely used or easily emulated) * basis is good, but insufficient, because iterators universe is more complicated than we think > This is not to say that the question should not be re-considered. Given the > StackOverflow experience in addition to that of the tracker and python-list > (and maybe python-ideas), a special exception might be made in relation to > points 1 and 3. --[offtopic about Python enhancements / proposals feedback]-- Yes, without SO I probably wouldn't trigger this at all. Because tracker doesn't help with raising importance - there are no votes, no feature proposals, no "stars". And what I "like" the most is that very "nice" resolution status - "committed/rejected" - which doesn't say anything at all. Python list? I try not to disrupt the frequency there. Python ideas? Too low participation level for gathering signals. There are many people that read, support, but don't want to reply (don't want to stand out or just lazy). There are many outside who don't want to be subscribed at all. There are 2000+ people spending time on Python conferences all over the world each year we see only a couple of reactions for every Python idea here. Quite often there are mistakes and omissions that would be nice to correct and you can't. So StackOverflow really helps here, but it is a Q&A tool, which is still much better than ML that are solely for chatting, brainstorming and all the crazy reading / writing stuff. They don't help to develop ideas collaboratively. Quite often I am just lost in amount of text to handle. --[/offtopic]-- > It regard to point 2: many 'proposals', including Anatoly's, neglect this > detail. But the function has to do *something* when seqlen % grouplen != 0. > So an 'idea' is not really a concrete programmable proposal until > 'something' is specified. > > Exception -- not possible for an itertool until the end of the iteration > (see below). To raise immediately for sequences, one could wrap grouper. > > def exactgrouper(sequence, k): # untested > if len(sequence) % k: > raise ValueError('Sequence length {} must be a multiple of group length > {}'.format(len(sequence), k) > else: > return itertools.grouper(sequence, k) Right. Iterator is not a sequence, because it doesn't know the length of its sequence. The method should not belong to itertools at all then. Python 3 is definitely become more complicated. I'd prefer to keep separated from iterator stuff, but it seems more harder with every iteration. > Of course, sequences can also be directly sequentially sliced (but should > the result be an iterable or sequence of blocks?). But we do not have a > seqtools module and I do not think there should be another method added to > the seq protocol. I'd expect strings chunked into strings and lists into lists. Don't want to know anything about protocols. > Fill -- grouper always does this, with a default of None. > > Truncate, Remainder -- grouper (zip_longest) cannot directly do this and no > recipes are given in the itertools docs. (More could be, see below.) > > Discussions on python-list gives various implementations either for > sequences or iterables. For the latter, one approach is "it = > iter(iterable)" followed by repeated islice of the first n items. Another is > to use a sentinal for the 'fill' to detect a final incomplete block (tuple > for grouper). > > def grouper_x(n, iterable): # untested > sentinal = object() > for g in grouper(n, iterable, sentinal): > if g[-1] != sentinal: > yield g > else: > # pass to truncate > # yield g[:g.index(sentinal) for remainer > # raise ValueError for delayed exception We need a simple function to split a sequence into chunks(). Now we face with the problem to apply that technique to a sequence of infinite length when a last element of infinite sequence is encountered. You might be thinking now that this is a reduction to absurdity. But I'd say it is an exit from the trap. Mathematically this problem can't be solved. I am not ignoring your solution - I think it's quite feasible, but isn't it an overcomplication? I mean 160 people out of 149 who upvoted the question are pretty happy with an answer that just outputs the last chunk as-is: http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-even… chunks('ABCDEFG', 3) --> 'ABC' 'DEF' 'G' And it is quite nice solution to me, because you're free to do anything you'd like if you expect you data to be odd: for chunk in chunks('ABCDEFG', size): if len(chunk) < size: raise Tail You can make a helper iterator out of it too. > --- > The above discussion of point 2 touches on point 4, which Raymond neglected > in the particular message above but which has come up before: What are the > allowed input and output types? An idea is not a programmable proposal until > the domain, range, and mapping are specified. Domain? Mapping? I am not ignoring existing knowledge and experience. I just don't want to complicate and don't see appropriate `import usecase` in current context, so I won't try to guess what this means. in string -> out list of strings in list -> out list of lists > Possible inputs are a specific sequence (string, for instance), any > sequence, any iterable. Possible outputs are a sequence or iterator of > sequence or iterator. The various python-list and stackoverflow posts > questions asks for various combinations. zip_longest and hence grouper takes > any iterable and returns an iterator of tuples. (An iterator of maps might > be more useful as a building block.) This is not what one usually wants with > string input, for instance, nor with range input. To illustrate: Allright. Got it. Sequences have a length and can be sliced with [i:j], iterator can't be sliced (and hence no chunks can be made). So this function doesn't belong to itertools - it is a missing string or sequence method. We can't have a chunk with an iterator, because iterator over a string decomposes it into a group of pieces with no reverse function. We can have a group and then join the group into something. But this requires the knowledge of appropriate join() function for the iterator, and probably not efficient. As there are no such function (must be that Mapping you referenced above) - the recomposition into chunks is impossible. > import itertools as it > > def grouper(n, iterable, fillvalue=None): > "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx" > args = [iter(iterable)] * n > return it.zip_longest(*args, fillvalue=fillvalue) > > print(*(grouper(3, 'ABCDEFG', 'x'))) # probably not wanted > print(*(''.join(g) for g in grouper(3, 'ABCDEFG', 'x'))) > # > ('A', 'B', 'C') ('D', 'E', 'F') ('G', 'x', 'x') > ABC DEF Gxx > > -- > What to do? One could easily write 20 different functions. So more thought > is needed before adding anything. -1 on the idea as is. I've learned a new English type of argument - "straw man" (I used to call this "hijacking"). This -1 doesn't belong to original idea. It belongs to proposal of itertools.chunks() with a long list of above points and completely different user stories (i.e. not "split string into chunks"). I hope you still +1 with 160 people on SO that think Python needs an easy way to chunk sequences. > For the doc, I think it would be helpful here and in most module subchapters > if there were a subchapter table of contents at the top (under 9.1 in this > case). Even though just 2 lines here (currently, but see below), it would > let people know that there *is* a recipes section. After the appropriate > tables, mention that there are example uses in the recipe section. Possibly > add similar tables in the recipe section. Unfortunately, it appeared that grouper() is not chunks(). It doesn't delivers list of list of chars given string as an input instead of list of chunks. > Another addition could be a new subsection on grouping (chunking) that would > discuss post-processing of grouper (as discussed above), as well as other > recipes, including ones specific to strings and sequences. It would > essentially be a short how-to. Call it 9.1.3 "Grouping, Blocking, or > Chunking Sequences and Iterables". The synonyms will help external > searching. A toc would let people who have found this doc know to look for > this at the bottom. This makes matters pretty ugly. In ideal language there should be less docs, not more.

4 3

Re: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size, fill=None)
by anatoly techtonik 05 Jul '12

05 Jul '12

On Fri, Jun 29, 2012 at 11:32 PM, Georg Brandl <g.brandl(a)gmx.net> wrote: > On 26.06.2012 10:03, anatoly techtonik wrote: >> >> Now that Python 3 is all about iterators (which is a user killer >> feature for Python according to StackOverflow - >> http://stackoverflow.com/questions/tagged/python) would it be nice to >> introduce more first class functions to work with them? One function >> to be exact to split string into chunks. >> >> itertools.chunks(iterable, size, fill=None) >> >> Which is the 33th most voted Python question on SO - >> >> http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-even… >> >> P.S. CC'ing to python-dev@ to notify about the thread in python-ideas. >> > > Anatoly, so far there were no negative votes -- would you care to go > another step and propose a patch? Was about to say "no problem", but in fact - there is. Sorry from whining from my side and thanks for nudging. The only thought that a simple task of copy/pasting relevant code from http://docs.python.org/library/itertools.html?highlight=itertools#recipes will require a few hours waiting of download (still not everybody has a high-speed internet) makes me switch to other less time consuming tasks before getting around to it. These tasks become more important in a few hours, and basically I've passed through this many times before. It then becomes quite hard to switch back. I absolutely don't mind someone else being credited for the idea, because ideas usually worthless without implementation. It will be interesting to design how the process could work in a separate thread. For now the best thing I can do (I don't risk even to mention anything with 3.3) is to copy/paste code from the docs here: from itertools import izip_longest def chunks(iterable, size, fill=None): """Split an iterable into blocks of fixed-length""" # chunks('ABCDEFG', 3, 'x') --> ABC DEF Gxx args = [iter(iterable)] * size return izip_longest(fillvalue=fill, *args) BTW, this doesn't work as expected (at least for strings). Expected is: chunks('ABCDEFG', 3, 'x') --> 'ABC' 'DEF' 'Gxx' got: chunks('ABCDEFG', 3, 'x') --> ('A' 'B' 'C') ('D' 'E' 'F') ('G' 'x' 'x') Needs more round tuits definitely.

4 4

Re: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size, fill=None)
by Raymond Hettinger 05 Jul '12

05 Jul '12

On Jun 30, 2012, at 10:44 PM, Stefan Behnel wrote: >> >> Another addition could be a new subsection on grouping (chunking) that >> would discuss post-processing of grouper (as discussed above), as well as >> other recipes, including ones specific to strings and sequences. It would >> essentially be a short how-to. Call it 9.1.3 "Grouping, Blocking, or >> Chunking Sequences and Iterables". The synonyms will help external >> searching. A toc would let people who have found this doc know to look for >> this at the bottom. > > If it really is such an important use case for so many people, I agree that > it's worth special casing it in the docs. It's not a trivial algorithmic > step from a sequential iterable to a grouped iterable. I'm not too keen on adding a section like this to the itertools docs. Instead, I would be open adding "further reading" section with external links to interesting iterator writeups in blogs, cookbooks, stack overflow answers, wikis, etc. If one of you wants to craft an elegant blog post on "Grouping, Blocking, or Chunking Sequences and Iterables", I would be happy to link to it. Raymond

3 3