Mailman 3 May 2013 - Python-ideas

Trie ABC?
by Geoffrey Sneddon 07 May '13

07 May '13

Currently there are a large number of trie implementations for Python, all with slightly different APIs. It would be nice to introduce a ABC for Tries to attempt to unify these. Why do people want tries in Python? Typically because they provide quick lookup of prefixes and keys starting with a prefix. html5lib uses a pluggable trie based around a pseudo-ABC for tokenizing entities, for example (see below for links). It would likely make sense to introduce Trie and MutableTrie ABCs inheriting from Mapping and (MutableMapping, Trie) respectively. It is suggested to add at least two mixins: longest_prefix(prefix) returning the longest prefix of "prefix" that is a member of the Trie. keys_with_prefix(prefix) (or maybe keys_starting_with, but I'm bike-shedding myself now!) returning an iterable of keys that start with with prefix. Some implementations simply override keys for this, adding an optional first argument of prefix. Many implementations also include something like has_keys_with_prefix returning len(keys_with_prefix(prefix)) > 0. A selection of existing trie implementations for Python: https://github.com/kmike/datrie https://github.com/buriy/python-chartrie https://github.com/kmike/hat-trie https://github.com/dhain/trie I can attempt to summarize the various APIs these provide if there is any interest whatsoever (it seems like there's little use if it's decided against on principle!). It would be nice to get an implementation into the stdlib if people are in favour of an ABC, but given there's no real singular solution that has been proven in the field I'm not rushing to force this. <http://bugs.python.org/issue9520> was a case of trying to get a trie implementation into the stdlib. There's something vaguely along the lines of the above ABC that I implemented for html5lib in <https://github.com/html5lib/html5lib-python/blob/master/html5lib/trie/_base…>, as well as a light-weight pure-Python implementation in <https://github.com/html5lib/html5lib-python/blob/master/html5lib/trie/py.py>. /gsnedders

5 9

PEP: Extended stat_result (First Draft)
by Pieter Nagel 07 May '13

07 May '13

Following our discussion of last week, here is a first draft of the PEP PEP: XXX Title: Extended stat_result Version: $Revision$ Last-Modified: $Date$ Author: Pieter Nagel <pieter(a)nagel.co.za> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 03-May-2013 Python-Version: 3.4 Abstract ======== This PEP proposes extending the result of ``os.stat()``, ``os.fstat()`` and ``os.lstat()`` calls with added methods such as ``is_file()``. These added methods will obviate the need to use the ``stat`` module to interpret the result of these calls. Motivation ========== Currently, there are two different mechanisms for interrogating the file types of filesystem paths, each with distinctly different appearance and performance characteristics. The first mechanism is a set of functions in the module ``os.path``, such as ``os.path.isfile()`` and ``os.path.isdir()``. These functions’ names express their semantics relatively directly, but performance-wise each call entails an ``os.stat()`` call, which could potentially be redundant if another ``os.*stat()`` call had been done earlier for the same path in order to query other similar properties of the path. The second mechanism is by first calling ``os.stat()``, ``os.fstat()`` or ``os.lstat()`` (henceforth collectively referred to as just “``os.*stat()``”) for a particular path, and then interpreting the result using functions in the ``stat`` module. Performance-wise, these only require a single ``os.*stat()`` call, no matter how many times different properties of the result object are interrogated. But the downside is that the names of the functions needed to interrogate the result object, such as ``stat.S_ISREG()``, are relatively opaque, and motivated more by a desire to conform to standards for the names of the underlying C macros than by a desire to be semantically meaningful in English or to be Pythonic. There are situations where the performance penalty of ``os.*stat()`` calls can be significant enough to take into consideration. For example, on some networked filesystems they can be quite slow. Another consideration is that each call releases the GIL, which can also have negative performance effects especially on multi-threaded code. The end result of all this is that performance-agnostic code can be written in a relatively straightforward way:: if os.path.isfile(f) or os.path.isdir(f): # do something Whereas in contrast, similar code that wishes to avoid the penalty of two potential calls to ``os.stat()``, will look radically different:: st = os.stat(f) if stat.S_ISREG(st.st_mode) or stat.S_ISDIR(st.st_mode): # do something The cost is even worse if one takes into account that the second code fragment still needs to take the nonexistence of ``f`` into account in order to be completely semantically equivalent to the first, and also has the extra cost of needing to import the ``stat`` module. This PEP proposes ameliorating the situation by adding higher-level predicates such as ``is_file()`` and ``is_dir()`` directly to the ``stat_result`` object, so that (assuming the file ``f`` exists) the second code example can become:: st = os.stat(f) if st.is_file() or st.is_dir(): # do something Specification ============= Added methods on ``stat_result`` -------------------------------- is_dir() Equivalent to ``bool(stat.S_ISDIR(self.st_mode))``. is_character_device() Equivalent to ``bool(stat.S_ISCHR(self.st_mode))``. is_block_device() Equivalent to ``bool(stat.S_ISBLK(self.st_mode))``. is_file() Equivalent to ``bool(stat.S_ISREG(self.st_mode))``. is_fifo() Equivalent to ``bool(stat.S_ISFIFO(self.st_mode))``. is_symbolic_link() Equivalent to ``bool(stat.S_ISLNK(self.st_mode))``. is_socket() Equivalent to ``bool(stat.S_ISSOCK(self.st_mode)``. same_stat(other) Equivalent to ``os.path.samestat(self, other)``. file_mode() This shall return ``stat.filemode(stat.S_IMODE(self.st_mode))``, i.e. a string of the form ‘-rwxrwxrwx’. permission_bits() This shall return ``stat.S_IMODE(self.st_mode)``. format() This shall return ``stat.S_IFMT(self.st_mode)``. Added functions in ``os.path`` ------------------------------ is_dir(f) This shall be an alias for the existing isdir(f). is_character_device(f) This shall return ``os.stat(f).is_character_device()``, or ``False`` if ``f`` does not exist. is_block_device(f) This shall return ``os.stat(f).is_block_device()``, or ``False`` if ``f`` does not exist. is_file() This shall be an alias for the existing isfile(f). is_fifo() This shall return ``os.stat(f).is_fifo()``, or ``False`` if ``f`` does not exist. is_symbolic_link() This shall return ``os.stat(f).is_symbolic_link()``, or ``False`` if ``f`` does not exist. is_socket() This shall return ``os.stat(f).is_socket()``, or ``False`` if ``f`` does not exist. Rationale ========= The PEP is strongly motivated by a desire for symmetry between functions in ``os.path`` and methods on ``stat_result``. Therefore, for each predicate function in ``os.path`` that is essentially just an interrogation of ``os.*stat()``, given an existing path, the similarly-named predicate method on ``stat_result`` should have the exact same semantics. This definition does not cover the case where the path being interrogated does not exist. In those cases, predicate functions in ``os.path``, such as ``os.path.isfile()``, will return ``False``, whereas ``os.*stat()`` will raise FileNotFoundError even before any ``stat_result`` is returned that could have been interrogated. This renders considerations of how the proposed new predicates on ``stat_result`` could have been symmetrical with functions in ``os.path``, if their ``stat_result`` had existed, moot, and this PEP does not propose doing anything about the situation (but see `Open Issues`_ below). Secondly, this definition refers to ‘similarly-named’ predicates instead of ‘identically-named’ predicates, because the names in ``os.path`` pre-date PEP 8 [#PEP-8]_, and are not compliant with it. This PEP takes the position that it is better that the new predicate methods on ``stat_result`` be named in compliance with PEP 8 [#PEP-8]_ (i.e. ``is_file()``), than that they be precisely identical to the names in ``os.path`` (i.e ``isfile()``). Note also that PEP 428 [#PEP-428]_ also specifies PEP-8 compliant names such as ``is_file()`` for the exact same concepts, and if PEP 428 [#PEP-428]_ should be accepted, the issue would be even more pertinent. Lastly, this PEP takes the notion of symmetry as far as adding methods and aliases to the existing ``os.path`` in order to be symmetrical with the added behaviour on ``stat_result``. But the author is least strongly convicted of this latter point, and may be convinced to abandon it. Backwards Compatibility ======================= This PEP neither removes current behavior of ``stat_result``, nor changes the semantics of any current behavior. Likewise, it adds functions and aliases for functions to ``os.path``, but does not remove or change any existing ones. Therefore, this PEP should not cause any backwards incompatibilities, except in the rare and esoteric cases where code is dependent on the *nonexistence* of the proposed new names. It is not deemed important remain compatible with code that mistakenly holds the Python Standard Library to be closed for new additions. Open Issues =========== Whether it is more desirable for the proposed added methods’ names to follow PEP 8 [#PEP-8]_ (i.e. ``is_file()`` etc.), or to mirror the pre-existing names in ``os.path`` (i.e. ``isfile()`` etc.) is still open for debate. The existing attributes on ``stat_result`` follow the pattern ``st_*`` in conformance to the relevant POSIX names for the fields of the C-level ``stat`` structure. The new names for the behaviours proposed here do not contain such an ``st_`` prefix (nor could they, for that would suggest a conformance with ``stat`` structure names which do not exist in POSIX). But the resulting asymmetry of names is annoying. Should aliases for the existing ``st_*`` names be added that omit the ``st_`` prefix? This PEP does not address a higher-lever mechanism for exposing the owner/group/other read/write/execute permissions. Is there a need for this? This PEP does not address a higher-lever mechanism for exposing the of the underlying ``st_flags`` field. Is there a need for this? This PEP proposes aliases and methods to make ``os.path`` conform more to the added ``stat_result`` methods proposed here. But is the impedance mismatch between ``isfile`` and ``is_file`` really that much of an issue to warrant this? As it stands, this PEP does not address the asymmetry between the existing ``os.path.isfile()`` etc. functions and the new proposed mechanism in the case where the underlying file does not exist. There is a way to handle this, though: an optional flag could be added to ``os.*stat()`` that would return a null object implementation of ``stat_result`` whenever the file does not exist. Then that null object could return ``False`` to ``is_file()`` etc., That means that the following code would behave identically, even when the file ``f`` does not exist:: if os.path.isfile(f) or os.path.isdir(f): # do something st = os.stat(f, null_if_missing=True) if st.is_file() or st.is_dir(): # do something Would this be a useful mechanism? Rejected Proposals ================== It has been proposed [#filetype]_ that a mechanism be added whereby ``stat_result`` could return some sort of type code identifying the file type. Originally these type codes were proposed as strings such as 'reg', 'dir', and the like, but others suggested enumerations instead. The author rejected that proposal to keep the current PEP focused on ameliorating existing asymmetries rather than adding new behavior, but is not opposed to the notion in principle (assuming enums are used instead of strings). Experience with creating the reference implementation for this PEP may yet change the author's mind. Concerns have been raised [#isdoor]_ about platform-specific stat flags (such as S_ISDOOR on Solaris) that Python does not currently support, and which could be added as part of this proposal. The author has rejected such proposals, yet again in order to keep the PEP focused. The author may, yet again, be persuaded otherwise. References ========== .. [#PEP-8] PEP 8, Style Guide for Python Code , Rossum, Warsaw (http://www.python.org/dev/peps/pep-0008) .. [#PEP-428] PEP 428, The pathlib module -- object-oriented filesystem paths , Pitrou (http://www.python.org/dev/peps/pep-0428) .. [#filetype] http://mail.python.org/pipermail/python-ideas/2013-May/020378.html .. [#isdoor] http://mail.python.org/pipermail/python-ideas/2013-May/020378.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Pieter Nagel

20 68

Re: [Python-ideas] Implement `Executor.map_as_completed`
by Yannick Hold-Geoffroy 05 May '13

05 May '13

Hello, It may be interesting to highlight that multiprocessing handles this by providing a imap_unordered function<http://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.…> . I provide an interface compatible to the futures (PEP-3148<http://www.python.org/dev/peps/pep-3148/>) in my parallel framework (SCOOP <http://scoop.googlecode.com/>) and I also feel the need for a simple way to iterate through results as they are completed. I understand that the *map()* function should be as generic as possible and finer grained concurrency should be explicitly handled by the user, but as Mr. Rachum said, a shortcut between the finest granularity (user have to submit futures manually) and generic serial-like map could be of use (It would help write DRY code since this is a really common use case). I am not necessarily for the creation of a new function implicitly doing an unordered iteration, though. Since futures.map returns an iterator, I think we should provide users with a function that wraps the futures.map and allow users to explicitly call it for unordered iteration. This could be very intuitive -- Like allowing futures.as_completed to accept futures.map() output. This could be done using the object-oriented philosophy (by keeping some internal state linking the map result to their related future) or functional philosophy (by returning an object behaving like the result with the addition of an identifier to the future). Internally, the function could either implement a callback returning the value to the user as soon as results are available or call as_completed on their related futures. Either way (new function or new wrapper) would greatly simplify some usual use cases of concurrent or parallel programs using or inspired by PEP-3148. Have a nice day, Yannick Hold

1 0

global and nonlocal with atributes
by João Bernardo 05 May '13

05 May '13

Hi, I couldn't find whether this was proposed or not, but seems interesting to me: Whenever there's an assignment inside a big function, we think it is a local variable, but it could have been defined as global on top and that can mess things up if not checked. So, I why not have attribute syntax on "global" and "nonlocal" keywords? Something that is written like: x = 10 def increment(): global x x += 1 could be replaced by x = 10 def increment(): global.x += 1 another example: def foo(x): def bar(): return nonlocal.x + 1 return bar These should generate the same "LOAD_GLOBAL" and "LOAD_DEREF" bytecodes, so "global" and "nonlocal" will not be objects nor passed as arguments. That way you can have both "global.x", "nonlocal.x" and "x" variables without conflict. x = 10 def f(x): def g(): # how can I set both "x" variables inside here? "Namespaces are one honking great idea -- let's do more of those!" -- João Bernardo

7 12

Implement `Executor.map_as_completed`
by Ram Rachum 05 May '13

05 May '13

I suggest a combination of `futures.as_completed` and `futures.Executor.map`. I think it should be a method `futures.Executor.map_as_completed`. It'll be the same as `.map`, but yield the results according to the order of completion. What do you think? (Background: This Stack Overflow question<http://stackoverflow.com/questions/16276423/pythons-concurrent-futures-iter…> .)

1 0

Extend the os.stat() result objects with methods like isfile() and isdir()
by Pieter Nagel 03 May '13

03 May '13

I propose adding methods like isfile(), isdir(), islink(), isfifo() and so on - basically everything that would currently be done via code like "stat.S_ISREG(s.st_mode)". Please indicate support or not, so I can know whether to draft a PEP and work on implementation. My motivation is twofold: Firstly, it would make code that needs to interpret stat() results using the existing S_ISREG etc. methods in the stat module look cleaner, more Pythonic, and less like C code manipulating bitmasks. Secondly, in a recent discussion on python-dev [1] the issue was raised that the stat() call can perform badly under certain situations, and that some form of caching of the result of stat() calls is therefore desirable. This proposal makes it easier to do one form of caching stat() results: the kind where the result is manually cached by storing it in some variable. Think of code such as: if os.path.isfile(f) or os.path.isdir(f): # do something This will indirectly cause two calls to stat(). Currently, if you want to manually cache that stat call, you'll need to write: s = os.stat(f) if stat.S_ISREG(s.st_mode) or stat.S_ISDIR(s.st_mode): # do something This not only looks more convoluted and requires an extra import of stat, but it also looks wildly different from the previous code even though it basically has the same semantics. Under my proposal, this could become: s = os.stat(f) if s.isfile() or s.isdir(): # do something This proposal is independent of the current PEP 428 Path object proposal. However, if accepted, users of PEP 428 Path objects will also benefit, since those can also return results of stat() calls. -- Pieter Nagel

12 20

chunks implementation
by anatoly techtonik 02 May '13

02 May '13

16+ hours work. print(list( chunks('sadfdfa', 3) )) print(list( chunks(range(8), 3) )) print(list( chunks([1,2,3,4,5,7], 3) )) ['sad', 'fdf', 'a'] [[0, 1, 2], [3, 4, 5], [6, 7]] [[1, 2, 3], [4, 5, 7]] Is it good? -- anatoly t.

3 5

Personal views/filters (summaries) for discussions
by anatoly techtonik 01 May '13

01 May '13

I find it really hard to track proposals, ideas and various deviations in mailing lists, which is especially actual for lists such as python-ideas. I bet other people experience this problem too. The typical scenario: 1. You make a proposal 2. The discussion continues 3. Part of the discussion is hijacked 4. Another part brings the problem you haven't seen 5. You don't have time to investigate the problem 6. Discussion continues 7. Thread quickly gets out of scope of daily emails 8. Contact lost Several week later you remember about the proposal: 9. You open the original proposal to notice a small novel 10. You start to reread 11. Got confused 13. Recall the details 14, Find a way out from irrelevant deviation 15. Encounter the problem 16. Spend what is left to investigate the problem 17. Run out of time The major problem I have is steps 9-15. Sometimes these take the most of the time. What would help to make all the collaboration here more productive are colored view/filters (summaries) for discussions. It would work like so: 00. The discussion is laid out as a single page 01. You define some aspect of discussion (name the filter) 02. You mark text related to the aspect 03. You save the markings. 04. You insert summaries and TODOs 05. Now you select the aspect 06. Irrelevant parts are grayed out 07. Additionally you can collapse grayed sections An ability to edit and enhance these filters will allow to devote a small bits of free time to analyze and summarize the discussion state instead of requiring a single big piece to reread the whole discussion. This way you can split the task of dealing with complexity over time, which I think is more than actual nowadays. IMO this process can be very beneficial for Python development. -- anatoly t.

5 8

docstring in decorator?
by Peter Norvig 01 May '13

01 May '13

We've all gotten used to the conventions for docstrings laid out in PEP 257. But I got to wondering: if we had had decorators back in the 20th century, would docstrings be in a decorator? I found out that PEP 318 says "Guido felt ... it was quite possible that a 'docstring' decorator could help move the docstring to outside the function body." Is this a good idea? Compare: Current practice: def square(x): """ Compute the square of the number x. For example, square(3) == (3 * 3) == 9. """ return x * x Docstring in decorator: @doc("""Compute the square of the number x. For example, square(3) == (3 * 3) == 9.""") def square(x): return x * x Advantages of this approach: * Matches practice of other languages, such as Java and C++, where (by most conventions) comments precede the first line of the function definition. * If you have a short function and want to write the header and body on one line, you can, and still have a docstring. * At least for top-level functions, we don't need the complicated rules (24-line-long 'trim' function from PEP 257) to determine what to do with leading blanks, because there are no leading blanks. (For nested functions we still have the complication.) * The docstring can be an expression -- it need not be a literal string. For example, we could apply a localization/internationalization function to each string. (I know this could also be done by a tool that hunts through modules and looks for docstrings, but the decorator could do it at definition time without need of an external tool.) * The docstring could be several comma-separated parts which would be ' '.join-ed together, as in print, making it easier to construct an interesting docstring: @doc("Compute the square", version_str, author_str) * Alternately, doc could take keyword args and do a setattr for each one: @doc("Compute the square", version="1.1", author="GvR") Disadvantage: * If it ain't broke, don't fix it. Recommendation: I'm not proposing any change to Python; no proposal is necessary because if people like this they can easily implement it themselves. (I am interested in people's reactions.) Implementation: def doc(*docstring_parts, **kwargs): """Return a decorator that attaches a docstring.""" def f(fn): docstr = ' '.join(map(str, docstring_parts)) fn.__doc__ = docstr for attr in kwargs: setattr(fn, attr, kwargs[attr]) return fn return f

4 3

Biggest Fake Conference in Computer Science
by eliswilson＠hushmail.com 01 May '13

01 May '13

Biggest Fake Conference in Computer Science We are researchers from different parts of the world and conducted a study on the world’s biggest bogus computer science conference WORLDCOMP http://sites.google.com/site/worlddump1 organized by Prof. Hamid Arabnia from University of Georgia, USA. We submitted a fake paper to WORLDCOMP 2011 and again (the same paper with a modified title) to WORLDCOMP 2012. This paper had numerous fundamental mistakes. Sample statements from that paper include: (1). Binary logic is fuzzy logic and vice versa (2). Pascal developed fuzzy logic (3). Object oriented languages do not exhibit any polymorphism or inheritance (4). TCP and IP are synonyms and are part of OSI model (5). Distributed systems deal with only one computer (6). Laptop is an example for a super computer (7). Operating system is an example for computer hardware Also, our paper did not express any conceptual meaning. However, it was accepted both the times without any modifications (and without any reviews) and we were invited to submit the final paper and a payment of $500+ fee to present the paper. We decided to use the fee for better purposes than making Prof. Hamid Arabnia richer. After that, we received few reminders from WORLDCOMP to pay the fee but we never responded. This fake paper is different from the two fake papers already published (see https://sites.google.com/site/worlddump4 for details) in WORLDCOMP. We MUST say that you should look at the above website if you have any thoughts of participating in WORLDCOMP. DBLP and other indexing agencies have stopped indexing WORLDCOMP’s proceedings since 2011 due to its fakeness. See http://www.informatik.uni-trier.de/~ley/db/conf/icai/index.html for of one of the conferences of WORLDCOMP and notice that there is no listing after 2010. See Section 2 of http://sites.google.com/site/dumpconf for comments from well-known researchers about WORLDCOMP. The status of your WORLDCOMP papers can be changed from scientific to other (i.e., junk or non-technical) at any time. Better not to have a paper than having it in WORLDCOMP and spoil the resume and peace of mind forever! Our study revealed that WORLDCOMP is money making business, using University of Georgia mask, for Prof. Hamid Arabnia. He is throwing out a small chunk of that money (around 20 dollars per paper published in WORLDCOMP’s proceedings) to his puppet (Mr. Ashu Solo or A.M.G. Solo) who publicizes WORLDCOMP and also defends it at various forums, using fake/anonymous names. The puppet uses fake names and defames other conferences to divert traffic to WORLDCOMP. He also makes anonymous phone calls and threatens the critiques of WORLDCOMP (See Item 7 of Section 5 of above website). That is, the puppet does all his best to get a maximum number of papers published at WORLDCOMP to get more money into his (and Prof. Hamid Arabnia’s) pockets. Prof. Hamid Arabnia makes a lot of tricks. For example, he appeared in a newspaper to fool the public, claiming him a victim of cyber-attack (see Item 8 in Section 5 of above website). Monte Carlo Resort (the venue of WORLDCOMP for more than 10 years, until 2012) has refused to provide the venue for WORLDCOMP’13 because of the fears of their image being tarnished due to WORLDCOMP’s fraudulent activities. That is why WORLDCOMP’13 is taking place at a different resort. WORLDCOMP will not be held after 2013. The draft paper submission deadline is over but still there are no committee members, no reviewers, and there is no conference Chairman. The only contact details available on WORLDCOMP’s website is just an email address! We ask Prof. Hamid Arabnia to publish all reviews for all the papers (after blocking identifiable details) since 2000 conference. Reveal the names and affiliations of all the reviewers (for each year) and how many papers each reviewer had reviewed on average. We also ask him to look at the Open Challenge (Section 6) at https://sites.google.com/site/moneycomp1 and respond if he has any professional values. Sorry for posting to multiple lists. Spreading the word is the only way to stop this bogus conference. Please forward this message to other mailing lists and people. We are shocked with Prof. Hamid Arabnia and his puppet’s activities at http://worldcomp-fake-bogus.blogspot.com Search Google using the keyword worldcomp fake for additional links.

3 3