Mailman 3 January 2013 - Python-ideas

PEP 3156 / Tulip question: write/send callback/future
by Eli Bendersky Jan. 19, 2013

Jan. 19, 2013

Hi, I'm looking through PEP 3156 and the Tulip code, and either something is missing or I'm not looking in the right places. I can't find any sort of callback / future return for asynchronous writes, e.g. in transport. Should there be no "data_sent" parallel to "data_received" somewhere? Or, alternatively, "write" returning some sort of future that can be checked later for status? For connections that aren't infinitely fast it's useful to know when the data was actually sent/written, or alternatively if an error has occurred. This is also important for when writing would actually block because of full buffers. boost::asio has such a handler for async_write. Eli

3 4

Re: [Python-ideas] Regarding 'const' and 'require' statements
by Harding, James Jan. 19, 2013

Jan. 19, 2013

Thank you everyone for your comments. I wish to retract my idea due to some killer issues. First, if a module had platform dependencies in defining constants, my scheme would not be able to handle that. The idea fails because many common situations would not work. Python should work for all. Second, this scheme would require some sort of time stamping of files where constants were taken from in order to see if a re-compile is necessary. The time needed for time-stamp checks would likely be more than any time saved by using constant names. Now, back to the shadows. Thank you, James Harding

1 0

Regarding 'const' and 'require' statements
by Harding, James Jan. 18, 2013

Jan. 18, 2013

Hello, There are so many replies that I am going to try and summarize responses with a lot of cut and paste in one post. Sorry if this is the wrong way to do it. > Do you have some concrete Python code which would clearly be improved by this proposal? Let me explain myself. I am a low-level programmer fascinated by Python's elegant syntax and how it is executed. We actually do little Python programming here but we do allow interaction between Python and our product and so I am not able to show any concrete code. I guess that makes me a crank but I am fine with that. At a low level, I look at what Python has to go through to execute statements and thoughts swirl through my mind as to how it could be improved. I finally cracked and made a post here with one of those improvements. >> const ST_MODE = 0 >> So, the compiler will ?replace any use of the identifier with? the constant value. Yes, the compiler will replace any use of the name with its value. A statement like: If c == ST_MODE: Would be treated by the compiler at compile-time as if it had seen: If c == 0: The name ST_MODE in this example is not a bindable name. The name only lives during compilation and is not accessible at run-time. It would not be stored in a dictionary (unless the magic syntax 'require module as *' were used that only confuses what I am trying to say). > name_prefix = "ST_" > foo = globals().get(name_prefix + "MODE") > > What do you expect the compiler to do in the above code? Since the name is not accessible at run-time, the above would produce an exception. Const names are only available at compile-time. > If I'd written his proposal I'd have probably termed these things "bind-once", generating names that may not be rebound. They would still > need to be carefully placed if the compiler were to have the option of constant folding i.e. they're need to be outside function and class > definitions, determinable from static analysis. These are "bind-never" names. The compiler would have to be able to see the definition when a module that uses them is being compiled. That is the reason for the require statement. The compiler does not normally look at the contents of other modules when parsing a source file. The require statement tells it to do so. >> The expression would be restricted to result in an immutable object >What is the purpose of this restriction? My thought is that a constant name should have the same value regardless of context. If I were to say something like "const A = B" then A is no longer a constant and when substituted depends on how B is interpreted within the current context (is it a global? A local? A nonlocal?). If I were to say "const A = [1,2,3]" then you need to worry about side effects. You would have to entirely clone the value at compile-time for each use rather than simply incrementing the reference count. >Is that the driving use-case for your suggestion? Compile-time efficiency? >If so, then I suspect that you're on the wrong track. As I understand it, >the sort of optimizations that PyPy can perform at runtime are far more >valuable than this sort of constant substitution. That is my basic track. If the existing tools handle this better than my idea should be discarded as not providing any significant improvement and adding additional baggage to the language. >k = ("Some value", "Another value") # for example >x = k >y = k >assert x is y # this always passes, no matter the value of k > >But if k is a const, it will fail, because the lines "x = k" and "y = k" >will be expanded at compile time: The restriction that constant names be immutable objects would allow their values to be placed in the constant pool for the function. In the above, if 'k' were a constant name then it would (hopefully) reside in a single location in the constant pool and the assignments to 'x' and 'y' would access the same constant pool location. >Another question: what happens if the constant expression can't be >evaluated until runtime? Constant expressions would be restricted to be compile-time constants. They would not be evaluated at run-time. >Compiler-enforced immutability is one of those really hard problems which, >if you manage to do flexibly and correctly, would be an academically >publishable result, not something you hack into the interpreter over a >weekend. I have to plead guilty here. I am not an academic and do not know all the implications of things. I do not follow research either and so am basically proposing this as a crank/hacker sort of person. >- multi-stage computations, so the program is partially-evaluated at >"compile" time and the `const` sections computed. This is also really hard. >Furthermore, if you want to be able to use bits of the standard library in >the early stages (you probably do, e.g. for things like min, max, len, >etc.) either you'd need to manually start annotating huge chunks of the >standard library to be available at "compile" time (a huge undertaking) or >you'll need an effect-tracking-system to do it for you. This is indeed a big worry. I would have had it such that a module could (but not required in any sense) be split into two parts. One part that is referenced at run-time using the existing import mechanism. This would not change. The second part of a module would be constants (and only constants) that are referenced at compile-time. There would be no requirement that modules change over to this new method. It would just mean that constants defined in the module are available to the compiler. That last statement is apparently not exactly true if I understand the comments about what PyPy optimizations do. The idea of the compiler accessing the source files for other modules does give me pause. Currently, compiling one module is fairly disjoint from other modules in that a change to one module does not require a re-compile of modules that use it even if 'constants' are changed. This is a good feature of Python and maybe something to boast about but I would worry if these ideas introduced bad practices. I don't think that there have been many cases of Python programmers saying: "I made a change to my module - you need to recompile your module to get the changes". I would like to thank you all for critiquing my ideas and pointing out its flaws with patience and respect. In many ways this was just an exercise in getting things off my chest because in the end I am just a crank. Thank you, James Harding

2 1

'const' and 'require' statements
by Harding, James Jan. 18, 2013

Jan. 18, 2013

Hello, I am new here but am itching with an idea. Here are two separate ideas but they are related so they shall both be presented at the same time. The first idea is for a 'const' statement for declaring constant names. Its syntax would be: 'const' identifier '=' expression The expression would be restricted to result in an immutable object such as 17, "green", or (1,2,3). The compiler would effectively replace any use of the identifier with this expression when seen. Some examples of constants might include: const ST_MODE = 0 const FontName = "Ariel" const Monday = 1 const Tuesday = Monday + 1 # may use previously defined const in expression. Compiler will fold constants (hopefully) Constant names would be limited in scope. A constant defined in a function would only have a life to the end of the function, for instance. Now why should there be such a syntax if the existing language already has a mechanism for effectively declaring constants, which it does? First, it opens possibilities for the compiler to do things like more constant folding and generally producing more efficient code. Second, since the compiler substitutes for the name at compile time, there is no chance for the name to be stepped on at run-time. Third, ideas such as PEP 3103 could be re-visited. One of the problems in PEP 3103 was that so often constants are represented by names and those names may be changed and/or the constant values in those names are not known until run-time. Constant names are fine but of limited use if they may only be used within the module they are declared in. This brings up the second idea of a 'require' statement. The import statement of Python is executed at run-time. This creates a disconnect between modules at compile time (which is a good thing) but gives the compiler no hint as to how to produce better code. What I propose is a 'require' statement with almost exactly the same syntax as the import and from statements but with the keyword 'require' substituted for 'module'. The word require was chosen because the require declaration from the BLISS language helped inspire this idea. C minded people might prefer using a word such as include be used instead. What the require statement would do is cause the module to be read in by the compiler and compiled when the statement is parsed. The contents of a required module could be restricted to only be const statements in order to avoid the many headaches this would produce. Examples: require font_data from stat_constants require ST_MODE from weekdays require * In the first example, the name 'font_data' would be a constant module to the compiler. An expression such as font_data.FontName would at compile-time reference the constant name FontName from the font_data module and substitute for it. In the second example, the constant name ST_MODE is added to the current scope. In the third example, all constant names defined in the module (except those with a '_' prefix) are added to the current scope. Since the names added are constant names and not variable names, it is OK to use require * at the function scope level. In order to help compatibility with existing uses and to avoid declaring constants twice, a require statement could use a 'as *' to both include constant names and assign them to a module's dictionary. For example, the file stat.py might do something like: require stat_constants as * This would add all the constant names defined in the stat_constants module and place them in the stat module's dictionary. For instance, if there is the line in stat_constant.py: const ST_MODE = 0 Then for stat.py the compiler will act as if it saw: ST_MODE = 0 Well, those are my two bits of ideas. Thank you, James Harding

11 14

Fwd: Boolean behavior of None
by Ilkka Pelkonen Jan. 17, 2013

Jan. 17, 2013

Hi all, I ran into an issue in expression evaluation with Python for Windows 2.7.3. Consider the following code: expected_result = (expected_string != 'TRUE') # Boolean element = find_element() # Can return None or an instance of Element flag = (element and element.is_visible()) if flag == expected_result: ..# Ok ..return # Otherwise perform some failure related stuff. This code does not work. What happens on the 'flag' assignment row, is that if 'element' is None, the expression returns None, not False. This makes the if comparison to fail if expected_result is False, since boolean False is not None. To me as a primarily C++ programmer it seems there could be two different changes here, either change the behavior of the 'and' expression, forcing it to return Boolean even if the latter part is not evaluated, and/or make the comparison "False == None" return True. Although potentially complex, I'd myself go for the first approach. It seems to me more logical that False != None than an 'and' expression returning non-boolean. Also the latter change might require people change their code, while the former should not require any modifications. This behavior probably results in lots of errors when people like me, used to more traditional languages, take on Python in a serious manner. I like the concept 'pythonic', and am trying to apply it to practice like above. Hoping to hear your thoughts, Regards, Ilkka Pelkonen

7 10

question about the Tulip effort
by Eli Bendersky Jan. 16, 2013

Jan. 16, 2013

Hi, I've been reading PEP 3156 and looking at the reference implementation ( http://code.google.com/p/tulip/). I'll be happy to contribute to the effort, and following are a couple of questions on how to do that. 1. Questions and clarifications should be sent to this list (python-ideas), correct? 2. Is there a list of tasks help would be needed with? Is it the the TODO file in tulip's root dir? 3. How/where to contribute patches? Eli

2 3

The async API of the future
by Guido van Rossum Jan. 16, 2013

Jan. 16, 2013

Work priorities don't allow me to spend another day replying in detail to the various emails on this topic, but I am still keeping up reading! I have read Greg's response to my comparison between Future+yield-based coroutines and his yield-from-based, Future-free coroutines, and after having written a small prototype, I am now pretty much convinced that Greg's way is superior. This doesn't mean you can't use generators or yield-from for other purposes! It's just that *if* you are writing a coroutine for use with a certain schedule, you must use yield and yield-from in accordance to the scheduler's rules. However, code you call can still use yield and yield-from for iteration, and you can still use for-loops. In particular, if f is a coroutine, it can still write "for x in g(): ..." where g is a generator meant to be an iterator. However if g were instead a coroutine, f should call it using "yield from g()", and f and g should agree on the interface of their scheduler. As to other topics, my current feeling is that we should try to separately develop requirements and prototype implementations of the I/O loop of the future, and to figure the loosest possible coupling between that and a coroutine scheduler (or any other type of scheduler). In particular, I think the I/O loop should not assume the event handlers are implemented using coroutines -- but if someone wants to write an awesome coroutine scheduler, they should be able to delegate all their I/O waiting needs to the I/O loop with very little trouble. To me, this means that the I/O loop probably should use "plain" callback functions (i.e., not Futures, Deferreds or coroutines). We should also standardize the interface to the I/O loop so that 3rd parties can plug in their own I/O loop -- I don't see an end to the debate whether the best C library for event handling is libevent, libev or libuv. While the focus of the I/O loop should be on single-threaded event handling, some standard interface should exist so that you can run certain code in a separate thread and wait for its completion -- I've found this handy when calling socket.getaddrinfo(), which may block. (Apparently async DNS lookups are really hard -- I read some complaints about libevent's DNS lookups, and IIUC many Firefox lockups are due to this.) But there may be other uses for this too. An issue in the design of the I/O loop is the strain between a ready-based and completion-based design. The typical Unix design (whether based on select or any of the poll variants) is usually ready-based; but on Windows, the only way to get high performance is to base it on IOCP, which is completion-based (i.e. you start a specific async operation, like writing N bytes, and the I/O loop tells you when it is done). I would like people to be able to write fast event handling programs on Windows too, and ideally the only change would be the implementation of the I/O loop. But I don't know how tenable that is given the dramatically different style used by IOCP and the need to use native Windows API for all async I/O -- it sounds like we could only do this if the library providing the I/O loop implementation also wrapped all I/O operations, andthat may be a bit much. Finally, there should also be some minimal interface so that multiple I/O loops can interact -- at least in the case where one I/O loop belongs to a GUI library. It seems this is a solved problem (as well solved as you can hope for) to Twisted, so we should just adopt their approach. -- --Guido van Rossum (python.org/~guido)

15 54

Adding '**' recursive search to glob.glob
by Paul Moore Jan. 15, 2013

Jan. 15, 2013

This may be simple enough to just be a feature request on the tracker, but I thought I'd post it here first to see if people thought it was a good idea. I'd like it if the glob module supported the (relatively common) facility to use ** to mean recursively search subdirectories. It's a reasonably straightforward patch, and offers a feature that is fairly difficult to implement in user code on top of the existing functionality. The syntax is supported in a number of places (for example the bash shell and things like Java Ant) so it will be relatively familiar to users. For people who don't know the syntax, "a/**/b" is equivalent to "a/*/b or a/*/*/b or a/*/*/*/b or ..." (for as many levels as needed). One obvious downside is that if used carelessly, it can make globbing pretty slow. So I'd propose that it be added as an optional extension enabled using a flag argument (glob(pat, allow_recursive=True)) which is false by default. That would also mean that backward compatibility should not be an issue. Any comments? Is this worth submitting a patch to the tracker? Paul.

12 21

Re: [Python-ideas] Adding '**' recursive search to glob.glob
by Yuval Greenfield Jan. 14, 2013

Jan. 14, 2013

http://bugs.python.org/issue13968 "Support recursive globs" Yuval

2 1

csv dialect enhancement
by rurpy＠yahoo.com Jan. 13, 2013

Jan. 13, 2013

There is a common dialect of CSV, often used in database applications [*1], that distinguishes between an empty (quoted) string, e.g., the second field in "abc","",3 and an empty field, e.g., the second field in "abc",,3 This distinction is needed to specify or tell the difference between 0-length strings and NULLs, when sending csv data to or receiving it from a database application. AFAICT, Python's csv module does not distinguish between empty fields and empty quoted strings. Both of the examples above, when parsed by csv.Reader, will return ['abc', '', 3] (or possibly '3' for the last item depending on options). Similarly, csv.Writer produces the same output csv text (nothing or a quoted empty string depending on Dialect.quoting) for row items '' or None. csv.Reader could distinguish between the above cases by using an empty string ('') to report an empty (quoted) string field, and None to report an empty field. Thus the second example would produce ['abc', None, 3] (or ...,'3'). Similarly, csv.Writer could produce alternate text (nothing or a quoted empty string) depending on whether a row item was None or an empty string. I propose that a new dialect attribute be added, "nulls" [*2], which when false (default) will cause csv to behave as it currently does. When true it will have the following effect: Reader: When two adjacent delimiters occur, or two white-space separated delimiters when Dialect.skipinitialspace is true, a value of None will be returned for that field. Writer: When a None is present in the the list of items being formatted, it will result in an empty output field (two adjacent delimiters) regardless of other options (eg a QUOTE_ALL setting.) Sniffer: Will set "nulls" to True when both adjacent delimiters and quoted empty strings are seen in the input text. (Perhaps this behaviour needs to be optional for backward compatibility reasons?) I think this will allow the csv module to generate the csv dialect(s) commonly used by databases applications. A specific use case: I am migrating data from a MS Access database to Postgresql. I run a tool that extracts table data from Access and correctly produces CSV files in the dialect used by Postgresql with some (nullable) column values having empty fields and other non- nullable column values having empty string fields. But I need to modify some values before import. So I write a Python program that parses the csv data, modifies some of it and writes it back out, using the csv module. But the result is that all empty fields and empty strings are written out identically as one or the other (the distinction is not preserved). Result is that information is lost and the output cannot be used. I would be able to do this if the csv module provide a "nulls" option as proposed above. AFAICT, Python's csv module does not distinguish between empty fields and empty quoted strings. Both of the examples above, when parsed by csv.Reader, will return ['abc', '', 3] (or possibly '3' for the last item depending on options). Similarly, csv.Writer produces the same output csv text (nothing or a quoted empty string depending on Dialect.quoting) for row items '' or None. csv.Reader could distinguish between the above cases by using an empty string ('') to report an empty (quoted) string field, and None to report an empty field. Thus the second example would produce ['abc', None, 3] (or ...,'3'). Similarly, csv.Writer could produce alternate text (nothing or a quoted empty string) depending on whether a row item was None or an empty string. I propose that a new dialect attribute be added, "nulls" [*2], which when false (default) will cause csv to behave as it currently does. When true it will have the following effect: Reader: When two adjacent delimiters occur, or two white-space separated delimiters when Dialect.skipinitialspace is true, a value of None will be returned for that field. Writer: When a None is present in the the list of items being formatted, it will result in an empty output field (two adjacent delimiters) regardless of other options (eg a QUOTE_ALL setting.) Sniffer: Will set "nulls" to True when both adjacent delimiters and quoted empty strings are seen in the input text. (Perhaps this behaviour needs to be optional for backward compatibility reasons?) I think this will allow the csv module to generate the csv dialect(s) required for databases applications. A specific use case: I am migrating data from a MS Access database to Postgresql. I run a tool that extracts table data from Access and produces CSV files in the dialect used by Postgresql with some (nullable) column values having empty fields and other non-nullable column values having empty string fields. But I need to modify some values before import. So I write a Python program that parses the csv data, modifies some of it and writes it out, using the csv module. But the result is that all empty fields and empty strings are written out identically as one or the other (the distinction is not preserved). Result is that information is lost and the output cannot be used. I would be able to do this if the csv module provide a "nulls" option as proposed above. ---- [*1] One of the two most important open-source databases, Postgresql, uses this dialect. See: http://www.postgresql.org/docs/9.2/interactive/sql-copy.html#AEN66692 I don't know about the other. [*2] I don't really care what the attribute name is; I chose "nulls" as a trial balloon because I wanted to avoid something with "none" in it to avoid confusion with QUOTE_NONE.

4 4