Mailman 3 August 2015 - Python-ideas

new feature for pip a moduletree
by shiva prasanth 18 Aug '15

18 Aug '15

pip list only shows the name of package which auther wants which cant be imported for import we have to search for internet in which package format auther has written what i want to introduce is a new argument for pip this is new output which will show the list of all available packages. and if there is a subpackage it should be represented as branch tree and if all the leafs should be modules it is justlike pstree command in linux. $pip moduletree name name==version packagename-----+----{subpackage1}-----+----{module1} | +----{module2} | +----{module3} +----{subpackage2}------+---{module1} +----{subpackage3}------+-----{module} +-----mainmodule1 +-----mainmodule2 $pip moduletree # will give the above output for the for each package in pip list $pip moduletree packagename --classes will output the availble class names in the leaf node

1 0

Map to Many Function
by Mark Tse 18 Aug '15

18 Aug '15

Currently, when the function for map() returns a list, the resulting object is an iterable of lists: >>> list(map(lambda x: [x, x], [1, 2, 3, 4])) [[1, 1], [2, 2], [3, 3], [4, 4]] However, a function to convert each element to multiple elements, similar to flatMap (Java) or SelectMany (C#) does not exist, for doing the following: >>> list(mapmany(lambda x: [x, x], [1, 2, 3, 4])) [1, 1, 2, 2, 3, 3, 4, 4] Proposal: new built-in method or standard library function to do mapmany. Sample use case: Library JSON data returns a list of authors, and each author has a list of books: { [ { 'author': 'name', 'books': ['book1', 'book2'] }, { 'author': 'name, 'books': ['book3', 'book4'] }, ... ] } allbooks = list(mapmany(lambda x: x['books'], json)) (First time posting - please redirect me to things I need to read if something's wrong with the format of this suggestion). Mark

11 16

pip moduletree new argument
by shiva prasanth 17 Aug '15

17 Aug '15

pip list only shows the name of package which auther wants which cant be imported for import we have to search for internet in which package format auther has written what i want to introduce is a new argument for pip this is new output which will show the list of all available packages. and if there is a subpackage it should be represented as branch tree and if all the leafs should be modules it is justlike pstree command in linux. $pip moduletree name name==version packagename-----+----{subpackage1}-----+----{module1} | +----{module2} | +----{module3} +----{subpackage2}------+---{module1} +----{subpackage3}------+-----{module} +-----mainmodule1 +-----mainmodule2 $pip moduletree # will give the above output for the for each package in pip list $pip moduletree packagename --classes will output the availble class names in the leaf node

1 0

Draft PEP: Automatic Globbing of Filenames in argparse on Windows
by Kef Schecter 14 Aug '15

14 Aug '15

PEP: XXX Title: Automatic Globbing of Filenames in argparse on Windows Version: $Revision$ Last-Modified: $Date$ Author: Kef Schecter <furrykef(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 14-Aug-2015 Python-Version: 3.6 Post-History: Abstract ======== This PEP proposes to add functionality to argparse to allow glob (wildcard) expressions to be handled automagically on Windows. Motivation ========== For many command-line tools, it is handy to be able to specify wildcards in order to operate on more than one file at a time. On Unix-like systems, this is handled automatically by the shell. On Windows, however, the default shell does not have this behavior, nor does Microsoft's PowerShell. Yet Windows users generally expect wildcards to work. For example, most built-in commands such as ``dir`` and ``type`` accept wildcard arguments, and have since the early days of MS-DOS. It is already possible for programmers to work around this issue, but it is a bit cumbersome and it is easy to make the behavior almost, but not quite, correct. Moreover, since Python has a "batteries included" philosophy, and this is a very common feature, it is the author's opinion that the correct functionality should be available out of the box. How It Must Be Done Currently ============================= :: if platform.system() == 'Windows': filenames = [] for filename in args.files: if '*' in filename or '?' in filename or '[' in filename: filenames += glob.glob(filename) else: filenames.append(filename) args.files = filenames Why This Is a Problem ===================== - Authors, especially those who use Unix-like systems, will usually not bother to add this code unless users specifically request it, and perhaps not even then. How often have you seen this code in a program? - It is easy to forget the platform check or not understand why it is necessary. Automatically globbing filenames on a Unix-like system is wrong because the shell is supposed to handle it already; on such a system, if the program sees a name like ``*.txt``, then it means the user explicitly specified the name of a file that, improbable as it may seem, has an asterisk in its filename. On Windows, filenames with the characters ``*`` and ``?`` in their name are not possible, so this is partially irrelevant on Windows even when using a Unix-like shell such as bash (but see `Square Brackets`_ below). - It is easy to forget to check the string for wildcard characters before passing it to glob.glob. If the user specifies a filename with no wildcards such as ``foo.txt``, and foo.txt does not exist, then glob.glob will silently ignore the file, giving the program no opportunity to print a message such as "No file named foo.txt". - glob.glob may not be quite the right function to use. See `Square Brackets`_ below. - It is boilerplate code that is applicable to a large number of programs without change, which suggests it belongs in a library. Solution ======== Add a keyword argument to argparse.ArgumentParser.add_argument called ``glob``. If it is true, it will automatically glob filenames using code much like the boilerplate code given earlier in `How It Must Be Done Currently`_. This argument is only meaningful when nargs is set to an appropriate value such as '+' or '*'. The default value of this argument should be False. This ensures backward compatibility with existing programs that assume wildcards are not expanded, such as a program that accepts a regex as an argument. A possibly better behavior might be to make this argument default to True (enabling the functionality automagically without the programmer needing to be aware of it) and only expand wildcard arguments that are not provided in quotes, similar to how Unix-like shells behave. However, there appears to be no simple way to tell whether an argument was supplied in quotes or not; the strings in sys.argv already have had the quotes removed. Square Brackets =============== It has been noted above that the characters ``*`` and ``?`` will never appear in filenames on Windows. However, the characters ``[`` and ``]``, which glob.glob uses for wildcards, **can** be used in filenames, and may not be especially uncommon. There are three possible ways of handling this: 1. Use a version of glob.glob without the wildcard functionality that ``[`` and ``]`` provide. This type of wildcard has never been standard for wildcard arguments to MS-DOS or Windows command-line programs. 2. Specify some kind of escaping mechanism; for example, ``\[foo\].txt`` would refer to a file that has ``[`` and ``]`` in its filename. This may not be intuitive behavior for Windows users. 3. Keep glob.glob's standard functionality. Programs using this feature will not be able to operate on files that have square brackets in their names. Of these, the first should adhere best to the principle of least surprise. Windows users do not expect square brackets to form wildcard expressions. If they want such functionality, they will probably already be using a shell such as bash that handles it for them. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

3 3

Making concurrent.futures.Futures awaitable
by Alex Grönholm 13 Aug '15

13 Aug '15

There's an open issue for adding support for awaiting for concurrent.futures.Futures here: http://bugs.python.org/issue24383 This is about writing code like this: async def handler(self): result = await some_blocking_api.do_something_cpu_heavy() await self.write(result) As it stands, without this feature, some boilerplate is required: from asyncio import wrap_future async def handler(self): result = await wrap_future(some_blocking_api.do_something_cpu_heavy()) await self.write(result) I wrote a patch (with tests by Yury Selivanov) that adds __await__() to concurrent.futures.Future and augments the asyncio Task class to handle concurrent Futures. My arguments on why we should add this: * it eliminates the boilerplate code, reducing complexity * it also makes concurrent Futures work with "yield from" style non-native coroutines * it does not interfere with any existing functionality * standard library components should work with each other

4 7

Yet More Unpacking Generalizations (or, Dictionary Literals as lvalues)
by Scott Sanderson 13 Aug '15

13 Aug '15

Hi All, Occasionally I find myself wanting to unpack the values of a dictionary into local variables of a function. This most often occurs when marshalling values to/from some serialization format. For example: def do_stuff_from_json(json_dict): actual_dict = json.loads(json_dict) foo = actual_dict['foo'] bar = actual_dict['bar'] # Do stuff with foo and bar. In the same spirit as allowing argument unpacking into tuples or lists, what I'd really like to be able write is something like: def do_stuff_from_json(json_dict): # Assigns variables in the **values** of the lefthand side by doing lookups # of the corresponding keys in the result of the righthand side expression. {'foo': foo, 'bar': bar} = json.loads(json_dict) Nearly all the arguments in favor of tuple/list unpacking also apply to this construct. In particular: 1. It makes the code more self-documenting, in that the left side of the expression looks more like the expected output of the right side. 2. The construct can be implemented more efficiently by the interpreter by using a dictionary analog of the UNPACK_SEQUENCE opcode (e.g. UNPACK_MAP). An interesting question that falls out of this idea is whether/how we should handle nested structures. I'd expect the rule to be that something like: {'toplevel': {'key1': key1, 'key2': key2}} = value would desugar into something equivalent to: TEMP = value['toplevel'] key1 = TEMP['key1'] key2 = TEMP['key2'] del TEMP while something like {'toplevel': (x, y)} = value would desugar into something like: (x, y) = value['toplevel'] At the bytecode level, I'd expect this to be implemented with a new instruction, analogous to the current UNPACK_SEQUENCE, which would pop N keys and a map from the stack, and push map[key] onto the stack for each popped key. We'd then recurse through the values left on the stack, storing them as we would store the sub-lvalues if they were in a standard assignment. Thus the code for something like: {'name': name, 'tuple': (x, y), 'dict': {'subkey': subvalue}} = values would translate into the following "pseudo-bytecode": LOAD_NAME 'values' # Push rvalue onto the stack. LOAD_CONST 'dict' # Push top-level keys onto the stack. LOAD_CONST 'tuple' LOAD_CONST 'name' UNPACK_MAP 3 # Unpack keys. Pops values and all keys from the stack. # TOS = values['name'] # TOS1 = values['tuple'] # TOS2 = values['dict'] STORE_FAST name # Terminal names are simply stored. UNPACK_SEQUENCE 2 # Push the two entries in values['tuple'] onto the stack. # TOS = values['tuple'][0] # TOS1 = values['tuple'][1] # TOS2 = values['dict'] STORE_FAST x STORE_FAST y LOAD_CONST 'subkey' # TOS = 'subkey' # TOS1 = values['dict'] UNPACK_MAP 1 # TOS = values['dict']['subkey'] STORE_FAST subvalue I'd be curious to hear others' thoughts on whether this seems like a reasonable idea. One open question is whether non-literals should be allowed as keys in dictionaries (the above still works as expected if the keys are allowed to be names or expressions; the LOAD_CONSTs would turn into whatever expression or LOAD_* is necessary to put the necessary value on the stack). Another question is if/how we should handle extra keys in right-hand side of the assignment (my guess is that we shouldn't do anything special with that case). -Scott P.S. I attempted to post this last night, but it seems to have not gone through. Apologies for the double post if I'm mistaken about that.

8 13

PEP 487
by Neil Girdhar 12 Aug '15

12 Aug '15

Has there been any progress with PEP 487? I am finding myself writing a lot of boilerplate because of Python's so-called "metaclass hell". What are the problems with PEP 487? Best, Neil

2 1

Re: [Python-ideas] [Python-Dev] PEP-498: Literal String Formatting
by Wes Turner 12 Aug '15

12 Aug '15

On Tue, Aug 11, 2015 at 12:52 PM, Wes Turner <wes.turner(a)gmail.com> wrote: > ... I'm now -1000 on this. > > ~"Make it hard to do wrong; or easy to do correctly" > > ... Here are these, (which should also not be used for porting shell > scripts to python): http://jinja.pocoo.org/docs/dev/templates/#expressions > So, again, I am -1000 on (both of these PEPs) because they are just another way of making it too easy to do the wrong thing. * #1 most prevalent security vulnerability: *1**CWE-89 <http://cwe.mitre.org/data/definitions/89.html>: Improper Neutralization of Special Elements used in an SQL Command ('SQL Injection')* * ORM with parametrization, quoting, escaping and lists of reserved words * SQLAlchemy * #2 most prevalent security vulnerability: *2**CWE-78 <http://cwe.mitre.org/data/definitions/78.html>: Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection')* * Command preparation library (which builds a tuple() for exec) * Sarge, subprocess.call(shell=False=0) - [ ] DOC: (Something like this COULD/SHOULD be in the % and str.format docs as well) > > On Tue, Aug 11, 2015 at 12:48 PM, Wes Turner <wes.turner(a)gmail.com> wrote: > >> >> On Tue, Aug 11, 2015 at 12:08 PM, Nick Coghlan <ncoghlan(a)gmail.com> >> wrote: >> >>> [off list] >>> >>> On 12 August 2015 at 01:28, Wes Turner <wes.turner(a)gmail.com> wrote: >>> > >>> > On Aug 11, 2015 10:19 AM, "Wes Turner" <wes.turner(a)gmail.com> wrote: >>> >> >>> >> >>> >> On Aug 11, 2015 10:10 AM, "Alexander Walters" < >>> tritium-list(a)sdamon.com> >>> >> wrote: >>> >> > >>> >> > This may seam like a simplistic solution to i18n, but why not just >>> add a >>> >> > method to string objects (assuming we implement f-strings) that >>> just returns >>> >> > the original, unprocessed string. If the string was not an >>> f-string, it >>> >> > just returns self. The gettext module can be modified, I think >>> trivially, >>> >> > to use the method instead of the string directly. >>> >> > >>> >> > Is this a horrible idea? >>> > >>> > - [ ] review all string interpolation (for "injection") >>> > * [ ] review every '%' >>> > * [ ] review every ".format()" >>> > * [ ] review every f-string (AND LOCALS AND GLOBALS) >>> > * every os.system, os.exec*, subprocess.Popen >>> > * every unclosed tag >>> > * every unescaped control character >>> > >>> > This would create work we don't need. >>> > >>> > Solution: __str_shell_ escapes, adds slashes, and quotes. __str__SQL__ >>> refs >>> > a global list of reserved words. >>> >>> Wes, we're not mind readers - I know you're trying to be concise to >>> save people time when reading, but these bullet-point-only posts are >>> *harder* to read than if you wrote out a full explanation of what you >>> meant. With this cryptic form, we have to try to guess the missing >>> pieces, which is slower and less certain than having them already >>> written out in the post. >>> >> >> ~"This is another way to make it easier to do the wrong thing; where a >> better solution (AND/OR DOCS ON ALL STRING INTERPOLATION) would be less >> likely to increase the ocurrence of CWE TOP 25 #1 and #2" >> >> printf is often dangerous and wrng because things aren't escaped (or >> scope is not controlled, or things are mutable) >> >> >> ~"Make it hard to do; or easy to do the right way" >> >> >>> >>> Regards, >>> Nick. >>> >>> -- >>> Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia >>> >> >> >

5 8

Outside the box string formatting idea
by Ron Adam 11 Aug '15

11 Aug '15

While the discussion of string formatting has focused on the concept of "format strings", it seems to me they are really format expressions. The expressions in this case are a bit like comprehensions in that they have special syntax/rules that is only valid in a small limited context. It may be possible to do that in this case as an actual expression rather than as a string and still get most of the benefits. Shorter more compressed isn't always better when it comes to readability. Consider this as a trial balloon. If there is clear consensus against it, no problem. It can be added to the rejected ideas section of PEP-498. Below are some examples to compare from the PEP-498. I strongly suggest posting the examples into idle or a syntax highlighting editor to really see what difference it makes in readability. (it won't run, but the highlighting will work.) I used % here, but that's just a place holder at this point, some other symbol could be used ... if this idea has any merit. Actually it could even be reduced to a parentheses with items separated by spaces. Examples from PEP: f'My name is {name}, my age next year is {age+1}, my anniversary is {anniversary:%A, %B %d, %Y}.' (% 'My name is ' name ', my age next year is ' (age+1) ', my anniversary is ' {anniversary ':%A, %B %d, %Y'} '.') # The {expres fmt_spec} sytnax is only valid in (% ...) expressions. f'He said his name is {name!r}.' (% 'He said his name is ' {name '!r'} '.') f'abc{expr1:spec1}{expr2!r:spec2}def{expr3:!s}ghi' (% 'abc' {expr1 'spec1'} {expr2 '!r:spec2'} 'def' expr3 'ghi') f'result={foo()}' (% 'result =' foo()) x = 10 y = 'hi' result = a' 'b' f'{x}' 'c' f'str<{y:^4}>' 'd' 'e' result = (% 'a' 'b' x 'c' '<' {y '^4'} '>' 'd' 'e') result = (% 'ab' x 'c<' {y '^4'} '>de') >>> f'{{k:v for k, v in [(1, 2), (3, 4)}}' '{k:v for k, v in [(1, 2), (3, 4)}' (% {k:v for k, v in [(1, 2), (3, 4)}) This evaluates to a dictionary first. Most expressions naturally evaluate from the inside out, left to right. So there's no conflict. It just converts the dict comprehension to a string. The features allowed in (% ...) 1. Implicit concatenation is allowed for expressions separated by a space. But only in an explicit and limited context. 2. Expressions and format specs can be combined in braces separated by a space. Expressions without format specs don't need them. 3. It maintains the separation of strings and expressions. It does require typing more quotes than a f'...' format string, but the separation of items by only spaces along with syntax highlighting makes it much more readable I think. It's a run time expression, and not an object that can be bound to a name, just like f-strings. It's result can though, that is the same as f-strings. It removes the apparent controversy of evaluating strings. Which f-strings really don't do, because they are compiled to something very similar to what this does. It's just much more obvious with this suggestion that that isn't a problem. Complex expressions need ()'s around them in the format expression to ensure it's a single value, but the parser may be able to tell a + b, from a b. This is the explicit run time expression would do what a f-string does. Reduce repetition of names, allow text and expressions to be interspersed in the order they are used with as little extra noise as possible. The f-strings go one step further and eliminate the extra parentheses and quotes. But at the cost of having them run together and loosing the syntax highlighting of expressions in an editor. It also will make it harder for external syntax checkers to work. It probably would need some ironing out of some the edge cases though. A {expr} evaluates as a set. Just use expr, or {expr ''}. Things like that. I think this is as close to an f-string as we could get and still keep the expressions out of the strings. Cheers, Ron

5 8

Re: [Python-ideas] String interpolation for all literal strings
by Jonathan Slenders 10 Aug '15

10 Aug '15

2015-08-07 9:33 GMT+02:00 Nick Coghlan <ncoghlan(a)gmail.com>: > If f-strings are always eagerly interpolated > prior to translation, then I can foresee a lot of complaints from > folks asking why this doesn't work right: > > print(_(f"This is a translated message with {a} and {b} interpolated")) > What if we decide to lazy-interpolate f-strings? A creation of such an f-string, creates an f-string object. (Which has a proper __repr__, so that it's transparent for those that don't care.) Calling the __str__ method of the f-string triggers the interpolation. This should give enough freedom to implement a working ugettext. (But I still don't see any reason to go beyond supporting more than just is simple variable name between the curly braces...)

1 0