Mailman 3 September 2015 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' … [View More]

51 165

proposal: "python -m foo" should bind sys.modules['foo']
by Cameron Simpson Jan. 18, 2017

Jan. 18, 2017

Hello all, This is a writeup of a proposal I floated here: https://mail.python.org/pipermail/python-list/2015-August/694905.html last Sunday. If the response is positive I wish to write a PEP. Briefly, it is a natural expectation in users that the command: python -m module_name ... used to invoke modules in "main program" mode on the command line imported the module as "module_name". It does not, it imports it as "__main__". An import within the program of "module_name" makes a new … [View More]instance of the module, which causes cognitive dissonance and has the side effect that now the program has two instances of the module. What I propose is that the above command line _should_ bind sys.modules['module_name'] as well as binding '__main__' as it does currently. I'm proposing that the python -m option have this effect (python pseudocode): % python -m module.name ... runs: # pseudocode, with values hardwired for clarity import sys M = new_empty_module(name='__main__', qualname='module.name') sys.modules['__main__'] = M sys.modules['module.name'] = M # load the module code from wherever (not necessarily a file - CPython # already must do this phase) M.execfile('/path/to/module/name.py') Specificly, this would have the following two changes to current practice: 1) the module is imported _once_, and bound to both its canonical name and also to __main__. 2) imported modules acquire a new attribute __qualname__ (analogous to the recent __qualname__ on functions). This is always the conanoical name of the module as resolved by the importer. For most modules __name__ will be the same as __qualname__, but for the "main" module __name__ will be '__main__'. This change has the following advantages: The current standard boilerplate: if __name__ == '__main__': ... invoke "main program" here ... continues to work unchanged. Importantly, if the program then issues "import module_name", it is already there and the existing instance is found and used. The thread referenced above outlines my most recent encounter with this and the trouble it caused me. Followup messages include some support for this proposed change, and some criticism. The critiquing article included some workarounds for this multiple module situation, but they were (1) somewhat dependent on modules coming from a file pathname and (2) cumbersome and require every end user to adopt these changes if affected by the situation. I'd like to avoid that. Cheers, Cameron Simpson <cs(a)zip.com.au> The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. - George Bernard Shaw [View Less]

5 13

solving multi-core Python
by Eric Snow June 4, 2016

June 4, 2016

tl;dr Let's exploit multiple cores by fixing up subinterpreters, exposing them in Python, and adding a mechanism to safely share objects between them. This proposal is meant to be a shot over the bow, so to speak. I plan on putting together a more complete PEP some time in the future, with content that is more refined along with references to the appropriate online resources. Feedback appreciated! Offers to help even more so! :) -eric -------- Python's multi-core story is murky at best. … [View More]Not only can we be more clear on the matter, we can improve Python's support. The result of any effort must make multi-core (i.e. parallelism) support in Python obvious, unmistakable, and undeniable (and keep it Pythonic). Currently we have several concurrency models represented via threading, multiprocessing, asyncio, concurrent.futures (plus others in the cheeseshop). However, in CPython the GIL means that we don't have parallelism, except through multiprocessing which requires trade-offs. (See Dave Beazley's talk at PyCon US 2015.) This is a situation I'd like us to solve once and for all for a couple of reasons. Firstly, it is a technical roadblock for some Python developers, though I don't see that as a huge factor. Regardless, secondly, it is especially a turnoff to folks looking into Python and ultimately a PR issue. The solution boils down to natively supporting multiple cores in Python code. This is not a new topic. For a long time many have clamored for death to the GIL. Several attempts have been made over the years and failed to do it without sacrificing single-threaded performance. Furthermore, removing the GIL is perhaps an obvious solution but not the only one. Others include Trent Nelson's PyParallels, STM, and other Python implementations.. Proposal ======= In some personal correspondence Nick Coghlan, he summarized my preferred approach as "the data storage separation of multiprocessing, with the low message passing overhead of threading". For Python 3.6: * expose subinterpreters to Python in a new stdlib module: "subinterpreters" * add a new SubinterpreterExecutor to concurrent.futures * add a queue.Queue-like type that will be used to explicitly share objects between subinterpreters This is less simple than it might sound, but presents what I consider the best option for getting a meaningful improvement into Python 3.6. Also, I'm not convinced that the word "subinterpreter" properly conveys the intent, for which subinterpreters is only part of the picture. So I'm open to a better name. Influences ======== Note that I'm drawing quite a bit of inspiration from elsewhere. The idea of using subinterpreters to get this (more) efficient isolated execution is not my own (I heard it from Nick). I have also spent quite a bit of time and effort researching for this proposal. As part of that, a number of people have provided invaluable insight and encouragement as I've prepared, including Guido, Nick, Brett Cannon, Barry Warsaw, and Larry Hastings. Additionally, Hoare's "Communicating Sequential Processes" (CSP) has been a big influence on this proposal. FYI, CSP is also the inspiration for Go's concurrency model (e.g. goroutines, channels, select). Dr. Sarah Mount, who has expertise in this area, has been kind enough to agree to collaborate and even co-author the PEP that I hope comes out of this proposal. My interest in this improvement has been building for several years. Recent events, including this year's language summit, have driven me to push for something concrete in Python 3.6. The subinterpreter Module ===================== The subinterpreters module would look something like this (a la threading/multiprocessing): settrace() setprofile() stack_size() active_count() enumerate() get_ident() current_subinterpreter() Subinterpreter(...) id is_alive() running() -> Task or None run(...) -> Task # wrapper around PyRun_*, auto-calls Task.start() destroy() Task(...) # analogous to a CSP process id exception() # other stuff? # for compatibility with threading.Thread: name ident is_alive() start() run() join() Channel(...) # shared by passing as an arg to the subinterpreter-running func # this API is a bit uncooked still... pop() push() poison() # maybe select() # maybe Note that Channel objects will necessarily be shared in common between subinterpreters (where bound). This sharing will happen when the one or more of the parameters to the function passed to Task() is a Channel. Thus the channel would be open to the (sub)interpreter calling Task() (or Subinterpreter.run()) and to the new subinterpreter. Also, other channels could be fed into such a shared channel, whereby those channels would then likewise be shared between the interpreters. I don't know yet if this module should include *all* the essential pieces to implement a complete CSP library. Given the inspiration that CSP is providing, it may make sense to support it fully. It would be interesting then if the implementation here allowed the (complete?) formalisms provided by CSP (thus, e.g. rigorous proofs of concurrent system models). I expect there will also be a _subinterpreters module with low-level implementation-specific details. Related Ideas and Details Under Consideration ==================================== Some of these are details that need to be sorted out. Some are secondary ideas that may be appropriate to address in this proposal or may need to be tabled. I have some others but these should be sufficient to demonstrate the range of points to consider. * further coalesce the (concurrency/parallelism) abstractions between threading, multiprocessing, asyncio, and this proposal * only allow one running Task at a time per subinterpreter * disallow threading within subinterpreters (with legacy support in C) + ignore/remove the GIL within subinterpreters (since they would be single-threaded) * use the GIL only in the main interpreter and for interaction between subinterpreters (and a "Local Interpreter Lock" for within a subinterpreter) * disallow forking within subinterpreters * only allow passing plain functions to Task() and Subinterpreter.run() (exclude closures, other callables) * object ownership model + read-only in all but 1 subinterpreter + RW in all subinterpreters + only allow 1 subinterpreter to have any refcounts to an object (except for channels) * only allow immutable objects to be shared between subinterpreters * for better immutability, move object ref counts into a separate table * freeze (new machinery or memcopy or something) objects to make them (at least temporarily) immutable * expose a more complete CSP implementation in the stdlib (or make the subinterpreters module more compliant) * treat the main interpreter differently than subinterpreters (or treat it exactly the same) * add subinterpreter support to asyncio (the interplay between them could be interesting) Key Dependencies ================ There are a few related tasks/projects that will likely need to be resolved before subinterpreters in CPython can be used in the proposed manner. The proposal could implemented either way, but it will help the multi-core effort if these are addressed first. * fixes to subinterpreter support (there are a couple individuals who should be able to provide the necessary insight) * PEP 432 (will simplify several key implementation details) * improvements to isolation between subinterpreters (file descriptors, env vars, others) Beyond those, the scale and technical scope of this project means that I am unlikely to be able to do all the work myself to land this in Python 3.6 (though I'd still give it my best shot). That will require the involvement of various experts. I expect that the project is divisible into multiple mostly independent pieces, so that will help. Python Implementations =================== They can correct me if I'm wrong, but from what I understand both Jython and IronPython already have subinterpreter support. I'll be soliciting feedback from the different Python implementors about subinterpreter support. C Extension Modules ================= Subinterpreters already isolate extension modules (and built-in modules, including sys). PEP 384 provides some help too. However, global state in C can easily leak data between subinterpreters, breaking the desired data isolation. This is something that will need to be addressed as part of the effort. [View Less]

26 130

Add appdirs module to stdlib
by Philipp A. Oct. 18, 2015

Oct. 18, 2015

When defining a place for config files, cache files, and so on, people usually hack around in a OS-dependent, misinformed, and therefore wrong way. Thanks to the tempfile API we at least don’t see people hardcoding /tmp/ too much. There is a beautiful little module that does things right and is easy to use: appdirs <https://pypi.python.org/pypi/appdirs> TI think this is a *really* good candidate for the stdlib since this functionality is useful for everything that needs a cache or … [View More]

21 31

Binary f-strings
by Eric V. Smith Oct. 8, 2015

Oct. 8, 2015

Now that f-strings are in the 3.6 branch, I'd like to turn my attention to binary f-strings (fb'' or bf''). The idea is that: >>> bf'datestamp:{datetime.datetime.now():%Y%m%d}\r\n' Might be translated as: >>> (b'datestamp:' + ... bytes(format(datetime.datetime.now(), ... str(b'%Y%m%d', 'ascii')), ... 'ascii') + ... b'\r\n') Which would result in: b'datestamp:20150927\r\n' The only real question is: what encoding to use for the second parameter to … [View More]bytes()? Since an object must return unicode from __format__(), I need to convert that to bytes in order to join everything together. But how? Here I suggest 'ascii'. Unfortunately, this would give an error if __format__ returned anything with a char greater than 127. I think we've learned that an API that only raises an exception with certain specific inputs is fragile. Guido has suggested using 'utf-8' as the encoding. That has some appeal, but if we're designing this for wire protocols, not all protocols will be using utf-8. Another idea would be to extend the "conversion char" from just 's', 'r', or 'a', which don't make much sense for bytes, to instead be a string that specifies the encoding. The default could be ascii, and if you want to specify something else: bf'datestamp:{datetime.datetime.now()!utf-8:%Y%m%d}\r\n' That would work for any encoding that doesn't have ':', '{', or '}' in the encoding name. Which seems like a reasonable restriction. And I might be over-generalizing here, but you'd presumably want to make the encoding a non-constant: bf'datestamp:{datetime.datetime.now()!{encoding}:%Y%m%d}\r\n' I think my initial proposal will be to use 'ascii', and not support any conversion characters at all for fb-strings, not even 's', 'r', and 'a'. In the future, if we want to support encodings other than 'ascii', we could then add !conversions mapping to encodings. My reasoning for using 'ascii' is that 'utf-8' could easily be an error for non-utf-8 protocols. And by using 'ascii', at least we'd give a runtime error and not put possibly bogus data into the resulting binary string. Granted, the tradeoff is that we now have a case where whether or not the code raises an exception is dependent upon the values being formatted. If 'ascii' is the default, we could later switch to 'utf-8', but we couldn't go the other way. The only place this is likely to be a problem is when formatting unicode string values. No other built-in type is going to have a non-ascii compatible character in its __format__, unless you do tricky things with datetime format_specs. Of course user-defined types can return any unicode chars from __format__. Once we make a decision, I can apply the same logic to b''.format(), if that's desirable. I'm open to suggestions on this. Thanks for reading. -- Eric. [View Less]

11 32

Pre-PEP Adding A Secrets Module To The Standard Library
by Steven D'Aprano Oct. 7, 2015

Oct. 7, 2015

Following on to the discussions about changing the default random number generator, I would like to propose an alternative: adding a secrets module to the standard library. Attached is a draft PEP. Feedback is requested. (I'm going to only be intermittently at the keyboard for the next day or so, so my responses may be rather slow.) -- Steve

19 69

Non-English names in the turtle module (with Spanish example)
by Al Sweigart Oct. 5, 2015

Oct. 5, 2015

I've created a prototype for how we could add foreign language names to the turtle.py module and erase the language barrier for non-English schoolkids. The Tortuga module has the same functionality as You can test it out by running "pip install tortuga" https://pypi.python.org/pypi/Tortuga Since Python 2 doesn't have simpledialog, I used the PyMsgBox pure-python module for the input boxes. This code is small enough that it could be added into turtle.py. (It's just used for simple tkinter … [View More]

3 2

PEP 505 (None coalescing operators) thoughts
by Jeff Hardy Oct. 5, 2015

Oct. 5, 2015

TL;DR: +1 for the idea -1 on the propagating member-access or index operators +1 on spelling it "or?" C# has had null-coalescing since about 2005, and it's one feature I miss in every other language that I use. I view null/None as a necessary evil, so getting rid of them as soon possible is a good thing in my book. Nearly every bit of Python I've ever written would have benefitted from it, if just to get rid of the "x if x is not None else []" mess. That said, I think the other (propagating) … [View More]

34 122

secrets module -- secret.keeper?
by Jim J. Jewett Oct. 3, 2015

Oct. 3, 2015

Will the secrets module offer any building blocks to actually protect a secret? e.g., an easy way to encrypt a file with a given password? an encrypted datastore? a getpass that works even in IDLE? -jJ

6 5

Re: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence
by Brett Cannon Oct. 2, 2015

Oct. 2, 2015

On Wed, 30 Sep 2015 at 09:38 Neil Girdhar <mistersheik(a)gmail.com> wrote: > In fairness, one is a superset of the other. You always get an Iterable. > You sometimes get a Sequence. It's a bit like multiplication? with > integers you get integers, with floats, you get floats. > No, it's not like multiplication. =) I hate saying this since I think it's tossed around too much, but int/float substitution doesn't lead to a Liskov substitution violation like substituting out a … [View More]

17 70

Fwd: Consider making enumerate a sequence if its argument is a sequence
by Alexander Belopolsky Sept. 30, 2015

Sept. 30, 2015

On Wed, Sep 30, 2015 at 11:28 AM, Neil Girdhar <mistersheik(a)gmail.com> wrote: > I found myself writing: > > for vertex, height in zip( > self.cache.height_to_vertex[height_slice], > range(height_slice.start, height_slice.stop)): > > I would have preferred: > > for height, vertex in enumerate( > self.cache.height_to_vertex)[height_slice]: > This does … [View More]

1 0

Submitting a job to an asyncio event loop
by Vincent Michel Sept. 30, 2015

Sept. 30, 2015

Hi, I noticed there is currently no standard solution to submit a job from a thread to an asyncio event loop. Here's what the asyncio documentation says about concurrency and multithreading: > To schedule a callback from a different thread, the BaseEventLoop.call_soon_threadsafe() method should be used. > Example to schedule a coroutine from a different thread: > loop.call_soon_threadsafe(asyncio.async, coro_func()) The issue with this method is the loss of the coroutine result.… [View More]

3 7

Consider making enumerate a sequence if its argument is a sequence
by Neil Girdhar Sept. 30, 2015

Sept. 30, 2015

What are the pros and cons of making enumerate a sequence if its argument is a sequence? I found myself writing: for vertex, height in zip( self.cache.height_to_vertex[height_slice], range(height_slice.start, height_slice.stop)): I would have preferred: for height, vertex in enumerate( self.cache.height_to_vertex)[height_slice]:

2 1

PEP 505 (None coalescing operators) thoughts
by Brian O'Neill Sept. 30, 2015

Sept. 30, 2015

> On 9/29/2015 9:20 AM, Rob Cliffe wrote: > Why not > > > > def __init__(self, vertices=None, edges=None, weights=None, > > source_nodes=None): > > self.vertices = vertices if vertices is not None else [] > > self.edges = edges if edges is not None else [] > > self.weights = weights if weights is not None else {} > > self.source_nodes = source_nodes if source_nodes is not None … [View More]

3 2

list as parameter for the split function
by Niilos Sept. 30, 2015

Sept. 30, 2015

Hello everyone, I was wondering how to split a string with multiple separators. For instance, if I edit some subtitle file and I want the string '00:02:34,452 --> 00:02:37,927' to become ['00', '02', '34', '452', '00', '02', '37', '927'] I have to use split too much time and I didn't find a "clean" way to do it. I imagined the split function with an iterator as parameter. The string would be split each time its substring is in the iterator. Here is the syntax I considered for this : … [View More]

9 9

PEP 505 (None coalescing operators) thoughts
by Brian O'Neill Sept. 29, 2015

Sept. 29, 2015

> > A further virtue of > > > > self.vertices = vertices or [] > > > > and the like is that they coerce falsy parameters of the wrong type to the falsy object of the correct type. > > E.g. if vertices is '' or 0, self.vertices will be set to [], whereas the ternary expression only tests > > > > for not-None so self.vertices will be set to a probably crazy value. > > Doesn't seem like a virtue to me, seems like it's probably hiding … [View More]

1 0

Using `or?` as the null coalescing operator
by Alessio Bogon Sept. 29, 2015

Sept. 29, 2015

I really like PEP 0505. The only thing that does not convince me is the `??` operator. I would like to know what you think of an alternative like `or?`: a_list = some_list or? [] a_dict = some_dict or? {} The rationale behind is to let `or` do its job with “truthy” values, while `or?` would require non-None values. The rest of the PEP looks good to me. I apologise in advance if this was already proposed and I missed it. Regards, Alessio

4 4

Maybe python should support arrow syntax for easier use async call ?
by 张沈鹏 Sept. 29, 2015

Sept. 29, 2015

# The example works with tornado dev verison & python3.5 import tornado from tornado.httpclient import AsyncHTTPClient from tornado.concurrent import Future from tornado.gen import convert_yielded from functools import wraps Future.__or__ = Future.add_done_callback def future(func): @wraps(func) def _(*args, **kwds): return convert_yielded(func(*args, **kwds)) return _ ############## @future async def ping(url): httpclient = AsyncHTTPClient() r = … [View More]

2 1

maxsplit in os.path.split
by Todd Sept. 28, 2015

Sept. 28, 2015

The "str.split" and "str.rsplit" methods have a useful "maxsplit" option, which lets you set the number of times to split, defaulting to -1 (which is "unlimited"). The corresponding "os.path.split", however, has no "maxsplit" option. It can only split once, which splits the last path segment (the "basename") from the rest (equivalent of "str.rsplit" with "maxsplit=1"). I think it would be useful if "os.path.split" also had a "maxsplit" option. This would default to "1" (the current behavior"… [View More]

3 2

PEP 504: Using the system RNG by default
by Nick Coghlan Sept. 25, 2015

Sept. 25, 2015

Hi folks, Based on the feedback in the recent threads, I've written a draft PEP that dispenses with the userspace CSPRNG idea, and instead proposes: * defaulting to using the system RNG for the module level random API in Python 3.6+ * implicitly switching to the deterministic PRNG if you call random.seed(), random.getstate() or random.setstate() (this implicit fallback would trigger a silent-by-default deprecation warning in 3.6, and a visible-by-default runtime warning after 2.7 goes EOL) * … [View More]

15 61

Using "||" (doubled pipe) as the null coalescing operator?
by Nick Coghlan Sept. 25, 2015

Sept. 25, 2015

This may just be my C programmer brain talking, but reading the examples in PEP 505 makes me think of the existing use of "|" as the bitwise-or operator in both Python and C, and "||" as the logical-or operator in C. Using || for None-coalescence would still introduce a third "or" variant into Python as PEP 505 proposes (for good reasons), but without introducing a new symbolic character that relates to "OR" operations: x | y: bitwise OR (doesn't short circuit) x or y: logical OR (… [View More]

7 12

Null coalescing operators
by Mark Haase Sept. 23, 2015

Sept. 23, 2015

StackOverflow has many questions <http://stackoverflow.com/search?q=%5Bpython%5D+null+coalesce> on the topic of null coalescing operators in Python, but I can't find any discussions of them on this list or in any of the PEPs. Has the addition of null coalescing operators into Python ever been discussed publicly? Python has an "or" operator that can be used to coalesce false-y values, but it does not have an operator to coalesce "None" exclusively. C# has nice operators for handling … [View More]

29 119

Re: [Python-ideas] Null coalescing operators
by Chris Barker - NOAA Federal Sept. 23, 2015

Sept. 23, 2015

Sent from my iPhone On Sep 22, 2015, at 6:21 PM, Fernando Perez <fperez.net(a)gmail.com In fact, that already happened once: before Python 2.4, our prefix for "magic functions" was the @ character, and when that was introduced as the decorator prefix, we had to scramble.� Note that the userbase of iPython was orders of magnitude smaller then -- changes like that would be a much bigger deal now. And while iPython was born in the scientific software community, and sees a lot (most) of its … [View More]

1 0

Re: [Python-ideas] add a single __future__ for py3?
by Chris Barker - NOAA Federal Sept. 23, 2015

Sept. 23, 2015

On Sep 22, 2015, at 6:43 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: On Sep 21, 2015, at 10:59, Gregory P. Smith <greg(a)krypto.org> wrote: I think people should stick with *from __future__ import absolute_import* regardless of what code they are writing. If the py3 way of handling Absolute vs relative import isn't better --- why is it in Py3???? Anyway, the point of a this is to get your py2 code working as similarly as possible on py3. So better or worse, or not all that … [View More]different, you still want that behavior. But again, it looks like this ship has sailed... Thanks for indulging me. -Chris They will eventually create a file innocuously called something like calendar.py (the same name as a standard library module) in the same directory as their main binary and their debugging of the mysterious failures they just started getting from the tarfile module will suddenly require leveling up to be able to figure it out. ;) But they'll get the same problems either way. If calendar.py isn't on sys.path, it won't interfere with tarfile. And if it is on sys.path, so it does interfere with tarfile, then it's already an absolute import, so enabling absolute_import doesn't help. I suppose if they've done something extra stupid, like putting a package directory on sys.path as well as putting something called calendar.py in that package and importing it with an unqualified import, then maybe it'll be easier for someone to explain all the details of everything they did wrong (including why they shouldn't have put the package on sys.path) if they're using absolute_imports, but beyond that, I don't see how it helps this case. -gps On Mon, Sep 21, 2015 at 8:18 AM Guido van Rossum <guido(a)python.org> wrote: > It's just about these four imports, right? > > > from __future__ import absolute_import > from __future__ import division > from __future__ import print_function > from __future__ import unicode_literals > > I think the case is overblown. > > - absolute_import is rarely an issue; the only thing it does (despite the > name) is give an error message when you attempt a relative import without > using a "." in the import. A linter can find this easily for you, and a > little discipline plus the right example can do a lot of good here. > > - division is important. > > - print_function is important. > > - unicode_literals is useless IMO. It breaks some things (yes there are > still APIs that don't take unicode in 2.7) and it doesn't nearly as much as > what would be useful -- e.g. repr() and <stream>.readline() still return > 8-bit strings. I recommend just using u-literals and abandoning Python 3.2. > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

1 0

add a single __future__ for py3?
by Chris Barker Sept. 21, 2015

Sept. 21, 2015

Hi all, the common advise, these days, if you want to write py2/3 compatible code, is to do: from __future__ import absolute_import from __future__ import division from __future__ import print_function from __future__ import unicode_literals https://docs.python.org/2/howto/pyporting.html#prevent-compatibility-regres… I'm trying to do this in my code, and teaching my students to do it to. but that's actually a lot of code to write. It would be nice to have a: from __future__ import py3 … [View More]

15 20

Re: [Python-ideas] [Python-Dev] Make stacklevel=2 by default in warnings.warn()
by Steven D'Aprano Sept. 20, 2015

Sept. 20, 2015

On Sat, Sep 19, 2015 at 11:55:44PM -0700, Nathaniel Smith wrote: > I don't have enough fingers to count how many times I've had to > explain how stacklevel= works to maintainers of widely-used packages > -- they had no idea that this was even a thing they were getting > wrong. Count me in that. I had no idea it was even a thing. -- Steve

1 0

new format spec for iterable types
by Wolfgang Maier Sept. 20, 2015

Sept. 20, 2015

Hi, in the parallel "format and print" thread, Andrew Barnert wrote: > For example, in a 5-line script I wrote last night, I've got > print(head, *names, sep='\t'). I could have used > print('\t'.join(chain([head], names)) instead--in fact, any use of > multi-argument print can be replaced by > print(sep.join(map(str, args)))--but that's less convenient, less > readable, and less likely to occur to novices. And there are plenty > of other alternatives, from > … [View More]

11 29

Structural type checking for PEP 484
by Guido van Rossum Sept. 18, 2015

Sept. 18, 2015

Jukka wrote up a proposal for structural subtyping. It's pretty good. Please discuss. https://github.com/ambv/typehinting/issues/11#issuecomment-138133867 -- --Guido van Rossum (python.org/~guido)

12 41

Bring line continuation to multi-level dictionary lookup
by John Wong Sept. 18, 2015

Sept. 18, 2015

Hi everyone. I work with APIs which have deep nested dictionary structure response. Imagine a simplified case: foo = {1: {2: {3: {4: {5: 6 } } } } Now imagine I need to get to 6: foo['1']['2']['3']['4']['5']['6'] This looks managable, but if the key name is long, then I certainly will end doing this to respect my style guide. To make it concrete, let's use something reallistic, a response call from AWS API: response = {'DescribeDBSnapshotsResponse': {'ResponseMetadata': {'RequestId': '… [View More]

5 5

Non-English names in the turtle module.
by Al Sweigart Sept. 16, 2015

Sept. 16, 2015

I've opened an issue for adding non-English names to the turtle module's function names: https://bugs.python.org/issue24990 This would effectively take this code: import turtle t = turtle.Pen() t.pencolor('green') t.forward(100) ...and have this code in French be completely equivalent: import turtle t = turtle.Plume() t.couleurplume('vert') t.avant(100) (Pardon my google-translate French.) This, of course, is terrible way for a software module to implement … [View More]

25 92

Add inspect.getenclosed to return/yield source code for nested classes and functions
by Un Do Sept. 16, 2015

Sept. 16, 2015

I propose adding a function into inspect module that will retrieve definitions of classes and functions (standard and lambdas) located inside another function/method. In my opinion this would a small but nice and useful addition to the standard library. It can be implemented using a couple of undocumented function from that module (findsource and getblock) without any performance drawbacks. Example: In [9]: print(getsource(function)) def function(): class inner_class(): … [View More]

1 0

Should our default random number generator be secure?
by Guido van Rossum Sept. 16, 2015

Sept. 16, 2015

I've received several long emails from Theo de Raadt (OpenBSD founder) about Python's default random number generator. This is the random module, and it defaults to a Mersenne Twister (MT) seeded by 2500 bytes of entropy taken from os.urandom(). Theo's worry is that while the starting seed is fine, MT is not good when random numbers are used for crypto and other security purposes. I've countered that it's not meant for that (you should use random.SystemRandom() or os.urandom() for that) but he … [View More]

28 125

Globally configurable random number generation
by Nick Coghlan Sept. 15, 2015

Sept. 15, 2015

This is an expansion of the random module enhancement idea I previously posted to Donald's thread: https://mail.python.org/pipermail/python-ideas/2015-September/035969.html I'll write it up as a full PEP later, but I think it's just as useful in this form for now. = Defining the problem = We're moving into an era where the easiest way to publish software is as a web application, with "deployment" to client systems done at runtime via a web browser. It's regularly the case that "learn to … [View More]program" classes (especially those aimed at adults picking up programming for the first time) will introduce folks to both a web development framework and how to deploy web applications on a developer focused service with a free hosting tier, like Heroku or OpenShift. It's also the case that we live in an era where there's a lot of well-intentioned-but-actually-bad advice on the internet when it comes to generating security sensitive tokens, and the folks receiving that advice through forums like Stack Overflow aren't necessarily ever going to see the "don't do that" guidance in the standard library's random module documentation, or the docs for the cryptography library, or the docs for a web framework like Flask, Django or Pyramid. One of the ways we know many of the folks doing web development often don't take admonitions in documentation seriously is because one of the most popular web servers for Python on these kinds of services is Django's "runserver", even though Django's docs specifically say only to use that for local development. It isn't OK to say "the developers deserve the consequences that come to them" as in many case, it isn't the developers that suffer the consequences, but the users of their applications. One reason we know weak RNGs can be a problem in practice is because the same kind of concern exists in PHP web applications, and https://media.blackhat.com/bh-us-12/Briefings/Argyros/BH_US_12_Argyros_PRNG… shows how the relative predictability of password reset tokens can be used to compromise administrator accounts. Rather than playing whackamole with individual web applications (many of which will be written by inexperienced developers), or attempting to demonstrate that a deterministic PRNG is "secure enough" for these use cases (when the research on PHP and deterministic PRNGs in general indicates that it isn't), it is proposed to migrate Python to a default random implementation that *is* known to be secure enough for these kinds of use cases. At the same time, deterministic random number generation is still desirable in many situations, and we also don't want to require that folks learning Python in the future be required to take a crash course in web application security theory first. Thus, it is also proposed that the abstraction used to present these differences to end users minimise the references to the underlying security concepts. A key outcome of this proposal is that it will retroactively upgrade a lot of existing instructions on the internet for generating default passwords and other sensitive tokens in Python from "actively harmful" to "not necessarily ideal, but at least not wrong if you're using Python 3.6+". This *is* a compatibility break for the sake of correcting default behaviours that are fine when developing applications for local use, but problematic from a network service security perspective, just as happened with the introduction of hash randomisation. Unlike the hash randomisation change, this one is readily addressed in old versions on a case by case basis, so it is only proposed to make the change in a future feature release of Python, not in any current maintenance releases. = Core abstraction = The core concept of this proposal involves classifying random number generators in Python as follows: * seedable * seedless * system These terms are chosen to make sense to folks that have *no idea* about the way different kinds of random number generator work and how that affects their security properties, but do know whether or not they need to be able to pass in a particular fixed seed in order to regenerate the same series of outputs. The guidance to Python users is then: * we use the seedless RNG by default as it provides the best balance of speed and security * if you need to be able to exactly reproduce output sequences, use the seedable RNG * if you know you're doing security sensitive work, use the system RNG directly to eliminate Python's seedless RNG as a potential source of vulnerabilities Importantly, there are relatively simple answers to the following two questions (which could be added to the Design FAQ): Q: Why isn't the seedable RNG the default random implementation (any more)? A: The same properties that make it possible to provide an explicit seed to the seedable RNG and get a predictable series of outputs make it inappropriate for tasks like generating session IDs and password reset tokens in web applications. Since folks continued to use the default RNG for those cases, even after years of the core development team, web framework developers and security engineers saying "Don't do that, use the system RNG instead", we eventually changed the default behaviour to just make those cases OK. Q: Why isn't the system RNG the default implementation? A: Due to the way operating systems work, calling into the kernel to get a random number is always going to be slower than generating one within the Python runtime. The default seedless generator provides most of the same benefits as using the system RNG directly, but is an order of magnitude faster as it doesn't need to call into the kernel as often. = Proposed change for Python 3.6 = * add a random.SeedlessRandom API that omits the seed(), getstate() and setstate() methods and uses a cryptographically secure PRNG internally (such as the ChaCha20 algorithm implemented by OpenBSD) * rename random.Random to random.SeedableRandom * make random.Random a subclass of SeedableRandom that deprecates seed(), getstate() and setstate() * deprecate the seed(), getstate() and setstate() methods on SystemRandom * expose the global SeedableRandom instance as random.seedable_random * expose a global SeedlessRandom instance as random.seedless_random * expose a global SystemRandom instance as random.system_random * provide a random.set_default_instance() API that makes it possible to specify the instance used by the module level methods * the module level seed(), getstate(), and setstate() functions will throw RuntimeError if the corresponding method is missing from the default instance In 3.6, "random.set_default_instance(random.seedless_random)" will opt in to the CSPRNG when using the module level functions process wide, while "from random import seedless_random as random" will do so on a module by module basis. "from random import system_random as random" also becomes available as a simple upgrade path for security sensitive modules. Appropriate helpers would be added to the six and future projects to allow single source Python 2/3 projects to easily cope with the change in behaviour when using the seeded RNG for its intended purposes. For many projects, compatibility code will consist of the following lines in a compatibility module: try: from random import seedable_random as random except ImportError: import random It would also be desirable for the seedless random number generator to be made available as a PyPI package for use on older Python versions. = Proposed change for Python 3.7 = * random.Random becomes an alias for random.SeedlessRandom * the default instance changes to be random.seedless_random In 3.7, "random.set_default_instance(random.seedable_random)" will opt back in to the deterministic PRNG when using the module level functions process wide, while "from random import seedable_random as random" will do so on a module by module basis. = Seedable random number generation = This is what we have today. The MT random implementation supports explicit seeding, state retrieval, and state restoration. It doesn't automatically mix in additional system entropy as it operates. This is the right choice for use cases like computer games, map generation, and randomising the order of test execution, as in these situations, it's desirable to be able to reproduce a past sequence exactly. = Seedless random number generators = This is the key proposed new addition: a cryptographically secure, non-deterministic, userspace PRNG. It's faster than the system RNG as it avoids the need to make a system API call. The "seedless" name comes from the fact that the inability to feed in a fixed seed is the most obvious API difference relative to deterministic RNGs, and hence provides a mental hook for people to remember which is which, without needing to know the relevant background security theory (which is arcane enough to be opaque even to developers with decades of experience and hence isn't something we want to be inflicting on folks in the process of learning to program). = System random number generator = The only proposed change here is providing a default instance to enable the "from random import system_random as random" pattern. Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia [View Less]

15 33

Python's Source of Randomness and the random.py module Redux
by Donald Stufft Sept. 15, 2015

Sept. 15, 2015

Ok, I reached out to Theo de Raadt to talk to him about what he was suggesting without Guido having to play messenger and forward fragments of the email conversation. I'm starting a new thread because this email is rather long, and I'm hoping to divorce it a bit from the back and forth about a proposal that wasn't exactly what Theo was suggesting that is being discussed in the other thread. Essentially, there are three basic types of uses of random (the concept, not the module). Those are: 1. … [View More]People/usecases who absolutely need deterministic output given a seed and for whom security properties don't matter. 2. People/usecases who absolutely need a cryptographically random output and for whom having a deterministic output is a downside. 3. People/usecases that fall somewhere in between where it may or may not be security sensitive or it may not be known if it's security sensitive. The people in group #1 are currently, in the Python standard library, best served using the MT random source as it provides exactly the kind of determinsm they need. The people in group #2 are currently, in the Python standard library, best served using os.urandom (either directly or via random.SystemRandom). However, the third case is the one that Theo's suggestion is attempting to solve. In the current landscape, the security minded folks will tell these people to use os.urandom/random.SystemRandom and the performance or otherwise less security minded folks will likely tell them to just use random.py. Leaving these people with a random that is not cryptographically safe. The questin then is, does it matter if #3 are using a cryptographically safe source of randomness? The answer is obviously that we don't know, and it's possible that the user doesn't know. In these cases it's typically best if we default to the more secure option and expect people to opt in to insecurity. In the case of randomness, a lot of languages (Python included) don't do that and instead they opt to pick the more peformant option first, often with the argument (as seen in the other thread) that if people need a cryptographically secure source of random, they'll know how to look for it and if they don't know how to look for it, then it's likely they'll have some other security problem. I think (and I believe Theo thinks) this sort of thinking is short sighted. Let's take an example of a web application, it's going to need session identifiers to put into a cookie, you'll want these to be random and it's not obvious on the tin for a non-expert that you can't just use the module level functions in the random module to do this. Another examples are generating API keys or a password. Looking on google, the first result for "python random password" is StackOverflow which suggests: ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(N)) However, it was later edited to, after that, include: ''.join(random.SystemRandom().choice(string.ascii_uppercase + string.digits) for _ in range(N)) So it wasn't obvious to the person who answered that question that the random module's module scoped functions were not appropiate for this use. It appears that the original answer lasted for roughly 4 years before it was corrected, so who knows how many people used that in those 4 years. The second result has someone asking if there is a better way to generate a random password in Python than: import os, random, string length = 13 chars = string.ascii_letters + string.digits + '!@#$%^&*()' random.seed = (os.urandom(1024)) print ''.join(random.choice(chars) for i in range(length)) This person obviously knew that os.urandom existed and that he should use it, but failed to correctly identify that the random module's module scoped functions were not what he wanted to use here. The third result has this code: import string import random def randompassword(): chars=string.ascii_uppercase + string.ascii_lowercase + string.digits size=8 return ''.join(random.choice(chars) for x in range(size,12)) I'm not going to keep pasting snippets, but going through the results it is clear that in the bulk of cases, this search turns up code snippets that suggest there is likely to be a lot of code out there that is unknownly using the random module in a very insecure way. I think this is a failing of the random.py module to provide an API that guides users to be safe which was attempted to be papered over by adding a warning to the documentation, however like has been said before, you can't solve a UX problem with documentation. Then we come to why might we want to not provide a safe random by default for the folks in the #3 group. As we've seen in the other thread, this basically boils down to the fact that for a lot of users they don't care about the security properties and they just want a fast random-esque value. This particular case is made stronger by the fact that there is a lot of code out there using Python's random module in a completely safe way that would regress in a meaningful way if the random module slowed down. The fact that speed is the primary reason not to give people in #3 a cryptographically secure source of random by default is where we come back to the meat of Theo's suggestion. His claim is that invoking os.urandom through any of the interfaces imposes a performance penalty because it has to round trip through the kernel crypto sub system for every request. His suggestion is essentially that we provide an interface to a modern, good, userland cryptographically secure source of random that is running within the same process as Python itself. One such example of this is the arc4random function (which doesn't actually provide ARC4 on OpenBSD, it provides ChaCha, it's not tied to one specific algorithm) which comes from libc on many platforms. According to Theo, modern userland CSPRNGs can create random bytes faster than memcpy which eliminates the argument of speed for why a CSPRNG shouldn't be the "default" source of randomness. Thus the proposal is essentially: * Provide an API to access a modern userland CSPRNG. * Provide an implementation of random.SomeKindOfRandom that utilizes this. * Move the MT based implementation of the random module to random.DeterministicRandom. * Deprecate the module scoped functions, instructing people to use the new random.SomeKindofRandom unless they need deterministic random, in which case use random.DeterministicRandom. This can of course be tweaked one way or the other, but that's the general idea translated into something actionable for Python. I'm not sure exactly how I feel about it, but I certainly do think that the current situation is confusing to end users and leaving them in an insecure state, and that a minimum we should move MT to something like random.DeterministicRandom and deprecate the module scoped functions because it seems obvious to me that the idea of a "default" random function that isn't safe is a footgun for users. As an additional consideration, there are security experts who believe that userland CSPRNGs should not be used at all. One of those is Thomas Ptacek who wrote a blog post [1] on the subject. In this, Thomas makes the case that a userland CSPRNG pretty much always depends on the cryptographic security of the system random, but that it itself may be broken which means you're adding a second, single point of failure where a mistake can cause you to get non-random data out of the system. I had asked Theo about this, and he stated that he disagreed with Thomas about never using a userland CSPRNG and in his opinion that blog post was mostly warning people away from using something like MT in the userland and away from /dev/random (which is often the cause of people reaching for MT because /dev/random blocks which makes programs even slower). It seems to boil down to, do we want to try to protect users by default or at least make it more obvious in the API which one they want to use (I think yes), and if so do we think that /dev/urandom is "fast enough" for most people in group #3 and if not, do we agree with Theo that a modern userland CSPRNG is safe enough to use, or do we agree with Thomas that it's not and if we think that it is, do we use arc4random and what do we do on systems that don't have a modern userland CSPRNG in their libc. [1] http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA [View Less]

24 100

Re: [Python-ideas] Should our default random number generator be secure?
by Tim Peters Sept. 14, 2015

Sept. 14, 2015

[Nathaniel Smith <njs(a)vorpus.org>] > Yeah, the independent-seed-for-each-thread approach works for any RNG, but > just like people feel better if they have a 100% certified guarantee that > the RNG output in a single thread will pass through every combination of > possible values (if you wait some cosmological time), they also feel better > if there is some 100% certified guarantee that the RNG values in two threads > will also be uncorrelated with each other. > … [View More]

3 4

Round division
by Serhiy Storchaka Sept. 13, 2015

Sept. 13, 2015

In Python there is a operation for floor division: a // b. Ceil division easy can be expressed via floor division: -((-a) // b). But round division is more complicated. This operation is needed in Fraction.__round__, in a number of methods in the datetime module (see _divide_and_round). Due to the complexity of the correct Python implementation, it is slower then just division. I propose to add special function in the math module. This not only will speed up Python implementation of the … [View More]

9 10

Re: [Python-ideas] Should our default random number generator be secure?
by M.-A. Lemburg Sept. 13, 2015

Sept. 13, 2015

On 10.09.2015 19:04, Xavier Combelle wrote: >> I think this is the major misunderstanding here: >> >> The random module never suggested that it generates pseudo-random data >> of crypto quality. >> >> I'm pretty sure people doing crypto will know and most others >> simply don't care :-) >> >> Evidence: We used a Wichmann-Hill PRNG as default in random >> for a decade and people still got their work done. Mersenne >> was added in … [View More]Python 2.3 and bumped the period from >> 6,953,607,871,644 (13 digits) to 2**19937-1 (6002 digits). > > It is not a evidence, I have an evidence of the opposite: > some people can and does use random.random() for generating session key or > csrf tokens and it's an insecure default. It all depends on what you consider "secure" or "secure enough" and points directly to another misunderstanding: that "secure" is a well-defined term :-) The random module seeds its global Random instance using urandom (if available on the system), so while the generator itself is deterministic, the seed used to kick off the pseudo-random series is not. For many purposes, this is secure enough. It's also easy to make the output of the random instance more secure by passing it through a crypto hash function. But back to the original question: What is "secure" ? In crypto terms, "secure" usually refers to "computationally infeasible to calculate before the sun goes dark" (to take one variant). More realistically, it can be defined as: Based on the public knowledge known today, it's impossible to run a program which allows converting the output of a crypto function back to its inputs within a reasonable time span. And this property will - based on today's knowledge - hold for at least the next 5-10 years. You may notice the many parameters in these definition attempts. It all depends on who you ask. With the advent of new technologies like quantum computers, it's not at all clear that any of those definitions will still hold in a couple of years. It's well possible that only quantum computers will be able to implement the necessary programs and it'll take a while for mobile phones to catch up and come with chips implementing those ;-) Now, leaving aside this bright future, what's reasonable today ? If you look at tools like untwister: https://github.com/bishopfox/untwister you can get a feeling for how long it takes to deduce the seed from an output sequence. Bare in mind, that in order to be reasonably sure that the seed is correct, the available output sequence has to be long enough. That's a known plain text attack, so you need access to lots of session keys to begin with. The tools is still running on an example set of 1000 32-bit numbers and it says it'll be done in 1.5 hours, i.e. before the sun goes down in my timezone. I'll leave it running to see whether it can find my secret key. Untwister is only slightly smarter than bruteforce. Given that MT has a seed size of 32 bits, it's not surprising that a tool can find the seed within a day. Perhaps it's time to switch to a better version of MT, e.g. a 64-bit version (with 64-bit internal state): http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt64.html or an even faster SIMD variant with better properties and 128 bit internal state: http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/SFMT/index.html Esp. the latter will help make brute force attacks practically impossible. Tim ? BTW: Looking at the sources of the _random module, I found that the seed function uses the hash of non-integers such as e.g. strings passed to it as seeds. Given the hash randomization for strings this will create non-deterministic results, so it's probably wise to only use 32-bit integers as seed values for portability, if you need to rely on seeding the global Python RNG. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 11 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2015-09-18: PyCon UK 2015 ... 7 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ [View Less]

4 13

Re: [Python-ideas] Python's Source of Randomness and the random.py module Redux
by random832＠fastmail.us Sept. 11, 2015

Sept. 11, 2015

On Fri, Sep 11, 2015, at 09:36, Steven D'Aprano wrote: > Yes, calling `random.choice` is *significantly better* than calling > `random.SomethingRandom().choice`. It's better for beginners, it's even > better for expert users whose random needs are small, and those whose > needs are greater shouldn't be using the later anyway. Why is it that people who need deterministic/seed based random aren't considered to be "those whose needs are greater"?

2 2

Re: [Python-ideas] Structural type checking for PEP 484
by Jukka Lehtosalo Sept. 11, 2015

Sept. 11, 2015

On Wed, Sep 9, 2015 at 10:48 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: > On Sep 9, 2015, at 21:34, Jukka Lehtosalo <jlehtosalo(a)gmail.com> wrote: > > I'm not sure if I fully understand what you mean by implicit vs. explicit > ABCs (and the static/runtime distinction). Could you define these terms and > maybe give some examples of each? > > > I just gave examples just one paragraph above. > > A (runtime) implicit ABC is something that uses a … [View More]__subclasshook__ > (usually implementing a structural check). So, for instance, any type that > implements __iter__ is-a Iterable, e.g., according to isinstance or > issubclass or @singledispatch, because that's what > Iterable.__subclasshook__ checks for. > > A (runtime) explicit ABC is something that isn't implicit, like Sequence: > no hook, so nothing is-a Sequence unless it either inherits the ABC or > registers with it. > > You're proposing a parallel but separate distinction at static typing > time. Any ABC that's a Protocol is checked based on a structural check; > otherwise, it's checked based on inheritance. > In my proposal I actually suggest that protocols shouldn't support isinstance or issubclass (these operations should raise an exception) by default. A protocol is free to override the default exception-raising __subclasshook__ to implement a structural check, and a static type checker would allow isinstance and issubclass for protocols that do this. I'll need to explain this idea in more detail, as clearly the current explanation is too easy to misundertand. Here's a concrete example: class X(Protocol): def f(self): ... class A: def f(self): print('f') if isinstance(A(), X): ... # Raise an exception, because no __subclasshook__ override in X Previously I toyed with the idea of having a default implementation of __subclasshook__ that actually does a structural check, but I'm no longer sure if that would be desirable, as it's difficult to come up with an implementation that does the right thing in all reasonable cases. For example, consider a structural type like this that people might want to use to work around the current limitations of Callable (it doesn't support keyword arguments, for example): class MyCallable(Protocol): def __call__(self, x, y): ... (This example has some other potential issues that I'm hand-waving away for now.) Now how would the default isinstance work? Preferably it should only accept callables that are compatible with the signature, but doing that check is pretty difficult for arbitrary functions and should probably be out of scope for the typing module. Just checking whether __call__ exists would be too general, as the programmer probably expects that he's able to call the method with the specific arguments the type suggests. Also, sometimes checking the argument names would be a good thing to do, but sometimes any names (as long the the number of arguments is compatible) would be fine. > This means it's now possible to create supertypes that are implicit at > runtime but explicit at static typing time (which might occasionally be > useful), or vice-versa (which I can't imagine why you'd ever want). > As I showed above, you wouldn't get the latter unless you really try very hard (consenting adults and all). > > Besides the obvious negatives in having two not-quite-compatible and > very-different-looking ways of expressing the same concept, this is going > to lead to people wanting to know why their type checker is complaining > about perfectly good code ("I tested that constant with isinstance, and it > really is-a Spammable, and the type checker is inferring its type properly, > and yet I get an error passing it to a function that wants a Spammable") or > allowing blatantly invalid code ("I annotated my function to only take > Spammable arguments, but someone is passing something that calls the > fallback implementation of my singledispatch function instead of the > Spammable overload"). > I agree that having the default nominal/explicit isinstance semantics for a protocol type would be a very bad idea. > > Maybe the solution is to expand your proposal a little: make Protocol > automatically create a __subclasshook__ (which you listed as an optional > idea in the proposal), and also change all of the existing stdlib implicit > ABCs to Protocols and scrap their manual hooks, and also update the > relevant documentation (e.g., the abc module and the data model section on > __subclasshook__) to recommend using Protocol instead of implementing a > manual hook if the only thing you want is structural subtyping. Of course > the backward compatibility isn't perfect (unless you want to manually munge > up collections.abc when typing is imported), and people using legacy > third-party code might need to add stubs (although that seems necessary > anyway). But for most people, everything should just work as people expect. > A type is either structurally typed or explicitly (via inheritance or > registration) types, both at static typing time and a runtime, and that's > always expressed by the name Protocol. (But for the rare cases when you > really need a type check that's looser at runtime, you can still write a > manual hook to handle that.) > > Yeah, this would be nice, but as I argued above, implementing a generic __subclasshook__ is actually quite tricky. Jukka [View Less]

1 0

BUG in standard while statement
by Stephan Sahm Sept. 11, 2015

Sept. 11, 2015

Dear all I found a BUG in the standard while statement, which appears both in python 2.7 and python 3.4 on my system. It usually won't appear because I only stumbled upon it after trying to implement a nice repeat structure. Look: ``` class repeat(object): def __init__(self, n): self.n = n def __bool__(self): self.n -= 1 return self.n >= 0 __nonzero__=__bool__ a = repeat(2) ``` the meaning of the above is that bool(a) returns True 2-times, and after … [View More]

9 10

Desperate need for enhanced print function
by Anand Krishnakumar Sept. 11, 2015

Sept. 11, 2015

Hi! This is my first time I'm sending an email to the python-ideas mailing list. I've got an enhancement idea for the built-in print function and I hope it is as good as I think it is. Imagine you have a trial.py file like this: a = 4 b = "Anand" print("Hello, I am " + b + ". My favorite number is " + str(a) + ".") OR print("Hello, I am ", b, ". My favorite number is ", a, ".") Well I've got an idea for a function named "print_easy" (The only valid name I could come up with right now). So … [View More]

17 47

Wheels For ...
by Sven R. Kunze Sept. 10, 2015

Sept. 10, 2015

Hi folks, currently, I came across http://pythonwheels.com/ during researching how to make a proper Python distribution for PyPI. I thought it would be great idea to tell other maintainers to upload their content as wheels so I approached a couple of them. Some of them already provided wheels. Happy being able to have built my own distribution, I discussed the issue at hand with some people and I would like to share my findings and propose some ideas: 1) documentation is weirdly split … [View More]

9 10

Re: [Python-ideas] Should our default random number generator be secure?
by Tim Peters Sept. 10, 2015

Sept. 10, 2015

[Tim, on parallel PRNGs] >> There are some clean and easy approaches to this based on >> crypto-inspired schemes, but giving up crypto strength for speed. If >> you haven't read it, this paper is delightful: >> >> http://www.thesalmons.org/john/random123/papers/random123sc11.pdf [Nathaniel Smith] > It really is! As AES acceleration instructions become more common > (they're now standard IIUC on x86, x86-64, and even recent ARM?), even > just using … [View More]

2 1

Re: [Python-ideas] Should our default random number generator be secure?
by Guido van Rossum Sept. 9, 2015

Sept. 9, 2015

---------- Forwarded message ---------- From: Theo de Raadt Date: Wed, Sep 9, 2015 at 10:42 AM Subject: Re: getentropy, getrandom, arc4random() To: guido(a)python.org been speaking to a significant go person. confirmed. it takes data out of that buffer, and does not zero it behind itself. obviously for performance reasons. same type of thing happens with MT-style engines. in practice, they can be would backwards. a proper stream cipher cannot be turned backwards. however, that's just an … [View More]

1 0

Re: [Python-ideas] Should our default random number generator be secure?
by Guido van Rossum Sept. 9, 2015

Sept. 9, 2015

---------- Forwarded message ---------- From: Theo de Raadt Date: Wed, Sep 9, 2015 at 10:36 AM Subject: Re: getentropy, getrandom, arc4random() To: guido(a)python.org > Yet another thing. Where do you see that Go and Swift have secure random as > a keyword? Searching for "golang random" gives the math/rand package as the > first hit, which has a note reminding the reader to use crypto/rand for > security work. yes, well, look at the other phrase it uses... that produces a … [View More]

1 0

Re: [Python-ideas] Should our default random number generator be secure?
by Guido van Rossum Sept. 9, 2015

Sept. 9, 2015

I'm just going to forward further missives by Theo. ---------- Forwarded message ---------- From: Theo de Raadt Date: Wed, Sep 9, 2015 at 9:59 AM Subject: Re: getentropy, getrandom, arc4random() To: guido(a)python.org > Thanks. And one last thing: unless Go and Swift, Python has no significant > corporate resources behind it -- it's mostly volunteers begging their > employers to let them work on Python for a few hours per week. i understand because I find myself in the same … [View More]situation. however i think you overstate the difficulty involved. high-availibility random is kind of a new issue. so final advice from me; feel free to forward as you like. i think arc4random would be a better API to call on the back side than getentropy/getrandom. arc4random can seed initialize with a single getentropy/getrandom call at startup. that is done automatically. you can then use arc4random's results to initialize the MT. in a system call trace, this will show up as one getentropy/getrandom at process startup, which gets both subsystems going. really cheap. in the case of longer run times, the userland arc4random PRNG folding reduces the system calls required. this helps older kernels with slower entropy creation, taking pressure off their subsystem. driving everyone towards this one API which is so high performance is the right goal. chacha arc4random is really fast. if you were to create such an API in python, maybe this is how it will go: say it becomes arc4random in the back end. i am unsure what advice to give you regarding a python API name. in swift, they chose to use the same prefix "arc4random" (id = arc4random(), id = arc4random_uniform(1..n)"; it is a little bit different than the C API. google has tended to choose other prefixes. we admit the name is a bit strange, but we can't touch the previous attempts like drand48.... I do suggest you have the _uniform and _buf versions. Maybe apple chose to stick to arc4random as a name simply because search engines tend to give above average advice for this search string? so arc4random is natively available in freebsd, macos, solaris, and other systems like andriod libc (bionic). some systems lack it: win32, glibc, hpux, aix, so we wrote replacements for libressl: https://github.com/libressl-portable/openbsd/tree/master/src/lib/libcrypto/… https://github.com/libressl-portable/portable/tree/master/crypto/compat the first is the base openbsd tree where we maintain/develop this code for other systems, the 2nd part is scaffold in libressl that makes this available to others. it contains arc4random for those systems, and supplies getentropy() stubs for challenged systems. we'll admit we haven't got solutions for every system known to man. we are trying to handle fork issues, and systems with very bad entropy feeding. that's free code. the heavy lifting is done, and we'll keep maintaining that until the end of days. i hope it helps. -- --Guido van Rossum (python.org/~guido) [View Less]

1 0

NuGet/Chocolatey feed for releases
by Alexander Walters Sept. 9, 2015

Sept. 9, 2015

It would be incredibly convenient, especially for users of AppVayor's continuous integration service, if there were a(n official) repository for chocolatey containing recent releases of python. The official Chocolatey gallery contains installers for the latest 2.7 and 3.4 (as of this post). What I am proposing would contain the most commonly used pythons in testing (2.6 2.7 3.3 3.4 and future releases). I am perfectly willing to set up a repo for my own use, but am posting this to see … [View More]

3 4

Re: [Python-ideas] Wheels For ...
by Laura Creighton Sept. 8, 2015

Sept. 8, 2015

In a message of Sun, 06 Sep 2015 15:31:16 -0400, Terry Reedy writes: >On 9/6/2015 1:33 PM, Sven R. Kunze wrote: >> >> With this post, I would like raise awareness of the people in charge of >> the Python infrastructure. > >pypa is in charge of packaging. https://github.com/pypa >I believe the google groups link is their discussion forum. They have one -- https://groups.google.com/forum/#!forum/pypa-dev but you can also get them at the disutils mailing list. https:… [View More]

3 4

PyParallel update
by Trent Nelson Sept. 6, 2015

Sept. 6, 2015

[CC'ing python-dev@ for those that are curious; please drop and keep follow-up discussion to python-ideas@] Hi folks, I've made a lot of progress on PyParallel since the PyCon dev summit (https://speakerdeck.com/trent/pyparallel-pycon-2015-language-summit); I fixed the outstanding breakage with generators, exceptions and whatnot. I got the "instantaneous Wiki search server" working[1] and implemented the entire TechEmpower Frameworks Benchmark Suite[2], including a PyParallel-friendly pyodbc … [View More]module, allowing database connections and querying in parallel. [1]: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/wiki… [2]: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/tefb… I set up a landing page for the project: http://pyparallel.org And there was some good discussion on reddit earlier this week: https://www.reddit.com/r/programming/comments/3jhv80/pyparallel_an_experime… I've put together some documentation on the project, its aims, and the key parts of the solution regarding the parallelism through simple client/server paradigms. This documentation is available directly on the github landing page for the project: https://github.com/pyparallel/pyparallel Writing that documentation forced me to formalize (or at least commit) to the restrictions/trade-offs that PyParallel would introduce, and I'm pretty happy I was basically able to boil it down into a single rule: Don't persist parallel objects. That keeps the mental model very simple. You don't need to worry about locking or ownership or races or anything like that. Just don't persist parallel objects, that's the only thing you have to remember. It's actually really easy to convert existing C code or Python code into something that is suitable for calling from within a parallel callback by just ensuring that rule isn't violated. It took about four hours to figure out how NumPy allocated stuff and add in the necessary PyParallel-aware tweaks, and not that much longer for pyodbc. (Most stuff "just works", though.) (The ABI changes would mean this is a Python 4.x type of thing; there are fancy ways we could avoid ABI changes and get this working on Python 3.x, but, eh, I like the 4.x target. It's realistic.) The other thing that clicked is that asyncio and PyParallel would actually work really well together for exploiting client-driven parallelism (PyParallel really is only suited to server-oriented parallelism at the moment, i.e. serving HTTP requests in parallel). With asyncio, though, you could keep the main-thread/single-thread client-drives-computation paradigm, but have it actually dispatch work to parallel.server() objects behind the scenes. For example, in order to process all files in a directory in parallel, asyncio would request a directory listing (i.e. issue a GET /) which the PyParallel HTTP server would return, it would then create non-blocking client connections to the same server and invoke whatever HTTP method is desired to do the file processing. You can either choose to write the new results from within the parallel context (which could then be accessed as normal files via HTTP), or you could have PyParallel return json/bytes, which could then be aggregated by asyncio. Everything is within the same process, so you get all the benefits that provides (free access to anything within scope, like large data structures, from within parallel contexts). You can synchronously call back into the main thread from a parallel thread, too, if you wanted to update a complex data structure directly. The other interesting thing that documentation highlights is the advantage of the split brain "main thread vs parallel thread" GC and non-GC allocators. I'm not sure if I've ever extolled the virtue of such an approach on paper or in e-mail. It's pretty neat though and allows us to avoid a whole raft of problems that need to be solved when you have a single GC/memory model. Next steps: once 3.5 is tagged, I'm going to bite the bullet and rebase. That'll require a bit of churn, so if there's enough interest from others, I figured we'd use the opportunity to at least get it building again on POSIX (Linux/OSX/FreeBSD). From there people can start implementing the missing bits for implementing the parallel machinery behind the scenes. The parallel interpreter thread changes I made are platform agnostic, the implementation just happens to be on Windows at the moment; don't let the Windows-only thing detract from what's actually being pitched: a (working, demonstrably-performant) solution to "Python's GIL problem". Regards, Trent. [View Less]

1 0

Re: [Python-ideas] Add appdirs module to stdlib
by Andrew Barnert Sept. 4, 2015

Sept. 4, 2015

Responses below, but first, another issue: Things like app data and prefs aren't a single directory. XDG has a notion of a search path rather than a single directory; Windows has a notion of separate search domains; OS X makes things as fun as possible by having both. For writing a new file, you're usually fine just writing to the first path in the default domain (as long as it exists or you can create it), but for reading files you're supposed to look in /etc or All Users or whatever if it's … [View More]not found there. Most cross-platform wrappers I've used in the past didn't deal with this automatically, and a lot of them didn't even make it easy to do manually. On Sep 1, 2015, at 14:19, Nathaniel Smith <njs(a)vorpus.org> wrote: > > On Sep 1, 2015 02:45, "Andrew Barnert via Python-ideas" > <python-ideas(a)python.org> wrote: >> >>> On Sep 1, 2015, at 01:00, Philipp A. <flying-sheep(a)web.de> wrote: >>> >>> When defining a place for config files, cache files, and so on, people usually hack around in a OS-dependent, misinformed, and therefore wrong way. >>> >>> Thanks to the tempfile API we at least don’t see people hardcoding /tmp/ too much. >>> >>> There is a beautiful little module that does things right and is easy to use: appdirs >> >> >> Is appdirs compatible with the OS X recommendations (as required by the App Store). Apple only gives you cache and app data directories; prefs are supposed to use NSDefaults API or emulate the file names and formats properly, and you have to be sensitive to the sandbox.) > > No, AFAICT it doesn't get this right -- it just hard-codes the OS X > directories. The biggest problem with most of the cross-platform libraries I've seen is that they assume there is a prefs directory, and on OS X, that really isn't true. If your app explicitly opens the exact same file that NSDefaults would have opened, you're breaking the rules. (Since the mandatory sandbox went into effect, this usually doesn't get you rejected from the App Store anymore, but before that it did.) And taking a quick look at appdirs, it has a user_config_dir that seems to mean exactly that. So, how can a stdlib library handle that? Meanwhile, it looks like appdirs expects you to give it an app name and company name to construct the paths. What happens if you give it names that don't match the ones in your bundle? You're opening files that belong to another app. Which is again violating Apple's rules. One more thing: I don't know if it's guaranteed that the right way of doing things on OS X (whether via Cocoa or CoreFoundation) won't spawn a background thread for you. After all, the APIs can talk to the sandbox service and sometimes even the iCloud service. Is that a problem for something in the stdlib? > It also didn't quite implement the XDG spec correctly > (there's some fallback behavior you're supposed to do if the magic > envvars don't make sense that it skips -- very unusual that this will > matter). And windows I'm not sure about -- the logic in appdirs looked > reasonable to me when I was reviewing this a few months ago, but there > seem to be a bunch of semi-contradictory standards and so it's hard to > know what's even "correct" in the tricky cases. > > All of this is probably as much an argument *for* providing the > functionality as a standard thing as it is against, > but any PEP here > probably needs to be thorough about citing the research to show that > it's actually getting the various platform standards correct. > > What makes it particularly difficult is that if you "fix a bug" in a > library like appdirs, so that it starts suddenly returning different > results on some computer somewhere, then what it looks like to the end > user is that their data/settings/whatever have suddenly evaporated and > whatever disk space was being used for caches never gets cleaned up > and so forth. Generally when applications change how they compute > these directories, they also include tricky migration logic to check > both the old and new names, move stuff over if needed, but I'm not > sure how a low-level library like this can support that usefully... And that's especially true in the case of Apple's standards, which also include specific rules about how you're supposed to do such a migration, and doing so requires stuff that can't be done from entirely inside the code. [View Less]

12 17

ideas for type hints for variable: beyond comments
by Ian Sept. 2, 2015

Sept. 2, 2015

mypy currently inspects the comment on the line of first assignment for the variables to be type hinted. It is logical that at some time python language will add support to allow these type hints to move from comments to the code as has happened for 'def' signatures. One logical syntax would be to move from i = 1 # Infer type int for i to i:int = 1 # no comment needed, but does not look attractive The first question that arises is 'is the type inference legal for the … [View More]

4 4

Re: [Python-ideas] ideas for type hints for variable: beyond comments
by Ian Sept. 1, 2015

Sept. 1, 2015

Chris Angelico wrote: " > It is logical that at some time python language will add support to allow > these type hints to move from comments to the code as has happened for 'def' > signatures. Potential problem: Function annotations are supported all the way back to Python 3.0, but any new syntax would be 3.6+ only. That's going to severely limit its value for quite some time. That doesn't mean new syntax can't be added (otherwise none ever would), but the bar is that much higher … [View More]- you'll need an extremely compelling justification. " My intent must not have been clear. I am not suggesting changing function annotations. I think function annotations as they are represent an addition that has been well thought out and is a very useful step which extends what is possible in a very useful way. Pep 484 as introduced in 3.5 allows this to be taken further. I am suggesting building on PEP 484 with a complimentary extensions for variables. If extensions are made in the same manner as function annotations, then the actual python code simply has hints added. Generating warnings or other steps is the domain of separate off-line checkers. I feel it is clear that at some time an extension to allow type-hints for variables, complimenting current function annotations will be added. I am just providing food for thought on how annotations for variables can be added. > The other question is 'what about globals and nonlocals?'. Currently > globals and nonlocals need a 'global' or 'nonlocal' statement to allow > assignment, but what if these values are not assigned in scope? "Not sure what you're talking about here. If they're not assigned in this scope, then presumably they have the same value they had from some other scope. You shouldn't need to declare that "len" is a function, inside every function that calls it. Any type hints should go where it's assigned, and nowhere else. " These are hints. Not a 'need'. The type hints may be desired in the code referencing the 'globals' or 'nonlocals', but not desired in the original context. The idea is to allow this, NOT to require or need a declaration. Hope this helps clarify what I am trying to suggest. > What if we allowed > global i:int > > or > > nonlocal i:int > > and even > > local i:int > > Permitting a new keyword 'local' to me might bring far more symmetry between > different cases. Hey, if you want C, you know where to find it :) > Use of the 'local' keyword in the global namespace could indicate a value > not accessible in other namespaces. "I'm not sure what "not accessible" would mean. If someone imports your module, s/he gains access to all your globals. Do you mean that it's "not intended for external access" (normally notated with a single leading underscore)? Or is this a new feature - some way of preventing other modules from using these? That might be useful, but that's a completely separate proposal. " Good point, the single _already does what I was thinking. I never think of using it in this specific case. I tend to associate it with hinting that an identifier is for internal use within a class. Not to get warnings about use from outside the global namespace of globals. > Personally I would like to go even further and allow some syntax to allow > (or disable) flagging the use of new variables without type hinting as > possible typos "If you're serious about wanting all your variables to be declared, then I think you want a language other than Python. There are such languages around (and maybe even compiling to Python byte-code, I'm not sure), but Python isn't built that way. Type hinting is NOT variable declaration, and never will be. (Though that's famous last words, I know, and I'm not the BDFL or even anywhere close to that. If someone pulls up this email in ten years and laughs in my face, so be it. It'd not be the first time I've been utterly confidently wrong!) " No, I am not serious about wanting all variables to be declared under normal circumstances. Again, as you say, this is about hinting and getting warnings. I think there are circumstances where hinting may be sufficiently useful that getting a warning for a missing hint would be desirable. This is not a suggestion to change python. But a suggestion to allow for specific situations without a change of how things normally happen. Thank you for taking the time to comment. It is appreciated and I hope I have been able to use your feedback to clarify. [View Less]

2 1

ideas for type hints for variable: beyond comments
by ia n Sept. 1, 2015

Sept. 1, 2015

mypy currently inspects the comment on the line of first assignment for the variables to be type hinted. It is logical that at some time python language will add support to allow these type hints to move from comments to the code as has happened for 'def' signatures. One logical syntax would be to move from i = 1 # Infer type int for i to i:int = 1 # no comment needed, but does not look attractive The first question that arises is 'is the type inference legal for the … [View More]

1 0