Mailman 3 June 2016 - Python-ideas

Strings can sometimes convert to bytes without an encoding
by Franklin? Lee June 15, 2016

June 15, 2016

Current behavior (3.5.1): >>> bytes('') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: string argument without an encoding Suggestion: If the character size is 1, the `bytes`/`bytearray` constructor doesn't need a specified encoding. High-level idea: If the string only has code points in range(128), encoding is optional (and useless anyway). The new error message could be TypeError: non-ASCII string argument without an encoding How: CPython strings store characters in an array, such that each character takes a single entry. With an entry per character, indexing is just a regular C array index operation. Since PEP 393, the size( )of the elements of the array is just the size needed for the largest character. Thus, CPython strings "know" whether or not they're ASCII. Other implementations without PEP 393 can do a scan of the code points to check the 0-127 condition during building. That means O(n) more checks, but in those implementations, the per-character checks are already necessary with an explicit encoding, since you'd need to see if that character needs encoding. (The `bytes` and ASCII-`str` could in fact share memory, given a few tweaks. But that's an implementation detail.)

11 14

Proposal for special name and qualname symbols
by Steven D'Aprano June 14, 2016

June 14, 2016

This is an new thread inspired by the "Quick idea: defining variables from functions that take the variable name" thread. For the benefit of those who haven't read that thread, in a nutshell, there are some functions or classes that find it useful or necessary to know their own name or qualified name. At the moment, apart from a few special cases where the compiler does the work behind the scenes (function and class definitions), the only way to do this is to hard-code the name/qualname as a string. The classic example from the standard library is namedtuple: Record = namedtuple('Record', fields) Proposal: we introduce two special symbols to represent the name and qualname at any point. They can be used anywhere an identifier can be used. We could use special dunder names, let's say __NAME__ and __QNAME__, but for now I'm going to use two special symbols: @ for the unqualified bare name; @@ for the qualified name. The namedtuple example would become: Record = namedtuple(@, fields) If your function takes its name parameter somewhere other than the first argument, that's not a problem: MyClass = make_a_class(spam, eggs, name=@, fnord=True) '@' takes on the name, or dotted name, on the left-hand side of an assignment: x = function(@) # @ == 'x' spam.x = function(@) # @ == 'spam.x' If there are multiple targets, and they are all simple or dotted names, @ is a tuple of names: a, b, c.d = function(@) # @ == ('a', 'b', 'c.d') a = b = c.d = function(@) # @ == ('a', 'b', 'c.d') Those two cases are indistinguishable by @. I'm not sure if that's a bug or a feature :-) For now, any other assignment target will be a syntax error. That restriction may be relaxed in the future, if necessary. spam['key'] = function(@) # SyntaxError @ is not restricted to appear inside function calls: element = mapping[@] # like mapping['element'] The qualified name @@ gets the module name: x = function(@@) # @@ = 'module.x' or, inside a class or function, the class/function name: class K: attr = @@ # 'module.K.attr' def method(self): x = @@ # 'module.K.method.x' Whether dunder names or symbols, assignment to them will be illegal: @ = "myname" # SyntaxError Questions and answers: Q: Why @ rather than $? A: $ is too similar to S for my liking, and I believe that Guido has said that $ will never be used for syntax in Python. Q: What's the use-case for qualified names? A: Classes need to know their qualified name for pickling. Q: Isn't this rather magical? A: Of course it is, but the `def` and `class` statements already make use of that compile magic, this just extends it to other things. Q: Are you sure about the module being part of the qualified name? A: I'm not wedded to that idea. If it's not necessary, take it out. Q: Wait a minute, isn't @ used for matrix multiplication now? And @@ may be used for matrix exponentiation in the future? A: Oh yeah, I forgot that. Damn. Guess we'll have to use the dunder names then. Q: But the dunder names are too long to type. A: Then think of your own symbols and propose them :-) -- Steve

18 27

bool.from_config_str()
by jab＠math.brown.edu June 14, 2016

June 14, 2016

Dear Python-Ideas, (New here... hope this idea contributes something!) Several times I've had to implement a silly function that converts the odd environment variable (or some other value from the outside world that, perforce, comes in as a string) to the boolean it actually represents. I say "silly" because, for other commonly-needed types of primitive values, we have concise, idiomatic ways to convert them from strings, leaving no need for wheel reinvention: int(os.getenv('NUM_FJORDS')) float(os.getenv('PRICE_LIMBURGER')) etc. So I thought it might be nice if we could do something like this for booleans, too: Python 3.6.0a1+ (default:0b18f7d262cc+, Jun 12 2016, 18:21:54) [GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> bool.from_config_str('false') False >>> bool.from_config_str('False') False >>> bool.from_config_str('True') True >>> bool.from_config_str('true') True >>> bool.from_config_str('0') False >>> bool.from_config_str('') False >>> bool.from_config_str('1') True I seized the opportunity to work up my first (tiny) patch for CPython which implements this, along with tests (please see attached patch, which applies cleanly against current tip). Is there any interest in this? If so, I'd be happy to make it work for bytes too, or to make any other changes that would help get it landed. Thanks for your consideration, and looking forward to your feedback. Josh

12 16

eval_literal
by jab＠math.brown.edu June 13, 2016

June 13, 2016

(Riffing off some discussion in another thread, I had another idea I wanted to throw out there.) Given that Eval Really Is Dangerous[1], has something like this ever been considered?: >>> int(str(42)) 42 >>> float(str(42.0)) 42.0 >>> bool(str(False)) # :( True >>> eval_literal('42') 42 >>> eval_literal('42.0') 42.0 >>> eval_literal('False') # :) False >>> eval_literal('', default=False) # shout out to PEP 463 / Michael Selik False i.e. An extremely limited version of eval, possibly just for literals or even literal atoms, that would make it safe? [1] http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html

5 5

Quick idea: defining variables from functions that take the variable name
by Guido van Rossum June 9, 2016

June 9, 2016

In mypy we have a need for type variables, which are created like this: from typing import TypeVar T = TypeVar('T') I just saw a lightning talk about sympy where they define symbols to be used in mathematical equations, like this: from sympy import Symbol x = Symbol('x') I'm sure this is not a new idea, but so far I've always thought that this is pretty esoteric and the approach here is good enough. But maybe we can actually do better, and whatever solution we come up with might also be useful for extracting attributes from an object or values from a mapping? -- --Guido van Rossum (python.org/~guido)

29 120

Allow __bytes__ to return NotImplemented
by Émanuel Barry June 7, 2016

June 7, 2016

I'm currently writing a class that can take either a str of bytes object, and is meant to work with either type. For my purpose, it makes sense to return the underlying bytes object when calling bytes() on it. However, defining __bytes__ on the class means that I also have to return bytes even if the underlying object is a str. The obvious way to fix this is to define two different classes, one which handles str and one which handles bytes. However, I was wondering if returning NotImplemented from __bytes__ would be a viable alternative. I'm not sure if there'd be any performance hit (there is probably some optimized C code that simply checks for the existence of __bytes__ in the namespace without calling it), but hopefully there aren't. Another alternative would be to define object.__bytes__, which one can call to fall back to the default behaviour. While I'm at it, maybe other methods could use that, too (thinking mainly of __int__, __float__, __abs__, __complex__, __round__, __index__, and maybe __len__ and __iter__). Thoughts? Is it reasonable to define a single class that can work with both str and bytes, or should I fix my code? -Emanuel

4 6

1 + True = 2
by Giampaolo Rodola' June 6, 2016

June 6, 2016

>>> 1 + True 2 >>> 1 + False 1 >>> I bumped into this today by accident and I can't recall ever being aware of this. Why isn't this a TypeError in Python 3? -- Giampaolo - http://grodola.blogspot.com

7 8

Re: [Python-ideas] Exposing regular expression bytecode
by Jonathan Goble June 5, 2016

June 5, 2016

OK, thanks for the comments, everyone. I'm glad to hear that people generally think this is a useful idea. Some specific replies: On Tue, Feb 16, 2016 at 4:22 AM, Chris Angelico <rosuav(a)gmail.com> wrote: > For what it's worth, I read your post with interest, but didn't have > anything substantive to reply - mainly because I don't use regexes > much. But it would be rather cool to be able to decompile a regex. > Imagine a regex pretty-printer: compile an arbitrary string, and if > it's a valid regex, decompile it to a valid source code form, using > re.VERBOSE. That could help _hugely_ with debugging, if the trick can > be pulled off. > > ChrisA That's exactly the type of tools I envision being made available by third parties. Depending on how much I get invested into this project, I may even write such a tool myself (though that's not guaranteed). On Tue, Feb 16, 2016 at 4:55 AM, Paul Moore <p.f.moore(a)gmail.com> wrote: > Sorry. I don't personally have any issue with the proposal, and it > sounds like a reasonable idea. I don't think it's likely to be > *hugely* controversial - although it will likely need a little care in > documenting the feature to ensure that we are clear that there's no > guarantees of backward compatibility that we don't want to commit to > on the newly - exposed data. And we should also ensure that by > exposing this information, we don't preclude changes such as the > incorporation of the regex module (I don't know if the regex module > has a bytecode implementation like the re module does). The regex implementation is indeed something I would need to investigate here, and will do so before I go too far. > The next step is probably simply to raise a tracker issue for this. I > know you said you have little C experience, but the big problem is > that it's unlikely that any of the core devs with C experience will > have the time or motivation to code up your idea. So without a working > patch, and someone willing and able to respond to comments on the > patch, it's not likely to progress. Tracker issue is already filed: http://bugs.python.org/issue26336 I actually filed the issue before I realized that the mailing lists were a better place to discuss it. > But if you are willing to dig into Python's C API yourself (and it > sounds like you are) there are definitely people who will help you. > You might want to join the core mentorship list (see > http://pythonmentors.com/) where you should get plenty of assistance. > This proposal sounds like a great "beginner" task, as well - so even > if you don't want to implement it yourself, still put it on the > tracker, and mark it as an "easy" change, and maybe some other > newcomer who wants a task to help them learn the C API will pick it > up. I'll look into the mentorship list; thanks for the link. As for marking it "easy", I don't seem to have the necessary permissions to change the Keywords field; perhaps you or someone else can set that flag for me? If so, I'd appreciate it. :-) > Hope that helps - thanks for the suggestion and sorry if it seems like > no-one was interested at first. It's an unfortunate fact of life > around here that things *do* take time to get people's interest. You > mention patience in one of your messages - that's definitely something > you'll need to cultivate, I'm afraid... :-) Patience is something I've been working on since I was a little kid. I'm 29 years old now, and it still eludes me from time to time. But yes, it's something I'll have to work on. :-P Also, I received a small patch off-list from Petr Viktorin implementing a getter for the code list (thanks, Petr). I'll need to test it, but from the little I know of the C API it looks like it will get me started in the right direction. Assuming that works, what's left is a public constructor for the regex type (to enable optimizers), a dis-like module, and docs and tests. I don't think this would be major enough to require a PEP, but of course being new here, I'm open to being told I'm wrong. :-)

4 7

solving multi-core Python
by Eric Snow June 3, 2016

June 3, 2016

tl;dr Let's exploit multiple cores by fixing up subinterpreters, exposing them in Python, and adding a mechanism to safely share objects between them. This proposal is meant to be a shot over the bow, so to speak. I plan on putting together a more complete PEP some time in the future, with content that is more refined along with references to the appropriate online resources. Feedback appreciated! Offers to help even more so! :) -eric -------- Python's multi-core story is murky at best. Not only can we be more clear on the matter, we can improve Python's support. The result of any effort must make multi-core (i.e. parallelism) support in Python obvious, unmistakable, and undeniable (and keep it Pythonic). Currently we have several concurrency models represented via threading, multiprocessing, asyncio, concurrent.futures (plus others in the cheeseshop). However, in CPython the GIL means that we don't have parallelism, except through multiprocessing which requires trade-offs. (See Dave Beazley's talk at PyCon US 2015.) This is a situation I'd like us to solve once and for all for a couple of reasons. Firstly, it is a technical roadblock for some Python developers, though I don't see that as a huge factor. Regardless, secondly, it is especially a turnoff to folks looking into Python and ultimately a PR issue. The solution boils down to natively supporting multiple cores in Python code. This is not a new topic. For a long time many have clamored for death to the GIL. Several attempts have been made over the years and failed to do it without sacrificing single-threaded performance. Furthermore, removing the GIL is perhaps an obvious solution but not the only one. Others include Trent Nelson's PyParallels, STM, and other Python implementations.. Proposal ======= In some personal correspondence Nick Coghlan, he summarized my preferred approach as "the data storage separation of multiprocessing, with the low message passing overhead of threading". For Python 3.6: * expose subinterpreters to Python in a new stdlib module: "subinterpreters" * add a new SubinterpreterExecutor to concurrent.futures * add a queue.Queue-like type that will be used to explicitly share objects between subinterpreters This is less simple than it might sound, but presents what I consider the best option for getting a meaningful improvement into Python 3.6. Also, I'm not convinced that the word "subinterpreter" properly conveys the intent, for which subinterpreters is only part of the picture. So I'm open to a better name. Influences ======== Note that I'm drawing quite a bit of inspiration from elsewhere. The idea of using subinterpreters to get this (more) efficient isolated execution is not my own (I heard it from Nick). I have also spent quite a bit of time and effort researching for this proposal. As part of that, a number of people have provided invaluable insight and encouragement as I've prepared, including Guido, Nick, Brett Cannon, Barry Warsaw, and Larry Hastings. Additionally, Hoare's "Communicating Sequential Processes" (CSP) has been a big influence on this proposal. FYI, CSP is also the inspiration for Go's concurrency model (e.g. goroutines, channels, select). Dr. Sarah Mount, who has expertise in this area, has been kind enough to agree to collaborate and even co-author the PEP that I hope comes out of this proposal. My interest in this improvement has been building for several years. Recent events, including this year's language summit, have driven me to push for something concrete in Python 3.6. The subinterpreter Module ===================== The subinterpreters module would look something like this (a la threading/multiprocessing): settrace() setprofile() stack_size() active_count() enumerate() get_ident() current_subinterpreter() Subinterpreter(...) id is_alive() running() -> Task or None run(...) -> Task # wrapper around PyRun_*, auto-calls Task.start() destroy() Task(...) # analogous to a CSP process id exception() # other stuff? # for compatibility with threading.Thread: name ident is_alive() start() run() join() Channel(...) # shared by passing as an arg to the subinterpreter-running func # this API is a bit uncooked still... pop() push() poison() # maybe select() # maybe Note that Channel objects will necessarily be shared in common between subinterpreters (where bound). This sharing will happen when the one or more of the parameters to the function passed to Task() is a Channel. Thus the channel would be open to the (sub)interpreter calling Task() (or Subinterpreter.run()) and to the new subinterpreter. Also, other channels could be fed into such a shared channel, whereby those channels would then likewise be shared between the interpreters. I don't know yet if this module should include *all* the essential pieces to implement a complete CSP library. Given the inspiration that CSP is providing, it may make sense to support it fully. It would be interesting then if the implementation here allowed the (complete?) formalisms provided by CSP (thus, e.g. rigorous proofs of concurrent system models). I expect there will also be a _subinterpreters module with low-level implementation-specific details. Related Ideas and Details Under Consideration ==================================== Some of these are details that need to be sorted out. Some are secondary ideas that may be appropriate to address in this proposal or may need to be tabled. I have some others but these should be sufficient to demonstrate the range of points to consider. * further coalesce the (concurrency/parallelism) abstractions between threading, multiprocessing, asyncio, and this proposal * only allow one running Task at a time per subinterpreter * disallow threading within subinterpreters (with legacy support in C) + ignore/remove the GIL within subinterpreters (since they would be single-threaded) * use the GIL only in the main interpreter and for interaction between subinterpreters (and a "Local Interpreter Lock" for within a subinterpreter) * disallow forking within subinterpreters * only allow passing plain functions to Task() and Subinterpreter.run() (exclude closures, other callables) * object ownership model + read-only in all but 1 subinterpreter + RW in all subinterpreters + only allow 1 subinterpreter to have any refcounts to an object (except for channels) * only allow immutable objects to be shared between subinterpreters * for better immutability, move object ref counts into a separate table * freeze (new machinery or memcopy or something) objects to make them (at least temporarily) immutable * expose a more complete CSP implementation in the stdlib (or make the subinterpreters module more compliant) * treat the main interpreter differently than subinterpreters (or treat it exactly the same) * add subinterpreter support to asyncio (the interplay between them could be interesting) Key Dependencies ================ There are a few related tasks/projects that will likely need to be resolved before subinterpreters in CPython can be used in the proposed manner. The proposal could implemented either way, but it will help the multi-core effort if these are addressed first. * fixes to subinterpreter support (there are a couple individuals who should be able to provide the necessary insight) * PEP 432 (will simplify several key implementation details) * improvements to isolation between subinterpreters (file descriptors, env vars, others) Beyond those, the scale and technical scope of this project means that I am unlikely to be able to do all the work myself to land this in Python 3.6 (though I'd still give it my best shot). That will require the involvement of various experts. I expect that the project is divisible into multiple mostly independent pieces, so that will help. Python Implementations =================== They can correct me if I'm wrong, but from what I understand both Jython and IronPython already have subinterpreter support. I'll be soliciting feedback from the different Python implementors about subinterpreter support. C Extension Modules ================= Subinterpreters already isolate extension modules (and built-in modules, including sys). PEP 384 provides some help too. However, global state in C can easily leak data between subinterpreters, breaking the desired data isolation. This is something that will need to be addressed as part of the effort.

26 130

pattern matching proof-of-concept
by Michael Selik June 3, 2016

June 3, 2016

My original dict unpacking proposal was very short and lacked a motivating usage. Toy examples made my proposal look unnecessarily verbose and suggested obvious alternatives with easy current syntax. Nested/recursive unpacking is much more troublesome, especially when combined with name-binding. I wrote an example to compare my proposal with current syntax. Example usage. https://github.com/selik/destructure/blob/master/examples/fips.py Implementation. https://github.com/selik/destructure/blob/master/destructure.py The design of my module I'm least happy with is the name-binding. I extended a SimpleNamespace to create an Erlang-style distinction between bound and unbound names. Though the API is a bit awkward, now that the module is built, I'm less enthusiastic about introducing new syntax. Funny how that works. I haven't yet decided how to add post-binding guards to the cases.

4 7