Mailman 3 July 2015 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' … [View More]

51 165

solving multi-core Python
by Eric Snow June 4, 2016

June 4, 2016

tl;dr Let's exploit multiple cores by fixing up subinterpreters, exposing them in Python, and adding a mechanism to safely share objects between them. This proposal is meant to be a shot over the bow, so to speak. I plan on putting together a more complete PEP some time in the future, with content that is more refined along with references to the appropriate online resources. Feedback appreciated! Offers to help even more so! :) -eric -------- Python's multi-core story is murky at best. … [View More]Not only can we be more clear on the matter, we can improve Python's support. The result of any effort must make multi-core (i.e. parallelism) support in Python obvious, unmistakable, and undeniable (and keep it Pythonic). Currently we have several concurrency models represented via threading, multiprocessing, asyncio, concurrent.futures (plus others in the cheeseshop). However, in CPython the GIL means that we don't have parallelism, except through multiprocessing which requires trade-offs. (See Dave Beazley's talk at PyCon US 2015.) This is a situation I'd like us to solve once and for all for a couple of reasons. Firstly, it is a technical roadblock for some Python developers, though I don't see that as a huge factor. Regardless, secondly, it is especially a turnoff to folks looking into Python and ultimately a PR issue. The solution boils down to natively supporting multiple cores in Python code. This is not a new topic. For a long time many have clamored for death to the GIL. Several attempts have been made over the years and failed to do it without sacrificing single-threaded performance. Furthermore, removing the GIL is perhaps an obvious solution but not the only one. Others include Trent Nelson's PyParallels, STM, and other Python implementations.. Proposal ======= In some personal correspondence Nick Coghlan, he summarized my preferred approach as "the data storage separation of multiprocessing, with the low message passing overhead of threading". For Python 3.6: * expose subinterpreters to Python in a new stdlib module: "subinterpreters" * add a new SubinterpreterExecutor to concurrent.futures * add a queue.Queue-like type that will be used to explicitly share objects between subinterpreters This is less simple than it might sound, but presents what I consider the best option for getting a meaningful improvement into Python 3.6. Also, I'm not convinced that the word "subinterpreter" properly conveys the intent, for which subinterpreters is only part of the picture. So I'm open to a better name. Influences ======== Note that I'm drawing quite a bit of inspiration from elsewhere. The idea of using subinterpreters to get this (more) efficient isolated execution is not my own (I heard it from Nick). I have also spent quite a bit of time and effort researching for this proposal. As part of that, a number of people have provided invaluable insight and encouragement as I've prepared, including Guido, Nick, Brett Cannon, Barry Warsaw, and Larry Hastings. Additionally, Hoare's "Communicating Sequential Processes" (CSP) has been a big influence on this proposal. FYI, CSP is also the inspiration for Go's concurrency model (e.g. goroutines, channels, select). Dr. Sarah Mount, who has expertise in this area, has been kind enough to agree to collaborate and even co-author the PEP that I hope comes out of this proposal. My interest in this improvement has been building for several years. Recent events, including this year's language summit, have driven me to push for something concrete in Python 3.6. The subinterpreter Module ===================== The subinterpreters module would look something like this (a la threading/multiprocessing): settrace() setprofile() stack_size() active_count() enumerate() get_ident() current_subinterpreter() Subinterpreter(...) id is_alive() running() -> Task or None run(...) -> Task # wrapper around PyRun_*, auto-calls Task.start() destroy() Task(...) # analogous to a CSP process id exception() # other stuff? # for compatibility with threading.Thread: name ident is_alive() start() run() join() Channel(...) # shared by passing as an arg to the subinterpreter-running func # this API is a bit uncooked still... pop() push() poison() # maybe select() # maybe Note that Channel objects will necessarily be shared in common between subinterpreters (where bound). This sharing will happen when the one or more of the parameters to the function passed to Task() is a Channel. Thus the channel would be open to the (sub)interpreter calling Task() (or Subinterpreter.run()) and to the new subinterpreter. Also, other channels could be fed into such a shared channel, whereby those channels would then likewise be shared between the interpreters. I don't know yet if this module should include *all* the essential pieces to implement a complete CSP library. Given the inspiration that CSP is providing, it may make sense to support it fully. It would be interesting then if the implementation here allowed the (complete?) formalisms provided by CSP (thus, e.g. rigorous proofs of concurrent system models). I expect there will also be a _subinterpreters module with low-level implementation-specific details. Related Ideas and Details Under Consideration ==================================== Some of these are details that need to be sorted out. Some are secondary ideas that may be appropriate to address in this proposal or may need to be tabled. I have some others but these should be sufficient to demonstrate the range of points to consider. * further coalesce the (concurrency/parallelism) abstractions between threading, multiprocessing, asyncio, and this proposal * only allow one running Task at a time per subinterpreter * disallow threading within subinterpreters (with legacy support in C) + ignore/remove the GIL within subinterpreters (since they would be single-threaded) * use the GIL only in the main interpreter and for interaction between subinterpreters (and a "Local Interpreter Lock" for within a subinterpreter) * disallow forking within subinterpreters * only allow passing plain functions to Task() and Subinterpreter.run() (exclude closures, other callables) * object ownership model + read-only in all but 1 subinterpreter + RW in all subinterpreters + only allow 1 subinterpreter to have any refcounts to an object (except for channels) * only allow immutable objects to be shared between subinterpreters * for better immutability, move object ref counts into a separate table * freeze (new machinery or memcopy or something) objects to make them (at least temporarily) immutable * expose a more complete CSP implementation in the stdlib (or make the subinterpreters module more compliant) * treat the main interpreter differently than subinterpreters (or treat it exactly the same) * add subinterpreter support to asyncio (the interplay between them could be interesting) Key Dependencies ================ There are a few related tasks/projects that will likely need to be resolved before subinterpreters in CPython can be used in the proposed manner. The proposal could implemented either way, but it will help the multi-core effort if these are addressed first. * fixes to subinterpreter support (there are a couple individuals who should be able to provide the necessary insight) * PEP 432 (will simplify several key implementation details) * improvements to isolation between subinterpreters (file descriptors, env vars, others) Beyond those, the scale and technical scope of this project means that I am unlikely to be able to do all the work myself to land this in Python 3.6 (though I'd still give it my best shot). That will require the involvement of various experts. I expect that the project is divisible into multiple mostly independent pieces, so that will help. Python Implementations =================== They can correct me if I'm wrong, but from what I understand both Jython and IronPython already have subinterpreter support. I'll be soliciting feedback from the different Python implementors about subinterpreter support. C Extension Modules ================= Subinterpreters already isolate extension modules (and built-in modules, including sys). PEP 384 provides some help too. However, global state in C can easily leak data between subinterpreters, breaking the desired data isolation. This is something that will need to be addressed as part of the effort. [View Less]

26 130

Learning from the shell in supporting asyncio background calls
by Nick Coghlan Aug. 21, 2015

Aug. 21, 2015

Hi folks, Based on the recent discussions Sven kicked off regarding the complexity of interacting with asyncio from otherwise synchronous code, I came up with an API design that I like inspired by the way background and foreground tasks in the POSIX shell work. My blog post about this design is at http://www.curiousefficiency.org/posts/2015/07/asyncio-background-calls.html, but the essential components are the following two APIs: def run_in_background(target, *, loop=None): """… [View More]Schedules target as a background task Returns the scheduled task. If target is a future or coroutine, equivalent to asyncio.ensure_future If target is a callable, it is scheduled in the default executor """ ... def run_in_foreground(task, *, loop=None): """Runs event loop in current thread until the given task completes Returns the result of the task. For more complex conditions, combine with asyncio.wait() To include a timeout, combine with asyncio.wait_for() """ ... run_in_background is akin to invoking a shell command with a trailing "&" - it puts the operation into the background, leaving the current thread to move on to the next operation (or wait for input at the REPL). When coroutines are scheduled, they won't start running until you start a foreground task, while callables delegated to the default executor will start running immediately. To actually get the *results* of that task, you have to run it in the foreground of the current thread using run_in_foreground - this is akin to bringing a background process to the foreground of a shell session using "fg". To relate this idea back to some of the examples Sven was discussing, here's how translating some old serialised synchronous code to use those APIs might look in practice: # Serial synchronous data loading def load_and_process_data(): data1 = load_remote_data_set1() data2 = load_remote_data_set2() return process_data(data1, data2) # Parallel asynchronous data loading def load_and_process_data(): future1 = asyncio.run_in_background(load_remote_data_set1_async()) future2 = asyncio.run_in_background(load_remote_data_set2_async()) data1 = asyncio.run_in_foreground(future1) data2 = asyncio.run_in_foreground(future2) return process_data(data1, data2) The application remains fundamentally synchronous, but the asyncio event loop is exploited to obtain some local concurrency in waiting for client IO operations. Regards, Nick. P.S. time.sleep() and asyncio.sleep() are rather handy as standins for blocking and non-blocking IO operations. I wish I'd remembered that earlier :) -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia [View Less]

8 16

Briefer string format
by Mike Miller Aug. 9, 2015

Aug. 9, 2015

Have long wished python could format strings easily like bash or perl do, ... and then it hit me: csstext += f'{nl}{selector}{space}{{{nl}' (This script included whitespace vars to provide a minification option.) I've seen others make similar suggestions, but to my knowledge they didn't include this pleasing brevity aspect. -Mike

35 163

Re: [Python-ideas] Concurrency Modules
by Sven R. Kunze Aug. 6, 2015

Aug. 6, 2015

Hi. I am back. First of all thanks for your eager participation. I would like to catch on on Steve's and Mark's examples as they seem to be very good illustrations of what issue I still have. Steve explained why asyncio is great and Mark explained why threading+multiprocessing is great. Each from his own perspective and focusing on the internal implementation details. To me, all approaches can now be fit into this sort of table. Please, correct me if it's wrong (that is very important): … [View More]# | code lives in | managed by --+---------------+------------- 1 | processes | os scheduler 2 | threads | os scheduler 3 | tasks | event loop But the original question still stands: Which one to use? Ignoring little details like 'shared state', 'custom prioritization', etc., they all look the same to me and to what it all comes down are these little nasty details people try to explain so eagerly. Not saying that is a bad thing but it has some implications on production code I do not like and in the following I am going to explain that. Say, we have decided for approach N because of some requirements (examples from here and there, guidelines given by smart people, customer needs etc.) and wrote hundred thousand lines of code. What if these requirements change 6 years in the future? What if the maintainer of approach N decided to change it in such a way that is not compatible with our requirements anymore? From what I can see there is no easy way 'back' to use another approach. They all have different APIs, basically for: 'executing a function and returning its precious result (the cake)'. asyncio gives us the flexibility to choose a prioritization mechanism. Nice to have, because we are now independent on the os scheduler. But do we really ever need that? What is wrong with the os scheduler? Would that not mean that Mark better switches to asyncio? We don't know if we ever would need that in project A and project B. What now? Use asyncio just in case? Preemptively? @Steve Thanks for that great explanation of how asyncio works and its relationship to threads/processes. But I still have a question: why can't we use threads for the cakes? (1 cake = 1 thread). Not saying that asyncio would be a bad idea to use here, but couldn't we accomplish the same functionality by using threads? I think, after we've settled the above questions, we should change the focus from How do they work internally and what are the tiny differences? (answered greatly by Mark) to When do I use which one? The latter question actually is what counts for production code. It actually is quite interesting to know and to ponder over all the differences, dependencies, corner cases etc. However, when it actually comes down to 'executing a piece of code and returning its result', you end up deciding which approach to choose. You won't implement all 3 different ways just because it is great to see all the nasty little details to click in. On Thursday, July 9, 2015 at 11:54:11 PM UTC+1, Sven R. Kunze wrote: > > In order to make a sound decision for the question: "Which one(s) do I > use?", at least the following items should be somehow defined clearly > for these modules: > > 1) relationship between the modules > 2) NON-overlapping usage scenarios > 3) future development intentions > 4) ease of usage of the modules => future syntax > 5) examples [View Less]

10 24

Reprs of classes and functions
by Serhiy Storchaka Aug. 5, 2015

Aug. 5, 2015

This idea is already casually mentioned, but sank deep into the threads of the discussion. Raise it up. Currently reprs of classes and functions look as: >>> int <class 'int'> >>> int.from_bytes <built-in method from_bytes of type object at 0x826cf60> >>> open <built-in function open> >>> import collections >>> collections.Counter <class 'collections.Counter'> >>> collections.Counter.fromkeys <bound method … [View More]

7 9

A different format for PI?
by Abe Dillon Aug. 1, 2015

Aug. 1, 2015

Is there a forum or something similar related to python-ideas? If there isn't, I think there should be. The mailing list format is restrictive. There's no good way to search past discussions and the digests I get are disorganized and difficult to follow. I'd like to contribute, but I don't know if my ideas are topics that have already been discussed in depth or if they're actually new. -Abe Dillon

6 7

fork
by Sven R. Kunze July 30, 2015

July 30, 2015

Hi everybody, well during the discussion of the concurrency capabilities of Python, I found this article reading worthwhile: http://chriskiehl.com/article/parallelism-in-one-line/ His statement "Mmm.. Smell those Java roots." basically sums the whole topic up for me. That is sequential code (almost plain English): for image in images: create_thumbnail(image) In order to have a start with parallelism and concurrency, we need to do the following: pool = Pool() pool.map(… [View More]

4 3

Additional datetime-like module
by Mike Miller July 29, 2015

July 29, 2015

(Apologies, I've gotten a bit lost in the recent PEP-431 discussion on -dev. To recap, it concerns changing datetime to use UTC internally.) When doing datetime calculations, I read two major use cases, one I'll call "simple" (aka naive), and the other "robust", that comes into play when multiple users, zones, dst, leaps, and/or calendar requirements surface. Instead of ignoring one case, or trying to shoehorn several into one module, I submit the cases could be broken into at least two … [View More]

7 10

Fwd: Loop manager syntax
by Andrew Barnert July 29, 2015

July 29, 2015

On Jul 29, 2015, at 13:16, Todd <toddrjen(a)gmail.com> wrote: > I thought it was better to first determine whether the idea had any merit at all before getting into details. If the idea turned out to be stupid then there isn't much point getting into details. I can't speak for anyone else, but I don't think the idea itself is stupid, or I wouldn't have responded at all. After all, it's not far from the way you do parallel loops via OMP in C, or in Cython. >> Meanwhile, if … [View More]mutating the dicts (that is, the inner function assigning to variables) doesn't affect the outer namespace, what about mutating the values in the dict? Do you deep-copy everything into the dicts, then "deep-update" back out later? If not, how does a line like "d[k] += 1" inside the loop end up working at all (where d is a global or nonlocal)? > > The contents of the dict are deep-copied when passed to the loop manager. Whether the loop manager deep copies those or not is up to it. Well, if it's already deep-copied once, then it doesn't really matter if you deep-copy it again; it's still separate objects. That means you can't use any variables in a loop whose values aren't deep-copyable (=picklable), or you'll get some kind of exception. It also means, as I pointed out above, that you need some kind of deep-update mechanism to get all the values back into the parent scope, not just the ones directly in variables. Since Python doesn't have any such mechanism, you have to invent one. And I don't know if there is any sensible algorithm for such a thing in general. (Maybe something where you version or timestamp all the values, but even that fails the i+=1 case, which is why it's not sufficient for STM without also adding rollback and retryable commits.) And meanwhile, it means you don't have the option of delayed vs. live updates as you suggested, it means you have the option of delayed vs. no updates at all. > It may be possible to make the dicts directly represent the namespaces, and thus have changes immediately reflected in the namespace. I thought keeping things well-isolated was more important, but it may not be worth the trouble. > >> Meanwhile, this scope semantic is very weird if it interacts with the rest of Python. For example, if you call a closure that was built outside, but you've modified one of the variables that closure uses, it'll be called the with old value, right? That would be more than a little confusing. > > Where and when did you modify it? Inside the loop, for example. That presumably means you've modified the deep copy. But, assuming closures can be deep-copied at all, they'll presumably have independent copies, not copies still linked to the variables you have (otherwise they're not deep copies). Which would be very different from normal semantics. For example: def spam(x): def adder(y): return x+y for i in range(2): x=10; print(adder(i)) spam(0) This is obviously silly code, but it's the shortest example I could think of on the spur of the moment that demonstrates most of the problems. This prints 10 then 11. But if you change it to "eggs for i in range(2):", there is no way that any eggs could be implemented (with your design) that gives you the same 10 and 11, because adder is going to be deep-copied with x=0, not x as a closure cell. Or, even if x _is_ a closure cell somehow, it's not the same thing as the x in the enclosing dict passed into the function, so assigning x=10 still doesn't affect it. Again, this is a silly toy example, but it demonstrates real problems that treating scopes as just dictionaries, and deep-copying them, create. >> And finally, how does this copy-and-merge work in parallel? Let's say you do x=0 outside the loop, then inside the loop you do x+=i. So, the first iteration starts with x=0, changes it to x=1, and you merge back x=1. The second iteration starts with x=0, changes it to x=2, and you merge back x=2. This seems like a guaranteed race, with no way either the interpreter or the Pool.__for__ mechanism could even have a possibility of ending up with 3, much less guarantee it. I think what you actually need here is not copied and merged scopes, but something more like STM, which is a lot more complicated. > > Again, this would be up to the loop handler. In the case of multiprocessing, each repetition will likely get x=0, and the last value (whatever that is) will be returned at the end. The documentation would need to say that the behavior in such cases is non-deterministic and shouldn't be counted on. But that makes parallel for basically useless unless you're doing purely-immutable code. If that's fine, you might as well not try to make mutability work in the first place, in which case you might as well just do comprehensions, in which case you might as well just call the map method. A design that's much more complicated, but still doesn't add any functionality in a usable way for the paradigm use case, doesn't seem worth it. I think you _could_ come up with something that actually works by exposing a bunch of new features for dealing with cells and frames from Python, which could lead to other cool functionality (again, like pure-Python greenlets), which I think may be worth exploring. >>> In cases where one or more of the namespaces doesn't make sense the corresponding dict would be empty. The method would return three dicts containing the local, nonlocal, and global namespaces, any or all of which could be empty. Returning a non-empty dict in a case where the corresponding namespace doesn't make sense would raise an exception. The interpreter would merge these dicts back into the corresponding namespaces. >>> >>> The function-like object would be passed four dicts corresponding to the same namespaces, and would return a tuple of three dicts corresponding to the same namespaces. The interpreter would again be responsible for initializing the function-like object's namespaces with the contents of the dicts and pulling out those namespaces at the end. >>> >> >> What does this "function-like object" look like? How do you call it with these dicts in such a way that their locals, nonlocals, and globals get set up as the variables the function was compiled to expect? (For just locals and globals, you can get the code object out, and exec that instead of calling the function, as I suggested--but you clearly don't want that, so what do you want instead?) Again, this is a new feature that I think would be more complicated, and more broadly useful, than your suggested feature, so it ought to be specified. > > I am not sure I am understanding the question. Can you please explain in more detail? Can you sketch out the API of what creating and calling these function-like objects looks like? It can't be just like defining and calling a function. Extracting the code object from a function (or compiling one) and calling exec with it are closer, but still not sufficient, since function objects, and exec, only have locals and globals from the Python side (to deal with nonlocals you have to drop down to C), and since the mechanism to update the outer scope doesn't exist (without dropping down to C). But if you can look at what's already there and show what you'd need to add, even if you can't work out how to implement it as a CPython patch or anything, that could still be a very useful idea. >>> In the case of yield, the returned tuple will have one additional element for the yielded value. >>> >> >> How you do distinguish between "exited normally" and "yielded None"? > > The first would return a length-3 tuple and the second would return a length-4 tuple with the last element being "None". OK, that makes sense. > >>> The interpreter would be in charge of remembering which yield it is at, but the function-like object would still be initialized with the namespaces provided by the method. So any loop handler that allows yielding will need to be able to get the correct values in the namespace, failing to do so will raise an exception. The function-like object always has an optional argument for injecting values into the yield, but passing anything to it when the function-like object is not at a yield that accepts a value would raise an exception. >>> >> >> Well, you also need to handle throwing exceptions into the function-like object. > > I am not sure what you mean by this. Generator objects have not only __next__ and send, but also throw. >> More importantly, if you think about this, you're proposing to have something like explicit generator states and/or continuations that can be manually passed around and called (and even constructed?). Together with your manual scopes (which can at least be merged into real scopes, if not allowing real scopes to be constructed), this gives you all kinds of cool things--I think you could write Stackless or greenlets in pure Python if you had this. But again, this is something we definitely don't have today, that has to be designed before you can just assume it for a new feature. > > I considered that possibility, but it seemed overly complicated. So at least in my proposal right now there is no way to get at the generator state. But if there's no way to get at the generator state, I don't see how you can implement what you want. What is the method holding? How does it know whether it called a function or a generator, and whether it needs to act as a function or a generator? >>> Returns and breaks will be exceptions, which contain the namespaces as extra data. >>> >> >> Which, as I said, obviously means that you've created a new compiler mode that, among other things, compiled returns and breaks into raises. > > Correct, which is why I call this a "function-like object" rather than a "function". Sure, but the point is that compiling any function that has a custom loop has to switch to a new mode that had to do various things differently than just compiling a suite, or an explicit function, or a comprehension. >>> How to deal with yields, breaks, and returns is up to the class designer. There is no reason all loop handlers would need to handle all possible loop behaviour. It would be possible to catch and re-raise break or return exceptions, or simply not handle them at all, in cases where they shouldn't be used. Similarly, a class could simply raise an exception if the function-like object tries to yield anything if yielding didn't make sense. >>> >> >> But how does the class know whether, e.g., yielding from the calling function makes sense? It doesn't know what function(s) it's going to be called from. > > I am not understanding the question. Either there is a sane way to deal with yields, in which case it would do so, or there isn't, in which case it wouldn't. Can you name a situation where allowing a yield would be sane in some cases but not in others? Yielding from inside a loop is a very common thing to do, so I think any proposal had better be able to handle it. The problem is that whether a function is a generator function or not is determined at compile time, not run time. A __for__ method can't know at compile time whether the loop suites (function-like objects) that will be passed to it will yield or not, or whether the functions that will call it will yield or not. (Especially since the same method may get called once with a suite that yields, and once with a suite that doesn't.) So, how can you write something that works? Either your method isn't a generator function, and therefore it can't do anything useful if the function-like object gives it a yield value, or it is, in which case you have the opposite problem. Or you need some new way to implement the generator protocol in a way that's explicit (i.e., doesn't actually use generators) in the middle layers but transparent at the compiler level and then again at the level of calling the function with the loop in it. Providing explicit generator state objects isn't the only way to solve the problem, but it is _a_ way to solve it, and it could be useful elsewhere. If you have something simpler than works, it may be worth doing that something simpler instead, but I can't think of anything. >> And, again, _how_ can it handle yielding if it decides to do so? You can't write a method that's sometimes a generator function and sometimes not (depending on whether some function-like object returns a special value or not). > > There are a couple of ways I thought of to deal with this. One is that there are two versions of each loop handler, one for generators and one for regular loops. The other is that loop handler can be passed an object that it would push values to yield. The latter would be very weird, because generators are driven from the outside, pull-based, and the inner function-like object is also going to be pull-based; putting a push-based interface in between will be misleading or confusing at best. >>> While loop managers would be similar, except instead of a variable name and iterator it would be passed a second function-like object for the conditional and a tuple of variable names used in the conditional. >>> >> >> And it has to manually do the LEGB rule to dynamically figure out where to look up those names? > > If it doesn't want to alter them, then no it just passes along the namespaces to the function-like object. If it wants to alter them, then yes. That is, what, two or three lines of code? But it's not two or three lines of code. I'm not even sure it's doable at all, e.g., because, again, assignments are handled mostly at compile time, not run time. >> Also, you realize that, even if it uses exactly the same rules as the normal interpreter, the effect will be different, because the interpreter applies the rule partly at compile time, not completely dynamically, right? > > Yes, if someone is using this feature they would need to be aware of the effects it has. Having loop modifiers that break all kinds of standard loop features and change the basic semantics of variable lookup doesn't seem like a good addition to the language. If it were _possible_ to do things right, but also possible to do them weirdly, you could just file that under "consenting adults", but if any loop manager that can be written is guaranteed to break a large chunk of Python and there's no way around that, I don't think it's a feature anyone would want. >>> This function-like object would return a namespace dict for the local namespace and a boolean for the result of the conditional. Ideally this namespace dict would be empty or None if it is identical to the input namespace. >>> >> Why is it returning a namespace dict? An expression can't assign to a variable, so the namespace will always be identical. >> >> An expression can, of course, call a mutating method, but unless you're suggesting another deep-copy/deep-update (or STM transaction) here, the value is already going to be mutated, and the dict won't help anyway. > > It is up to the loop manager whether to deep copy or not. It would, of course, be possible for the loop manager to determine whether anything has mutated, but I felt this approach would be more consistent. What use are you envisioning for taking a dict and returning a different dict, other than a deep copy, and a subsequent deep update (and who does that deep update if not the loop manager)? >>> > And, beyond the problems with concurrency, you have cross-process problems. For example, how do you pickle a live scope from one interpreter, pass it to another interpreter, and make it work on the first interpreter's scope? >>> >>> That is the whole point of passing namespace dicts around. >>> >> >> OK, think about this: >> >> I have a list a=[0]*8. >> >> Now, for i in range(8), I set a[i] = i**2. >> >> If I do that with a pool for, each instance gets its own copy of the scope with its own copy of a, which is modifies. How does the interpreter or the Pool.__for__ merge those 8 separate copies of a back in those 8 separate namespaces back into the original namespace? > > It would need to check if any of the elements are not the same as before (using "is"). It's going to walk the whole chain of everything reachable from the dict? I'm not sure how you even do that in general (you could make assumptions about how the copy protocol is implemented for each type, but if those assumptions were good enough, we wouldn't have the whole copy protocol in the first place...). And what is it comparing them to? If it's got deep copies--which it will--everything is guaranteed to not the same (except maybe for ints and other simple immutable objects that the interpreter is allowed to collapse, but in that case it's just a coincidence). >>> > > However, there are other uses as well. For example, Python has no "do...while" structure, because nobody has come up with a clean way to do it (and probably nobody ever will). However, under this proposal it would be possible for a third-party package to implement a "while" loop manager that can provide this functionality: >>> > >>> > If someone can come up with a clean way to write this do object (even ignoring the fact that it appears to be a weird singleton global object--unless, contrary to other protocols, this one allows you do define the magic methods as @classmethods and then knows how to call them appropriately), why hasn't anyone come up with a clean way of writing a do...while structure? How would it be easier this way? >>> >>> It would be easier because it can be uglier. The bar for new statements is necessarily much, much, much higher than for third-party packages. I certainly wouldn't propose loop handlers solely or even primarily to allow do...while loops, this is more of a side benefit and an example of the sorts of variations on existing loop behaviour that would be possible. >>> >>> > > I think, like context managers, this would provide a great deal of flexibility to the language and allow a lot of useful behaviors. Of course the syntax and details are just strawmen examples at this point, there may be much better syntaxes. But I think the basic idea of being able to control a loop in a manner like this is important. >>> > >>> > The major difference between this proposal and context managers is that you want to be able to have the loop manager drive the execution of its suite, while a context manager can't do that; it just has __enter__ and __exit__ methods that get called before and after the suite is executed normally. That's how it avoids all of the problems here. Of course it still uses a change to the interpreter to allow the __exit__ method to get called as part of exception handling, but it's easy to see how you could have implemented it as a source transformation into a try/finally, in which case it wouldn't have needed any new interpreter functionality at all. >>> >>> Yes, that is why I said it was similar "in principle". The implementation is different, but I think the concepts have a lot in common. >>> >> >> But my point is that they're very different even in principle. One just supplies functions to get called around the suite, the other tries to control the way the suite is executed, using some kind of novel mechanism that still hasn't been designed. > Perhaps it is better to say this is more similar to decorators, then. But it's not really similar to decorators either. A decorator is just like any normal higher-order function. It takes a function, it returns a function, that's it. No monkeying with scopes or capturing and resuming yields or anything of the sort you're proposing. And that's my point: you seem to think your proposal is just a simple thing that will fit on top of Python 3.5, but it actually relies on a whole slew of new things that still need to be invented before it makes sense. Or, to put it another way, if you intentionally designed a proposal to demonstrate why Scheme continuations and environments are cooler than anything Python has, I think it would look a lot like this... (And the usual counter is that Python doesn't have everything to let you implement custom control flow at the language level because allowing that is actually a _bad_ thing for readability, not a feature.) [View Less]

1 0

Syntax for key-value iteration over mappings
by Petr Viktorin July 29, 2015

July 29, 2015

Hello, Currently, the way to iterate over keys and values of a mapping is to call items() and iterate over the resulting view:: for key, value in a_dict.items(): print(key, value) I believe that looping over all the data in a dict is a very imporant operation, and I find myself writing this quite often. Every time I do, it seems it's boilerplate; it looks a like a workaround rather than a preferred way of doing things. In dict comprehensions and literals, key-value pairs are … [View More]

16 40

Loop manager syntax
by Todd July 29, 2015

July 29, 2015

Following the discussion of the new "async" keyword, I think it would be useful to provide a generic way to alter the behavior of loops. My idea is to allow a user to take control over the operation of a "for" or "while" loop. The basic idea is similar to context managers, where an object implementing certain magic methods, probably "__for__" and "__while__", could be placed in front of a "for" or "while" statement, respectively. This class would then be put in charge of carrying out the … [View More]

5 9

Class introspection by pydoc vs. attributes on meta classes
by Ronald Oussoren July 26, 2015

July 26, 2015

Hi, Pydoc (and AFAIK loads of other introspection tools as well) currently ignores attributes on metaclasses. I wonder if it would be better to teach pydoc (and possibly inspect as well) about those, and thought it would be better to ask here before starting on a patch. That is, given the following definitions: class Meta (type): def hidden_class(cls): print('H', cls.__name__) class MyObject (metaclass=Meta): @classmethod def public_class(cls): print('P', cls.… [View More]

2 1

Mitigating 'self.' Method Pollution
by Michael Hewitt July 24, 2015

July 24, 2015

Last night I made a post to the neopythonic blog proposing a Python 3.x feature that Guido asked me to forward to this alias. For the full background, see the link to my post below. For brevity, I will simply submit the proposal here. The specific problem I am addressing is the pollution of Python methods by 'self.' to reference fields. Here is the proposal: The name of the first parameter to a method can be used to scope subsequent variable references similar to the behavior of 'global'. … [View More]

26 62

Add SMTPS support to logging.handlers.SMTPHandler
by jakedrummond＠verizon.net July 24, 2015

July 24, 2015

Example of modification: old: smtp = smtplib.SMTP(self.mailhost, port, timeout=self._timeout) new: if self.smtps: smtp = smtplib.SMTP_SSL(self.mailhost, port, *self.smtps, timeout=self._timeout) else: smtp = smtplib.SMTP(self.mailhost, port, timeout=self._timeout) old: if self.secure is not None: smtp.ehlo() smtp.starttls(*self.secure) smtp.ehlo() new: smtp.ehlo() if self.tls: smtp.starttls(*self.tls) smtp.ehlo()

1 0

Fwd: Re: Secure unpickle
by Ryan Gonzalez July 23, 2015

July 23, 2015

A further idea: hashes. Each Pickle database (or whatever it's called) would contain a hash made up of: a) The types used to pickle the data. b) The hash of the data itself, prefixed with 2 bytes that have some sort of hard-to-get meaning (the length of the call stack?). c) The seconds since epoch, or another 64-bit value. The three values would likely be merged via bitwise or. This has the advantage that there are three different elements making up the hash, some of which are harder to … [View More]

6 10

Secure unpickle
by Neil Girdhar July 23, 2015

July 23, 2015

I've heard it said that pickle is a security hole, and so it's better to write your own serialization routine. That's unfortunate because pickle has so many advantages such as automatically tying into copy/deepcopy. Would it be possible to make unpickle secure, e.g., by having the caller create a context in which all calls to unpickle are limited to unpickling a specific set of types? (When these types unpickle their sub-objects, they could potentially limit the set of types further.)

3 3

Re: [Python-ideas] Briefer string format
by Mike Miller July 20, 2015

July 20, 2015

Thanks for thinking about it, Chris. After several mentions of u'' and b'' I was waiting patiently for '' and r''. I guess r'' does nothing but what does '' or "" do at runtime? "\tcommand line --params\n" --> " command line --params " It looks like there is some runtime conversion happening there, though I don't know the implementation details. On Mon, 20 Jul 2015 10:51:28 +1000, Chris Angelico wrote: > Will an f"..." format string be mergeable with other strings? All the … [View More]

3 2

Disallow "00000" as a synonym for "0"
by Neil Girdhar July 20, 2015

July 20, 2015

As per this question: http://stackoverflow.com/questions/31447694/why-does-python-3-allow-00-as-a… It seems like Python accepts "000000000" to mean "0". Whatever the historical reason, should this be deprecated?

9 14

namedtuple fields with default values
by Russell Kaplan July 17, 2015

July 17, 2015

I'm using a namedtuple to keep track of several fields, only some of which ever need to be specified during instantiation. However, there is no Pythonic way to create a namedtuple with fields that have default values. >>> Foo = namedtuple('Foo', ['bar', 'optional_baz']) >>> f = Foo('barValue') # Not passing an argument for every field will cause a TypeError Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: __new__() takes exactly … [View More]

6 5

Should a single/double quote followed by a left parenthesis raise SyntaxError?
by Jacob Niehus July 17, 2015

July 17, 2015

I recently forgot the '%' between a format string and its tuple of values and got "TypeError: 'str' object is not callable." The error makes sense, of course, because function calling has higher precedence than anything else in the expression, so while: >>> '%s' 'abc' yields '%sabc', >>> '%s' ('abc') yields a TypeError. My question is whether this should be caught as a syntax error instead of a runtime error. I can't think of any possible case where a single/double quote … [View More]

8 16

Concurrency Modules
by Sven R. Kunze July 11, 2015

July 11, 2015

Hi, that's a follow up on the discussion started on python-dev ('The importance of the async keyword') and this issue http://bugs.python.org/issue24571 . After discussing the whole topic and reading it up further, it became clear to me what's actually missing in Python. That is a definitive guide of why/when a certain concurrency module is supposed to be used: Currently, I can name 4 modules of which I know that they more or less deal with the topic: - concurrent - threading - … [View More]

10 13

Rename python-dev?
by Ryan Gonzalez July 10, 2015

July 10, 2015

This is a crazy idea that popped into my head recently. What if python-dev were to be renamed to something like python-internals? (Yeah, that's a bad name, but my naming skills suck...) Basically, something that gets across the idea that it's for development of Python, not in Python. Python-dev would visibly disappear, but mails sent to python-dev would be redirected to the new name. BTW, in reality, it's not an uncommon mistake. I know of a few mailing lists built for development *with* a … [View More]

6 5