Mailman 3 June 2015 - Python-ideas

Re: [Python-ideas] millisecond and microsecond times without floats
by random832＠fastmail.us 26 Jun '15

26 Jun '15

On Mon, Jun 22, 2015, at 19:15, Paul Sokolovsky wrote: > > > Hello from MicroPython, a lean Python implementation > scaling down to run even on microcontrollers > (https://github.com/micropython/micropython). > > Our target hardware base oftentimes lacks floating point support, and > using software emulation is expensive. So, we would like to have > versions of some timing functions, taking/returning millisecond and/or > microsecond values as integers. What about having a fixed-point decimal numeric type to be used for this purpose? Allowing time (and stat, and the relevant functions of the datetime module) to return any real numeric type rather than being required to use float would be a useful extension.

3 3

Re: [Python-ideas] natively logging sys.path modifications
by Chris Angelico 25 Jun '15

25 Jun '15

On Thu, Jun 25, 2015 at 4:26 AM, anatoly techtonik <techtonik(a)gmail.com> wrote: > That object will be broken if somebody decides to use assignment: > > sys.path = [] > > And as far as I know it is not possible to prevent this case or guard > against this replacement. So what you want is for the sys module to log all assignments to a particular attribute, AND for all mutations of that attribute to be logged as well. That sounds like two completely separate problems to be solved, but neither is fundamentally impossible (although you'd need to fiddle with the sys module itself to do the other). I suggest you investigate ways of solving this that require zero core code changes, as those ways will work on all existing Python versions. Then once you run up against an actual limitation, you'll have a better argument for code changes. ChrisA

1 0

web API to get a list of all module in stdlib
by anatoly techtonik 24 Jun '15

24 Jun '15

Hi, People from core-workflow are not too active about the idea, so I finally found a time to repost it here. The original is here: http://comments.gmane.org/gmane.comp.python.devel.core-workflow/130 The idea is that docs.python.org site should export the list of Python modules shipped in stdlib for particular Python version in a machine readable format. There are recipes like these to get the list of modules: https://stackoverflow.com/questions/6463918/how-can-i-get-a-list-of-all-the… But they give only the modules enabled for specific interpreter/platform. Not the list of modules that is included in de-facto standard for this stdlib version. This is need for processing information, for all Python versions, so instead parsing HTML tables, it would be more useful to directly fetch csv or json. That way anybody can quickly validate the processing algorithm without wasting time on extracting and normalizing the data. I see the data as the necessary step to organize a work around "externally evolving standard library", so a way to query it should be somewhat sustainable and obvious. Docs looks like an obvious way yo do so, like: https://docs.python.org/2.7.2/dataset/modules.json -- anatoly t.

4 5

natively logging sys.path modifications
by anatoly techtonik 24 Jun '15

24 Jun '15

Hi, sys.path is kind of important thing for troubleshooting. It may worth to ship it with a logging mechanism that allows to quickly dump who added what and (optionally) why. I see the log as a circular memory buffer of limited size, to say 256 entries, that contains tuples in the following format: path, who, where, why path -- actual path added to sys.path who -- the context - package.module:function or - package.module:class.method or - package.module:__toplevel__ where -- full filename and line number to the instruction why -- advanced API may allow to set this field -- anatoly t.

3 2

PyParallel update (was: solving multi-core Python)
by Trent Nelson 24 Jun '15

24 Jun '15

On Sat, Jun 20, 2015 at 03:42:33PM -0600, Eric Snow wrote: > Furthermore, removing the GIL is perhaps an obvious solution but not > the only one. Others include Trent Nelson's PyParallels, STM, and > other Python implementations.. So, I've been sprinting relentlessly on PyParallel since Christmas, and recently reached my v0.0 milestone of being able to handle all the TEFB tests, plus get the "instantaneous wiki search" thing working too. The TEFB (Techempower Framework Benchmarks) implementation is here: https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed3… (The aim was to have it compete in this: https://www.techempower.com/benchmarks/#section=data-r10, but unfortunately they broke their Windows support after round 9, so there's no way to get PyParallel into the official results without fixing that first.) The wiki thing is here: https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed3… I particularly like the wiki example as it leverages a lot of benefits afforded by PyParallel's approach to parallelism, concurrency and asynchronous I/O: - Load a digital search trie (datrie.Trie) that contains every Wikipedia title and the byte-offset within the wiki.xml where the title was found. (Once loaded the RSS of python.exe is about 11GB; the trie itself has about 16 million items in it.) - Load a numpy array of sorted 64-bit integer offsets. This allows us to do a searchsorted() (binary search) against a given offset in order to derive the next offset. - Once we have a way of getting two byte offsets, we can use ranged HTTP requests (and TransmitFile behind the scenes) to efficiently read random chunks of the file asynchronously. (Windows has a huge advantage here -- there's simply no way to achieve similar functionality on POSIX in a non-blocking fashion (sendfile can block, a disk read() can block, a memory reference into a mmap'd file that isn't in memory will page fault, which will block).) The performance has far surpassed anything I could have imagined back during the async I/O discussions in September 2012, so, time to stick a fork in it and document the experience, which is what I'll be working on in the coming weeks. In the mean time: - There are installers available here for those that wish to play around with the current state of things: http://download.pyparallel.org/ - I wrote a little helper thing that diffs the hg tree against the original v3.3.5 tag I based the work off and committed the diffs directly -- this provides a way to review the changes that were made in order to get to the current level of functionality: https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed3… (It only includes files that existed in the v3.3.5 tag, I don't include diffs for new files I've added.) It's probably useful reviewing the diffs after perusing pyparallel.h: https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed3… ....as you'll see lots of guards in place in most of the diffs. E.g.: Py_GUARD() -- make sure we never hit this from a parallel context Px_GUARD() -- make sure we never hit this from a main thread Py_GUARD_OBJ(o) -- make sure object o is always a main thread object Px_GUARD_OBJ(o) -- make sure object o is always a parallel object PyPx_GUARD_OBJ(o) -- if we're a parallel context, make sure it's a parallel object, if we're a main thread, make sure it's a main thread object. If you haven't heard of PyParallel before, this might be a good place to start: https://speakerdeck.com/trent/. The core concepts haven't really changed since here (re: parallel contexts, main thread, main thread objects, parallel thread objects): https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploit… Basically, if we're a main thread, "do what we normally do", if we're a parallel thread, "divert to a thread-safe alternative". And a final note: I like the recent async additions. I mean, it's unfortunate that the new keyword clashes with the module name I used to hide all the PyParallel trickery, but I'm at the point now where calling something like this from within a parallel context is exactly what I need: async f.write(...) async cursor.execute(...) I've been working on PyParallel on-and-off now for ~2.5 years and have learned a lot and churned out a lot of code -- documenting it all is actually somewhat daunting (where do I start?!), so, if anyone has specific questions about how I addressed certain things, I'm more than happy to elicit more detail on specifics. Trent.

2 2

Responsive signal handling
by Devin Jeanpierre 23 Jun '15

23 Jun '15

On the topic of obscure concurrency trivia, signal handling in Python is not very friendly, and I'm wondering if anyone is interested in reviewing changes to fix it. The world today: handlers only run in the main thread, but if the main thread is running a bytecode (e.g. a call to a C function), it will wait for that first. For example, signal handlers don't get run if you are in the middle of a lock acquisition, thread join, or (sometimes) a select call, until after the call returns (which may take a very long time). This makes it difficult to be responsive to signals without acrobatics, or without using a library that does those acrobatics for you (such as Twisted.) Being responsive to SIGTERM and SIGINT is, IMO, important for running programs in the cloud, since otherwise they may be forcefully killed by the job manager, causing user-facing errors. (It's also annoying as a command line user when you can't kill a process with anything less than SIGKILL.) I would stress that, by and large, the signal module is a trap that most people get wrong, and I don't think the stdlib solution should be the way it is. (e.g. it looks to me like gunicorn gets it wrong, and certainly asyncio did in early versions.) One could implement one or both of the following: - Keep only running signal handlers in the main thread, but allow them to run even in the middle of a call to a C function for as many C functions as we can. This is not possible in general, but it can be made to work for all blocking operations in the stdlib. Operations that run in C but just take a long time, or that are part of third-party code, will continue to inhibit responsiveness. - Run signal handlers in a dedicated separate thread. IMO this is generally better than running signal handlers in the main thread, because it eliminates the separate concept of "async-safe" and just requires "thread-safe". So you can use regular threading synchronization primitives for safety, instead of relying on luck / memorized lists of atomic/re-entrant operations. Something still needs to run in the main thread though, for e.g. KeyboardInterrupt, so this is not super straightforward. Also, it could break any code that really relies on signal handlers running in the main thread. Either approach can be turned into a library, albeit potentially hackily in the first case. -- Devin

8 13

Pathlib additions & changes
by David Townshend 22 Jun '15

22 Jun '15

Hi Recently I've been trying out pathlib in some real code, and while it is a vast improvement on messing around with os, os.path and shutil there are a couple of suggestions I'd like to make. ** TL;DR: Add Path.copy, add Path.remove (replaces Path.rmdir and Path.unlink) and add flags to several methods. Details ====== 1. Add a copy method, i.e. source_path.copy(target_path), which by default should behave like shutil.copy2. 2. Why bother with the distinction between Path.unlink and Path.rmdir? Obviously they apply to different types of paths, but to a user, either way you just want to remove whatever is at the path, so perhaps just have a single Path.remove instead (but see point 3 below). 3. There are several other minor irritations where a common pattern requires several lines or the use of a lower-level library such as shutil. For example: * Mkdir where path exists, but we don't care (common pattern on scripts) if not path.exists(): path.mkdir(parent=True) * Recursively remove a directory (no sane way using pathlib alone) shutil.rmtree(str(path)) * Move a file, creating parents if necessary py> if not target.parent.exists(): target.parent.mkdir(parents=true) source.rename(target) There are others, but these are a couple that spring to mind. There are three options. Either we add a bunch of specific functions for each of these (e.g. Path.rmtree, Path.rename_with_mkdir, etc), or we add a whole lot of boolean arguments (e.g. Path.rename(make_parents=True), or we use flags, e.g. Path.rename(flags=MAKE_PARENTS) Using flags is, IMHO the neatest solution, and could replace some boolean arguments already included. What follows is a suggestion of where flags might be useful, including the new methods suggested above. I haven't put a huge amount of thought into these, wanting to just get the general idea on the table, so I'm sure that upon closer inspection some won't make much sense or could be better named. chmod: RECURSIVE | DONT_FOLLOW_SYMLINKS copy: WITH_STATS | MAKE_PARENTS | OVERWRITE_EXISTING | IGNORE_EXISTING iterdir: RECURSIVE (maybe not worth it because of globbing) lchmod: RECURSIVE (Could be dropped in favour of chmod(flags=DONT_FOLLOW_SYMLINKS)) lstat: (Could be dropped in favour of stat(flags=DONT_FOLLOW_SYMLINKS)) mkdir: MAKE_PARENTS | OVERWRITE_EXISTING | IGNORE_EXISTING remove: RECURSIVE rename: MAKE_PARENTS | OVERWRITE_EXISTING | IGNORE_EXISTING replace: (Could be dropped in favour of rename(flags=OVERWRITE_EXISTING) ) rmdir: (Could be dropped in favour of remove) stat: DONT_FOLLOW_SYMLINKS touch: MAKE_PARENTS | IGNORE_EXISTING unlink: (Could be dropped in favour of remove) Regards David

4 4

Re: [Python-ideas] Python-ideas Digest, Vol 103, Issue 100
by jean-charles.douet＠laposte.net 21 Jun '15

21 Jun '15

Hello, My dix cents may not seem to be neither very clear, nor too argumented, but I wanted to let you know that there seem to be already interesting pythonic studies about how to release the so-called GIL and about, all, for which pragmatic use cases. Reading the following article, I discovered that Python by itself seems not to be a language, but language specification : http://www.toptal.com/python/why-are-there-so-many-pythons Thus, it justifies that PyPy is a more general approach than CPython, which looks like a particular case of Python, even if the most frequently used (?) Now, there is a specific study of PyPy aimed at removing the GIL, called "Software Transactional Memory". Here it is : http://doc.pypy.org/en/latest/stm.html Hope it helps : Best regards, Jean-Charles. ----- Mail original ----- De: python-ideas-request(a)python.org À: python-ideas(a)python.org Envoyé: Dimanche 21 Juin 2015 08:41:24 Objet: Python-ideas Digest, Vol 103, Issue 100 Send Python-ideas mailing list submissions to python-ideas(a)python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/python-ideas or, via email, send a message with subject or body 'help' to python-ideas-request(a)python.org You can reach the person managing the list at python-ideas-owner(a)python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Python-ideas digest..." Today's Topics: 1. Re: solving multi-core Python (Nathaniel Smith) 2. Re: solving multi-core Python (Nick Coghlan) 3. Re: solving multi-core Python (Wes Turner) ---------------------------------------------------------------------- Message: 1 Date: Sat, 20 Jun 2015 22:25:07 -0700 From: Nathaniel Smith <njs(a)pobox.com> To: Eric Snow <ericsnowcurrently(a)gmail.com> Cc: python-ideas <python-ideas(a)python.org> Subject: Re: [Python-ideas] solving multi-core Python Message-ID: <CAPJVwBkjoK31m2-ynrGF_AmYFL0ULL3LdX6r7+d+B7RienQh7A(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" On Jun 20, 2015 3:54 PM, "Eric Snow" <ericsnowcurrently(a)gmail.com> wrote: > > > On Jun 20, 2015 4:08 PM, "Nathaniel Smith" <njs(a)pobox.com> wrote: > > > > On Jun 20, 2015 2:42 PM, "Eric Snow" <ericsnowcurrently(a)gmail.com> wrote: > > > > > > tl;dr Let's exploit multiple cores by fixing up subinterpreters, > > > exposing them in Python, and adding a mechanism to safely share > > > objects between them. > > > > This all sounds really cool if you can pull it off, and shared-nothing threads do seem like the least impossible model to pull off. > > Agreed. > > > But "least impossible" and "possible" are different :-). From your email I can't tell whether this plan is viable while preserving backcompat and memory safety. > > I agree that those issues must be clearly solved in the proposal before it can be approved. I'm confident the approach I'm pursuing will afford us the necessary guarantees. I'll address those specific points directly when I can sit down and organize my thoughts. I'd love to see just a hand wavy, verbal proof-of-concept walking through how this might work in some simple but realistic case. To me a single compelling example could make this proposal feel much more concrete and achievable. > > Suppose I have a queue between two subinterpreters, and on this queue I place a list of dicts of user-defined-in-python objects, each of which holds a reference to a user-defined-via-the-C-api object. What happens next? > > You've hit upon exactly the trickiness involved and why I'm thinking the best approach initially is to only allow *strictly* immutable objects to pass between interpreters. Admittedly, my description of channels is very vague.:) There are a number of possibilities with them that I'm still exploring (CSP has particular opinions...), but immutability is a characteristic that may provide the simplest *initial* approach. Going that route shouldn't preclude adding some sort of support for mutable objects later. There aren't really many options for mutable objects, right? If you want shared nothing semantics, then transmitting a mutable object either needs to make a copy, or else be a real transfer, where the sender no longer has it (cf. Rust). I guess for the latter you'd need some new syntax for send-and-del, that requires the object to be self contained (all mutable objects reachable from it are only referenced by each other) and have only one reference in the sending process (which is the one being sent and then destroyed). > Keep in mind that by "immutability" I'm talking about *really* immutable, perhaps going so far as treating the full memory space associated with an object as frozen. For instance, we'd have to ensure that "immutable" Python objects like strings, ints, and tuples do not change (i.e. via the C API). This seems like a red herring to me. It's already the case that you can't legally use the c api to mutate tuples, ints, for any object that's ever been, say, passed to a function. So for these objects, the subinterpreter setup doesn't actually add any new constraints on user code. C code is always going to be *able* to break memory safety so long as you're using shared-memory threading at the c level to implement this stuff. We just need to make it easy not to. Refcnts and garbage collection are another matter, of course. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150620/c2b41fe0…> ------------------------------ Message: 2 Date: Sun, 21 Jun 2015 16:31:33 +1000 From: Nick Coghlan <ncoghlan(a)gmail.com> To: Nathaniel Smith <njs(a)pobox.com> Cc: Eric Snow <ericsnowcurrently(a)gmail.com>, python-ideas <python-ideas(a)python.org> Subject: Re: [Python-ideas] solving multi-core Python Message-ID: <CADiSq7cv538UBK9BE3e8eAakFB=njwHB-qnMu3m=qzLADzpsOg(a)mail.gmail.com> Content-Type: text/plain; charset=UTF-8 On 21 June 2015 at 15:25, Nathaniel Smith <njs(a)pobox.com> wrote: > On Jun 20, 2015 3:54 PM, "Eric Snow" <ericsnowcurrently(a)gmail.com> wrote: >> >> >> On Jun 20, 2015 4:08 PM, "Nathaniel Smith" <njs(a)pobox.com> wrote: >> > >> > On Jun 20, 2015 2:42 PM, "Eric Snow" <ericsnowcurrently(a)gmail.com> >> > wrote: >> > > >> > > tl;dr Let's exploit multiple cores by fixing up subinterpreters, >> > > exposing them in Python, and adding a mechanism to safely share >> > > objects between them. >> > >> > This all sounds really cool if you can pull it off, and shared-nothing >> > threads do seem like the least impossible model to pull off. >> >> Agreed. >> >> > But "least impossible" and "possible" are different :-). From your email >> > I can't tell whether this plan is viable while preserving backcompat and >> > memory safety. >> >> I agree that those issues must be clearly solved in the proposal before it >> can be approved. I'm confident the approach I'm pursuing will afford us the >> necessary guarantees. I'll address those specific points directly when I >> can sit down and organize my thoughts. > > I'd love to see just a hand wavy, verbal proof-of-concept walking through > how this might work in some simple but realistic case. To me a single > compelling example could make this proposal feel much more concrete and > achievable. I was one of the folks pushing Eric in this direction, and that's because it's a possibility that was conceived of a few years back, but never tried due to lack of time (and inclination for those of us that are using Python primarily as an orchestration tool and hence spend most of our time on IO bound problems rather than CPU bound ones): http://www.curiousefficiency.org/posts/2012/07/volunteer-supported-free-thr… As mentioned there, I've at least spent some time with Graham Dumpleton over the past few years figuring out (and occasionally trying to address) some of the limitations of mod_wsgi's existing subinterpreter based WSGI app separation: https://code.google.com/p/modwsgi/wiki/ProcessesAndThreading#Python_Sub_Int… The fact that mod_wsgi can run most Python web applications in a subinterpreter quite happily means we already know the core mechanism works fine, and there don't appear to be any insurmountable technical hurdles between the status quo and getting to a point where we can either switch the GIL to a read/write lock where a write lock is only needed for inter-interpreter communications, or else find a way for subinterpreters to release the GIL entirely by restricting them appropriately. For inter-interpreter communication, the worst case scenario is having to rely on a memcpy based message passing system (which would still be faster than multiprocessing's serialisation + IPC overhead), but there don't appear to be any insurmountable barriers to setting up an object ownership based system instead (code that accesses PyObject_HEAD fields directly rather than through the relevant macros and functions seems to be the most likely culprit for breaking, but I think "don't do that" is a reasonable answer there). There's plenty of prior art here (including a system I once wrote in C myself atop TI's DSP/BIOS MBX and TSK APIs), so I'm comfortable with Eric's "simple matter of engineering" characterisation of the problem space. The main reason that subinterpreters have never had a Python API before is that they have enough rough edges that having to write a custom C extension module to access the API is the least of your problems if you decide you need them. At the same time, not having a Python API not only makes them much harder to test, which means various aspects of their operation are more likely to be broken, but also makes them inherently CPython specific. Eric's proposal essentially amounts to three things: 1. Filing off enough of the rough edges of the subinterpreter support that we're comfortable giving them a public Python level API that other interpreter implementations can reasonably support 2. Providing the primitives needed for safe and efficient message passing between subinterpreters 3. Allowing subinterpreters to truly execute in parallel on multicore machines All 3 of those are useful enhancements in their own right, which offers the prospect of being able to make incremental progress towards the ultimate goal of native Python level support for distributing across multiple cores within a single process. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia ------------------------------ Message: 3 Date: Sun, 21 Jun 2015 01:41:21 -0500 From: Wes Turner <wes.turner(a)gmail.com> To: Eric Snow <ericsnowcurrently(a)gmail.com> Cc: Nathaniel Smith <njs(a)pobox.com>, Python-Ideas <python-ideas(a)python.org> Subject: Re: [Python-ideas] solving multi-core Python Message-ID: <CACfEFw_1JVpUmZwFVkye-fbEsfH_NXVSo6WDMi1azhnXdY6PcA(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Exciting! * http://zero-buffer.readthedocs.org/en/latest/api-reference/#zero_buffer.Buf… * https://www.google.com/search?q=python+channels * https://docs.python.org/2/library/asyncore.html#module-asyncore * https://chan.readthedocs.org/en/latest/ * https://goless.readthedocs.org/en/latest/ * other approaches to the problem (with great APIs): * http://celery.readthedocs.org/en/latest/userguide/canvas.html#chords * http://discodb.readthedocs.org/en/latest/ On Jun 20, 2015 5:55 PM, "Eric Snow" <ericsnowcurrently(a)gmail.com> wrote: > > On Jun 20, 2015 4:08 PM, "Nathaniel Smith" <njs(a)pobox.com> wrote: > > > > On Jun 20, 2015 2:42 PM, "Eric Snow" <ericsnowcurrently(a)gmail.com> > wrote: > > > > > > tl;dr Let's exploit multiple cores by fixing up subinterpreters, > > > exposing them in Python, and adding a mechanism to safely share > > > objects between them. > > > > This all sounds really cool if you can pull it off, and shared-nothing > threads do seem like the least impossible model to pull off. > > Agreed. > > > But "least impossible" and "possible" are different :-). From your email > I can't tell whether this plan is viable while preserving backcompat and > memory safety. > > I agree that those issues must be clearly solved in the proposal before it > can be approved. I'm confident the approach I'm pursuing will afford us > the necessary guarantees. I'll address those specific points directly when > I can sit down and organize my thoughts. > > > > > Suppose I have a queue between two subinterpreters, and on this queue I > place a list of dicts of user-defined-in-python objects, each of which > holds a reference to a user-defined-via-the-C-api object. What happens next? > > You've hit upon exactly the trickiness involved and why I'm thinking the > best approach initially is to only allow *strictly* immutable objects to > pass between interpreters. Admittedly, my description of channels is very > vague.:) There are a number of possibilities with them that I'm still > exploring (CSP has particular opinions...), but immutability is a > characteristic that may provide the simplest *initial* approach. Going > that route shouldn't preclude adding some sort of support for mutable > objects later. > > Keep in mind that by "immutability" I'm talking about *really* immutable, > perhaps going so far as treating the full memory space associated with an > object as frozen. For instance, we'd have to ensure that "immutable" > Python objects like strings, ints, and tuples do not change (i.e. via the C > API). The contents of involved tuples/containers would have to be likewise > immutable. Even changing refcounts could be too much, hence the idea of > moving refcounts out to a separate table. > > This level of immutability would be something new to Python. We'll see if > it's necessary. If it isn't too much work it might be a good idea > regardless of the multi-core proposal. > > Also note that Barry has a (rejected) PEP from a number of years ago about > freezing objects... That idea is likely out of scope as relates to my > proposal, but it certainly factors in the problem space. > > -eric > > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150621/e4aecde0…> ------------------------------ Subject: Digest Footer _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas ------------------------------ End of Python-ideas Digest, Vol 103, Issue 100 **********************************************

1 0

Keyword-only arguments?
by Amber Yust 21 Jun '15

21 Jun '15

One thing that has been a source of bugs and frustration in the past is the inability to designate a named keyword argument that cannot be passed as a positional argument (short of **kwargs and then keying into the dict directly). Has there been any previous discussion on the possibility of a means to designate named arguments as explicitly non-positional? Not a solid proposal, but to capture the essential difference of what I'm thinking of, along the lines of... def foo(bar, baz=None, qux: None): where bar is a required positional argument, baz is an optional argument that can have a value passed positionally or by name, and qux is an optional argument that must always be passed by keyword. Such a means would help avoid cases where a misremembered function signature results in a subtle and likely unnoticed bug due to unintended parameter/argument mismatch. (It's possible that this has been discussed before - a cursory search of python-ideas didn't bring up any direct discussion, but I may have missed something. If you have a link to prior discussion, please by all means point me at it!)

12 17

Re: [Python-ideas] Python-ideas Digest, Vol 103, Issue 15
by Andrew Barnert 15 Jun '15

15 Jun '15

On Jun 1, 2015, at 20:41, u8y7541 The Awesome Person <surya.subbarao1(a)gmail.com> wrote: > > I think you're right. I was also considering ... "editing" my Python distribution. If they didn't implement my suggestion for correcting floats, at least they can fix this, instead of making people hack Python for good results! If you're going to reply to digests, please learn how to reply inline instead of top-posting (and how to trim out all the irrelevant stuff). It's next to impossible to tell which part of which of the messages you're replying to even in simple cases like this one, with only 4 messages in the digest. >> On Mon, Jun 1, 2015 at 8:10 PM, <python-ideas-request(a)python.org> wrote: >> Send Python-ideas mailing list submissions to >> python-ideas(a)python.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://mail.python.org/mailman/listinfo/python-ideas >> or, via email, send a message with subject or body 'help' to >> python-ideas-request(a)python.org >> >> You can reach the person managing the list at >> python-ideas-owner(a)python.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of Python-ideas digest..." >> >> >> Today's Topics: >> >> 1. Re: Python Float Update (Steven D'Aprano) >> 2. Re: Python Float Update (Andrew Barnert) >> 3. Re: Python Float Update (Steven D'Aprano) >> 4. Re: Python Float Update (Andrew Barnert) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Tue, 2 Jun 2015 11:37:48 +1000 >> From: Steven D'Aprano <steve(a)pearwood.info> >> To: python-ideas(a)python.org >> Subject: Re: [Python-ideas] Python Float Update >> Message-ID: <20150602013748.GD932(a)ando.pearwood.info> >> Content-Type: text/plain; charset=us-ascii >> >> On Mon, Jun 01, 2015 at 05:52:35PM +0300, Joonas Liik wrote: >> >> > Having some sort of decimal literal would have some advantages of its own, >> > for one it could help against this sillyness: >> > >> > >>> Decimal(1.3) >> > Decimal('1.3000000000000000444089209850062616169452667236328125') >> >> Why is that silly? That's the actual value of the binary float 1.3 >> converted into base 10. If you want 1.3 exactly, you can do this: >> >> > >>> Decimal('1.3') >> > Decimal('1.3') >> >> Is that really so hard for people to learn? >> >> >> > I'm not saying that the actual data type needs to be a decimal ( >> > might well be a float but say shove the string repr next to it so it can be >> > accessed when needed) >> >> You want Decimals to *lie* about what value they have? >> >> I think that's a terrible idea, one which would lead to a whole set of >> new and exciting surprises when using Decimal. Let me try to predict a >> few of the questions on Stackoverflow which would follow this change... >> >> Why is equality so inaccurate in Python? >> >> py> x = Decimal(1.3) >> py> y = Decimal('1.3') >> py> x, y >> (Decimal('1.3'), Decimal('1.3')) >> py> x == y >> False >> >> Why does Python insert extra digits into numbers when I multiply? >> >> py> x = Decimal(1.3) >> py> x >> Decimal('1.3') >> py> y = 10000000000000000*x >> py> y - 13000000000000000 >> Decimal('0.444089209850062616169452667236328125') >> >> >> > ..but this is one really common pitfall for new users, i know its easy to >> > fix the code above, >> > but this behavior is very unintuitive.. you essentially get a really >> > expensive float when you do the obvious thing. >> >> Then don't do the obvious thing. >> >> Sometimes there really is no good alternative to actually knowing what >> you are doing. Floating point maths is inherently hard, but that's not >> the problem. There are all sorts of things in programming which are >> hard, and people learn how to deal with them. The problem is that people >> *imagine* that floating point is simple, when it is not and can never >> be. We don't do them any favours by enabling that delusion. >> >> If your needs are light, then you can ignore the complexities of >> floating point. You really can go a very long way by just rounding the >> results of your calculations when displaying them. But for anything more >> than that, we cannot just paper over the floating point complexities >> without creating new complexities that will burn people. >> >> You don't have to become a floating point guru, but it really isn't >> onerous to expect people who are programming to learn a few basic >> programming skills, and that includes a few basic coping strategies for >> floating point. >> >> >> >> -- >> Steve >> >> >> ------------------------------ >> >> Message: 2 >> Date: Mon, 1 Jun 2015 19:21:47 -0700 >> From: Andrew Barnert <abarnert(a)yahoo.com> >> To: Andrew Barnert <abarnert(a)yahoo.com> >> Cc: Nick Coghlan <ncoghlan(a)gmail.com>, python-ideas >> <python-ideas(a)python.org> >> Subject: Re: [Python-ideas] Python Float Update >> Message-ID: <5E8271BF-183E-496D-A556-81C407977FFE(a)yahoo.com> >> Content-Type: text/plain; charset=us-ascii >> >> On Jun 1, 2015, at 19:00, Andrew Barnert <abarnert(a)yahoo.com> wrote: >> > >> >> On Jun 1, 2015, at 18:27, Andrew Barnert via Python-ideas <python-ideas(a)python.org> wrote: >> >> >> >>> On Jun 1, 2015, at 17:08, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> >>> >> >>> On 2 Jun 2015 08:44, "Andrew Barnert via Python-ideas" >> >>> <python-ideas(a)python.org> wrote: >> >>>> But the basic idea can be extracted out and Pythonified: >> >>>> >> >>>> The literal 1.23 no longer gives you a float, but a FloatLiteral, which is either a subclass of float, or an unrelated class that has a __float__ method. Doing any calculation on it gives you a float. But as long as you leave it alone as a FloatLiteral, it has its literal characters available for any function that wants to distinguish FloatLiteral from float, like the Decimal constructor. >> >>>> >> >>>> The problem that Python faces that Swift doesn't is that Python doesn't use static typing and implicit compile-time conversions. So in Python, you'd be passing around these larger values and doing the slow conversions at runtime. That may or may not be unacceptable; without actually building it and testing some realistic programs it's pretty hard to guess. >> >>> >> >>> Joonas's suggestion of storing the original text representation passed >> >>> to the float constructor is at least a novel one - it's only the idea >> >>> of actual decimal literals that was ruled out in the past. >> >> >> >> I actually built about half an implementation of something like Swift's LiteralConvertible protocol back when I was teaching myself Swift. But I think I have a simpler version that I could implement much more easily. >> >> >> >> Basically, FloatLiteral is just a subclass of float whose __new__ stores its constructor argument. Then decimal.Decimal checks for that stored string and uses it instead of the float value if present. Then there's an import hook that replaces every Num with a call to FloatLiteral. >> >> >> >> This design doesn't actually fix everything; in effect, 1.3 actually compiles to FloatLiteral(str(float('1.3')) (because by the time you get to the AST it's too late to avoid that first conversion). Which does actually solve the problem with 1.3, but doesn't solve everything in general (e.g., just feed in a number that has more precision than a double can hold but less than your current decimal context can...). >> >> >> >> But it just lets you test whether the implementation makes sense and what the performance effects are, and it's only an hour of work, >> > >> > Make that 15 minutes. >> > >> > https://github.com/abarnert/floatliteralhack >> >> And as it turns out, hacking the tokens is no harder than hacking the AST (in fact, it's a little easier; I'd just never done it before), so now it does that, meaning you really get the actual literal string from the source, not the repr of the float of that string literal. >> >> Turning this into a real implementation would obviously be more than half an hour's work, but not more than a day or two. Again, I don't think anyone would actually want this, but now people who think they do have an implementation to play with to prove me wrong. >> >> >> and doesn't require anyone to patch their interpreter to play with it. If it seems promising, then hacking the compiler so 2.3 compiles to FloatLiteral('2.3') may be worth doing for a test of the actual functionality. >> >> >> >> I'll be glad to hack it up when I get a chance tonight. But personally, I think decimal literals are a better way to go here. Decimal(1.20) magically doing what you want still has all the same downsides as 1.20d (or implicit decimal literals), plus it's more complex, adds performance costs, and doesn't provide nearly as much benefit. (Yes, Decimal(1.20) is a little nicer than Decimal('1.20'), but only a little--and nowhere near as nice as 1.20d). >> >> >> >>> Aside from the practical implementation question, the main concern I >> >>> have with it is that we'd be trading the status quo for a situation >> >>> where "Decimal(1.3)" and "Decimal(13/10)" gave different answers. >> >> >> >> Yes, to solve that you really need Decimal(13)/Decimal(10)... Which implies that maybe the simplification in Decimal(1.3) is more misleading than helpful. (Notice that this problem also doesn't arise for decimal literals--13/10d is int vs. Decimal division, which is correct out of the box. Or, if you want prefixes, d13/10 is Decimal vs. int division.) >> >> >> >>> It seems to me that a potentially better option might be to adjust the >> >>> implicit float->Decimal conversion in the Decimal constructor to use >> >>> the same algorithm as we now use for float.__repr__ [1], where we look >> >>> for the shortest decimal representation that gives the same answer >> >>> when rendered as a float. At the moment you have to indirect through >> >>> str() or repr() to get that behaviour: >> >>> >> >>>>>> from decimal import Decimal as D >> >>>>>> 1.3 >> >>> 1.3 >> >>>>>> D('1.3') >> >>> Decimal('1.3') >> >>>>>> D(1.3) >> >>> Decimal('1.3000000000000000444089209850062616169452667236328125') >> >>>>>> D(str(1.3)) >> >>> Decimal('1.3') >> >>> >> >>> Cheers, >> >>> Nick. >> >>> >> >>> [1] http://bugs.python.org/issue1580 >> >> _______________________________________________ >> >> Python-ideas mailing list >> >> Python-ideas(a)python.org >> >> https://mail.python.org/mailman/listinfo/python-ideas >> >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> ------------------------------ >> >> Message: 3 >> Date: Tue, 2 Jun 2015 13:00:40 +1000 >> From: Steven D'Aprano <steve(a)pearwood.info> >> To: python-ideas(a)python.org >> Subject: Re: [Python-ideas] Python Float Update >> Message-ID: <20150602030040.GF932(a)ando.pearwood.info> >> Content-Type: text/plain; charset=utf-8 >> >> Nicholas, >> >> Your email client appears to not be quoting text you quote. It is a >> conventional to use a leading > for quoting, perhaps you could configure >> your mail program to do so? The good ones even have a "Paste As Quote" >> command. >> >> On with the substance of your post... >> >> On Mon, Jun 01, 2015 at 01:24:32PM -0400, Nicholas Chammas wrote: >> >> > I guess it?s a non-trivial tradeoff. But I would lean towards considering >> > people likely to be affected by the performance hit as doing something ?not >> > common?. Like, if they are doing that many calculations that it matters, >> > perhaps it makes sense to ask them to explicitly ask for floats vs. >> > decimals, in exchange for giving the majority who wouldn?t notice a >> > performance difference a better user experience. >> >> Changing from binary floats to decimal floats by default is a big, >> backwards incompatible change. Even if it's a good idea, we're >> constrained by backwards compatibility: I would imagine we wouldn't want >> to even introduce this feature until the majority of people are using >> Python 3 rather than Python 2, and then we'd probably want to introduce >> it using a "from __future__ import decimal_floats" directive. >> >> So I would guess this couldn't happen until probably 2020 or so. >> >> But we could introduce a decimal literal, say 1.1d for Decimal("1.1"). >> The first prerequisite is that we have a fast Decimal implementation, >> which we now have. Next we would have to decide how the decimal literals >> would interact with the decimal module. Do we include full support of >> the entire range of decimal features, including globally configurable >> precision and other modes? Or just a subset? How will these decimals >> interact with other numeric types, like float and Fraction? At the >> moment, Decimal isn't even part of the numeric tower. >> >> There's a lot of ground to cover, it's not a trivial change, and will >> definitely need a PEP. >> >> >> > How many of your examples are inherent limitations of decimals vs. problems >> > that can be improved upon? >> >> In one sense, they are inherent limitations of floating point numbers >> regardless of base. Whether binary, decimal, hexadecimal as used in some >> IBM computers, or something else, you're going to see the same problems. >> Only the specific details will vary, e.g. 1/3 cannot be represented >> exactly in base 2 or base 10, but if you constructed a base 3 float, it >> would be exact. >> >> In another sense, Decimal has a big advantage that it is much more >> configurable than Python's floats. Decimal lets you configure the >> precision, rounding mode, error handling and more. That's not inherent >> to base 10 calculations, you can do exactly the same thing for binary >> floats too, but Python doesn't offer that feature for floats, only for >> Decimals. >> >> But no matter how you configure Decimal, all you can do is shift the >> gotchas around. The issue really is inherent to the nature of the >> problem, and you cannot defeat the universe. Regardless of what >> base you use, binary or decimal or something else, or how many digits >> precision, you're still trying to simulate an uncountably infinite >> continuous, infinitely divisible number line using a finite, >> discontinuous set of possible values. Something has to give. >> >> (For the record, when I say "uncountably infinite", I don't just mean >> "too many to count", it's a technical term. To oversimplify horribly, it >> means "larger than infinity" in some sense. It's off-topic for here, >> but if anyone is interested in learning more, you can email me off-list, >> or google for "countable vs uncountable infinity".) >> >> Basically, you're trying to squeeze an infinite number of real numbers >> into a finite amount of memory. It can't be done. Consequently, there >> will *always* be some calculations where the true value simply cannot be >> calculated and the answer you get is slightly too big or slightly too >> small. All the other floating point gotchas follow from that simple >> fact. >> >> >> > Admittedly, the only place where I?ve played with decimals extensively is >> > on Microsoft?s SQL Server (where they are the default literal >> > <https://msdn.microsoft.com/en-us/library/ms179899.aspx>). I?ve stumbled in >> > the past on my own decimal gotchas >> > <http://dba.stackexchange.com/q/18997/2660>, but looking at your examples >> > and trying them on SQL Server I suspect that most of the problems you show >> > are problems of precision and scale. >> >> No. Change the precision and scale, and some *specific* problems goes >> away, but they reappear with other numbers. >> >> Besides, at the point that you're talking about setting the precision, >> we're really not talking about making things easy for beginners any >> more. >> >> And not all floating point issues are related to precision and scale in >> decimal. You cannot divide a cake into exactly three equal pieces in >> Decimal any more than you can divide a cake into exactly three equal >> pieces in binary. All you can hope for is to choose a precision were the >> rounding errors in one part of your calculation will be cancelled by the >> rounding errors in another part of your calculation. And that precision >> will be different for any two arbitrary calculations. >> >> >> >> -- >> Steve >> >> >> ------------------------------ >> >> Message: 4 >> Date: Mon, 1 Jun 2015 20:10:29 -0700 >> From: Andrew Barnert <abarnert(a)yahoo.com> >> To: Steven D'Aprano <steve(a)pearwood.info> >> Cc: "python-ideas(a)python.org" <python-ideas(a)python.org> >> Subject: Re: [Python-ideas] Python Float Update >> Message-ID: <79C16144-8BF7-4260-A356-DD4E8D97BAAD(a)yahoo.com> >> Content-Type: text/plain; charset=us-ascii >> >> On Jun 1, 2015, at 18:58, Steven D'Aprano <steve(a)pearwood.info> wrote: >> > >> >> On Tue, Jun 02, 2015 at 10:08:37AM +1000, Nick Coghlan wrote: >> >> >> >> It seems to me that a potentially better option might be to adjust the >> >> implicit float->Decimal conversion in the Decimal constructor to use >> >> the same algorithm as we now use for float.__repr__ [1], where we look >> >> for the shortest decimal representation that gives the same answer >> >> when rendered as a float. At the moment you have to indirect through >> >> str() or repr() to get that behaviour: >> > >> > Apart from the questions of whether such a change would be allowed by >> > the Decimal specification, >> >> As far as I know, GDAS doesn't specify anything about implicit conversion from floats. As long as the required explicit conversion function (which I think is from_float?) exists and does the required thing. >> >> As a side note, has anyone considered whether it's worth switching to IEEE-754-2008 as the controlling specification? There may be a good reason not to do so; I'm just curious whether someone has thought it through and made the case. >> >> > and the breaking of backwards compatibility, >> > I would really hate that change for another reason. >> > >> > At the moment, a good, cheap way to find out what a binary float "really >> > is" (in some sense) is to convert it to Decimal and see what you get: >> > >> > Decimal(1.3) >> > -> Decimal('1.3000000000000000444089209850062616169452667236328125') >> > >> > If you want conversion from repr, then you can be explicit about it: >> > >> > Decimal(repr(1.3)) >> > -> Decimal('1.3') >> > >> > ("Explicit is better than implicit", as they say...) >> > >> > Although in fairness I suppose that if this change happens, we could >> > keep the old behaviour in the from_float method: >> > >> > # hypothetical future behaviour >> > Decimal(1.3) >> > -> Decimal('1.3') >> > Decimal.from_float(1.3) >> > -> Decimal('1.3000000000000000444089209850062616169452667236328125') >> > >> > But all things considered, I don't think we're doing people any favours >> > by changing the behaviour of float->Decimal conversions to implicitly >> > use the repr() instead of being exact. I expect this strategy is like >> > trying to flatten a bubble under wallpaper: all you can do is push the >> > gotchas and surprises to somewhere else. >> > >> > Oh, another thought... Decimals could gain yet another conversion >> > method, one which implicitly uses the float repr, but signals if it was >> > an inexact conversion or not. Explicitly calling repr can never signal, >> > since the conversion occurs outside of the Decimal constructor and >> > Decimal sees only the string: >> > >> > Decimal(repr(1.3)) cannot signal Inexact. >> > >> > But: >> > >> > Decimal.from_nearest_float(1.5) # exact >> > Decimal.from_nearest_float(1.3) # signals Inexact >> > >> > That might be useful, but probably not to beginners. >> >> I think this might be worth having whether the default constructor is changed or not. >> >> I can't think of too many programs where I'm pretty sure I have an exactly-representable decimal as a float but want to check to be sure... but for interactive use in IPython (especially when I'm specifically trying to explain to someone why just using Decimal instead of float will/will not solve their problem) I could see using it. >> >> ------------------------------ >> >> Subject: Digest Footer >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas(a)python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> >> >> ------------------------------ >> >> End of Python-ideas Digest, Vol 103, Issue 15 >> ********************************************* > > > > -- > -Surya Subbarao

17 23