Python-ideas September 2014

python-ideas@python.org

70 participants
36 discussions

Specify number of items to allocate for array.array() constructor

by Sven Rahmann

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

4 months, 2 weeks

Implicit string literal concatenation considered harmful?

by Guido van Rossum

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' operator, so there's really no reason to support 'a' 'b' any more. (The reason was always rather flimsy; I copied it from C but the reason why it's needed there doesn't really apply to Python, as it is mostly useful inside macros.) Would it be reasonable to start deprecating this and eventually remove it from the language? -- --Guido van Rossum (python.org/~guido)

2 years, 3 months

including psutil in the standard library?

by Stefano Borini

I am wondering if it would be possible to include psutil (https://pypi.python.org/pypi/psutil ) in the standard library, and if not, what would be needed. I am not a developer of it, but I am using psutil at work with good success. it provides a good deal of services for querying and managing processes in a cross platform way. Any thoughts?

5 years, 7 months

Negative limit for traceback.*

by Thomas Allen

I'd like to propose adding the ability to extract last n stack trace entries from a traceback using functions from traceback module. Simply put, passing limit=-n would make functions to return (or print or yield) last abs(n) entries. No-brainer implementation for extract_tb: extracted = list(_extract_tb_iter(tb, limit=(limit, None)[limit < 0])) if limit is not None and limit < 0: return extracted[limit:] return extracted The motivation: http://stackoverflow.com/q/25472412/2301450

5 years, 8 months

Better integration of multiprocessing with asyncio

by Dan O'Reilly

I think it would be helpful for folks using the asyncio module to be able to make non-blocking calls to objects in the multiprocessing module more easily. While some use-cases for using multiprocessing can be replaced with ProcessPoolExecutor/run_in_executor, there are others that cannot; more advanced usages of multiprocessing.Pool aren't supported by ProcessPoolExecutor (initializer/initargs, contexts, etc.), and other multiprocessing classes like Lock and Queue have blocking methods that could be made into coroutines. Consider this (extremely contrived, but use your imagination) example of a asyncio-friendly Queue: import asyncio import time def do_proc_work(q, val, val2): time.sleep(3) # Imagine this is some expensive CPU work. ok = val + val2 print("Passing {} to parent".format(ok)) q.put(ok) # The Queue can be used with the normal blocking API, too. item = q.get() print("got {} back from parent".format(item)) def do_some_async_io_task(): # Imagine there's some kind of asynchronous I/O # going on here that utilizes asyncio. asyncio.sleep(5) @asyncio.coroutine def do_work(q): loop.run_in_executor(ProcessPoolExecutor(), do_proc_work, q, 1, 2) do_some_async_io_task() item = yield from q.coro_get() # Non-blocking get that won't affect our io_task print("Got {} from worker".format(item)) item = item + 25 yield from q.coro_put(item) if __name__ == "__main__": q = AsyncProcessQueue() # This is our new asyncio-friendly version of multiprocessing.Queue loop = asyncio.get_event_loop() loop.run_until_complete(do_work(q)) I have seen some rumblings about a desire to do this kind of integration on the bug tracker (http://bugs.python.org/issue10037#msg162497 and http://bugs.python.org/issue9248#msg221963) though that discussion is specifically tied to merging the enhancements from the Billiard library into multiprocessing.Pool. Are there still plans to do that? If so, should asyncio integration with multiprocessing be rolled into those plans, or does it make sense to pursue it separately? Even more generally, do people think this kind of integration is a good idea to begin with? I know using asyncio is primarily about *avoiding* the headaches of concurrent threads/processes, but there are always going to be cases where CPU-intensive work is going to be required in a primarily I/O-bound application. The easier it is to for developers to handle those use-cases, the better, IMO. Note that the same sort of integration could be done with the threading module, though I think there's a fairly limited use-case for that; most times you'd want to use threads over processes, you could probably just use non-blocking I/O instead. Thanks, Dan

5 years, 9 months

Python 2's re module should take longs

by Ryan Gonzalez

This works: re.search('(abc)', 'abc').group(1) but this doesn't: re.search('(abc)', 'abc').group(1L) The latter raises "IndexError: no such group". Shouldn't that technically work? -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/

5 years, 9 months

Implicit submodule imports

by Thomas Gläßle

Hey folks, What do you think about making it easier to use packages by automatically importing submodules on attribute access. Consider this example: >>> import matplotlib >>> figure = matplotlib.figure.Figure() AttributeError: 'module' object has no attribute 'figure' For the newcomer (like me some months ago) it's not obvious that the solution is to import matplotlib.figure. Worse even: it may sometimes/later on work, if the submodule has been imported from another place. How, I'd like it to behave instead (in pseudo code, since `package` is not a python class right now): class package: def __getattr__(self, name): try: return self.__dict__[name] except KeyError: # either try to import `name` or raise a nicer error message The automatic import feature could also play nicely when porting a package with submodules to or from a simple module with namespaces (as suggested in [1]), making this transition seemless to any user. I'm not sure about potential problems from auto-importing. I currently see the following issues: - harmless looking attribute access can lead to significant code execution including side effects. On the other hand, that could always be the case. - you can't use attribute access anymore to test whether a submodule is imported (must use sys.modules instead, I guess) In principle one can already make this feature happen today, by replacing the object in sys.modules - which is kind of ugly and has probably more flaws. This would also be made easier if there were a module.__getattr__ ([2]) or "metaclass" like feature for modules (which would be just a class then, I guess). Sorry, if this has come up before and I missed it. Anyhow, just interested if anyone else considers this a nice feature. Best regards, Thomas [1] https://mail.python.org/pipermail/python-ideas/2014-September/029341.html [2] https://mail.python.org/pipermail/python-ideas/2012-April/014957.html

5 years, 9 months

`numbers.Natural`

by Ram Rachum

I wish the `numbers` module would include a `Natural` class that would simply check whether the number is integral and positive.

5 years, 9 months

use gmtime(0) epoch in functions that use mktime()

by Akira Li

Python uses "seconds since the epoch" term to describe time.time() return value. POSIX term is "seconds since the Epoch" (notice the capitalization) where Epoch is 1970-01-01 00:00:00+00:00. C99 term is "calendar time" -- the encoding of the calendar time returned by the time() function is unspecified. Python documentation defines `epoch` as: The :dfn:`epoch` is the point where the time starts. On January 1st of that year, at 0 hours, the "time since the epoch" is zero. For Unix, the epoch is 1970. To find out what the epoch is, look at ``gmtime(0)``. time module documentation specifies calendar.timegm() as the inverse of time.gmtime() while timegm() uses the fixed 1970 Epoch instead of gmtime(0) epoch. datetime.astimezone() (local_timezone()) passes Unix timestamp [1970] to time.localtime() that may expect timestamp with different epoch [gmtime(0)]. email.util.mktime_tz() uses both mktime() [gmtime(0)] and timegm() [1970]. mailbox.py uses both time.time() [gmtime(0)] and timegm() [1970]. http.cookiejar uses both EPOCH_YEAR=1970 and datetime.utcfromtimestamp() [gmtime(0) epoch] for "seconds since epoch". It seems 1970 Epoch is used for file times on Windows (os.stat()) but os.path.getatime() refers to "seconds since epoch" [gmtime(0) epoch]. date{,time}.{,utc}fromtimestamp(), datetime.timestamp() docs equates "POSIX timestamp" [1970 Epoch] and time.time()'s returned value [gmtime(0) epoch]. datetime.timestamp() is inconsistent if gmtime(0) is not 1970. It uses mktime() for naive datetime objects [gmtime(0) epoch]. But it uses POSIX Epoch for aware datetime objects. Correct me if I'm wrong here. --- Possible fixes: Say in the `epoch` definition that stdlib doesn't support gmtime(0).tm_year != 1970. OR don't use mktime() if 1970 Epoch is used e.g., create an aware datetime object in the local timezone instead and use it to compute the result with 1970 Epoch. OR add *analog* of TZ=UTC time.mktime() and use it in stdlib where necessary. Looking at previous attempts (e.g., [1], [2]) to implement timegm(), the problem seems over-constrained. A different name could be used, to avoid wrong expectations e.g., datetime could use `(aware_datetime_object - gmtime0_epoch) // sec` [1] http://bugs.python.org/issue6280, [2] http://bugs.python.org/issue1667546 OR set EPOCH_YEAR=gmtime(0).tm_year instead of 1970 in calendar.timegm(). It may break backward compatibility if there is a system with non-1970 epoch. Deal on a case-by-case basis with other places where 1970 Epoch is used. And drop "POSIX timestamp" [1970 Epoch] and use "seconds since the epoch" [gmtime(0) epoch] in the datetime documentation. Change internal EPOCH year accordingly. What is Python-ideas opinion about it? -- Akira

5 years, 9 months

"continue with" for dynamic iterable injection

by Cathal Garvey

Hello all, First time post, so go gentle. :) During a twitter conversation with @simeonvisser, an idea emerged that he suggested I share here. The conversation began with me complaining that I'd like a third mode of explicit flow control in Python for-loops; the ability to repeat a loop iteration in whole. The reason for this was that I was parsing data where a datapoint indicated the end of a preceding sub-group, so the datapoint was both a data structure indicator *and* data in its own right. So I'd like to have iterated over the line once, branching into the flow control management part of an if/else, and then again to branch into the data management part of the same if/else. Yes, this is unnecessary and just a convenience for parsing that I'd like to see. During the discussion though, a much more versatile solution presented itself: repurposing the `continue` keyword to `continue with`, similarly to the repurposing of `yield` with `yield from`. This would avoid keyword bloat, but it would be explicit and intuitive to use, and would allow a pretty interesting extension of iteration with the for-loop. The idea is that `continue with` would allow injection of any datatype or variable, which becomes the next iterated item. So, saying `continue with 5` means that the next *x* in a loop structured as `for x in..` would be 5. This would satisfy my original, niche-y desire to re-iterate over something, but would also more broadly allow dynamic injection of *any* value into a for-loop iteration. As an extension to the language, it would present new and exciting ways to iterate infinitely by accident. It would also present new and exciting ways for people to trip up on mutable data; one way to misuse this is to iterate over data, modifying the data, then iterating over it again. The intention, rather, is that the repeated iteration allows re-iteration over unaltered data in response to a mid-iteration change in state (i.e., in my case, parsing a token, changing parser state, and then parsing the token again because it also carries informational content). However, bad data hygiene has always been part of Python, because it's a natural consequence of the pan-mutability that makes it such a great language. So, given that the same practices that an experienced Python dev learns to use would apply in this case, I don't think it adds anything that an experienced dev would trip over. As a language extension, I've never seen something like "continue with" in another language. You can mimic it with recursive functions and `cons`-ing the injection, but as a flow-control system for an imperative language, nope..and Python doesn't encourage or facilitate recursion anyway (no TCE). I'd love thoughts on this. It's unnecessary; but then, once something's declared turing complete you can accuse any additional feature of being unnecessary. I feel, more to the point, that it's in keeping with the Python philosophy and would add a novel twist to the language not seen elsewhere. best, Cathal -- Twitter: @onetruecathal, @formabiolabs Phone: +353876363185 Blog: http://indiebiotech.com miniLock.io: JjmYYngs7akLZUjkvFkuYdsZ3PyPHSZRBKNm6qTYKZfAM

5 years, 9 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas September 2014