Mailman 3 July 2015 - Python-ideas

@contextlib.contextmanager(with_context=True) - passing a context to contextmanager
by Giampaolo Rodola' July 10, 2015

July 10, 2015

contextlib.contextmanager in Python 3 added the ability to define functions which work both as decorators and context managers, which is great and one of the features I appreciate the most about Python 3 (seriously). I am facing a use case where I find contextlib.contextmanager has some limitations though. I have a decorator which looks like this: def wrap_exceptions(fun): """Decorator which translates bare OSError and IOError exceptions into NoSuchProcess and AccessDenied. """ … [View More] @functools.wraps(fun) def wrapper(self, *args, **kwargs): # here "self" is the Process class instance try: return fun(self, *args, **kwargs) except EnvironmentError as err: if err.errno in (errno.ENOENT, errno.ESRCH): raise NoSuchProcess(self.pid, self._name) if err.errno in (errno.EPERM, errno.EACCES): raise AccessDenied(self.pid, self._name) raise return wrapper ...and I use it like this: class Process: @wrap_exceptions def exe(): ... The two key things about this decorator are: - it's designed to be used with class methods - it has a reference to the method's class instance (self) I would like to push this a bit further and make wrap_exceptions() work also a contextmanager, which is what contextlib.contextmanager should allow me to do in an easy way. As contextlib.contextmanager stands right it won't allow my use case though as there's no way to pass a reference of the class instance (Process) to wrap_exceptions. So here is my proposal: what if we add a new "with_context" argument to contextlib.contextmanager? The resulting code would look like this: @contextlib.contextmanager(with_context=True) def wrap_exceptions(ctx): # ctx is the Process class instance try: yield except EnvironmentError as err: pid = ctx.pid name = ctx.name if err.errno in (errno.ENOENT, errno.ESRCH): raise NoSuchProcess(pid, name) if err.errno in (errno.EPERM, errno.EACCES): raise AccessDenied(pid, name) raise class Process: @wrap_exceptions() def exe(): ... class Process: def exe(self): with wrap_exceptions(self): ... It must be noted that: - when with_context=True and wrap_exceptions is used as a decorator it can only be used to decorate class methods and not regular functions - when used as a decorator "self" is automatically passed as the first argument for wrap_exceptions I'm not sure if this is actually possible as I haven't gone through contextlib.contextmanager in details (it's quite magical). Thoughts? -- Giampaolo - http://grodola.blogspot.com [View Less]

2 1

Reporting unmatched parentheses in SyntaxError messages?
by Nick Coghlan July 9, 2015

July 9, 2015

One of the more opaque error messages new Python users can encounter is a syntax error due to unmatched parentheses: File "/home/me/myfile.py", line 11 data = func() ^ SyntaxError: invalid syntax While I have no idea how we could implement it, I'm wondering if that might be clearer if the error message instead looked more like this: File "/home/me/myfile.py", line 11 data = func() ^ SyntaxError: invalid syntax (Unmatched '(' on line 10) Or, … [View More]

11 13

OrderedDict.peekitem()
by Kale Kundert July 7, 2015

July 7, 2015

Today I was trying to use collections.OrderedDict to manage a LIFO queue, and I was surprised to realize that OrderedDict doesn't provide a way to look at its first or last item. There is an OrderedDict.popitem() method, which removes and returns either the first or last item, but it's not hard to imagine cases where you would want to see what's on the queue without popping it right away. My proposal is to add a peekitem() method to OrderedDict. This method would have the same signature and … [View More]

12 24

Re: [Python-ideas] [Python-Dev] PEP 493: Redistributor guidance for Python 2.7 HTTPS
by Stephen J. Turnbull July 7, 2015

July 7, 2015

Cross-posted to redirect discussion. Replies directed to Python-Ideas. Erik Bray writes on Python-Dev: > On Mon, Jul 6, 2015 at 6:21 AM, Antoine Pitrou <solipsis(a)pitrou.net> wrote: > > On Mon, 6 Jul 2015 14:22:46 +1000, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > >> > >> The main change from the last version discussed on python-ideas > > > > Was it discussed there? That list has become totally useless, I've > > stopped … [View More]

3 2

Re: [Python-ideas] [Python-Dev] PEP 493: Redistributor guidance for Python 2.7 HTTPS
by Guido van Rossum July 7, 2015

July 7, 2015

On Tue, Jul 7, 2015 at 11:24 AM, Mark Lawrence <breamoreboy(a)yahoo.co.uk> wrote: > From https://mail.python.org/mailman/listinfo/python-ideas > > <quote> > This list is to contain discussion of speculative language ideas for > Python for possible inclusion into the language. If an idea gains traction > it can then be discussed and honed to the point of becoming a solid > proposal to put to python-dev as appropriate. > </quote> > > Relative to … [View More]

1 0

Indexable dict and set (was: OrderedDict.peekitem())
by Grant Jenks July 7, 2015

July 7, 2015

>>> On Mon, Jul 6, 2015 at 3:30 PM, Neil Girdhar <mistersheik(a)gmail.com> >>> wrote: >>> SortedDict >>> (http://www.grantjenks.com/docs/sortedcontainers/sorteddict.html) >>> manages to support indexing. Can OrderedDict do the same thing? >> >> On Mon, Jul 6, 2015 at 5:59 PM, Eric Snow <ericsnowcurrently(a)gmail.com> >> wrote: >> >> Don't forget that, as the docs describe, an "OrderedDict is a dict >> … [View More]

1 0

Fwd: [Python-Dev] An yocto change proposal in logging module to simplify structured logs support
by Ludovic Gasc July 6, 2015

July 6, 2015

Hi Python-Ideas ML, To resume quickly the idea: I wish to add "extra" attribute to LogMessage, to facilitate structured logs generation. For more details with use case and example, you can read message below. Before to push the patch on bugs.python.org, I'm interested in by your opinions: the patch seems to be too simple to be honest. Regards. -- Ludovic Gasc (GMLudo) http://www.gmludo.eu/ ---------- Forwarded message ---------- From: Guido van Rossum <guido(a)python.org> Date: 2015-… [View More]05-24 23:44 GMT+02:00 Subject: Re: [Python-Dev] An yocto change proposal in logging module to simplify structured logs support To: Ludovic Gasc <gmludo(a)gmail.com> Ehh, python-ideas? On Sun, May 24, 2015 at 10:22 AM, Ludovic Gasc <gmludo(a)gmail.com> wrote: > Hi, > > 1. The problem > > For now, when you want to write a log message, you concatenate the data > from your context to generate a string: In fact, you convert your > structured data to a string. > When a sysadmin needs to debug your logs when something is wrong, he must > write regular expressions to extract interesting data. > > Often, he must find the beginning of the interesting log and follow the > path. Sometimes, you can have several requests in the same time in the log, > it's harder to find interesting log. > In fact, with regular expressions, the sysadmin tries to convert the log > lines strings to structured data. > > 2. A possible solution > > You should provide a set of regular expressions to your sysadmins to help > them to find the right logs, however, another approach is possible: > structured logs. > Instead of to break your data structure to push in the log message, the > idea is to keep the data structure, to attach that as metadata of the log > message. > For now, I know at least Logstash and Journald that can handle structured > logs and provide a query tool to extract easily logs. > > 3. A concrete example with structured logs > > As most Web developers, we build HTTP daemons used by several different > human clients in the same time. > In the Python source code, to support structured logs, you don't have a > big change, you can use "extra" parameter for that, example: > > [handle HTTP request] > LOG.debug('Receive a create_or_update request', extra={'request_id': > request.request_id, > > 'account_id': account_id, > > 'aiohttp_request': request, > > 'payload': str(payload)}) > [create data in database] > LOG.debug('Callflow created', extra={'account_id': account_id, > 'request_id': > request.request_id, > 'aiopg_cursor': cur, > 'results': row}) > > Now, if you want, you can enhance the structured log with a custom logging > Handler, because the standard journald handler doesn't know how to handle > aiohttp_request or aiopg_cursor. > My example is based on journald, but you can write an equivalent version > with python-logstash: > #### > from systemdream.journal.handler import JournalHandler > > class Handler(JournalHandler): > # Tip: on a system without journald, use socat to test: > # socat UNIX-RECV:/run/systemd/journal/socket STDIN > def emit(self, record): > if record.extra: > # import ipdb; ipdb.set_trace() > if 'aiohttp_request' in record.extra: > record.extra['http_method'] = > record.extra['aiohttp_request'].method > record.extra['http_path'] = > record.extra['aiohttp_request'].path > record.extra['http_headers'] = > str(record.extra['aiohttp_request'].headers) > del(record.extra['aiohttp_request']) > if 'aiopg_cursor' in record.extra: > record.extra['pg_query'] = > record.extra['aiopg_cursor'].query.decode('utf-8') > record.extra['pg_status_message'] = > record.extra['aiopg_cursor'].statusmessage > record.extra['pg_rows_count'] = > record.extra['aiopg_cursor'].rowcount > del(record.extra['aiopg_cursor']) > super().emit(record) > #### > > And you can enable this custom handler in your logging config file like > this: > [handler_journald] > class=XXXXXXXXXX.utils.logs.Handler > args=() > formatter=detailed > > And now, with journalctl, you can easily extract logs, some examples: > Logs messages from 'lg' account: > journalctl ACCOUNT_ID=lg > All HTTP requests that modify the 'lg' account (PUT, POST and DELETE): > journalctl ACCOUNT_ID=lg HTTP_METHOD=PUT > HTTP_METHOD=POST HTTP_METHOD=DELETE > Retrieve all logs from one specific HTTP request: > journalctl REQUEST_ID=130b8fa0-6576-43b6-a624-4a4265a2fbdd > All HTTP requests with a specific path: > journalctl HTTP_PATH=/v1/accounts/lg/callflows > All logs of "create" function in the file "example.py" > journalctl CODE_FUNC=create CODE_FILE=/path/example.py > > If you already do a troubleshooting on a production system, you should > understand the interest of this: > In fact, it's like to have SQL queries capabilities, but it's logging > oriented. > We use that since a small time on one of our critical daemon that handles > a lot of requests across several servers, it's already adopted from our > support team. > > 4. The yocto issue with the Python logging module > > I don't explain here a small part of my professional life for my pleasure, > but to help you to understand the context and the usages, because my patch > for logging is very small. > If you're an expert of Python logging, you already know that my Handler > class example I provided above can't run on a classical Python logging, > because LogRecord doesn't have an extra attribute. > > extra parameter exists in the Logger, but, in the LogRecord, it's merged > as attributes of LogRecord: > https://github.com/python/cpython/blob/master/Lib/logging/__init__.py#L1386 > > It means, that when the LogRecord is sent to the Handler, you can't > retrieve the dict from the extra parameter of logger. > The only way to do that without to patch Python logging, is to rebuild by > yourself the dict with a list of official attributes of LogRecord, as is > done in python-logstash: > > https://github.com/vklochan/python-logstash/blob/master/logstash/formatter.… > At least to me, it's a little bit dirty. > > My quick'n'dirty patch I use for now on our CPython on production: > > diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py > index 104b0be..30fa6ef 100644 > --- a/Lib/logging/__init__.py > +++ b/Lib/logging/__init__.py > @@ -1382,6 +1382,7 @@ class Logger(Filterer): > """ > rv = _logRecordFactory(name, level, fn, lno, msg, args, exc_info, > func, > sinfo) > + rv.extra = extra > if extra is not None: > for key in extra: > if (key in ["message", "asctime"]) or (key in > rv.__dict__): > > At least to me, it should be cleaner to add "extra" as parameter > of _logRecordFactory, but I've no idea of side effects, I understand that > logging module is critical, because it's used everywhere. > However, except with python-logstash, to my knowledge, extra parameter > isn't massively used. > The only backward incompatibility I see with a new extra attribute of > LogRecord, is that if you have a log like this: > LOG.debug('message', extra={'extra': 'example'}) > It will raise a KeyError("Attempt to overwrite 'extra' in LogRecord") > exception, but, at least to me, the probability of this use case is near to > 0. > > Instead of to "maintain" this yocto patch, even it's very small, I should > prefer to have a clean solution in Python directly. > > Thanks for your remarks. > > Regards. > -- > Ludovic Gasc (GMLudo) > http://www.gmludo.eu/ > > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) [View Less]

4 6

Re: [Python-ideas] Should iPython Notebook replace Idle
by Todd July 6, 2015

July 6, 2015

On Jul 4, 2015 5:16 AM, "Jason Swails" <jason.swails(a)gmail.com> wrote: > > > > On Fri, Jul 3, 2015 at 10:01 PM, Sayth Renshaw <flebber.crue(a)gmail.com> wrote: >> >> In future releases of Python should ipython Notebooks replace idle as the default tool for new users to learn python? >> >> >> This would as I see it have many benefits? >> >> 1. A nicer more usual web interface for new users. >> 2. Would allow the python … [View More]documentation and tutorials to be distributed as ipython notebooks which would allow new users to play and interact with the tutorials as they proceed. No download separate code retyping just edit run and play. >> 3. Would allow teachers to setup notebooks knowing that all users have the same default environment, no need for setting up virtualenvs etc. >> 4. Strengthen the learning base and for new python developers as a whole. >> >> Thoughts? > > > IPython and IDLE are different. IPython is *just* an interactive Python interpreter with a ton of tweaks and enhancements. IDLE, by contrast, is both an upscale interpreter (not *nearly* as feature-complete as IPython), but it's also an IDE. AFAICT, IPython does not do this. > > Also, look at the IPython dependencies for its core functionalities: > > - jinja2 > - sphinx > - pyzmq > - pygments > - tornado > - PyQt | PySide > > None of these are part of the Python standard library. By contrast, IDLE is built entirely with stdlib components (tkinter for the GUI). AFAIK, nothing in the stdlib depends on anything outside of it. And addition to the Python stdlib imposes some pretty serious restrictions on a library. If the IPython team agreed to release their tools with the stdlib instead of IDLE, they'd have to give up a lot of control over their project: > > - License > - Release schedule > - Development environment > > Everything gets swallowed into Python. I can't imagine this ever happening. > It is certainly true that IDLE and IPython do not cover the same use-cases, and it almost certainly true that putting the IPython notebook into the standard library is infeasible. That being said, one thing that IPython and other shells have shown is that it is possible to make a much more powerful python shell. So I don't think it is out of the realm of possibility to take a hard look at the current python shell and see where and how it can be made more useful. The IPython shell is one of many places we could look for ideas. More out-there, but it probably isn't completely impossible for python to provide some sort of native notebook-like interface, or at least some sort of interface that makes it convenient for third parties to make such notebook interfaces. [View Less]

9 17

Pass a function as the argument "step" of range()
by Pierre Quentel July 6, 2015

July 6, 2015

In languages such as Javascript, the incrementation of a for loop counter can be done by an operation, for instance : for(i=1; i<N; i*=2) would iterate on the powers of 2 lesser than N. To achieve the same thing in Python we currently can't use range() because it increments by an integer (the argument "step"). An option is to build a generator like : def gen(N): i = 1 while i<=N: yield i i *= 2 then we can iterate on gen(N). My proposal is that besides an … [View More]

14 49

Enabling access to the AST for Python code
by Ben Hoyt July 4, 2015

July 4, 2015

Hi Python Ideas folks, (I previously posted a similar message on Python-Dev, but it's a better fit for this list. See that thread here: https://mail.python.org/pipermail/python-dev/2015-May/140063.html) Enabling access to the AST for compiled code would make some cool things possible (C# LINQ-style ORMs, for example), and not knowing too much about this part of Python internals, I'm wondering how possible and practical this would be. Context: PonyORM (http://ponyorm.com/) allows you to write … [View More]

15 29

Are there asynchronous generators?
by Adam Bartoš July 3, 2015

July 3, 2015

Hello, I had a generator producing pairs of values and wanted to feed all the first members of the pairs to one consumer and all the second members to another consumer. For example: def pairs(): for i in range(4): yield (i, i ** 2) biconsumer(sum, list)(pairs()) -> (6, [0, 1, 4, 9]) The point is I wanted the consumers to be suspended and resumed in a coordinated manner: The first producer is invoked, it wants the first element. The coordinator implemented by biconsumer … [View More]function invokes pairs(), gets the first pair and yields its first member to the first consumer. Then it wants the next element, but now it's the second consumer's turn, so the first consumer is suspended and the second consumer is invoked and fed with the second member of the first pair. Then the second producer wants the next element, but it's the first consumer's turn… and so on. In the end, when the stream of pairs is exhausted, StopIteration is thrown to both consumers and their results are combined. The cooperative asynchronous nature of the execution reminded me asyncio and coroutines, so I thought that biconsumer may be implemented using them. However, it seems that it is imposible to write an "asynchronous generator" since the "yielding pipe" is already used for the communication with the scheduler. And even if it was possible to make an asynchronous generator, it is not clear how to feed it to a synchronous consumer like sum() or list() function. With PEP 492 the concepts of generators and coroutines were separated, so asyncronous generators may be possible in theory. An ordinary function has just the returning pipe – for returning the result to the caller. A generator has also a yielding pipe – used for yielding the values during iteration, and its return pipe is used to finish the iteration. A native coroutine has a returning pipe – to return the result to a caller just like an ordinary function, and also an async pipe – used for communication with a scheduler and execution suspension. An asynchronous generator would just have both yieling pipe and async pipe. So my question is: was the code like the following considered? Does it make sense? Or are there not enough uses cases for such code? I found only a short mention in https://www.python.org/dev/peps/pep-0492/#coroutine-generators, so possibly these coroutine-generators are the same idea. async def f(): number_string = await fetch_data() for n in number_string.split(): yield int(n) async def g(): result = async/await? sum(f()) return result async def h(): the_sum = await g() As for explanation about the execution of h() by an event loop: h is a native coroutine called by the event loop, having both returning pipe and async pipe. The returning pipe leads to the end of the task, the async pipe is used for cummunication with the scheduler. Then, g() is called asynchronously – using the await keyword means the the access to the async pipe is given to the callee. Then g() invokes the asyncronous generator f() and gives it the access to its async pipe, so when f() is yielding values to sum, it can also yield a future to the scheduler via the async pipe and suspend the whole task. Regards, Adam Bartoš [View Less]

12 27