I am proposing namespace context managers with implementing `__enter__` and
`__exit__` on dict objects. It would make closures possible in python with
a pythonic syntax.
a = 4
namespace = {}
with namespace:
a = 3
assert a == 4
assert namespace["a"] == 3
Hello list,
I sent an email to this list two or three months ago about the same idea.
In that discussion, there were both skepticism and support. Since I had
some time during the previous long weekend, I have made my idea more
concrete and I thought I would try with the list again, after having run it
through some of you privately.
GOAL: To have some parsing primitives in the stdlib so that other modules
in the stdlib itself can make use of. This would alleviate various security
issues we have seen throughout the years.
With that goal in mind, I opine that any parsing library for this purpose
should have the following characteristics:
#. Can be expressed in code. My opinion is that it is hard to review
generated code. Code review is even more crucial in security contexts.
#. Small and verifiable. This helps build trust in the code that is meant
to plug security holes.
#. Less evolving. Being in the stdlib has its drawback that is development
velocity. The library should be theoretically sound and stable from the
beginning.
#. Universal. Most of the times we'll parse left-factored context-free
grammars, but sometimes we'll also want to parse context-sensitive grammars
such as short XML fragments in which end tags must match start tags.
I have implemented a tiny (~200 SLOCs) package at
https://gitlab.com/nam-nguyen/parser_compynator that demonstrates something
like this is possible. There are several examples for you to have a feel of
it, as well as some early benchmark numbers to consider. This is far
smaller than any of the Python parsing libraries I have looked at, yet more
universal than many of them. I hope that it would convert the skeptics ;).
Finally, my request to the list is: Please debate on: 1) whether we want a
small (even private, underscore prefixed) parsing library in the stdlib to
help with tasks that are a little too complex for regexes, and 2) if yes,
how should it look like?
I also welcome comments (naming, uses of operator overloading, features,
bikeshedding, etc.) on the above package ;).
Thanks!
Nam
(This is my first time posting on any Python list; I've tried to search for this idea and didn't find it but if I looked in the wrong places/this has already been discussed I apologize and feel free to tell me!)
Say you have a list and you want to perform some operation on each item in the list - but you don't need to store the result in a list.
There are three simple ways of doing this, at least as far as I know: ([print(item)] could be any expression, just using it as an example)
```
lst = [1, 2, 3, 4]
#1
for item in lst:
print(item)
# 2
[print(item) for item in lst]
# 3
for item in lst: print(item)
```
#1 - In my opinion, this should be a one line operation so #1 is not ideal.
#2 - It also shouldn't require storing results in array, to save time/memory, so #2 is out.
#3 - I think #3 is just not good syntax, it seems very unpythonic to me - it breaks the norm that blocks go on their own lines. It does seem the best of the three though and I know my assessment is kind of subjective.
I'm wondering if a possible alternative syntax could be a for-expression, like there's if-expressions, which always evaluates to None:
```
print(item) for item in lst
```
A more practical example of when this would be useful is extending list-2 with a modified version of list-1 - this syntax would avoid creating an intermediate list (not sure if the way lists are implemented in python removes this advantage by the way it resizes lists though).
```
lst1 = [1, 2, 3]
lst2 = [4, 5, 6]
lst1.append(item * 2) for item in lst1
```
Hello all,
I would like to briefly share my thoughts on non-standard evaluation (NSE),
why this is useful, and how, potentially, it can be added to Python. In
most languages, functions have access only to the *value* of their
arguments, and not to the expressions that yielded those values. However,
sometimes it is useful to have access to the expressions as well. For
example, let plot(x, y, ...) be a function that draws the scatter plot of y
against x and adds labels for the x- and y-axes. Currently, there is no way
to distinguish plot(x, *z*, ...) from plot(x, *y*, ...) from within the
function. As such, one needs to pass the names of the variables to the plot
function to be used as the axes' labels. Using NSE, though, the function
can look at the expressions in the method call and use those as the default
value for labels. In R, this idea is used widely to make the syntax simpler.
In the following, I sketch how I think this can be implemented:
1. Let BoundExpression be a class containing an ast.Expression as well
as locals and globals dictionaries. BoundExpression can also have an eval
method that evaluates its expression using its locals and globals
dictionaries.
2. Let `x` be the short form for BoundExpression("x", locals(),
globals()). Then, the plot function can be implemented in a way that
plot(`x`, `y`) draws scatter plot of x and y *and also,* labels the axes
correctly. No need to provide labels explicitly anymore.
3. A more challenging idea is to let developers decide whether their
functions need NSE or not. For example, when a function is defined as def
f(`x`), for any method call like f(x), the first argument should be wrapped
with a BoundExpression instance.
I would appreciate any feedback on these ideas.
Best,
Nima
In asyncio, when a task awaits for another task (or future), it can be cancelled right after the awaited task finished (before the callback have been processed). Thus, if the awaited task has consumed data, the data is lost.
For instance, with the following code:
import asyncio
available_data = []
data_ready = asyncio.Future()
def feed_data(data):
global data_ready
available_data.append(data)
data_ready.set_result(None)
data_ready = asyncio.Future()
async def consume_data():
while not available_data:
await asyncio.shield(data_ready)
return available_data.pop()
async def wrapped_consumer():
task = asyncio.ensure_future(consume_data())
return await task
If I perform those exact steps:
async def test():
task = asyncio.ensure_future(wrapped_consumer())
await asyncio.sleep(0)
feed_data('data')
await asyncio.sleep(0)
task.cancel()
await asyncio.sleep(0)
print ('task', task)
print ('available_data', available_data)
loop = asyncio.get_event_loop()
loop.run_until_complete(test())
Then I can see that the task has been cancelled despite the data being consumed. Since the result of `wrapped_consumer` cannot be retrieved, the data is forever lost.
task <Task cancelled coro=<wrapped_consumer() done, defined at <ipython-input-1-de4ad193b1d0>:17>>
available_data []
This side effect does not happen when awaiting a coroutine, but coroutine are not as flexible as tasks. It happens when awaiting a `Future`, a `Task`, or any function like `asyncio.wait`, `asyncio.wait_for` or `asyncio.gather` (which all inherit from or use `Future`). There is then no way to do anything equivalent to:
stop_future = asyncio.Future()
async def wrapped_consumer2():
task = asyncio.ensure_future(consume_data())
try:
await asyncio.wait([task, stop_future])
finally:
task.cancel()
await asyncio.wait([task])
if not task.cancelled():
return task.result()
else:
raise RuntimeError('stopped')
This is due to the Future calling the callback asynchronously:
https://github.com/python/cpython/blob/3.6/Lib/asyncio/futures.py#L214
for callback in callbacks:
self._loop.call_soon(callback, self)
I propose to create synchronous versions of those, or a `synchronous_callback` parameter, that turns the callbacks of `Future` synchronous. I've experimented a simple librairy `syncio` with CPython 3.6 to do this (it is harder to patch later versions due to the massive use of private methods).
Basically, needs to:
1) replace the `Future._schedule_callbacks` method by a synchronous version
2) fix `Task._step` to not fail when cleaning `_current_tasks` (https://github.com/python/cpython/blob/3.6/Lib/asyncio/tasks.py#L245)
3) rewrite all the functions to use synchronous futures instead of normal ones
With that librairy, the previous functions are possible and intuitive
import syncio
async def wrapped_consumer():
task = syncio.ensure_sync_future(consume_data())
return await task
stop_future = asyncio.Future()
async def wrapped_consumer2():
task = syncio.ensure_sync_future(consume_data())
try:
await syncio.sync_wait([task, stop_future])
finally:
task.cancel()
await asyncio.wait([task])
if not task.cancelled():
return task.result()
else:
raise RuntimeError('stopped')
No need to use `syncio` anywhere else in the code, which makes it totally transparent for the end user.
This "library" can be found here: https://github.com/aure-olli/aiokafka/blob/216b4ad3b4115bc9fa463e44fe2363...
It implements `SyncFuture`, `SyncTask`, `ensure_sync_future`, `sync_wait`, `sync_wait_for`, `sync_gather` and `sync_shield`. It works with CPython 3.6 only.
To conclude:
- asynchronous callbacks are preferrable in most case, but do not provide a coherent cancelled status in specific cases
- implementing a version with synchronous callback (or a `synchronous_callback` parameter) is rather easy (however step 2 need to be clarified, probably a cleaner way to fix this)
- it is totally transparent for the end user, as synchronous callback are totally compatible with asynchronous ones
Ouch! That's what happens when you don't proof read your emails :S
Replace --verbose with --print_operation, the idea of the snippet was to
show an example for keyword parameters.
Python allows you to write code in tight and readable form. Consider the
following example.
with connect() as stream: # connect() or __enter__() can fail.
for data in stream: # __next__() can fail
write(data) # write() can fail
The problem is that different lines can raise an exception of the same
type (for example OSError). We want to catch and handle exceptions
raised when open a connection, when read a data and when write a data in
different ways. Currently you need to expand so convenient Python
statements "with" and "for" (see PEP 343)
_mgr = connect()
_enter = type(_mgr).__enter__
_exit = type(_mgr).__exit__
_value = _enter(_mgr)
_exc = True
try:
stream = _value
_it = iter(stream)
while True:
try:
data = next(_it)
except StopIteration:
break
write(data)
except:
_exc = False
if not _exit(_mgr, *sys.exc_info()):
raise
finally:
if _exc:
_exit(_mgr, None, None, None)
and then add "try ... except" around corresponding explicit calls of
`__enter__()` and `next()`.
try:
_mgr = connect()
_enter = type(_mgr).__enter__
_exit = type(_mgr).__exit__
_value = _enter(_mgr)
_exc = True
except OSError:
handle_connection_error()
else:
try:
stream = _value
try:
_it = iter(stream)
except OSError:
handle_read_error()
else:
while True:
try:
data = next(_it)
except StopIteration:
break
except OSError:
handle_read_error()
break
try:
write(data)
except OSError:
handle_write_error()
except:
_exc = False
if not _exit(_mgr, *sys.exc_info()):
raise
finally:
if _exc:
_exit(_mgr, None, None, None)
Does not it look ugly?
I propose to add "except" clause to "for" and "with" statement to catch
exceptions in the code that can't be wrapped with "try ... except".
for VAR in EXPR:
BLOCK
except EXC:
HANDLER
should be equivalent to
try:
_it = iter(EXPR)
except EXC:
HANDLER
else:
while True:
try:
VAR = next(_it)
except StopIteration:
break
except EXC:
HANDLER
break
BLOCK
and
with EXPR as VAR:
BLOCK
except EXC:
HANDLER
try:
_mgr = EXPR
_enter = type(_mgr).__enter__
_exit = type(_mgr).__exit__
_value = _enter(_mgr)
_exc = True
except EXC:
HANDLER
else:
try:
VAR = _value
BLOCK
except:
_exc = False
if not _exit(_mgr, *sys.exc_info()):
raise
finally:
if _exc:
_exit(_mgr, None, None, None)
And correspondingly for asynchronous versions "async for" and "async with".
So you will be able to add errors handling like in:
with connect() as stream:
for data in stream:
try:
write(data)
except OSError:
handle_write_error()
except OSError:
handle_read_error()
except OSError:
handle_connection_error()
Maybe the def __main__() argument is already a dead horse, given the number of discussions it has created that have ended nowhere, but I think one argument in favour of its implementation would be including argument parsing in it, for example:
# main.py
def __run__(first_num, second_num, print_operation=False):
"""
Adds two numbers.
positional arguments:
- first_num: the first number.
- second_num: the second number.
optional arguments:
- print_operation: prints the sum operation to stdout.
"""
result = int(first_num) + int(second_num)
if print_operation:
print(f'{first_num} + {second_num} = {result}')
else:
print(result)
$ python main.py -h
Adds two numbers.
positional parameters:
- first_num: the first number.
- second_num: the second number.
optional arguments:
- print_operation: prints the sum operation to stdout.
$ python main.py 1 2 --verbose
1 + 2 = 3
$ python main.py 1 2
3
The -h flag would print the function docstring. We could also add type hints for casting (I'm not entirely sure how feasible this is):
def __run__(first_num: int, second_num: int, print_operation=False):
...
result = first_num + second_num # no need for casting here
...
Since __main__ is already a built-in function, I abstained from using it as the designated function name (I picked __run__, but any other suggestions would be welcome, also thought of __entrypoint__, __exec__). Thoughts?
In asyncio, when a task awaits for another task (or future), it can be
cancelled right after the awaited task finished. Thus, if the awaited task
has consumed data, the data is lost.
For instance, with the following code:
import asyncio
available_data = []
data_ready = asyncio.Future()
def feed_data(data):
global data_ready
available_data.append(data)
data_ready.set_result(None)
data_ready = asyncio.Future()
async def consume_data():
while not available_data:
await asyncio.shield(data_ready)
return available_data.pop()
async def wrapped_consumer():
task = asyncio.ensure_future(consume_data())
return await task
If I perform those exact steps:
async def test():
task = asyncio.ensure_future(wrapped_consumer())
await asyncio.sleep(0)
feed_data('data')
await asyncio.sleep(0)
task.cancel()
await asyncio.sleep(0)
print ('task', task)
print ('available_data', available_data)
loop = asyncio.get_event_loop()
loop.run_until_complete(test())
Then I can see that the task has been cancelled despite the data being
consumed. Since the result of `wrapped_consumer` cannot be retrieved, the
data is forever lost.
task <Task cancelled coro=<wrapped_consumer() done, defined at
<ipython-input-1-de4ad193b1d0>:17>>
available_data []
This side effect does not happen when awaiting a coroutine, but coroutines
are not as flexible as tasks (unless manipulated as a generator). It
happens when awaiting a `Future`, a `Task`, or any function like
`asyncio.wait`, `asyncio.wait_for` or `asyncio.gather` (which all inherit
from or use `Future`). There is then no way to do anything equivalent to:
stop_future = asyncio.Future()
async def wrapped_consumer2():
task = asyncio.ensure_future(consume_data())
try:
await asyncio.wait([task, stop_future])
finally:
task.cancel()
if not task.cancelled():
return task.result()
else:
raise RuntimeError('stopped')
This is due to the Future calling the callback asynchronously:
https://github.com/python/cpython/blob/3.6/Lib/asyncio/futures.py#L214
for callback in callbacks:
self._loop.call_soon(callback, self)
I propose to create synchronous versions of those, or a
`synchronous_callback` parameter, that turns the callbacks of `Future`
synchronous. I've experimented a simple library `syncio` with CPython 3.6
to do this (it is harder to patch later versions due to the massive use of
private methods).
Basically, needs to:
1) replace the `Future._schedule_callbacks` method by a synchronous version
2) fix `Task._step` to not fail when cleaning `_current_tasks` (
https://github.com/python/cpython/blob/3.6/Lib/asyncio/tasks.py#L245)
3) rewrite all the functions to use synchronous futures instead of normal
ones
With that library, the previous functions are possible and intuitive
import syncio
async def wrapped_consumer():
task = syncio.ensure_sync_future(consume_data())
return await task
stop_future = asyncio.Future()
async def wrapped_consumer2():
task = syncio.ensure_sync_future(consume_data())
try:
await syncio.sync_wait([task, stop_future])
finally:
task.cancel()
if not task.cancelled():
return task.result()
else:
raise RuntimeError('stopped')
No need to use `syncio` anywhere else in the code, which makes it totally
transparent for the end user. `wrapped_consumer` and `wrapped_consumer2`
are now cancelled if and only if the data hasn't been consumed, whatever is
the order of the steps (and the presence of `asyncio.sleep`).
This "library" can be found here:
https://github.com/aure-olli/aiokafka/blob/3acb88d6ece4502a78e230b234f47b...
It implements `SyncFuture`, `SyncTask`, `ensure_sync_future`, `sync_wait`,
`sync_wait_for`, `sync_gather` and `sync_shield`. It works with CPython 3.6.
To conclude:
- asynchronous callbacks are preferable in most cases, but do not provide a
coherent cancelled status in specific cases
- implementing a version with synchronous callback (or a
`synchronous_callback` parameter) is rather easy (however step 2 need to be
clarified, probably a cleaner way to fix this)
- it is totally transparent for the end user, as synchronous callbacks are
totally compatible with asynchronous ones
I think it looks very fine when you type {1, 2, 3} * {"a", "b", "c"} and
get set(itertools.product({1, 2, 3}, {"a", "b", "c"})). So i am proposing
set multiplication implementation as cartesian product.
>>>
