Hello,
I'm thinking of this idea that we have a pseudo-operator called
"Respectively" and shown maybe with ;
Some examples first:
a;b;c = x1;y1;z1 + x2;y2;z2
is equivalent to
a=x1+x2
b=y1+y2
c=z1+z2
So it means for each position in the statement, do something like
respectively. It's like what I call a vertical expansion, i.e. running
statements one by one.
Then there is another unpacking operator which maybe we can show with $
sign and it operates on lists and tuples and creates the "Respectively"
version of them.
So for instance,
vec=[]*10
$vec = $u + $v
will add two 10-dimensional vectors to each other and put the result in vec.
I think this is a syntax that can make many things more concise plus it
makes component wise operation on a list done one by one easy.
For example, we can calculate the inner product between two vectors like
follows (inner product is the sum of component wise multiplication of two
vectors):
innerProduct =0
innerProduct += $a * $b
which is equivalent to
innerProduct=0
for i in range(len(a)):
...innerProduct += a[i]+b[i]
For example, let's say we want to apply a function to all element in a
list, we can do:
f($a)
The $ and ; take precedence over anything except ().
Also, an important thing is that whenever, we don't have the respectively
operator, such as for example in the statement above on the left hand side,
we basically use the same variable or value or operator for each statement
or you can equivalently think we have repeated that whole thing with ;;;;.
Such as:
s=0
s;s;s += a;b;c; * d;e;f
which result in s being a*d+b,c*e+d*f
Also, I didn't spot (at least for now any ambiguity).
For example one might think what if we do this recursively, such as in:
x;y;z + (a;b;c);(d;e;f);(g;h;i)
using the formula above this is equivalent to
(x;x;x);(y;y;y);(z;z;z)+(a;b;c);(d;e;f);(g;h;i)
if we apply print on the statement above, the result will be:
x+a
x+b
x+c
y+d
y+e
y+f
z+g
z+h
z+i
Beware that in all of these ; or $ does not create a new list. Rather, they
are like creating new lines in the program and executing those lines one by
one( in the case of $, to be more accurate, we create for loops).
I'll appreciate your time and looking forward to hearing your thoughts.
Cheers,
Moj
Hi,
Thank you for all feedback on my PEP 511. It looks like the current
blocker point is the unclear status of "language extensions": code
tranformers which deliberately changes the Python semantics. I would
like to discuss how we should register them. I think that the PEP 511
must discuss "language extensions" even if it doesn't have to propose
a solution to make their usage easier. It's an obvious usage of code
transformers. If possible, I would like to find a compromise to
support them, but make it explicit that they change the Python
semantics.
By the way, I discussed with Joseph Jevnik who wrote codetransformer
(bytecode transformer) and lazy_python (AST transformer). He wrote me:
"One concern that I have though is that transformers are registered
globally. I think that the decorators in codetransformer do a good job
of signalling to reader the scope of some new code generation."
Currently, the PEP 511 doesn't provide a way to register a code
transformer but only use it under some conditions. For example, if
fatoptimizer is registered, all .pyc files will be called
file.cpython-36.fat-0.pyc even if fatoptimizer was disabled.
I propose to change the design of sys.set_code_transformers() to use
it more like a registry similar to the codecs registry
(codecs.register), but different (details below). A difference is that
the codecs registry uses a mapping (codec name => codec functions),
whereas sys.set_code_transformers() uses an ordered sequence (list) of
code transformers. A sequence is used because multiple code
transformers can be applied sequentially on a single .py file.
Petr Viktorin wrote that language extensions "target specific modules,
with which they're closely coupled: The modules won't run without the
transformer. And with other modules, the transformer either does
nothing (as with MacroPy, hopefully), or would fail altogether (as
with Hy). So, they would benefit from specific packages opting in. The
effects of enabling them globally range from inefficiency (MacroPy) to
failures or needing workarounds (Hy)."
Problem (A): solutions proposed below don't make code tranformers
mandatory. If a code *requires* a code transformer and the code
transformer is not registered, Python doesn't complain. Do you think
that it is a real issue in practice? For MacroPy, it's not a problem
in practice since functions must be decorated using a decorator from
the macropy package. If importing macropy fails, the module cannot be
imported.
Problem (B): proposed solutions below adds markers to ask to enable a
specific code transformer, but a code transformer can decide to always
modify the Python semantics without using such marker. According to
Nick Coghlan, code transformers changing the Python semantics *must*
require a marker in the code using them. IMHO it's the responsability
of the author of the code transformer to use markers, not the
responsability of Python.
Code transformers should maybe return a flag telling if they changed
the code or not. I prefer a flag rather than comparing the output to
the input, since the comparison can be expensive, especially for a
deep AST tree. Example:
class Optimizer:
def ast_optimizer(self, tree, context):
# ...
return modified, tree
*modified* must be True if tree was modified.
There are several options to decide if a code transformer must be used
on a specific source file.
(1) Add a check_code() and check_ast() functions to code transformers.
The code transformer is responsible to decide if it wants to transform
the code or not. Python doesn't use the code transformer if the check
method returns False.
Examples:
* MacroPy can search for the "import macropy" statement (of "from
macropy import ...") in the AST tree
* fatoptimizer can search for "__fatoptimizer__ = {'enabled': False}"
in the code: if this variable is found, the optimizer is completly
skipped
(2) Petr proposed to extend importlib to pass a code transformer when
importing a module.
importlib.util.import_with_transformer(
'mypackage.specialmodule', MyTransformer())
IMHO this option is too specific: it's restricted to importlib
(py_compile, compileall and interactive interpreter don't have the
feature). I also dislike the API.
(3) Petr also proposed "a special flag in packages":
__transformers_for_submodules__ = [MyTransformer()]
I don't like having to get access to MyTransformer. The PEP 511
mentions an use case where the transformed code is run *without*
registering the transformer. But this issue can easily be fixed by
using the string to identify the transformer in the registery (ex:
"fat") rather than its class.
I'm not sure that putting a flag on the package (package/__init__.py?)
is a good idea. I would prefer to enable language extensions on
individual files to restrict their scope.
(4) Sjoerd Job Postmus proposed something similar but using a comment
and not for packages, but any source file:
#:Transformers modname.TransformerClassName,
modname.OtherTransformerClassName
The problem is that comments are not stored in the AST tree. I would
prefer to use AST to decide if an AST transformer should be used or
not.
Note: I'm not really motived to extend the AST to start to include
comments, or even code formatting (spaces, newlines, etc.).
https://pypi.python.org/pypi/redbaron/ can be used if you want to
transform a .py file without touching the format. But I don't think
that AST must go to this direction. I prefer to keep AST simple.
(5) Nick proposed (indirectly) to use a different filename (don't use
".py") for language extensions.
This option works with my option (2): the context contains the
filename which can be used to decide to enable or not the code
transformer.
I understand that the code transformer must also install an importlib
hook to search for other filenames than only .py files. Am I right?
(6) Nick proposed (indirectly) to use an encoding cookie "which are
visible as a comment in the module header".
Again, I dislike this option because comments are not stored in AST.
Victor
On Thu, Jan 28, 2016 at 3:26 AM, Mirmojtaba Gharibi <
mojtaba.gharibi(a)gmail.com> wrote:
>
>
> On Wed, Jan 27, 2016 at 4:15 PM, Andrew Barnert <abarnert(a)yahoo.com>
> wrote:
>
>> On Jan 27, 2016, at 09:55, Mirmojtaba Gharibi <mojtaba.gharibi(a)gmail.com>
>> wrote:
>>
>> On Wed, Jan 27, 2016 at 2:29 AM, Andrew Barnert <abarnert(a)yahoo.com>
>> wrote:
>>
>>> On Jan 26, 2016, at 22:19, Mirmojtaba Gharibi <mojtaba.gharibi(a)gmail.com>
>>> wrote:
>>>
>>> Yes, I'm aware sequence unpacking.
>>> There is an overlap like you mentioned, but there are things that can't
>>> be done with sequence unpacking, but can be done here.
>>>
>>> For example, let's say you're given two lists that are not necessarily
>>> numbers, so you can't use numpy, but you want to apply some component-wise
>>> operator between each component. This is something you can't do with
>>> sequence unpacking or with numpy.
>>>
>>>
>>> Yes, you can do it with numpy.
>>>
>>> Obviously you don't get the performance benefits when you aren't using
>>> "native" types (like int32) and operations that have vectorizes
>>> implementations (like adding two arrays of int32 or taking the dot product
>>> of float64 matrices), but you do still get the same elementwise operators,
>>> and even a way to apply arbitrary callables over arrays, or even other
>>> collections:
>>>
>>> >>> firsts = ['John', 'Jane']
>>> >>> lasts = ['Smith', 'Doe']
>>> >>> np.vectorize('{1}, {0}'.format)(firsts, lasts)
>>> array(['Smith, John', 'Doe, Jane'], dtype='<U11)
>>>
>>> I think the form I am suggesting is simpler and more readable.
>>
>>
>> But the form you're suggesting doesn't work for vectorizing arbitrary
>> functions, only for operator expressions (including simple function calls,
>> but that doesn't help for more general function calls). The fact that numpy
>> is a little harder to read for cases that your syntax can't handle at all
>> is hardly a strike against numpy.
>>
>
> I don't need to vectorize the functions. It's already being done.
> Consider the ; example below:
> a;b = f(x;y)
> it is equivalent to
> a=f(x)
> b=f(y)
> So in effect, in your terminology, it is already vectorized.
> Similar example only with $:
> a=[0,0,0,0]
> x=[1,2,3,4]
> $a=f($x)
> is equivalent to
> a=[0,0,0,0]
> x=[1,2,3,4]
> for i in range(len(a)):
> ...a[i]=f(x[i])
>
>
>
>>
>> And, as I already explained, for the cases where your form _does_ work,
>> numpy already does it, without all the sigils:
>>
>> c = a + b
>>
>> c = a*a + 2*a*b + b*b
>>
>> c = (a * b).sum()
>>
>> It also works nicely over multiple dimensions. For example, if a and b
>> are both arrays of N 3-vectors instead of just being 3-vectors, you can
>> still elementwise-add them just with +; you can sum all of the results with
>> sum(axis=1); etc. How would you write any of those things with your
>> $-syntax?
>>
>> I'm happy you brought vectorize to my attention though. I think as soon
>> you make the statement just a bit complex, it would become really
>> complicated with vectorize.
>>
>>
>> For example lets say you have
>> x=[1,2,3,4,5,...]
>> y=['A','BB','CCC',...]
>> p=[2,3,4,6,6,...]
>> r=[]*n
>>
>> $r = str(len($y*$p)+$x)
>>
>>
>> As a side note, []*n is always just []. Maybe you meant [None for _ in
>> range(n)] or [None]*n? Also, where does n come from? It doesn't seem to
>> have anything to do with the lengths of x, y, and p. So, what happens if
>> it's shorter than them? Or longer? With numpy, of course, that isn't a
>> problem--there's no magic being attempted on the = operator (which is good,
>> because = isn't an operator in Python, and I'm not sure how you'd even
>> properly define your design, much less implement it); the operators just
>> create arrays of the right length.
>>
>> n I just meant symbolically to be len(x). So please replace n with
> len(x). I didn't mean to confuse you. sorry.
>
>
>> Anyway, that's still mostly just operators. You _could_ wrap up an
>> operator expression in a function to vectorize, but you almost never want
>> to. Just use the operators directly on the arrays.
>>
>> So, let's try a case that has even some minimal amount of logic, where
>> translating to operators would be clumsy at best:
>>
>> @np.vectorize
>> def sillyslice(y, x, p):
>> if x < p: return y[x:p]
>> return y[p:x]
>>
>> r = sillyslice(y, x, p)
>>
>> Being a separate function provides all the usual benefits: sillyslice is
>> reusable, debuggable, unit-testable, usable as a first-class object, etc.
>> But forget that; how would you do this at all with your $-syntax?
>>
>> Since you didn't answer any of my other questions, I'll snip them and
>> repost shorter versions:
>>
>> * what's wrong with using numpy? Nothing. What's wrong even with for loop
>> or assembly for that matter? I didn't argue that it's not possible to
>> achieve these things with assembly.
>> * what's wrong with APL or J or MATLAB? Not sure how relevant it is to
>> our core of conversation. Skipping this.
>> * what's wrong with making the operators elementwise instead of wrapping
>> the objects in some magic thing? The fact that whenever you
>> * what is the type of that magic thing anyway? It has no type. I refer
>> you to my very first email. In that email I exactly explained what it
>> means. It's at best a psuedo macro or something like that. It exactly is
>> equivalent when you write
>>
> a;b=f(x;y)
> to
> a=f(x)
> b=f(y)
>
> In other words, if I could interpret my code before python interpreter
> interpret it, I would convert the first to the latter.
>
>
I think a small part of the confusion is that there are at least four
separate (albeit related) use cases. They all use default arguments
for the current workarounds, but they don't have the same ideal
solution.
(1) Auxiliary variables
def f(x, _len=len): ...
This is often a micro-optimization; the _len keyword really shouldn't
be overridden. Partly because it shouldn't be overridden, having it
in the signature is just ugly.
This could be solved with another separator in the signature, such as
; or a second () or a new keyword ...
def f(x, aux _len=len): ...
def f(x, once _len=len): ...
def f(x; _len=len):...
def f(x)(_len=len): ...
def f(x){_len=len}: ...
But realistically, that _len isn't ugly *just* because it shouldn't be
overridden; it is also inherently ugly. I would prefer that something
like Victor's FAT optimizer just make this idiom obsolete.
(2) immutable bindings
once X
final Y
const Z
This is pretty similar to the auxiliary variables case, except that it
tends to be desired more outside of functions. The immutability can
be worth documenting on its own, but it feels too much like a typing
declaration, which raises questions of "why *this* distinction for
*this* variable?"
So again, I think something like Victor's FAT optimizer (plus comments
when immutability really is important) is a better long-term solution,
but I'm not as sure as I was for case 1.
(3) Persistent storage
def f(x, _cached_results={}): ...
In the past, I've managed to convince myself that it is good to be
able to pass in a different cache ... or to turn the function into a
class, so that I could get to self, or even to attach attributes to
the function itself (so that rebinding the name to another function in
a naive manner would fail, rather than produces bugs). Those
convincings don't stick very well, though.
This was clearly at least one of the motivations of some people who
asked about static variables.
I still think it might be nice to just have a way of easily opening a
new scope ... but then I have to explain why I can't just turn the
function into a class...
So in the end, I suspect this use case is better off ignored, but I am
pretty sure it will lead to some extra confusion if any of the others
are "solved" in a way that doesn't consider it.
(4) Current Value Capture
This is the loop variable case that some have thought was the only
case under consideration.
I don't have anything to add to Andrew Barnert's
https://mail.python.org/pipermail/python-ideas/2016-January/037980.html
but do see Steven D'Aprano's
https://mail.python.org/pipermail/python-ideas/2016-January/038047.html
for gotchas even within this use case.
-jJ
Ben Darnell (Tornado lead) brought up a good use case for allowing
@overload in regular Python files.
There's some discussion (some old, some new) here:
https://github.com/ambv/typehinting/issues/72
I now propose to allow @overload in non-stub (i.e. .py) files, but with the
following rule: a series of @overload-decorated functions must be followed
by an implementation function that's not @overload-decorated. Calling an
@overload-decorated function is still an error (I propose NotImplemented).
Due to the way repeated function definitions with the same name replace
each other, leaving only the last one active, this should work. E.g. for
Tornado's utf8() the full definition would look like this:
@overloaddef utf8(value: None) -> None: ...@overloaddef utf8(value:
bytes) -> bytes: ...@overloaddef utf8(value: str) -> bytes: ... # or
(unicode)->bytes, in PY2def utf8(value):
# Real implementation goes here.
NOTE: If you are trying to understand why we can't use a stub file here or
why we can't solve this with type variables or unions, please read the
issue and comment there if things are not clear. Here on python-ideas I'd
like to focus on seeing whether this amendment is non-controversial (apart
from tea party members who just want to repeal PEP 484 entirely :-).
I know that previously we wanted to come up with a complete solution for
multi-dispatch based on type annotations first, and there are philosophical
problems with using @overload (though it can be made to work using
sys._getframe()). The proposal here is *not* that solution. If you call the
@overload-decorated function, it will raise NotImplemented. (But if you
follow the rule, the @overload-decorated function objects are inaccessible
so this would only happen if you forgot or misspelled the final,
undecorated implementation function).
--
--Guido van Rossum (python.org/~guido)
The official asyncio documentation includes this note:
"""
Note: In this documentation, some methods are documented as
coroutines, even if they are plain Python functions returning a
Future. This is intentional to have a freedom of tweaking the
implementation of these functions in the future. If such a function is
needed to be used in a callback-style code, wrap its result with
ensure_future().
"""
Despite the note, this still causes confusion. See for example
https://mail.python.org/pipermail/python-list/2016-January/702342.html
As of Python 3.5, "awaitable" is a thing, and as of Python 3.5.1,
ensure_future is supposed to accept any awaitable. Would it be better
then to document these methods as returning awaitables rather than as
coroutines?
o/
While translating the Python Documentation in French [1][2], I
discovered that we're not the only country doing it, there is also Japan
[3][4], and Spain [5]. It's possible there's other but I didn't find
them (and it's the problem).
But there's only a few way for users to find the translations (hearing
about them, or explicitly searching for them on a search engine, which
they won't do, obviously expecting a link from the english version if
they exists).
So here is my idea: Why not linking translations from the main
documentation?
I know that's not directly supported by Sphinx doc [6], but separate
sphinx build, blindly (with hardcoded links) linking themselves, may
work (like readthedoc is probably doing). The downside of those links is
that we'll sometime link to untranslated parts, but those parts may be
marked as untranslated [7] to encourage new translators to help.
Thoughts?
[1] http://www.afpy.org/doc/python/3.5/
[2] https://github.com/afpy/python_doc_fr
[3] http://docs.python.jp/3/
[4] https://github.com/python-doc-ja/python-doc-ja
[5] http://docs.python.org.ar/tutorial/3/index.html
[6] https://github.com/sphinx-doc/sphinx/issues/2252
[7] https://github.com/sphinx-doc/sphinx/issues/1246
--
Julien Palard
On 24 January 2016 at 07:12, Stefan Krah <skrah.temporarily(a)gmail.com> wrote:
> Nick Coghlan <ncoghlan@...> writes:
>> You're already going to have to allow this for single lines to handle
>> Py2 compatible annotations, so it seems reasonable to also extend it
>> to handle overloading while you're still figuring out a native syntax
>> for that.
>
> I find that https://pypi.python.org/pypi/multipledispatch looks quite
> nice:
>
>>>> from multipledispatch import dispatch
>>>> @dispatch(int, int)
> ... def add(x, y):
> ... return x + y
> ...
>>>> @dispatch(float, float)
> ... def add(x, y):
> ... return x + y
> ...
>>>> add(1, 2)
> 3
>>>> add(1.0, 2.0)
> 3.0
>>>> add(1.0, 2)
> Traceback (most recent call last):
> File
> [cut because gmane.org is inflexible]
> line 155, in __call__
> func = self._cache[types]
> KeyError: (<class 'float'>, <class 'int'>)
Right, the Blaze folks have been doing some very nice work in that
area. One of the projects building on multipledispatch is the Odo
network of data conversion operations: https://github.com/blaze/odo
They do make the somewhat controversial design decision to make
dispatch operations process global by default [1], rather than scoping
by module. On the other hand, the design also makes it easy to define
your own dispatch namespace, so the default orthogonality with the
module system likely isn't a problem in practice, and the lack of
essential boilerplate does make it very easy to use in contexts like
an IPython notebook.
There is one aspect that still requires runtime stack introspection
[2], and that's getting access to the class scope in order to
implicitly make method dispatch specific to the class defining the
methods. It's the kind of thing that makes me wonder whether we should
be exposing a thread-local variable somewhere with a "class namespace
stack" that made it possible to:
- tell that you're currently running in the context of a class definition
- readily get access to the namespace of the innermost class currently
being defined
Cheers,
Nick.
[1] http://multiple-dispatch.readthedocs.org/en/latest/design.html#namespaces-a…
[2] https://github.com/mrocklin/multipledispatch/blob/master/multipledispatch/c…
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia