Mailman 3 February 2012 - Python-ideas

Re: [Python-ideas] [Python-Dev] matrix operations on dict :)
by Mark Janssen 10 Feb '12

10 Feb '12

On Wed, Feb 8, 2012 at 9:54 AM, julien tayon <julien(a)tayon.net> wrote: > 2012/2/7 Mark Janssen <dreamingforward(a)gmail.com>: > > On Mon, Feb 6, 2012 at 6:12 PM, Steven D'Aprano <steve(a)pearwood.info > > wrote: > > > I have the problem looking for this solution! > > > > The application for this functionality is in coding a fractal graph (or > > "multigraph" in the literature). This is the most powerful structure > that > > Computer Science has ever conceived. If you look at the evolution of > data > > structures in compsci, the fractal graph is the ultimate. From lists to > > trees to graphs to multigraphs. The latter elements can always encompass > > the former with only O(1) extra cost. > > { "a" : 1 } + { "a" : { "b" : 1 } } == KABOOM. This a counter example > proving it does not handle all structures. > > Okay, I guess I did not make myself very clear. What I'm proposing probably will (eventually) require changes to the "object" model of Python and may require (or want) the addition of the "compound" data-type (as in python's predecessor ABC). The symbol that denotes a compound would be the colon (":") and associates a left hand side with right-hand side value, a NAME with a VALUE. A dictionary would (then) be a SET of these. (Voila! things have already gotten simplified.) Eventually, I also think this will seque and integrate nicely into Mark Shannon's "shared-key dict" proposal (PEP 410). The compound data-type would act as the articulation point, around which the recursive, fractal data structure would revolve: much like the decimal point forms in (non-integer) number. (In theory, you could even do a reciprocal or __INVERT__ operation on this type.) OR perhaps a closer comparison is whatever separates the imaginary from the real part in a complex number on the complex plane -- by virtue of such, creates two orthogonal and independent spaces. The same we want to do with this new fractal dictionary type. While in the abstract one might think to allow any arbitrary data-type for right-hand-side values, in PRACTICE, integers are sufficient. The reason is thus. In the fractal data type, you simply need to define the bottom-most and top-most layers of the fractalset abstraction, which is the same as saying the relationship between the *atomic* and the *group* -- everything in-between will be taken care of by the power of the type itself. It makes sense to use a maximally atomic, INTEGER data type (starting with the UNIT 1) for the bottom most level., and a maximally abstract top-most level -- this is simply an abstract grouping type (i.e. a collection). I'm going to suggest a SET is the most abstract (i.e. sufficient) because it does not impose an order and for reasons regarding the CONFLATION rule (RULE1). The CONFLATION rule is thus: items of the same name are combined ({'a':1, 'a':3} ==> {'a':4}, and non-named (atomic) items are summed. To simplify representation, values should be conflated as much as possible, the idea is maximizing reduction. This rule separates a set from a list, because non-unique items will be conflated into one. Such a set or grouping should be looked at as an arbitrary n-dimensional space. An interesting thing to think about is how this space can be mapped into a unique 1-dimensional, ordered list and vice versa. Reflectively, a list can be converted uniquely into this fractal set thusly: All non-integer, non-collection items will be considered NAMES and counted. If an item is another list, it will recurse and create another set. If a set it will simply add it, as is. These rules could be important in object serialization (we'll call this EXPANSION). In any case, for sake of your example. In the above KABOOM example, unnamed, atomic elements can just be considered ANONYMOUS (using None as the key). In this case, the new dict becomes: { "a" : 1 } + { "a" : { "b" : 1 } } ==> { "a" : {None: 1, "b" : 1 } } , OR if have a compound data-type, we can remove the redundant pseudo-name: { "a" : { 1, "b" : 1 } }. Furthermore we can assume a default value of 1 for non-valued "names", so we could express this more simply: { 'a' } + { 'a" : { 'b' } } ==> { ''a': { 1, 'b' } } No ambiguity! as long as we determine a convention. As noted, one element is named, and the other is not. Consider unnamed values within a grouping like a GAS and *named* values as a SOLID. You're adding them into the same room where they can co-exist just fine. No confusion! To clarify the properties of this fractal data type more clearly: there is only 1 key in the the second, inner set ('b'). We can remove the values() method as they will always be the atomic INTEGER type and conflate to a single number. We'll call this other thing, this property "mass"; in this case = 2.) The use of physical analog is helpful and will inform the definition. (Could one represent a python CLASS heirarchy more simply with this fractalset object somehow....?) Further definitions: RULE2: When an atomic is added to a compound, a grouping must be created: 1 + "b" : 1 = { None : 1, "b" : 1 } RULE3: Preserve groupings where present: 'b' : 7 + { 'b' : 1 } = { 'b' : 8 } I think this might be sufficient. Darn, I hope it makes some sense.... mark

3 3

Re: [Python-ideas] Python 3000 TIOBE -3% (Massimo Di Pierro)
by Daniel Greenfeld 10 Feb '12

10 Feb '12

> Message: 1 > Date: Thu, 9 Feb 2012 15:41:22 -0600 > From: Massimo Di Pierro <massimo.dipierro(a)gmail.com> > To: Nick Coghlan <ncoghlan(a)gmail.com> > Cc: python-ideas <python-ideas(a)python.org> > Subject: Re: [Python-ideas] Python 3000 TIOBE -3% > Message-ID: <90C8316C-1AB0-4759-B3DF-0FB07477FF08(a)gmail.com> > Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes > > First of all all the Python developers are doing an amazing job, and > none of the comments should be taken as a critique but only as a > suggestion. I completely agree with Massimo again. :-) > > On Feb 9, 2012, at 3:34 PM, Nick Coghlan wrote: > [...] >> In the meantime, if the python.org packages for Windows aren't up to >> scratch (and they aren't in many ways), *use the commercially backed >> ones* (or one of the other sumo distributions that are out there). >> Don't tell your students to grab the raw installers directly from >> python.org, redirect them to the free rebuilds from ActiveState or >> Enthought, or go all out and get them to install something like >> Python(X, Y). > > This is what I do now. I tell my students if they have trouble to > Enthought. Yet there are issues with license and 32 (free) vs 64 bits > (not free). Long term I do not think this what we should encourage. I think it is odd to encourage users to go to use open source distros, but if they have installation problems (which is really common - Massimo/Titus/Audrey/Zed/etc seem to back me up here) to recommend 'somewhere' to go to commercial-but-free distros. If we should be pointing new users to ActiveState or Enthought, maybe we should just change the python.org default installers to what they provide. Tell you what, I'll take this matter off-list and bring it up with Jesse Noller and the rest of the board working on the python.org RFP. -- 'Knowledge is Power' Daniel Greenfeld http://pydanny.blogspot.com

4 6

Optional key to `bisect`'s functions?
by Masklinn 10 Feb '12

10 Feb '12

The ``bisect`` stuff is pretty neat, although probably underused (especially the insorts), but their usefulness is limited by the requirement that the lists directly contain sortable items, as opposed to ``sorted`` or ``list.sort``. It's possible to "use" them by copy/pasting the (Python) functions into the project/library code and adding either a custom key directly or a key function, but while this can still yield an order-of-magnitude speed gain over post-sorting sequences, it's cumbersome and it loses the advantage of _bisect's accelerators. Therefore, I believe it would be pretty neat to add an optional ``key=`` keyword (only?) argument, with the same semantics as in ``sorted``. It would make ``bisect`` much easier to use especially in stead of append + sorted combinations. The key should work for both insertion functions and bisection search ones. Thoughts?

7 18

Re: [Python-ideas] Python 3000 TIOBE -3%
by Sturla Molden 10 Feb '12

10 Feb '12

On 09.02.2012 20:34, Matt Joiner wrote: > Linux user here. I'm not sure that IOCP solve the I/O concurrency > issue anyway, it's just as convoluted as polling from memory. On Linux processes are so light-weight that you can fork (os.fork) instead of spawning threads. Threads are typically needed for Java, Solaris and Windows, where forking is either slow or not possible. But if you need really scalable I/O on Linux, consider select/poll or epoll. And on FreeBSD and Mac there is kqueue. Sturla

1 0

Re: [Python-ideas] Python-ideas Digest, Vol 63, Issue 23
by Daniel Greenfeld 10 Feb '12

10 Feb '12

> 1. Re: Python 3000 TIOBE -3% (Massimo Di Pierro) > 2. Re: Python 3000 TIOBE -3% (Guido van Rossum) > 3. Re: Python 3000 TIOBE -3% (Sturla Molden) > 4. Re: Python 3000 TIOBE -3% (Guido van Rossum) > Date: Thu, 9 Feb 2012 12:25:18 -0600 > From: Massimo Di Pierro <massimo.dipierro(a)gmail.com> > To: Steven D'Aprano <steve(a)pearwood.info> > Cc: python-ideas <python-ideas(a)python.org> >> Massimo Di Pierro wrote: >>> Here is another data point: >>> http://redmonk.com/sogrady/2012/02/08/language-rankings-2-2012/ >>> Unfortunately the TIOBE index does matter. I can speak for python >>> in education and trends I seen. >>> Python is and remains the easiest language to teach but it is no >>> longer true that getting Python to run is easer than alternatives >>> (not for the average undergrad student). >> >> Is that a commentary on Python, or the average undergrad student? > > I teach so the average student is my benchmark. Please do not > misunderstand. While some may be lazy, but the average CS undergrad is > not stupid but quite intelligent. They just do not like wasting time > with setups and I sympathize with that. Batteries included is the > Python motto. I'm going to delurk from this list and really back up Massimo here. It's not precisely his issue, but it's close enough to count. While we love our Linux and BSD variants, and OS X usage is growing, the truth of the matter is that the clear majority of people learning Python at the entry level do so on Windows. And I can assure you having attended many of the tutorials given by PyLadies and other groups, the part that took the most amount of time was ensuring a correct installation on Windows. It's not just a matter of getting the installation onto the machine, it's a matter of making sure the paths are set correctly so they can follow code examples trivially. In fact, at PyLadies tutorial events they would give literally special party hats to teachers who could get Python running under ideal conditions under Windows. And still it ate a lot of time and caused frustration. Frustration that gets shared with management and other people. I'm well aware that this matter of installation has been 'addressed'. There is a complex PEP to handle different version installs on Windows. I can go and click on a small link on the home page of python.org and download one-click installer that DOESN'T set up Windows paths. I can follow instructions 'somewhere' and get the paths set up, but I shouldn't have too. Students should be able to one-click Python and have it just work. Yet, for all the times I've been told it's fixed or we've complained about it and been told "It's getting fixed!", it is still an ongoing problem. If I were a Windows developer I would fix it today. So perhaps this should become a GSOC project of high priority: One click install of Python on Windows on a version hosted by python.org. Note: People suggest virtual machines or Vagrant. This works on new machines, but you try getting any of that working on an old Windows machine in a room of 40-100 students waiting on installation. Providing laptops for the event are also completely out of budget for most of these events. In order to make this issue as clear as possible, I'm going to quote Audrey Roy: "The number one thing that Python educators struggle with on entry level tutorials is Windows installations of Python. Ask me, ask Zed Shaw, ask any of the PyLadies." -- 'Knowledge is Power' Daniel Greenfeld http://pydanny.blogspot.com

2 1

matrix operations on dict :)
by julien tayon 09 Feb '12

09 Feb '12

Hello, Proposing vector operations on dict, and acknowledging there was an homeomorphism from rooted n-ary trees to dict, was inducing the possibility of making matrix of dict / trees. Since, linear algebrae on dict was coldly welcomed, I waited to have some code to back me up to push my reasoning furhter, and it happily worked the way books predicted. This was the reasoning : - dict <=> vector - Vectors + linear algebrae <=> matrix - Most of Rooted Trees <=> dict( dict( ... ) ) ** - Matrix * Vector = Vector2 <=> Matrix * tree1 = Tree2 ** see here for a coded explanation http://readthedocs.org/docs/vectordict/en/latest/intro.html#homeomorphism-b… for a sample of API, code, and result see here. http://readthedocs.org/docs/vectordict/en/latest/matrix.html#api dict of dict might not be the best way to make trees but, having matrix operations on dict is being able to transform trees in trees natively. The module is still quite a proof of concept, and it is not the implementation I advocate, but rather the idea. Because : isn't transforming trees into trees quite a recurrent task in modern Computer Science with key value database ? Plus matrix * tree being side effect free, it is a good candidate for a canonical way to tranform tree in a parallelisable way. And by the way I implemented matrix as vectordict so ... we have matrix operations on matrix. ^_^ (Brace thourself, InceptionMatrix are coming) For the «not faint of heart» that are able to read un Perlish un PEP8 code : http://pypi.python.org/pypi/VectorDict/0.3.0 my 2 euro cents (which of course worth more than 2 US cents <:o) ), Cheers, PS : I am not sure that using defaultdict as a backend was the best idea of the century, but keys appearing in a dict after an addition where not very much in my idea of how a normal python dict should behave. PPS : I will -if I still have time- code sets operations on dict, issubset, diff, union, intersection. These are quite easy, but so unfun. Since I am not very gifted at explaining, I prefer to code and show the result later. -- Julien Tayon

9 17

Re: [Python-ideas] [Python-Dev] matrix operations on dict :)
by Mark Janssen 09 Feb '12

09 Feb '12

On Wed, Feb 8, 2012 at 9:54 AM, julien tayon <julien(a)tayon.net> wrote: > 2012/2/7 Mark Janssen <dreamingforward(a)gmail.com>: > > On Mon, Feb 6, 2012 at 6:12 PM, Steven D'Aprano <steve(a)pearwood.info > > wrote: > > > > I have the problem looking for this solution! > > > { "a" : 1 } + { "a" : { "b" : 1 } } == KABOOM. This a counter example > proving it does not handle all structures. > > Ah, but I already anticipated this. One just has to decide the relationship between the *group* and the *atomic*. (These are key words that you can find out about at pangaia.sf.net "grouping model"). Admittedly, this might be arbitrary, but once decided you get the full power of the recursive data structure. It's kind of like defining the base case of factorial. The math (in my world) simply decided that factorial(0)=1 as the convention of "an empty product" (Wikipedia::Factorial). But, in theory, it should work and provide considerable power. Since it's all arbitrary one shouldn't get hung up too much on which convention is adopted, even though it will have to be followed thereafter. But "practice beats purity", as they say... :) mark

2 2

Add a recursive function to the glob package
by Yuval Greenfield 08 Feb '12

08 Feb '12

Many times I've wanted glob to give me all the "*.zip" or "*.py" or "*.h" files in a directory *and subdirectories* ever since I started using python 7 years ago. I don't know if I'm the only one or not but here's a patch: http://bugs.python.org/issue13968 I'd love to hear feedback on the notion and implementation, Yuval Greenfield

2 1

A key parameter for heapq.merge
by Simon Sapin 07 Feb '12

07 Feb '12

Hi, According to its own documentation, the merge() function in the heapq module is similar to sorted(). However, it has none of the key and reverse parameters that sorted() has. I think they could be just as useful as in sorted(). http://docs.python.org/dev/library/heapq.html#heapq.merge http://docs.python.org/dev/library/functions.html#sorted First of all what do you think of the idea? I’m working on a patch for a key parameter. I think it can be pretty straightforward, but I’ll measure if the "no key" case becomes slower (calls to lambda x: x) At worst we can always duplicate the loop. However, I am not sure how to implement reverse. Not all values have an "opposite value" that reverses order, _nsmallest and _nlargest are quite different, and merge uses neither. Anyway, if I get this to work, it will be my first contribution to CPython. I’m trying to follow the Developer’s guide, but is there something else I should be aware of? http://docs.python.org/devguide/ Regards, -- Simon Sapin

4 6

unpacking context managers in WITH statement
by Yury Selivanov 07 Feb '12

07 Feb '12

Hello, With the removal of "contextlib.nested" in python 3.2 nothing was introduced to replace it. However, I found it pretty useful, despite the fact that it had its own quirks. These quirks can (at least partially) be addressed by allowing unpacking syntax in the context manager. Consider the following snipped of code: ctxs = () if args.profile: ctxs += (ApplicationProfilerContext(),) if args.logging: ctxs += (ApplicationLoggingContext(),) with *ctxs: Application.run() As of now, without "nested" we have either option of reimplementing it, or to write lots of ugly code with nested 'try..except's. So the feature was taken out, but nothing replaced it. What do you think guys? Thanks, Yury

6 13