Hi all, Next #pypy-sync meeting on freenode.net is tomorrow Thursday at 1pm as usual. Regular topics -------------- * activity reports (3 prepared lines of info) * resolve conflicts/blockers Here is my draft proposal of Topics of the week ------------------ A quick point about IRC logging I (Chris) have set up an eggdrop server on tismerysoft.de. This server logs #pypy, #pypy-funding (password), #pypy-sync, #pypy-tb. Let me know tomorrow if it should log something else. To make this service more reliable, I would like to add a second server. I could use another one of mine, but this makes no sense in using the same network. I'd like to ask which other server to use, maybe codespeak, maybe Armin's new server. Anyway, I'd like to take responsibility and the rights to set that up, since I did it before. Participation on the CCC congress in Berlin There is a CCC congress in Berlin 2005–12–27 to 2005–12–30. See http://wiki.chaostreff.ch/index.php/Main_Page They told us on IRC that they are interested in talks from us. This appeared to me to be quite unspecific, they seem to be interested in scientific talks. For PyPy, I anyway thing that it would be good not to emphasize too much on Python itself, but the general aspectsof translation which are possible to tackle with out framework. Se also Armin's draft in pypy/doc. We should collect who wants to give a talk or just attend this congress. We also might talk about how we manage claims about giving a talk. Practice has been to say it somewhere at some time often enough, but I'm not sure if this is a pleasant strategy for everybody to continue with. I would like to ask people who would like to attend in any way to inform me early, because I probably can provide help in finding locals who can share a room. Current optimization activities and allocation of work There are currently several ongoings on optimization, where I can't remember that they were assigned officially in any way. Practice seems to be to justpick an issue which has a good chance to provide a 200 % speedup and then go for it. I am personally not really happy with this and would like to ask about opinions. My proposal is to have a more informal way to decide about activities, by issuing a short IRC meeting and giving everybody a chance to ask for participation. We should also be open to discuss reasons for an individual's rejection. This might be painful, but setting facts by just doing things is a worse practice IMHO. Other activities, follow-up on sync meeting of last week As a conclusion from last week's meeting, docs and reports appear to reduce to a meeting of Holger, Carl, Samuele and Armin one day before the Paris sprint. I would like to ask again, if this was meant this way. There is quite some time left until Paris. Can we talk again about what we want to work on in the meantime, if it is not just optimization? -- The next topic is related, maybe this is a merger. Making money out of PyPy Jacob asked me on the last sprint, if it is possible to make money out of PyPy right now. I passed this around once, but would like to insist. I am personally interested to make PyPy move away a bit from its purely academical status and to think a little bit about how we can create practical applications in the near future, which allow to grow marketable services for our members, in order to reduce PyPy's dependency from the EU sponsoring. Please propose additions and changes, I just listed what immediately came to my mind. I'm also happy if you say some topics are too big and should be moved to another meeting. See you tomorrow at 1 pm. sincerely -- chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On Thu, 15 Sep 2005, Christian Tismer wrote:
Hey all, I've only been reading these updates, and I haven't actually played with PyPy in while, so I could be way off here. But I do run a business and know a little bit about marketing, and I head up another python compiler project (pirate) so I know a little bit about the market demand for these things... :) Basically, there are really that many applications of this technology in its current state. The stated goal for this project is just to make a faster version of python, but can you really charge for that given that pypy is already open source? One option could be to find people who need python to run fast and get them to pay you to handle the cases that apply to *their project* up front, but I really don't think this is a viable market. It's much cheaper for the client to just write a C extension or port the whole thing to C. Another option is that you could use your technology created in pypy to create new *frontends* for pypy, and sell the service of taking legacy code in other languages, and translating that into call trees, and turning it magically into python or lisp or c... This seems like it could be a much more lucrative service, but it's a whole new research project and it takes you away from your core focus on python. (On the other hand, a company that did this would have a massive incentive to sponsor pypy development... Maybe there's a company that already does this kind of thing with some other technology... Like, say, people just rewriting things by hand? Another idea has to do with the sprints. I already told Holger this a while back, but I think you guys have one of the most impressive project management styles around, and the sprint idea seems like a great amount of fun. What if you capitalized on *that*? A week in europe to work on an open source project? That would be an awesome vacation! And to get to learn about compilers and python along the way? What I'm saying is you could market the sprints as a sort of training package for big companies interested in trying out python. They send their developers and pay you guys to train them. Even if the companies don't care about python, their developers *might* and that could be a great reward. Wasn't there a post just like this from someone in the army a while back? Market demand. :) Heck, do your next sprint as a partnership with these guys: http://www.geekcruises.com/ Anyway, it might be a crazy idea, but if you could get it working, it would capitalize on what you're already doing rather than force you to come up with some other devlepoment project off to the side... Sincerely, Michal J Wallace Sabren Enterprises, Inc. ------------------------------------- contact: michal@sabren.com hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ -------------------------------------
Hi Michal, On Wed, Sep 14, 2005 at 22:17 -0400, Michal Wallace wrote:
PyPy is not primarily - at least not only - about making Python faster although many people like to view it this way :-) PyPy is a lot about flexibility to produce custom implementations of Python for use in particular environments. Regarding persistence, security (e.g. sandboxing) and deployment there is a lot that we can possiblly do with PyPy implementations. I am sure that such possibilities will become clearer next year.
A build tool is part of the current EU project and of our plans.
We are doing this regularly :-)
:-)
There are many possibilities but only so much time and focus to pursue them. Bea also reported quite some interest from other projects and entities in the EU context. We got invitations to Afrika and other places and i guess we pursue them :-) Thanks for your kind advice, holger
holger krekel wrote: [snip]
PyPy is not primarily - at least not only - about making Python faster although many people like to view it this way :-)
This correction is a bit confusing... It is not surprising that people view "a faster Python" as a primary goal of your project, as that's the message that's been spread repeatedly by you all over the years: "The PyPy project aims at producing a flexible and fast Python implementation. The guiding idea is to translate a Python-level description of the Python language itself to lower level languages. Rumors have it that the secret goal is being faster-than-C which is nonsense, isn't it?" and "In the next step of the project, we will generate C code or machine code from the source of Pypy, thereby reducing the speed penalty. Later in the project, we will introduce optimisation steps that should make Pypy run faster than CPython." and "Psyco is a Python extension module which can massively speed up the execution of any Python code. ... The future of Psyco now lies in the PyPy project, which according to plan will provide a good base for a Python interpreter with better and well-integrated Psyco-like techniques as soon as 2006." I certainly hope that your "not primarily - at least not only" doesn't indicate that the goal of making Python faster is slowly becoming less important to you all, as I'd very much like a faster Python, and more flexibility of implementation is, while of tremendous personal interest to me, not one of much business interest in the foreseeable future, which means it's unlikely I'll get to play with PyPy much unless it offers performance benefits. I realize that of course that flexibility has always been a primary goal of PyPy as well; obviously speed isn't the *only* goal of the project, but the flexibility goal now dominates and a faster Python is indeed not of primary importance to the project, I would suggest you revise the previous statements, which make it rather likely people will get this wrong impression. Regards, Martijn
torsdagen den 15 september 2005 12.42 skrev Martijn Faassen:
I think you misunderstand Holger. Making Python faster is _a_ goal with Pypy, but it is not _the_ goal. The most important goals, as I see it are to produce something that is flexible and which has a clear implementation. However, this becomes an academic exercise if we produce something that is slower than competing implementations of Python. We want to build something that produces actual benefits in real world applications. Jacob
Jacob Hallén wrote:
That's the point. Not prioritizing speed is fine, but it does not allow us to be remarkably slower. Being of competitive speed and then more flexible is a must. If we are slower, the market will not consider us at all. ciao - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Christian Tismer wrote: [snip]
That's true, and that's a useful clarification. It's still different from the impression you're giving out, where you're strongly implying at least to me that speed will be significantly *faster* eventually with PyPy. Then again, it's forgivable that some hype is used to attract interest to the project. Regards, Martijn
Martijn Faassen wrote:
I thik I'm the worst speed junkie in the project, and I'm holding my breath a little until I'm convinced that we can be fast. ATM I'm quite vague about that, because of the complexity of crucial parts of our system, and the fact that I don't understand how we are going to translate that into speedy code without very high level optimization techniques. I simply can't tell whether we can reach this without major hand-rewriting or whether we can gather the missing speed factor by applying JIT technology. I always thought that the initial big step of creating an initial binary of reasonable speed would be an easy, first step. But it turned out to be a huge step which is by far not in the shape it should be. This amount of my misestimation makes me nervous and stops me from praising PyPy's speed all too loudly. On the other hand, we have extremely efficient results on the low level side. RPython programs have been shown to become possibly very fast. This is an area where I have no doubt making promises, and that's why I proposed to put some more effort there and at least provide a tool to produce fast extension modules from RPython, soon. This is also the reason why I want to make RPython more complete and easier to use. For sure we never have put any hype onto PyPy just to make it more attractive. We learned that flexibility and speed are not trivial to marry, and we have proven our claims about flexibility, but unfortunately not yet on speed. I would really hate to retract this claim. ATM, there is simply no prognosis possible. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Hello Christian and pypy team ;-) At 17:16 2005-09-15 +0200, you (Christian Tismer) wrote:
Well, keep the faith ;-) A couple of comments: 1) ISTM that "speed" needs to be split at least into speed of translation and speed of the resulting code/interpreter pair, though interactively one experiences the combined effect. (BTW, by that '/' I mean to highlight the fact that output code could be raw machine code packaged for OS linking/loading and where the "interpreter" is a CPU, vs. byte code for a VM such as found in cpython or jython etc. Also it could be pointed out that the target pair could eventually be on another platform entirely, such as a cell phone or game device, or I suppose a vending machine control system etc.) 1a) Re speed of translation: I note that translation has multiple phases/passes, so I wonder if this would be amenable to pipelining, so as to be able to take automatic advantage of multiple processors, either in the same box, or cooperating via fast network links if there were command line options for such delegation? Could one pickle state and pass it on to the next phase? Could you factor the passes as separate programs to connect in a unix-os-level pipeline? In which case I guess parallel make utilities could possibly come into play. Just thoughts... 1b) (1a) Brings the thought that the same interpreter will have a lot of unused baggage for any particular use, if it is a monolith. Which again brings up the translation-time/platform vs run-time/platform distinctions. I.e. space may be as important as speed in the target run-time environment, but I may be willing to crunch overnight translating on a cluster to make something that will fit and run acceptably on the target platform. BTW, is the issue of ROM-ability entirely low-level, or is it some kind of concern higher up? 1c) Will the different components of an interpreter be directable to dlls/sos and be able to load lazily/be-excluded-if-not-needed/whatever e.g. for good interactive startup, but also be configurable to load monolithically? IWT this is make-file stuff, but a makefile can't select pieces if they were welded together by the way they were developed, so I am wondering how that aspect is happening. 2) ISTM that you guys will be the first to notice what meta-info will be useful to pass into the translator(s) to achieve various results, so I wonder what analogies to e.g., C's "volatile" declaration you may want, and what vehicle you will want for representing and transmitting such information. Pragmas? more -*- cookies -*- ? If you come up with something really good, isn't it likely regular python will adopt it, as well as vice versa? The same goes for language syntax changes that don't hide behind a '#' -- IOW, will you guys wind up with some recommendations for language changes? I don't think you should be burdened with writing PEPs and haggling about syntactic sugar, but IWT you guys will discover optimization obstacles that no one else is in a position to see (or maybe even comprehend ;-), but that you could communicate by saying, "If we had a way to say this about such-and-so in python, we could optimize such-and-so much better" and that could help the language evolve in a win-win way. Is there a way to get a peek into your ongoing thoughts without asking you to hang all your unfinished paintings in the window when you go home from the studio each night? 2a) ISTM there will need to be module-specific meta-info, not just -O or -OO etc command line options. And since module translation occurs primarily at import time, there is the opportunity to pass dynamically determined meta-info to the import process as well, not just letting it see meta-info embedded in the source file or refer to interpreter startup command line parameter info. (I guess you could also package meta-info in various config files, or consolidate such info in one file for members of a package, etc. but I wonder if it's not dangerous to go further down the path of tangling abstract hierarchies with platform specific file system properties). So I'm wondering, if you see useful mods/variations to the semantics of import, or other python aspects, how would such ideas get back to python, so the appropriate language PEPs can get under way (though not prematurely) ? 3) I'm wondering how much of pypy translation technology could be factored out to apply to any language translation, and whether there could be a family of porting tool spinoffs eventually. I guess I rambled, as usual ;-) Anyhow, I am sure EU pypy funding is well spent. I hope they will renew their commitment when the time comes. Best wishes to you all. Regards, Bengt Richter
Hi Bengt, On Thu, Sep 15, 2005 at 06:46:30PM -0700, Bengt Richter wrote:
I don't think either of this is likely. We are not suggesting language changes or seeing the need for some. And for RPython we can mostly deal with regular Python syntax with a couple of additional specific objects that don't require any kind of syntax; for example, as in C there is sometimes a good reason to use signed or unsigned integers but Python doesn't have the distinction, we have defined a class "r_uint" which works like an unsigned integer with the same wrap-around semantics as C. The translator knows that it can be turned into a simple "unsigned int" in C. There is no need for more advanced support. In general we try hard to keep language changes separated from PyPy, with the exception of new built-in types providing extra features that are not really possible in CPython, just like Stackless did -- e.g. the thunk object space.
A lot of it. That's part of our goal: making it applicable to different languages. If anyone shows up with a Javascript parser, I'm sure we could together hack (and translate (and later have a JIT from)) a basic Javascript interpreter within a few weeks :-) A bientot, Armin.
--- Armin Rigo <arigo@tunes.org> wrote: ...
Actually, I think the currently useful thing might be a way to compile (any decent language, Python for choice) INTO Javascript (with reasonable efficiency), so one could write AJAX pages without actually having to code in Javascript...;-) Alex
Hi Alex! On Sun, Sep 25, 2005 at 18:42 -0700, Alex Martelli wrote:
that's a second possibility but there are two crucial perequesites regarding current PyPy: - a translation approach that focuses more on high-level languages (instead of the current RTyper "C" one) - a browser/DOM simulation layer in Python to allow rapid prototyping. I believe the idea with a PyPy/JS Interpreter is more interesting, at least from the PyPy perspective. Btw, being faster than current JS interpreters is probably easier than being faster than CPython :-) Such a JS-interpreter implementation could be integrated with browsers allowing JS and Python directly in the browser. That being said, it might actually be worthwhile to compile RPython to Javascript which should be reasonably fast for small programs. One possibility is to just create flowgraphs from a python program and then transform the flowgraph to javascript source code (without much annotation). I guess only real experiments would shed more light. cheers, holger
--- holger krekel <hpk@trillke.net> wrote:
Yep, the former would surely require lots of work (though maybe not the latter).
I believe the idea with a PyPy/JS Interpreter is more interesting, at least from the PyPy perspective. Btw,
Possibly, but I advisedly used the adjective "useful" rather than saying "interesting";-) JS's only advantage is one of *deployment* -- browsers have JS interpreters, so you can write AJAX code and dramatically improve the user experience. "Sweating blood" is a charitable description of what it takes, but several examples prove it can work in RL. I just find myself daydreaming that I could do the coding in Python (or almost any other VHLL) rather than JS, that's all;-)
being faster than current JS interpreters is probably easier than being faster than CPython :-)
Yes, good point.
Any Python implementation might be integrated with browsers, but it just seems to take a long time happening (even for Firefox, which should allow it most easily)...;-)
True, annotation wouldn't help much here (translating among VHLLs). Alex
On Sep 26, 2005, at 10:49 AM, Alex Martelli wrote:
Using MochiKit at a target on top of JS would make it even easier, since comparators, iterators, and repr are already there with Python semantics (more or less). You'll probably have to throw out multiple inheritance, anything that depends on id() or hash(), and some other features JS can't really do. -bob
On 26 Sep 2005 at 8:56, holger krekel wrote:
Hello everyone. Python and prototypes is one area where I have some personal experience. How well RPython will translate to Javascript depends on how fully RPython supports new-style classes. Javascript objects have single inheritance only. So problems arise immediately for multiple inheritance. An object space is needed to implement new-style classes on top of Javascript prototypes. All python arithmetic operations, attribute accesses and method calls become Javascript function calls. This adds a level of indirection to an already slow interpreter. Finally Javascript has no 'goto' statement. So a switch statement within a loop is needed to handle the spaghetti code produced by tranforming a flowgraph. How much different is this from an interpreter loop? So a Python program compiled to Javascript is essentially a Python interpreter that runs in Javascript. How worthwhile is this? Of course if RPython only supports single inheritance and improvements to flowgraph transformation will recover Python's statement structure then a closer translation may be possible. Lenard Lindstrom <len-l@telus.net>
Alex Martelli wrote:
I took a stab at this once based on somebody else's code... http://davidf.sjsoft.com/files/py2js I found some Python constructs ended up being tricky to translate to JavaScript (as the rest of the thread said). So the idea here was to be able to start with some Python code, get it into basic JavaScript syntax, but then you'd probably have to munge it anyway Cheers David
On Thursday 29 September 2005 12:13, David Fraser wrote:
I've just used your py2js to start playing around with python javascript integration on the web application server level. For people who have too much time on their hand: http://www.sdiehl.de/python required to run: python2.4 and cherrypy2.1 If these are installed, the example should run out of the box. To give some not so offtopic comment: I had this idea as well that a (fast) pypy could be the ideal base for web browser development. Since code written in python is so much easier to understand than, say, c++, it would be so much easier to get css standard conformance right, or Javascript implementation, or whatever. This could lead to much faster development cycles. But, of course, this would depend on pypy running not only as fast as CPython but considerably faster. --- Stephan
Jacob Hallén wrote:
I'm not sure how to understand Holger. He said PyPy is "not primarily - at least not only" about making a faster Python, and was in the context of correcting someone. I figured that perhaps the PyPy project is not bringing the message out as well as possible, whatever this message is.
So, is making a faster Python a *primary* goal of PyPy or not? I understand that the way to go about this is to produce a clear, flexible implementation, and that this is also cool, and also a primary goal by itself, but is making a faster Python *also* a primary goal or not? If it's a primary goal, why the correction? If not, the message you all are putting across might need some clarification, as you're certainly implying to me in the texts I quoted that it is a primary goal. Regards, Martijn
Hi Martijn, since I am in a very nitpicking mode... Martijn Faassen wrote:
Note that it says "fexible and fast": fexible comes first. And we explicitely say that the faster-than-C goal is nonsense.
"reducing the speed penalty" does not imply being very fast. It just means being not so slow.
"a good base" does not necessarily mean as fast as Psyco is now.
Holger's statement is very important. At the moment most people have the notion that PyPy is mainly about speed. But the really exciting fact about PyPy is its flexibility, therefore it makes sense to advertise the flexibility goals a bit (even if that means diminishing the speed goal in one single mail to pypy-dev). Of course, achieving high speed is a worthwhile goal but this goal can be reached _because of_ the flexibility of PyPy (not the other way round). :-) Cheers, Carl Friedrich
Hey, Carl Friedrich Bolz wrote:
since I am in a very nitpicking mode...
I'm nitpicking too, but even in the first announcement of PyPy, before it even had this name, the claim that the thing was going to be faster than CPython was made: As Armin Rigo of PSYCO fame takes part in the effort, we are confident that MinimalPython will eventually run faster than today's CPython.
Holger's statement is very important. At the moment most people have the notion that PyPy is mainly about speed.
What I am saying is that it's due to the PyPy project's communications that this notion has spread. To correct someone who picked up on this notion is therefore a bit confusing.
I know that, and I'll stop nitpicking now. I just thought it was a bit weird to start correcting an impression in people that seems a legitimate interpretation of previous communication of this project. Then again, an emphasis on flexibility is justified. What is exciting about PyPy seems to depend on ones perspective too. To the PyPy hackers themselves, the coolest thing is the flexibility allowing all sorts of fun experimentations with syntaxes and semantics and ways Python code is executed. To the wider world outside Python language hackers however the *results* of this flexibility of PyPy is what is truly interesting. The flexibility by itself is not interesting at all from that perspective. Since the topic of this thread was making money out of PyPy, this is a relevant thing to talk about. For non Python core hackers, the biggest potential gains in PyPy that I heard about so far would be: * increased performance * making Python better in an increasingly concurrent world (threading/GIL/stackless, etc). That ties into increased performance too, in multi processing environments. Regards, Martijn
Hi there Just a short comment: The fact that we have different perspectives and views on what has been communicated about the project goal indicates to me that there is/or was an unclear communication about this ;-) Thanks for raising attention to this Martijn. Cheers Bea On Thu, 15 Sep 2005, Martijn Faassen wrote:
Beatrice Düring Change Maker Tel: 031- 7750940 Järntorget 3 Mobil: 0734- 22 89 06 413 04 Göteborg E-post: bea@changemaker.nu www.changemaker.nu "Alla dessa måsten och alldaglighet. Allt detta som binder vår verklighet i bojor av skam och rep utav tvång. Alla dessa kedjor som binder vår sång. Jag skall slita dem alla i små, små stycken och möjligtvis av resterna göra mig smycken." - hemlig
Carl Friedrich Bolz wrote:
...
I don't think that order is relevant in a mathematical "and". For me it means "flexible without being slow" and "fast without being inflexible". This is the matter of optimization. We have seen examples of both. Python is a quite fast, inflexible implementation. PyPy is a very flexible but yet slow implementation. The proof of concept, that we can have both at the same time, is still not there. And I don't support the nonsense part at all. Psyco has proven that it can make CPython much faster under circumstances. The same factor can be at least expected for PyPy. But if PyPy does not jump over its current factor 40-50 hurdle, this acceleration might still not outperform CPython. ...
The impression of mostly being about speed might also come from the fact that speed is the thing that PyPy lacks at most, hurtingly. And it is clear if a project is started in the most flexible way possible from the ground, with no consideration of speed at all. We will need to find yet another stone of the wise to morph the code reasonably. As said, I very much hope it, while there is currently no indicator that we are close to it. We have to prove that last point about flexibility. The gap lies in the needed transformations, which need to be invented. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On Thu, 15 Sep 2005, holger krekel wrote:
On Wed, Sep 14, 2005 at 22:17 -0400, Michal Wallace wrote:
Hi Holger, What I said is that the stated goal is to make a faster version of python. I actually looked it up before I wrote that: """ The PyPy project aims at producing a flexible and fast Python implementation. The guiding idea is to translate a Python-level description of the Python language itself to lower level languages. Rumors have it that the secret goal is being faster-than-C which is nonsense, isn't it? more... """ -- http://codespeak.net/pypy/dist/pypy/doc/news.html As Martijn Faassen said, it's not surprising that people have the impression that pypy is all about speed. I think part of the problem is that people understand the word "fast". What that means is obvious. It's like python, only it runs faster! "Flexible" doesn't mean anything to me. Python is already flexible, so what does that mean? Well, you've got these different backends. Maybe that's what it means. And the people who made stackless and psyco and green threads are on the team, so maybe it's about those things. Who knows? The site doesn't say so if we want to figure out the status of those things, we have to think about it.
PyPy is a lot about flexibility to produce custom implementations of Python for use in particular environments. Regarding persistence,
Persistence. What does that mean? Is that like Squeak, where you can save the state of the whole interpreter as a single image? If so, that's pretty neat! It ought to be right up front on the site.
security (e.g. sandboxing)
Also neat. So another benefit is that you can make programs that run other programs safely? I know some guys from twisted like that stuff, because they want to make games where people can run their own scripts.
and deployment
I don't know what this means at all.
there is a lot that we can possiblly do with PyPy implementations. I am sure that such possibilities will become clearer next year.
Cool. I guess I was thinking more about what you could do to make money *now*, given the current state of the codebase... Sounds like one big part of that would be telling people more about the potential so they can say "I want that! I'll pay to make it happen!" :) Another thing I suspect is part of pypy but isn't really right up front is the whole continuations thing. Yes, CPS dos make an appearance int he top news article on the news page, but that'll just get buried as you post more news. Basically "flexible" is a feature, and not a very well defined one. Features don't sell well. What sells are benefits, and "fast" is a benefit. So if it's not the primary benefit, it's time to fix your message. :)
I don't understand. What I'm saying is that since you can translate python->lisp or python->python, then maybe you can use the same technology to translate cobol->python or cobol->java or whatever using the cool internal technology that pypy has. This would be a major research project and well outside the bounds of the pypy project. However, if I recall, there are already companies that do this kind of thing and pypy's internals might be of use to them and their clients. That's all I was saying. I'm not sure what build tools has to do with that. ;)
You have sprints regularly, but are you marketing them to companies as a reward or training for top employees? Last I heard, at least some of the sprints were limited to experienced PyPy developers, right? Obviously you don't want 500 newbies running around, but... <shrug>
Well, now that I hear the talk that there are other benefits than speed, I'd say you'd have a much better chance of making money if you capitalized on that. Also, all of this is about the top line, or revenue revenue. What about the bottom line? The bottom line is your profit: income minus expenses. PyPy development is very expensive. As I said, I love the idea of the sprints. I think they're great. They're also expensive. I don't know for sure that they're expensive to host, but they're certainly expensive to attend - especially for people who live on another continent. Lowering the cost of participating in pypy won't put any money in your pockets, but if the goal of the money is to have more man hours put into developing pypy (and I don't know what the goal is, but that is my guess) then there are many other ways you can get more smart people working on it. For example, I clicked on "issues" on the pypy site. Turns out these are not just issues, but also a sort of "to-do" list. Are any of these things that a newbie could do without going to a sprint? Maybe with some guidance. For example, to take an issue at random... #118 : "pickling of ll flowgraphs" I'm sure to you guys, pickling of ll flowgraphs makes a lot of sense. But to me as an outsider, I'm not sure what it means. Does ll mean llvm or low level? And what prevents you from pickling it? Can you write a test case that would pass if pickling ll flowgraphs worked? To an outsider like me, PyPy works by genius-level black magic. So if I wanted to help out and get ll flowgraph pickles working, I'd have a huge learning curve. I also don't know if it's really important or not. So basically if I want to help out with some code over the weekend, I look at the task here and I see a huge learning curve and then after all that work, I don't know if the task is really important or not. So what I'm saying is that it's difficult for people to pick up pypy. The impression is that you have to wait for a sprint and then go to europe if you want to help out. So I imagine that recruiting help is very expensive in terms of time. The point, then, is that one way to reach your goals without needing people to give you more money is to make it easier for people to give you their *time*. I'm working on a similar problem for pirate and at my own company, and I'd be willing to help with making the to-do list more approachable to newbies if you guys are interested. Sincerely, Michal J Wallace Sabren Enterprises, Inc. ------------------------------------- contact: michal@sabren.com hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ -------------------------------------
Hi Michal, there is a lot in your mail that i'd like to make some comments on. But I'd rather wait until we meet at some sprint to discuss some of that :-) On Thu, Sep 15, 2005 at 13:36 -0400, Michal Wallace wrote:
It's also a matter of good timing to bring more people into the project. For example, the Paris sprint is a quite good opportunity because we are heading for new stuff there where people can get involved and being helped in a more focused way.
The default issue tracker view separates issues into "easy", "medium" and "hard" to tackle and displays the easiest ones first.
That's a 'medium' issue and thus not really suited for newcomers.
I agree that it's good to try to describe issues assuming as few pre-knowledge as possible.
OK. But aren't there also a number of issues where it's not so hard to guess about their usefulness?
Sure. We actively mentor (or at least try to) new people and i hold that our documentation is not the worst. The fact is that writing a full Python interpreter implementation and a full compiler are - by themselves - not trivial tasks. Add some challenging goals (including making the EU project part of the project work) to that and you have current PyPy :-)
you are welcome! Going over the to-do list and adding useful information (maybe grabbed from someone at IRC) could certainly help. cheers, holger
Hi there Just a short comment to your email Michael (thanks for your thoughts onj this matter BTW ;-): One of the current major discussions about how to capitalize on Open Source as a business model is just that - learning/training. So I think you are on to something there Michael - both regarding the technical aspects and the project management/developer process aspects. Sprinting combines them both ;-) Cheers Bea On Wed, 14 Sep 2005, Michal Wallace wrote:
Beatrice Düring Change Maker Tel: 031- 7750940 Järntorget 3 Mobil: 0734- 22 89 06 413 04 Göteborg E-post: bea@changemaker.nu www.changemaker.nu "Alla dessa måsten och alldaglighet. Allt detta som binder vår verklighet i bojor av skam och rep utav tvång. Alla dessa kedjor som binder vår sång. Jag skall slita dem alla i små, små stycken och möjligtvis av resterna göra mig smycken." - hemlig
On Thu, 15 Sep 2005, Beatrice During wrote:
Just a short comment to your email Michael (thanks for your thoughts onj this matter BTW ;-):
:)
Hi Bea, I recently did a "sprint" in the form of the pyweek game challenge. We hung out in IRC for most of the week and made a game. (Or at least a game engine, before we ran out of time!) To me it was an exercise in project management but it was also a recruiting tool: I wanted to see what it was like to work with certain people before I spent the money to actually hire them. It worked well, and I plan to continue using that model to screen potential hires in the future... Maybe a company with more cash in the bank could use pypy in a similar way? Sincerely, Michal J Wallace Sabren Enterprises, Inc. ------------------------------------- contact: michal@sabren.com hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ -------------------------------------
Hi there On Thu, 15 Sep 2005, Michal Wallace wrote:
What a great application - do you mind if I spread the word about this..? Cheers Bea Beatrice Düring Change Maker Tel: 031- 7750940 Järntorget 3 Mobil: 0734- 22 89 06 413 04 Göteborg E-post: bea@changemaker.nu www.changemaker.nu "Alla dessa måsten och alldaglighet. Allt detta som binder vår verklighet i bojor av skam och rep utav tvång. Alla dessa kedjor som binder vår sång. Jag skall slita dem alla i små, små stycken och möjligtvis av resterna göra mig smycken." - hemlig
Hi Bea, hi Michal, On Thu, Sep 15, 2005 at 20:58 +0200, Beatrice During wrote:
I'd rather like to think that it's a nice side effect of sprints that you can get to know people who are interesting to hire. But the main reason for that is that you meet people who are doing things out of their own interest. If you start proposing or inviting to sprints for hiring purposes they might easily become more of a "working contest" or exam situation and loose much of the fun that you get from collaborating out of free interest. cheers, holger
Hello, On Thu, Sep 15, 2005 at 09:59:14PM +0200, holger krekel wrote:
I would also have a problem with asking people to come contribute to a project that I benefit from without them getting paid. Doing free software does not imply feeding from air... -- Nicolas Chauvat logilab.fr - services en informatique avancée et gestion de connaissances
Christian Tismer <tismer@stackless.com> writes:
I'll try to remember when it starts, this time...
The starship? I think you already have an account there :)
Participation on the CCC congress in Berlin
Is the event in German? I guess from the fact that the website it, the event probably is too...
It seems to me that rather than saying "inlining will make us faster" or "string interning will make us faster" or whatever, some effort needs to go into finding out why we are slow.
Well, my uninformed opinion would be "no, not right now".
Lots of people are interested (though maybe they shouldn't be) in a non-GIL implementation of Python. Offering different concurrency models is a really big selling point of PyPy, I think. Cheers, mwh -- That one is easily explained away as massively intricate conspiracy, though. -- Chris Klein, alt.sysadmin.recovery
On Thu, Sep 15, 2005 at 10:04:09AM +0100, Michael Hudson wrote:
This is what I'm working on (though from an academic context :), and I've been in somewhat of a holding pattern recently - doing some small experiments with CPython, reading and following mailing lists/IRC, trying to figure out when PyPy is finished enough that I can start doing work. Of course, if there's something I can do to help PyPy get to this point, I'll help. Has anyone written anything about the (at least near-term) plans for PyPy in the area of concurrency/locking? -- Nicholas Riley <njriley@uiuc.edu> | <http://www.uiuc.edu/ph/www/njriley>
Hi Nicholas, On Thu, Sep 15, 2005 at 11:18:15AM -0500, Nicholas Riley wrote:
Interesting. We should be about to consider alternate locking strategies right now. The current status is: we use GIL locking, and didn't even finish that -- the GIL is not released around system calls. Considering other locking strategies, I think the following one is possible. It would incur a small general run-time overhead, but it's of course all PyPy is about (whatever opinions might be heard around here :-) : avoiding as much as possible hard-wired design decisions and leaving compromizes to the choice of the user (or at least, for now, the person who translates and compiles PyPy). The naive locking idea is to put a cheap lock on each object, and then each space operation would first acquire the lock on its arguments and release it when it finishes. Recursive calls to the interpreter (i.e. calls to space operations that call back app-level Python code) would also release all the locks held by that thread. The locks should be re-entrants, i.e. not block if already held by the same thread. These locks can probably be optimized using some assumptions about our use case: we could for example acquire more locks when entering a space operation, but delay all releasing to the end of the bytecode (or even to the end of the next 'checkinterval' bytecodes), at which point we would bulk-release all locks held by the current thread. The above scheme is rather easy to implement as a proxy object space, doing the locking and then calling the standard object space. Other ideas are possible too; I'd be interested to hear about yours, what you've been playing with? A bientot, Armin.
On Thu, 15 Sep 2005, Christian Tismer wrote:
Hey all, I've only been reading these updates, and I haven't actually played with PyPy in while, so I could be way off here. But I do run a business and know a little bit about marketing, and I head up another python compiler project (pirate) so I know a little bit about the market demand for these things... :) Basically, there are really that many applications of this technology in its current state. The stated goal for this project is just to make a faster version of python, but can you really charge for that given that pypy is already open source? One option could be to find people who need python to run fast and get them to pay you to handle the cases that apply to *their project* up front, but I really don't think this is a viable market. It's much cheaper for the client to just write a C extension or port the whole thing to C. Another option is that you could use your technology created in pypy to create new *frontends* for pypy, and sell the service of taking legacy code in other languages, and translating that into call trees, and turning it magically into python or lisp or c... This seems like it could be a much more lucrative service, but it's a whole new research project and it takes you away from your core focus on python. (On the other hand, a company that did this would have a massive incentive to sponsor pypy development... Maybe there's a company that already does this kind of thing with some other technology... Like, say, people just rewriting things by hand? Another idea has to do with the sprints. I already told Holger this a while back, but I think you guys have one of the most impressive project management styles around, and the sprint idea seems like a great amount of fun. What if you capitalized on *that*? A week in europe to work on an open source project? That would be an awesome vacation! And to get to learn about compilers and python along the way? What I'm saying is you could market the sprints as a sort of training package for big companies interested in trying out python. They send their developers and pay you guys to train them. Even if the companies don't care about python, their developers *might* and that could be a great reward. Wasn't there a post just like this from someone in the army a while back? Market demand. :) Heck, do your next sprint as a partnership with these guys: http://www.geekcruises.com/ Anyway, it might be a crazy idea, but if you could get it working, it would capitalize on what you're already doing rather than force you to come up with some other devlepoment project off to the side... Sincerely, Michal J Wallace Sabren Enterprises, Inc. ------------------------------------- contact: michal@sabren.com hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ -------------------------------------
Hi Michal, On Wed, Sep 14, 2005 at 22:17 -0400, Michal Wallace wrote:
PyPy is not primarily - at least not only - about making Python faster although many people like to view it this way :-) PyPy is a lot about flexibility to produce custom implementations of Python for use in particular environments. Regarding persistence, security (e.g. sandboxing) and deployment there is a lot that we can possiblly do with PyPy implementations. I am sure that such possibilities will become clearer next year.
A build tool is part of the current EU project and of our plans.
We are doing this regularly :-)
:-)
There are many possibilities but only so much time and focus to pursue them. Bea also reported quite some interest from other projects and entities in the EU context. We got invitations to Afrika and other places and i guess we pursue them :-) Thanks for your kind advice, holger
holger krekel wrote: [snip]
PyPy is not primarily - at least not only - about making Python faster although many people like to view it this way :-)
This correction is a bit confusing... It is not surprising that people view "a faster Python" as a primary goal of your project, as that's the message that's been spread repeatedly by you all over the years: "The PyPy project aims at producing a flexible and fast Python implementation. The guiding idea is to translate a Python-level description of the Python language itself to lower level languages. Rumors have it that the secret goal is being faster-than-C which is nonsense, isn't it?" and "In the next step of the project, we will generate C code or machine code from the source of Pypy, thereby reducing the speed penalty. Later in the project, we will introduce optimisation steps that should make Pypy run faster than CPython." and "Psyco is a Python extension module which can massively speed up the execution of any Python code. ... The future of Psyco now lies in the PyPy project, which according to plan will provide a good base for a Python interpreter with better and well-integrated Psyco-like techniques as soon as 2006." I certainly hope that your "not primarily - at least not only" doesn't indicate that the goal of making Python faster is slowly becoming less important to you all, as I'd very much like a faster Python, and more flexibility of implementation is, while of tremendous personal interest to me, not one of much business interest in the foreseeable future, which means it's unlikely I'll get to play with PyPy much unless it offers performance benefits. I realize that of course that flexibility has always been a primary goal of PyPy as well; obviously speed isn't the *only* goal of the project, but the flexibility goal now dominates and a faster Python is indeed not of primary importance to the project, I would suggest you revise the previous statements, which make it rather likely people will get this wrong impression. Regards, Martijn
torsdagen den 15 september 2005 12.42 skrev Martijn Faassen:
I think you misunderstand Holger. Making Python faster is _a_ goal with Pypy, but it is not _the_ goal. The most important goals, as I see it are to produce something that is flexible and which has a clear implementation. However, this becomes an academic exercise if we produce something that is slower than competing implementations of Python. We want to build something that produces actual benefits in real world applications. Jacob
Jacob Hallén wrote:
That's the point. Not prioritizing speed is fine, but it does not allow us to be remarkably slower. Being of competitive speed and then more flexible is a must. If we are slower, the market will not consider us at all. ciao - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Christian Tismer wrote: [snip]
That's true, and that's a useful clarification. It's still different from the impression you're giving out, where you're strongly implying at least to me that speed will be significantly *faster* eventually with PyPy. Then again, it's forgivable that some hype is used to attract interest to the project. Regards, Martijn
Martijn Faassen wrote:
I thik I'm the worst speed junkie in the project, and I'm holding my breath a little until I'm convinced that we can be fast. ATM I'm quite vague about that, because of the complexity of crucial parts of our system, and the fact that I don't understand how we are going to translate that into speedy code without very high level optimization techniques. I simply can't tell whether we can reach this without major hand-rewriting or whether we can gather the missing speed factor by applying JIT technology. I always thought that the initial big step of creating an initial binary of reasonable speed would be an easy, first step. But it turned out to be a huge step which is by far not in the shape it should be. This amount of my misestimation makes me nervous and stops me from praising PyPy's speed all too loudly. On the other hand, we have extremely efficient results on the low level side. RPython programs have been shown to become possibly very fast. This is an area where I have no doubt making promises, and that's why I proposed to put some more effort there and at least provide a tool to produce fast extension modules from RPython, soon. This is also the reason why I want to make RPython more complete and easier to use. For sure we never have put any hype onto PyPy just to make it more attractive. We learned that flexibility and speed are not trivial to marry, and we have proven our claims about flexibility, but unfortunately not yet on speed. I would really hate to retract this claim. ATM, there is simply no prognosis possible. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Hello Christian and pypy team ;-) At 17:16 2005-09-15 +0200, you (Christian Tismer) wrote:
Well, keep the faith ;-) A couple of comments: 1) ISTM that "speed" needs to be split at least into speed of translation and speed of the resulting code/interpreter pair, though interactively one experiences the combined effect. (BTW, by that '/' I mean to highlight the fact that output code could be raw machine code packaged for OS linking/loading and where the "interpreter" is a CPU, vs. byte code for a VM such as found in cpython or jython etc. Also it could be pointed out that the target pair could eventually be on another platform entirely, such as a cell phone or game device, or I suppose a vending machine control system etc.) 1a) Re speed of translation: I note that translation has multiple phases/passes, so I wonder if this would be amenable to pipelining, so as to be able to take automatic advantage of multiple processors, either in the same box, or cooperating via fast network links if there were command line options for such delegation? Could one pickle state and pass it on to the next phase? Could you factor the passes as separate programs to connect in a unix-os-level pipeline? In which case I guess parallel make utilities could possibly come into play. Just thoughts... 1b) (1a) Brings the thought that the same interpreter will have a lot of unused baggage for any particular use, if it is a monolith. Which again brings up the translation-time/platform vs run-time/platform distinctions. I.e. space may be as important as speed in the target run-time environment, but I may be willing to crunch overnight translating on a cluster to make something that will fit and run acceptably on the target platform. BTW, is the issue of ROM-ability entirely low-level, or is it some kind of concern higher up? 1c) Will the different components of an interpreter be directable to dlls/sos and be able to load lazily/be-excluded-if-not-needed/whatever e.g. for good interactive startup, but also be configurable to load monolithically? IWT this is make-file stuff, but a makefile can't select pieces if they were welded together by the way they were developed, so I am wondering how that aspect is happening. 2) ISTM that you guys will be the first to notice what meta-info will be useful to pass into the translator(s) to achieve various results, so I wonder what analogies to e.g., C's "volatile" declaration you may want, and what vehicle you will want for representing and transmitting such information. Pragmas? more -*- cookies -*- ? If you come up with something really good, isn't it likely regular python will adopt it, as well as vice versa? The same goes for language syntax changes that don't hide behind a '#' -- IOW, will you guys wind up with some recommendations for language changes? I don't think you should be burdened with writing PEPs and haggling about syntactic sugar, but IWT you guys will discover optimization obstacles that no one else is in a position to see (or maybe even comprehend ;-), but that you could communicate by saying, "If we had a way to say this about such-and-so in python, we could optimize such-and-so much better" and that could help the language evolve in a win-win way. Is there a way to get a peek into your ongoing thoughts without asking you to hang all your unfinished paintings in the window when you go home from the studio each night? 2a) ISTM there will need to be module-specific meta-info, not just -O or -OO etc command line options. And since module translation occurs primarily at import time, there is the opportunity to pass dynamically determined meta-info to the import process as well, not just letting it see meta-info embedded in the source file or refer to interpreter startup command line parameter info. (I guess you could also package meta-info in various config files, or consolidate such info in one file for members of a package, etc. but I wonder if it's not dangerous to go further down the path of tangling abstract hierarchies with platform specific file system properties). So I'm wondering, if you see useful mods/variations to the semantics of import, or other python aspects, how would such ideas get back to python, so the appropriate language PEPs can get under way (though not prematurely) ? 3) I'm wondering how much of pypy translation technology could be factored out to apply to any language translation, and whether there could be a family of porting tool spinoffs eventually. I guess I rambled, as usual ;-) Anyhow, I am sure EU pypy funding is well spent. I hope they will renew their commitment when the time comes. Best wishes to you all. Regards, Bengt Richter
Hi Bengt, On Thu, Sep 15, 2005 at 06:46:30PM -0700, Bengt Richter wrote:
I don't think either of this is likely. We are not suggesting language changes or seeing the need for some. And for RPython we can mostly deal with regular Python syntax with a couple of additional specific objects that don't require any kind of syntax; for example, as in C there is sometimes a good reason to use signed or unsigned integers but Python doesn't have the distinction, we have defined a class "r_uint" which works like an unsigned integer with the same wrap-around semantics as C. The translator knows that it can be turned into a simple "unsigned int" in C. There is no need for more advanced support. In general we try hard to keep language changes separated from PyPy, with the exception of new built-in types providing extra features that are not really possible in CPython, just like Stackless did -- e.g. the thunk object space.
A lot of it. That's part of our goal: making it applicable to different languages. If anyone shows up with a Javascript parser, I'm sure we could together hack (and translate (and later have a JIT from)) a basic Javascript interpreter within a few weeks :-) A bientot, Armin.
--- Armin Rigo <arigo@tunes.org> wrote: ...
Actually, I think the currently useful thing might be a way to compile (any decent language, Python for choice) INTO Javascript (with reasonable efficiency), so one could write AJAX pages without actually having to code in Javascript...;-) Alex
Hi Alex! On Sun, Sep 25, 2005 at 18:42 -0700, Alex Martelli wrote:
that's a second possibility but there are two crucial perequesites regarding current PyPy: - a translation approach that focuses more on high-level languages (instead of the current RTyper "C" one) - a browser/DOM simulation layer in Python to allow rapid prototyping. I believe the idea with a PyPy/JS Interpreter is more interesting, at least from the PyPy perspective. Btw, being faster than current JS interpreters is probably easier than being faster than CPython :-) Such a JS-interpreter implementation could be integrated with browsers allowing JS and Python directly in the browser. That being said, it might actually be worthwhile to compile RPython to Javascript which should be reasonably fast for small programs. One possibility is to just create flowgraphs from a python program and then transform the flowgraph to javascript source code (without much annotation). I guess only real experiments would shed more light. cheers, holger
--- holger krekel <hpk@trillke.net> wrote:
Yep, the former would surely require lots of work (though maybe not the latter).
I believe the idea with a PyPy/JS Interpreter is more interesting, at least from the PyPy perspective. Btw,
Possibly, but I advisedly used the adjective "useful" rather than saying "interesting";-) JS's only advantage is one of *deployment* -- browsers have JS interpreters, so you can write AJAX code and dramatically improve the user experience. "Sweating blood" is a charitable description of what it takes, but several examples prove it can work in RL. I just find myself daydreaming that I could do the coding in Python (or almost any other VHLL) rather than JS, that's all;-)
being faster than current JS interpreters is probably easier than being faster than CPython :-)
Yes, good point.
Any Python implementation might be integrated with browsers, but it just seems to take a long time happening (even for Firefox, which should allow it most easily)...;-)
True, annotation wouldn't help much here (translating among VHLLs). Alex
On Sep 26, 2005, at 10:49 AM, Alex Martelli wrote:
Using MochiKit at a target on top of JS would make it even easier, since comparators, iterators, and repr are already there with Python semantics (more or less). You'll probably have to throw out multiple inheritance, anything that depends on id() or hash(), and some other features JS can't really do. -bob
On 26 Sep 2005 at 8:56, holger krekel wrote:
Hello everyone. Python and prototypes is one area where I have some personal experience. How well RPython will translate to Javascript depends on how fully RPython supports new-style classes. Javascript objects have single inheritance only. So problems arise immediately for multiple inheritance. An object space is needed to implement new-style classes on top of Javascript prototypes. All python arithmetic operations, attribute accesses and method calls become Javascript function calls. This adds a level of indirection to an already slow interpreter. Finally Javascript has no 'goto' statement. So a switch statement within a loop is needed to handle the spaghetti code produced by tranforming a flowgraph. How much different is this from an interpreter loop? So a Python program compiled to Javascript is essentially a Python interpreter that runs in Javascript. How worthwhile is this? Of course if RPython only supports single inheritance and improvements to flowgraph transformation will recover Python's statement structure then a closer translation may be possible. Lenard Lindstrom <len-l@telus.net>
Alex Martelli wrote:
I took a stab at this once based on somebody else's code... http://davidf.sjsoft.com/files/py2js I found some Python constructs ended up being tricky to translate to JavaScript (as the rest of the thread said). So the idea here was to be able to start with some Python code, get it into basic JavaScript syntax, but then you'd probably have to munge it anyway Cheers David
On Thursday 29 September 2005 12:13, David Fraser wrote:
I've just used your py2js to start playing around with python javascript integration on the web application server level. For people who have too much time on their hand: http://www.sdiehl.de/python required to run: python2.4 and cherrypy2.1 If these are installed, the example should run out of the box. To give some not so offtopic comment: I had this idea as well that a (fast) pypy could be the ideal base for web browser development. Since code written in python is so much easier to understand than, say, c++, it would be so much easier to get css standard conformance right, or Javascript implementation, or whatever. This could lead to much faster development cycles. But, of course, this would depend on pypy running not only as fast as CPython but considerably faster. --- Stephan
Jacob Hallén wrote:
I'm not sure how to understand Holger. He said PyPy is "not primarily - at least not only" about making a faster Python, and was in the context of correcting someone. I figured that perhaps the PyPy project is not bringing the message out as well as possible, whatever this message is.
So, is making a faster Python a *primary* goal of PyPy or not? I understand that the way to go about this is to produce a clear, flexible implementation, and that this is also cool, and also a primary goal by itself, but is making a faster Python *also* a primary goal or not? If it's a primary goal, why the correction? If not, the message you all are putting across might need some clarification, as you're certainly implying to me in the texts I quoted that it is a primary goal. Regards, Martijn
Hi Martijn, since I am in a very nitpicking mode... Martijn Faassen wrote:
Note that it says "fexible and fast": fexible comes first. And we explicitely say that the faster-than-C goal is nonsense.
"reducing the speed penalty" does not imply being very fast. It just means being not so slow.
"a good base" does not necessarily mean as fast as Psyco is now.
Holger's statement is very important. At the moment most people have the notion that PyPy is mainly about speed. But the really exciting fact about PyPy is its flexibility, therefore it makes sense to advertise the flexibility goals a bit (even if that means diminishing the speed goal in one single mail to pypy-dev). Of course, achieving high speed is a worthwhile goal but this goal can be reached _because of_ the flexibility of PyPy (not the other way round). :-) Cheers, Carl Friedrich
Hey, Carl Friedrich Bolz wrote:
since I am in a very nitpicking mode...
I'm nitpicking too, but even in the first announcement of PyPy, before it even had this name, the claim that the thing was going to be faster than CPython was made: As Armin Rigo of PSYCO fame takes part in the effort, we are confident that MinimalPython will eventually run faster than today's CPython.
Holger's statement is very important. At the moment most people have the notion that PyPy is mainly about speed.
What I am saying is that it's due to the PyPy project's communications that this notion has spread. To correct someone who picked up on this notion is therefore a bit confusing.
I know that, and I'll stop nitpicking now. I just thought it was a bit weird to start correcting an impression in people that seems a legitimate interpretation of previous communication of this project. Then again, an emphasis on flexibility is justified. What is exciting about PyPy seems to depend on ones perspective too. To the PyPy hackers themselves, the coolest thing is the flexibility allowing all sorts of fun experimentations with syntaxes and semantics and ways Python code is executed. To the wider world outside Python language hackers however the *results* of this flexibility of PyPy is what is truly interesting. The flexibility by itself is not interesting at all from that perspective. Since the topic of this thread was making money out of PyPy, this is a relevant thing to talk about. For non Python core hackers, the biggest potential gains in PyPy that I heard about so far would be: * increased performance * making Python better in an increasingly concurrent world (threading/GIL/stackless, etc). That ties into increased performance too, in multi processing environments. Regards, Martijn
Hi there Just a short comment: The fact that we have different perspectives and views on what has been communicated about the project goal indicates to me that there is/or was an unclear communication about this ;-) Thanks for raising attention to this Martijn. Cheers Bea On Thu, 15 Sep 2005, Martijn Faassen wrote:
Beatrice Düring Change Maker Tel: 031- 7750940 Järntorget 3 Mobil: 0734- 22 89 06 413 04 Göteborg E-post: bea@changemaker.nu www.changemaker.nu "Alla dessa måsten och alldaglighet. Allt detta som binder vår verklighet i bojor av skam och rep utav tvång. Alla dessa kedjor som binder vår sång. Jag skall slita dem alla i små, små stycken och möjligtvis av resterna göra mig smycken." - hemlig
Carl Friedrich Bolz wrote:
...
I don't think that order is relevant in a mathematical "and". For me it means "flexible without being slow" and "fast without being inflexible". This is the matter of optimization. We have seen examples of both. Python is a quite fast, inflexible implementation. PyPy is a very flexible but yet slow implementation. The proof of concept, that we can have both at the same time, is still not there. And I don't support the nonsense part at all. Psyco has proven that it can make CPython much faster under circumstances. The same factor can be at least expected for PyPy. But if PyPy does not jump over its current factor 40-50 hurdle, this acceleration might still not outperform CPython. ...
The impression of mostly being about speed might also come from the fact that speed is the thing that PyPy lacks at most, hurtingly. And it is clear if a project is started in the most flexible way possible from the ground, with no consideration of speed at all. We will need to find yet another stone of the wise to morph the code reasonably. As said, I very much hope it, while there is currently no indicator that we are close to it. We have to prove that last point about flexibility. The gap lies in the needed transformations, which need to be invented. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On Thu, 15 Sep 2005, holger krekel wrote:
On Wed, Sep 14, 2005 at 22:17 -0400, Michal Wallace wrote:
Hi Holger, What I said is that the stated goal is to make a faster version of python. I actually looked it up before I wrote that: """ The PyPy project aims at producing a flexible and fast Python implementation. The guiding idea is to translate a Python-level description of the Python language itself to lower level languages. Rumors have it that the secret goal is being faster-than-C which is nonsense, isn't it? more... """ -- http://codespeak.net/pypy/dist/pypy/doc/news.html As Martijn Faassen said, it's not surprising that people have the impression that pypy is all about speed. I think part of the problem is that people understand the word "fast". What that means is obvious. It's like python, only it runs faster! "Flexible" doesn't mean anything to me. Python is already flexible, so what does that mean? Well, you've got these different backends. Maybe that's what it means. And the people who made stackless and psyco and green threads are on the team, so maybe it's about those things. Who knows? The site doesn't say so if we want to figure out the status of those things, we have to think about it.
PyPy is a lot about flexibility to produce custom implementations of Python for use in particular environments. Regarding persistence,
Persistence. What does that mean? Is that like Squeak, where you can save the state of the whole interpreter as a single image? If so, that's pretty neat! It ought to be right up front on the site.
security (e.g. sandboxing)
Also neat. So another benefit is that you can make programs that run other programs safely? I know some guys from twisted like that stuff, because they want to make games where people can run their own scripts.
and deployment
I don't know what this means at all.
there is a lot that we can possiblly do with PyPy implementations. I am sure that such possibilities will become clearer next year.
Cool. I guess I was thinking more about what you could do to make money *now*, given the current state of the codebase... Sounds like one big part of that would be telling people more about the potential so they can say "I want that! I'll pay to make it happen!" :) Another thing I suspect is part of pypy but isn't really right up front is the whole continuations thing. Yes, CPS dos make an appearance int he top news article on the news page, but that'll just get buried as you post more news. Basically "flexible" is a feature, and not a very well defined one. Features don't sell well. What sells are benefits, and "fast" is a benefit. So if it's not the primary benefit, it's time to fix your message. :)
I don't understand. What I'm saying is that since you can translate python->lisp or python->python, then maybe you can use the same technology to translate cobol->python or cobol->java or whatever using the cool internal technology that pypy has. This would be a major research project and well outside the bounds of the pypy project. However, if I recall, there are already companies that do this kind of thing and pypy's internals might be of use to them and their clients. That's all I was saying. I'm not sure what build tools has to do with that. ;)
You have sprints regularly, but are you marketing them to companies as a reward or training for top employees? Last I heard, at least some of the sprints were limited to experienced PyPy developers, right? Obviously you don't want 500 newbies running around, but... <shrug>
Well, now that I hear the talk that there are other benefits than speed, I'd say you'd have a much better chance of making money if you capitalized on that. Also, all of this is about the top line, or revenue revenue. What about the bottom line? The bottom line is your profit: income minus expenses. PyPy development is very expensive. As I said, I love the idea of the sprints. I think they're great. They're also expensive. I don't know for sure that they're expensive to host, but they're certainly expensive to attend - especially for people who live on another continent. Lowering the cost of participating in pypy won't put any money in your pockets, but if the goal of the money is to have more man hours put into developing pypy (and I don't know what the goal is, but that is my guess) then there are many other ways you can get more smart people working on it. For example, I clicked on "issues" on the pypy site. Turns out these are not just issues, but also a sort of "to-do" list. Are any of these things that a newbie could do without going to a sprint? Maybe with some guidance. For example, to take an issue at random... #118 : "pickling of ll flowgraphs" I'm sure to you guys, pickling of ll flowgraphs makes a lot of sense. But to me as an outsider, I'm not sure what it means. Does ll mean llvm or low level? And what prevents you from pickling it? Can you write a test case that would pass if pickling ll flowgraphs worked? To an outsider like me, PyPy works by genius-level black magic. So if I wanted to help out and get ll flowgraph pickles working, I'd have a huge learning curve. I also don't know if it's really important or not. So basically if I want to help out with some code over the weekend, I look at the task here and I see a huge learning curve and then after all that work, I don't know if the task is really important or not. So what I'm saying is that it's difficult for people to pick up pypy. The impression is that you have to wait for a sprint and then go to europe if you want to help out. So I imagine that recruiting help is very expensive in terms of time. The point, then, is that one way to reach your goals without needing people to give you more money is to make it easier for people to give you their *time*. I'm working on a similar problem for pirate and at my own company, and I'd be willing to help with making the to-do list more approachable to newbies if you guys are interested. Sincerely, Michal J Wallace Sabren Enterprises, Inc. ------------------------------------- contact: michal@sabren.com hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ -------------------------------------
Hi Michal, there is a lot in your mail that i'd like to make some comments on. But I'd rather wait until we meet at some sprint to discuss some of that :-) On Thu, Sep 15, 2005 at 13:36 -0400, Michal Wallace wrote:
It's also a matter of good timing to bring more people into the project. For example, the Paris sprint is a quite good opportunity because we are heading for new stuff there where people can get involved and being helped in a more focused way.
The default issue tracker view separates issues into "easy", "medium" and "hard" to tackle and displays the easiest ones first.
That's a 'medium' issue and thus not really suited for newcomers.
I agree that it's good to try to describe issues assuming as few pre-knowledge as possible.
OK. But aren't there also a number of issues where it's not so hard to guess about their usefulness?
Sure. We actively mentor (or at least try to) new people and i hold that our documentation is not the worst. The fact is that writing a full Python interpreter implementation and a full compiler are - by themselves - not trivial tasks. Add some challenging goals (including making the EU project part of the project work) to that and you have current PyPy :-)
you are welcome! Going over the to-do list and adding useful information (maybe grabbed from someone at IRC) could certainly help. cheers, holger
Hi there Just a short comment to your email Michael (thanks for your thoughts onj this matter BTW ;-): One of the current major discussions about how to capitalize on Open Source as a business model is just that - learning/training. So I think you are on to something there Michael - both regarding the technical aspects and the project management/developer process aspects. Sprinting combines them both ;-) Cheers Bea On Wed, 14 Sep 2005, Michal Wallace wrote:
Beatrice Düring Change Maker Tel: 031- 7750940 Järntorget 3 Mobil: 0734- 22 89 06 413 04 Göteborg E-post: bea@changemaker.nu www.changemaker.nu "Alla dessa måsten och alldaglighet. Allt detta som binder vår verklighet i bojor av skam och rep utav tvång. Alla dessa kedjor som binder vår sång. Jag skall slita dem alla i små, små stycken och möjligtvis av resterna göra mig smycken." - hemlig
On Thu, 15 Sep 2005, Beatrice During wrote:
Just a short comment to your email Michael (thanks for your thoughts onj this matter BTW ;-):
:)
Hi Bea, I recently did a "sprint" in the form of the pyweek game challenge. We hung out in IRC for most of the week and made a game. (Or at least a game engine, before we ran out of time!) To me it was an exercise in project management but it was also a recruiting tool: I wanted to see what it was like to work with certain people before I spent the money to actually hire them. It worked well, and I plan to continue using that model to screen potential hires in the future... Maybe a company with more cash in the bank could use pypy in a similar way? Sincerely, Michal J Wallace Sabren Enterprises, Inc. ------------------------------------- contact: michal@sabren.com hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ -------------------------------------
Hi there On Thu, 15 Sep 2005, Michal Wallace wrote:
What a great application - do you mind if I spread the word about this..? Cheers Bea Beatrice Düring Change Maker Tel: 031- 7750940 Järntorget 3 Mobil: 0734- 22 89 06 413 04 Göteborg E-post: bea@changemaker.nu www.changemaker.nu "Alla dessa måsten och alldaglighet. Allt detta som binder vår verklighet i bojor av skam och rep utav tvång. Alla dessa kedjor som binder vår sång. Jag skall slita dem alla i små, små stycken och möjligtvis av resterna göra mig smycken." - hemlig
Hi Bea, hi Michal, On Thu, Sep 15, 2005 at 20:58 +0200, Beatrice During wrote:
I'd rather like to think that it's a nice side effect of sprints that you can get to know people who are interesting to hire. But the main reason for that is that you meet people who are doing things out of their own interest. If you start proposing or inviting to sprints for hiring purposes they might easily become more of a "working contest" or exam situation and loose much of the fun that you get from collaborating out of free interest. cheers, holger
Hello, On Thu, Sep 15, 2005 at 09:59:14PM +0200, holger krekel wrote:
I would also have a problem with asking people to come contribute to a project that I benefit from without them getting paid. Doing free software does not imply feeding from air... -- Nicolas Chauvat logilab.fr - services en informatique avancée et gestion de connaissances
Christian Tismer <tismer@stackless.com> writes:
I'll try to remember when it starts, this time...
The starship? I think you already have an account there :)
Participation on the CCC congress in Berlin
Is the event in German? I guess from the fact that the website it, the event probably is too...
It seems to me that rather than saying "inlining will make us faster" or "string interning will make us faster" or whatever, some effort needs to go into finding out why we are slow.
Well, my uninformed opinion would be "no, not right now".
Lots of people are interested (though maybe they shouldn't be) in a non-GIL implementation of Python. Offering different concurrency models is a really big selling point of PyPy, I think. Cheers, mwh -- That one is easily explained away as massively intricate conspiracy, though. -- Chris Klein, alt.sysadmin.recovery
On Thu, Sep 15, 2005 at 10:04:09AM +0100, Michael Hudson wrote:
This is what I'm working on (though from an academic context :), and I've been in somewhat of a holding pattern recently - doing some small experiments with CPython, reading and following mailing lists/IRC, trying to figure out when PyPy is finished enough that I can start doing work. Of course, if there's something I can do to help PyPy get to this point, I'll help. Has anyone written anything about the (at least near-term) plans for PyPy in the area of concurrency/locking? -- Nicholas Riley <njriley@uiuc.edu> | <http://www.uiuc.edu/ph/www/njriley>
Hi Nicholas, On Thu, Sep 15, 2005 at 11:18:15AM -0500, Nicholas Riley wrote:
Interesting. We should be about to consider alternate locking strategies right now. The current status is: we use GIL locking, and didn't even finish that -- the GIL is not released around system calls. Considering other locking strategies, I think the following one is possible. It would incur a small general run-time overhead, but it's of course all PyPy is about (whatever opinions might be heard around here :-) : avoiding as much as possible hard-wired design decisions and leaving compromizes to the choice of the user (or at least, for now, the person who translates and compiles PyPy). The naive locking idea is to put a cheap lock on each object, and then each space operation would first acquire the lock on its arguments and release it when it finishes. Recursive calls to the interpreter (i.e. calls to space operations that call back app-level Python code) would also release all the locks held by that thread. The locks should be re-entrants, i.e. not block if already held by the same thread. These locks can probably be optimized using some assumptions about our use case: we could for example acquire more locks when entering a space operation, but delay all releasing to the end of the bytecode (or even to the end of the next 'checkinterval' bytecodes), at which point we would bulk-release all locks held by the current thread. The above scheme is rather easy to implement as a proxy object space, doing the locking and then calling the standard object space. Other ideas are possible too; I'd be interested to hear about yours, what you've been playing with? A bientot, Armin.
participants (18)
-
Alex Martelli -
Armin Rigo -
Beatrice During -
Bengt Richter -
Bob Ippolito -
Carl Friedrich Bolz -
Christian Tismer -
David Fraser -
holger krekel -
hpk＠trillke.net
-
Jacob Hallén -
Lenard Lindstrom -
Martijn Faassen -
Michael Hudson -
Michal Wallace -
Nicholas Riley -
Nicolas Chauvat -
Stephan Diehl