Hi there Martijn Faassen skrev:

Hi there,

I posted this to a longer thread the other day but have received no answers to it whatsoever. Perhaps it got lost, so I'll do a repost. I spent a bit of time writing this post, and I ask some questions in it that I think should be answered. Perhaps by telling me I don't understand what I'm talking about or that my analysis is entirely incorrect. Perhaps it was simply too long?

Whatever it is, I do think the PyPy community needs to be aware of how technical decisions have an impact on attracting or repelling potential contributors. "answering by silence", which happened so far, is one way to make potential contributors worried. :)

Although I can´t answer your questions I do appreciate the time and effort you and others have spent on this issue during the last couple of days, and I am sure the PyPy core gang appreciates your input. The current plan is to meet at EuroPython in Vilnius to have strategic technical discussions regarding direction and among them decisions regarding the issues you raise in this email. I interpret the silence as people feeling that they can´t answer this until such a discussion (???) but that does not excuse the total silence - hence this email (that does not give you the answers either). As you might have noticed - this topic is also on the agenda as a sprint topic for the PyPy Vilnius sprint at which time we hope to have made some decisions. Since you are coming to EuroPython I look forward to meet you again and discuss this and make sure that a PyPy strategic session takes place which gives you answers - answers you might like - or not like ;-) I hope this response does not frustrate you too much.... Cheers Bea

I only understand a little of the technical discussion. I understand that RCTypes has problems, and that people anticipate having to break it and reimplement it in a different way in order to solve these problems. I understand that one feature that RCTypes offers, namely the ability to test your RPython code on CPython, might also go away.

There are currently, as far as I can see it, two reasons to use RPython for CPython extensions:

a) benefit from performance gains by rewriting your CPython code into RPython. This is what we want to do with the page template interpreter. In case of language implementations, this might lead to the ability to use other PyPy technology later (such as JIT generation and so on). Some others might purely be interested in performance though and aren't implementing a language.

b) better ways to write extensions that use C libraries in CPython. The typical Pyrex use case, though it can do the former as well. This would use RCTypes.

Use case a) probably has less interested people than use case b). Eventually JIT technologies will help here, which will make use case a) less important.

I happen to be interested in use case a) though. It is quite exciting to me that we could have a faster template language interpreter implementation that we can use in CPython today, with the potential benefits of PyPy for interpreter translation in the future as well. We are basically language implementors, one of your core target audiences. We just feel more comfortable doing work if we know we have payoffs in the near future, i.e. use in CPython. If we were implementing this template language in isolation and had no hope we could deploy it in production any time soon, we would be less interested.

I am actually quite interested in use case b) as well. While developing lxml, I've done quite a bit of work with Pyrex and I can see how RPython and RCTypes might potentially help projects like that. I've been taking a wait and see attitude here though, as the technology needed more maturing.

As I understand it, the proposed changes wouldn't affect use case a). Or is the ability to generate CPython extensions also going to be broken due to this work? Please let us know, as if CPython support is going to be broken, we would likely stop our efforts for the time being.

Use case b) would certainly be broken due to this work. What's more, I understand that testing your extension code in CPython first wouldn't work anymore (at least if it uses CTypes). This is a very nice feature of RPython that Pyrex doesn't have. You mentioned that there are potential ways to reimplement this feature again, but I'm not sure whether you're committing to such implementation work.

If I were writing C extension modules with PyPy I'd like to know things like:

* what are the reasons this change is going to be made?

* would this benefit just core PyPy or would it benefit extension module writers too?

* when do you expect the new way to be finalized?

* will you make a real commitment to make the "Test in CPython first" feature work again? Or is did you just sketched out possible ways to implement this in the new system and you hope some contributor is going to do the work?

* will you make a commitment to make this new way useful and supported for extension module writers, or should they invest their time in some other technology? I.e. what are the changes a future shift will make me change things again?

* How well documented will this be for CPython extension developers? Will you market this feature to developers? Things like this indicate your commitment in another way. :)

Please note that I'm not actually *asking* for anything but information. I'm not telling the PyPy developers they should make commitments and do various kinds of work. That's up to you. I'm just trying to guess what other developers might want to know.

Of course, the quality of your communication and the answers you actually give will make people decide whether to continue to use PyPy or move on. This will affect the way your open source community will grow and develop.

I would urge the PyPy developers to consider investing in the open source community by supporting features like this and promoting them. This would mean serving admittedly self-interested developers with short term goals. The promise of immediate payoff is a great way to attract people to an open source project. If there is the hope of continuing payoff (and PyPy has tremendous potential here), some of those new people can be expected to become valuable contributors to the project as a whole. And yes, I'm self-interested in saying all of this - that's my point.

Regards,

Martijn

_______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev

On 6/20/07, Martijn Faassen <faassen@startifact.com> wrote:

There are currently, as far as I can see it, two reasons to use RPython for CPython extensions:

a) benefit from performance gains by rewriting your CPython code into RPython. This is what we want to do with the page template interpreter. In case of language implementations, this might lead to the ability to use other PyPy technology later (such as JIT generation and so on). Some others might purely be interested in performance though and aren't implementing a language.

b) better ways to write extensions that use C libraries in CPython. The typical Pyrex use case, though it can do the former as well. This would use RCTypes.

While this is perhaps only an instance of your first use case (although not really performance-oriented), using RPython code as a CPython extension module is how I was planning on Safelisp being used (a small safe language written in RPython, meant for embedding). I can imagine a similar situation for PyPy's Python interpreter itself, so one could "sneak" PyPy-using Python code into their production system by importing the PyPy Python interpreter as a CPython module. -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/

On Wed, 20 Jun 2007 12:23:06 +0200 Martijn Faassen <faassen@startifact.com> wrote: ...

I happen to be interested in use case a) though. It is quite exciting to me that we could have a faster template language interpreter

What is this interpreter you refer to here ? Simon.

Hi there Martijn Faassen skrev:

Hi there,

I posted this to a longer thread the other day but have received no answers to it whatsoever. Perhaps it got lost, so I'll do a repost. I spent a bit of time writing this post, and I ask some questions in it that I think should be answered. Perhaps by telling me I don't understand what I'm talking about or that my analysis is entirely incorrect. Perhaps it was simply too long?

Whatever it is, I do think the PyPy community needs to be aware of how technical decisions have an impact on attracting or repelling potential contributors. "answering by silence", which happened so far, is one way to make potential contributors worried. :)

Although I can´t answer your questions I do appreciate the time and effort you and others have spent on this issue during the last couple of days, and I am sure the PyPy core gang appreciates your input. The current plan is to meet at EuroPython in Vilnius to have strategic technical discussions regarding direction and among them decisions regarding the issues you raise in this email. I interpret the silence as people feeling that they can´t answer this until such a discussion (???) but that does not excuse the total silence - hence this email (that does not give you the answers either). As you might have noticed - this topic is also on the agenda as a sprint topic for the PyPy Vilnius sprint at which time we hope to have made some decisions. Since you are coming to EuroPython I look forward to meet you again and discuss this and make sure that a PyPy strategic session takes place which gives you answers - answers you might like - or not like ;-) I hope this response does not frustrate you too much.... Cheers Bea

I only understand a little of the technical discussion. I understand that RCTypes has problems, and that people anticipate having to break it and reimplement it in a different way in order to solve these problems. I understand that one feature that RCTypes offers, namely the ability to test your RPython code on CPython, might also go away.

There are currently, as far as I can see it, two reasons to use RPython for CPython extensions:

a) benefit from performance gains by rewriting your CPython code into RPython. This is what we want to do with the page template interpreter. In case of language implementations, this might lead to the ability to use other PyPy technology later (such as JIT generation and so on). Some others might purely be interested in performance though and aren't implementing a language.

b) better ways to write extensions that use C libraries in CPython. The typical Pyrex use case, though it can do the former as well. This would use RCTypes.

Use case a) probably has less interested people than use case b). Eventually JIT technologies will help here, which will make use case a) less important.

I happen to be interested in use case a) though. It is quite exciting to me that we could have a faster template language interpreter implementation that we can use in CPython today, with the potential benefits of PyPy for interpreter translation in the future as well. We are basically language implementors, one of your core target audiences. We just feel more comfortable doing work if we know we have payoffs in the near future, i.e. use in CPython. If we were implementing this template language in isolation and had no hope we could deploy it in production any time soon, we would be less interested.

I am actually quite interested in use case b) as well. While developing lxml, I've done quite a bit of work with Pyrex and I can see how RPython and RCTypes might potentially help projects like that. I've been taking a wait and see attitude here though, as the technology needed more maturing.

As I understand it, the proposed changes wouldn't affect use case a). Or is the ability to generate CPython extensions also going to be broken due to this work? Please let us know, as if CPython support is going to be broken, we would likely stop our efforts for the time being.

Use case b) would certainly be broken due to this work. What's more, I understand that testing your extension code in CPython first wouldn't work anymore (at least if it uses CTypes). This is a very nice feature of RPython that Pyrex doesn't have. You mentioned that there are potential ways to reimplement this feature again, but I'm not sure whether you're committing to such implementation work.

If I were writing C extension modules with PyPy I'd like to know things like:

* what are the reasons this change is going to be made?

* would this benefit just core PyPy or would it benefit extension module writers too?

* when do you expect the new way to be finalized?

* will you make a real commitment to make the "Test in CPython first" feature work again? Or is did you just sketched out possible ways to implement this in the new system and you hope some contributor is going to do the work?

* will you make a commitment to make this new way useful and supported for extension module writers, or should they invest their time in some other technology? I.e. what are the changes a future shift will make me change things again?

* How well documented will this be for CPython extension developers? Will you market this feature to developers? Things like this indicate your commitment in another way. :)

Please note that I'm not actually *asking* for anything but information. I'm not telling the PyPy developers they should make commitments and do various kinds of work. That's up to you. I'm just trying to guess what other developers might want to know.

Of course, the quality of your communication and the answers you actually give will make people decide whether to continue to use PyPy or move on. This will affect the way your open source community will grow and develop.

I would urge the PyPy developers to consider investing in the open source community by supporting features like this and promoting them. This would mean serving admittedly self-interested developers with short term goals. The promise of immediate payoff is a great way to attract people to an open source project. If there is the hope of continuing payoff (and PyPy has tremendous potential here), some of those new people can be expected to become valuable contributors to the project as a whole. And yes, I'm self-interested in saying all of this - that's my point.

Regards,

Martijn

_______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev

On 6/20/07, Martijn Faassen <faassen@startifact.com> wrote:

There are currently, as far as I can see it, two reasons to use RPython for CPython extensions:

a) benefit from performance gains by rewriting your CPython code into RPython. This is what we want to do with the page template interpreter. In case of language implementations, this might lead to the ability to use other PyPy technology later (such as JIT generation and so on). Some others might purely be interested in performance though and aren't implementing a language.

b) better ways to write extensions that use C libraries in CPython. The typical Pyrex use case, though it can do the former as well. This would use RCTypes.

While this is perhaps only an instance of your first use case (although not really performance-oriented), using RPython code as a CPython extension module is how I was planning on Safelisp being used (a small safe language written in RPython, meant for embedding). I can imagine a similar situation for PyPy's Python interpreter itself, so one could "sneak" PyPy-using Python code into their production system by importing the PyPy Python interpreter as a CPython module. -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/

On Wed, 20 Jun 2007 12:23:06 +0200 Martijn Faassen <faassen@startifact.com> wrote: ...

I happen to be interested in use case a) though. It is quite exciting to me that we could have a faster template language interpreter

What is this interpreter you refer to here ? Simon.