* Manlio Perillo [2006-12-15 13:33:55 +0100]:

manlio@synapsis:~/projects/tests/nevow$ python static2.py 9.85 usec/pass

manlio@synapsis:~/projects/tests/django$ python static2.py 13.31 usec/pass

Is this python 2.4 or 2.5? The string handling speed-ups in 2.5 have a fairly substantial impact on Nevow rendering speed, I believe.

Nevow for every data object calls inevow.IContainer.child. The problem can be here, since it is called 25 times (the number of rows in the table)

That's easy enough to fix, just don't use data specials :) -- mithrandi, i Ainil en-Balandor, a faer Ambar

On Fri, 15 Dec 2006 15:43:44 +0100, Manlio Perillo wrote:

Jean-Paul Calderone ha scritto:

...

On Fri, 15 Dec 2006 11:56:55 +0100, Manlio Perillo wrote: [...]

...

Is it possible to use that same model used by django.template?

You pass a context (that is a dictionary usable like a stack: push/pop) to the render function. All the data directive will search the data in the context (like djando, first dictionary lookup, then attribute lookup, and so).

As an example n:data='x.y' ==> ctx['x']['y']. A rend.Page.renderHTTP can build a context from the instance dictionary.

Optimization can be a complicated undertaking.

I'm not asking for optimization.

You are. You pointed out something which you thought was slow. You suggested a different way to accomplish something much the same. You place greater value on one than the other because of performance reasons. That's what "optimization" means.

I'm asking for a possible rewriting of how rend.Page (and flat) works.

The context is being removed, so that will happen. Since the way you described continues to expose the context as an application level API, I don't think that is how the rewrite will go. Jean-Paul

On Fri, 15 Dec 2006 16:11:46 +0100, Manlio Perillo wrote:

Jean-Paul Calderone ha scritto:

...

On Fri, 15 Dec 2006 15:43:44 +0100, Manlio Perillo wrote:

...

Jean-Paul Calderone ha scritto:

...

On Fri, 15 Dec 2006 11:56:55 +0100, Manlio Perillo wrote: [...]

...

Is it possible to use that same model used by django.template?

You pass a context (that is a dictionary usable like a stack: push/pop) to the render function. All the data directive will search the data in the context (like djando, first dictionary lookup, then attribute lookup, and so).

As an example n:data='x.y' ==> ctx['x']['y']. A rend.Page.renderHTTP can build a context from the instance dictionary.

Optimization can be a complicated undertaking.

I'm not asking for optimization.

You are. You pointed out something which you thought was slow.

Well, its a fact (unless my tests are wrong).

I didn't say it wasn't.

...

You suggested a different way to accomplish something much the same.

I'm not sure if it will be the same.

It's a means of passing objects to different parts of Python code responsible for rendering a template. That's what the existing context code is for. They're basically the same thing.

...

You place greater value on one than the other because of performance reasons. That's what "optimization" means.

...

I'm asking for a possible rewriting of how rend.Page (and flat) works.

The context is being removed, so that will happen.

Ok, but how is going this branch of development?

It's not really going anywhere at the moment, presumably because everyone has more important things to work on.

...

Since the way you described continues to expose the context as an application level API, I don't think that is how the rewrite will go.

I'm talking about a different context.

Um, okay.

A context that holds data.

I don't know what that means. I don't think you can put anything except data into a data structure.

context = {'table': ..., 'header': ..., 'request': request} return flat.flatten(node, context)

Now stan use object adaptation (locate/remember) for obtaing the data requested by a data directive.

There will be no context nor any locate or remember methods in the rewritten system. They are a bad idea, completely independent from performance concerns. Jean-Paul

On Fri, 15 Dec 2006 16:41:01 +0100, Manlio Perillo wrote:

[snip]

...

There will be no context nor any locate or remember methods in the rewritten system. They are a bad idea, completely independent from performance concerns.

Ok, this is what i was trying to say. Can you please post an example on how to use the new system?

The plan is that render methods will take a request and a tag as arguments. There isn't a plan for data factories macro factories. If you look at nevow.page.Element, you will see what the replacement for nevow.rend.Fragment looks like. The replacement for nevow.rend.Page should be somewhat analogous. Jean-Paul

On Fri, 15 Dec 2006 17:22:46 +0100, Manlio Perillo wrote:

Jean-Paul Calderone ha scritto:

...

On Fri, 15 Dec 2006 16:41:01 +0100, Manlio Perillo wrote:

...

[snip]

...

There will be no context nor any locate or remember methods in the rewritten system. They are a bad idea, completely independent from performance concerns.

Ok, this is what i was trying to say. Can you please post an example on how to use the new system?

The plan is that render methods will take a request and a tag as arguments. There isn't a plan for data factories macro factories.

Wait. Does this means that I can no more render a table with the sequence renderer?

Most likely it does not mean this. Jean-Paul

On Fri, 15 Dec 2006 18:11:11 +0100, Manlio Perillo wrote:

Jean-Paul Calderone ha scritto:

...

On Fri, 15 Dec 2006 17:22:46 +0100, Manlio Perillo [...]

...

Wait. Does this means that I can no more render a table with the sequence renderer?

Most likely it does not mean this.

Please, let me understand. How will I (and I should now) render a table using as much XHML code as possible?

Maybe like this: http://twistedmatrix.com/trac/browser/sandbox/exarkun/merit/trunk/merit/reso... I have not explored much further than that yet. If you have any suggestions, feel free to make them. Jean-Paul

On Fri, 15 Dec 2006 08:43:44 -0600, Manlio Perillo wrote:

Jean-Paul Calderone ha scritto:

...

On Fri, 15 Dec 2006 11:56:55 +0100, Manlio Perillo wrote: [...]

...

Is it possible to use that same model used by django.template?

You pass a context (that is a dictionary usable like a stack: push/pop) to the render function. All the data directive will search the data in the context (like djando, first dictionary lookup, then attribute lookup, and so).

As an example n:data='x.y' ==> ctx['x']['y']. A rend.Page.renderHTTP can build a context from the instance dictionary.

Optimization can be a complicated undertaking.

I'm not asking for optimization. I'm asking for a possible rewriting of how rend.Page (and flat) works.

Re-writing rend,Page and flat to improve execution speed would most certainly be considered optimization. I'm not sure why you think otherwise; could you please explain? Note that I am confident that we'd all like Nevow to be faster, but optimization requires a lot more than just a cursory analysis. I get the impression from reading your post that you think your request is simple/trivial, and perhaps it seems that way, but there is a lot more to this. What you have done, so far, is to measure raw serialization speed, and in so doing, you've found that Django is faster. That part is fine, but the conclusion you've drawn is questionable. Do you know if the difference in speed actually makes any difference in a real web application? I'll bet that other factors, such as database access and network latency generally account for the majority of time that it takes for a request to be serviced. Please understand, I am not trying to offend you or tell you that Nevow doesn't need improving; everyone agrees that it could be a lot better. I am just trying to point out that your observations regarding serialization speed have not been made in a context that demonstrates a serious need to rewrite rend.Page and flat.flatten. Put another way, there are lots of automobiles in the world that go faster than my automobile, but unless my automobile is so slow that it overwhelms all other factors (traffic, for example), I don't really need to fix it, and could better spend my time on other things, like finding a better route to work. Hope this helps, L. Daniel Burr

On Fri, 15 Dec 2006 09:34:07 -0600, Manlio Perillo wrote:

L. Daniel Burr ha scritto:

...

[...] Re-writing rend,Page and flat to improve execution speed would most certainly be considered optimization.

Ok, lets consider it an optimization.

...

Note that I am confident that we'd all like Nevow to be faster, but optimization requires a lot more than just a cursory analysis. I get the impression from reading your post that you think your request is simple/trivial, and perhaps it seems that way, but there is a lot more to this.

My first question was: "Is it possible to use that same model used by django.template?"

Sure, but you posed that question for a reason. You didn't ask it because you thought it would be fun, you asked it because you found django to be faster, and wanted to use the same technique as django in order to make nevow faster.

I have posted my test results only later.

See above. You wouldn't have asked the question if you didn't want an optimization to be performed.

...

What you have done, so far, is to measure raw serialization speed, and in so doing, you've found that Django is faster. That part is fine, but the conclusion you've drawn is questionable. Do you know if the difference in speed actually makes any difference in a real web application? I'll bet that other factors, such as database access and network latency generally account for the majority of time that it takes for a request to be serviced.

The first test I have done was to compare the two web servers (Nevow + Twisted Web and Django + Lighttpd), obtaining the random data from a database.

Again, Nevow was much slower than Django.

Ok, that's great. Can you please provide the numbers? What was the concurrency level used?

...

Please understand, I am not trying to offend you or tell you that Nevow doesn't need improving; everyone agrees that it could be a lot better. I am just trying to point out that your observations regarding serialization speed have not been made in a context that demonstrates a serious need to rewrite rend.Page and flat.flatten.

I ask again: Nevow uses a lot of object adaptation. Is this really necessary?

Please google for posts to this list regarding adaptation and various rendering-speed concerns by one Andrea Arcangeli. In his case, he found Cheetah to be much faster than Nevow, just as you found Django to be much faster than Nevow. The long and short of it is that yes, object adaptation is more expensive that not using any adaptation at all. The real question is, do you care enough to become familiar with the branch in which the context-removal changes are being made, and to help in moving that work forward?

Regards Manlio Perillo

Hope this helps, L. Daniel Burr

On Fri, 15 Dec 2006 09:51:59 -0600, Manlio Perillo wrote:

L. Daniel Burr ha scritto:

...

On Fri, 15 Dec 2006 08:43:44 -0600, Manlio Perillo wrote: [...]

...

What you have done, so far, is to measure raw serialization speed, and in so doing, you've found that Django is faster.

One more thing.

This is not only a question Django vs Nevow.

I fully understand that. You could have used Cheetah, or Clearsilver, or any of a number of template engines, and found similar results.

If you read one of my previous messages, you will find that using data directives for rendering a table will slow down the rendering by a factor equal to 4.5.

Yes, I understand that too. I will point out that I have *never* used a data directive in any of my applications. I'm not sure why you feel that you need them. The fact that they are going away at some point is a very good thing, in my opinion. Given that data directives are going away, why even worry about this?

Regards Manlio Perillo

Hope this helps, L. Daniel Burr

On Fri, 15 Dec 2006 10:31:55 -0600, Manlio Perillo wrote:

L. Daniel Burr ha scritto:

...

[...] Yes, I understand that too. I will point out that I have *never* used a data directive in any of my applications. I'm not sure why you feel that you need them.

Because I want to write XHTML code as much as possible. This is possible with Djando and it is "fast", why it should not be possible with Nevow?

...

The fact that they are going away at some point is a very good thing, in my opinion.

If data directive are going away and I will no more be able to write:

<table> <caption>RandomTable</caption> <thead> <tr n:render="sequence" n:data="header"> <th n:pattern="item" n:render="string" /> </tr> </thead> <tbody n:render="sequence" n:data="table"> <tr n:pattern="item" n:render="sequence"> <td n:pattern="item" n:render="string" /> </tr> </tbody> </table>

You can still have patterns, and you can still render tables.

I will migrate to twisted web2 + django template, where I can write:

<table> <caption>RandomTable</caption> <thead> <tr> {% for i in header %} <th>{{ i }}</li> {% endfor %} </tr> </thead> <tbody> {% for i in table %} <tr> {% for j in i %} <td>{{ j }}</td> {% endfor %} </tr> {% endfor %} </tbody> </table>

I will note that this is most definitely *not* XHTML, so your statement "...want to write XHTML code as much as possible." doesn't make sense. The nevow template would probably pass XHTML validation, but this Django template surely won't, so I don't understand your point. Moreover, you've embedded iteration logic in the Django template, so if some idea of "XHTML purity" is driving you, the the Django approach, while faster, is less pure.

However I want to be sure. Maybe I'm using Nevow in the wrong way.

How do you render a table like the one in the example?

You could do something like this (untested): template.xhtml: <table xmlns="http://www.w3.org/1999/xhtml" xmlns:n="http://nevow.com/ns/nevow/0.1"> <caption>RandomTable</caption> <thead> <tr n:render="headers"> <th n:pattern="item"> </th> </tr> </thead> <tfoot/> <tbody n:render="rows"> <tr n:pattern="item"> <td n:pattern="item"> </td> </tr> </tbody> </table> from nevow import flat, loaders, page class MyTable(page.Element): docFactory = loaders.xmlfile('template.xhtml') def headers(self, request, tag): headerPattern = tag.patternGenerator('item') columns = [] for i in range(5): header = headerPattern() header.fillSlots('content', i) columns.append(header) return tag.clear()[columns] def rows(self, request, tag): rowPattern = tag.patternGenerator('item') rows = [] for i in range(10): row = rowPattern() cells = [] cellPattern = row.patternGenerator('item') for j in range(5): cell = cellPattern() cell.fillSlots('content', j) cells.append(cell) rows.append(row.clear()[cells]) return tag.clear()[rows] page.renderer(headers, rows) This is one possible way to do it, and it works now, with the current nevow release. Note that the template is generally similar to your original one, and we've kept the looping out of the template. The most important thing is to stop thinking in terms of data types (render="sequence", render="string"), and just write render methods that deal with app-level concepts (headers, rows). An important point to make here, is that the code I provided above is based on the same code used to implement the "sequence" directive, so there is really nothing magical going on.

Thanks and regards Manlio Perillo

Hope this helps, L. Daniel Burr

On Fri, 15 Dec 2006 12:34:53 -0600, Manlio Perillo wrote: [...]

The point is that in the template it should be "visible" the XHTML that will result.

As an example, this code:

<ul nevow:pattern="sums-container"> </ul> <li nevow:pattern="sums-element"> </li>

<ul nevow:pattern="methods-container"> </ul> <li nevow:pattern="methods-element"> </li>

<div> Wallclock duration totals (seconds) </div>

give me no clue on how the renderized page will look.

Sure, but that is nothing like the example I provided you. In my example, it is clear that the header row contains 1 or more th elements, and that the tbody element will contain 1 or more tr elements, each having 1 or more td elements. I agree that your example above doesn't read as clearly, but I'm sure you can write that in a style similar to what I used for the table example.

...

Moreover, you've embedded iteration logic in the Django template, so if some idea of "XHTML purity" is driving you, the the Django approach, while faster, is less pure.

Nevow has a better solution: the sequence renderer. But it will go away.

The sequence renderer is basically doing the same thing as the code I used in the "headers" and "rows" methods. We aren't talking about some large class/function that is complicated to reproduce. Adding a sequence renderer that complies with the upcoming changes to nevow is not going to be difficult. Even if no direct substitute is provided, nothing is stopping you from adding your own, or better yet, being the person to contribute a new sequence renderer. I think it would be a good thing to do.

...

[...] The most important thing is to stop thinking in terms of data types (render="sequence", render="string"), and just write render methods that deal with app-level concepts (headers, rows). An important point to make here, is that the code I provided above is based on the same code used to implement the "sequence" directive, so there is really nothing magical going on.

Ok, but what about reusability? Now for every table (in theory) I have to reimplement the sequence logic.

See above. The existing sequence renderer is just a few lines of code, not a magical entity. Why reimplement it for each table?

Well, I'm sure that it is possible to write generic/reusable code for rendering tables.

Absolutely. The code I provided in my example *is* a generic table element, that you could stick in any page. Just add some methods for querying your DB and you have a reusable element, that will render arbitrary data in tabular format.

...

Hope this helps,

Yes, thanks. I will try to use slots instead of data.

Slots are a good solution, because they work today, and will allow you to switch to the context-less version of nevow when it is released, without any major breakage.

P.S. How tested/stable is the new(?) page module?

I have used page.Element in a production environment for months now. I switched from using rend.Fragment to page.Element as soon as it was available, and things have worked perfectly, at least for me.

Regards Manlio Perillo

Hope this helps, L. Daniel Burr

On Sat, 16 Dec 2006 03:31:53 -0600, Manlio Perillo wrote:

L. Daniel Burr ha scritto:

...

On Fri, 15 Dec 2006 12:34:53 -0600, Manlio Perillo wrote: [...]

...

...

P.S. How tested/stable is the new(?) page module?

I have used page.Element in a production environment for months now. I switched from using rend.Fragment to page.Element as soon as it was available, and things have worked perfectly, at least for me.

...

One more question: is formal compatible with page.Element?

Not at this time. As far as I know, formal is still using rend.Fragment for its Field and Group objects, among other things. Changing formal to use page.Element is not terribly difficult, but somebody would need to see if the formal developers are interested in changing at this time. I think formal will be more likely to change once the context-removal work is actually finished in nevow; until then, I doubt there is any real urgency.

Thanks and regards Manlio Perillo

L. Daniel Burr