Oren Tirosh <oren-py-d@hishome.net> writes:

It overloads the meaning of the error handling argument in an unintuitive way. It gets to the point where it's much more than just error handling - it's actually extending the functionality of the codec.

Isn't that precisely the meaning fo "to handle"? 3 : to act on or perform a required function with regard to <handle the day's mail> It produces a replacement text, just in the same way as "ignore" or "replace" produce replacement texts.

Why implement yet another name-based registry?

Namespaces are one honking great idea -- let's do more of those!

There must be a simpler way to do it.

Propose one.

What are the use cases? Maybe a simple extension to charmap would be enough for all the practical cases?

The primary use case is XML: how do you efficiently use xml charrefs. Notice that you can *not* use the charmap codec, since the underlying encoding may not be based on the charmap codec. In addition, it allows to give a more detailed analysis of an encoding error, as it exposes the string position where the error occurs. This allows to determine a "best" encoding (i.e. one that needs the fewest amounts of exceptions, or the one that has the longest sequences of same encodings).

Me too. But if you really don't want it to be rejected you should try to find a way to make it simpler.

Can you please elaborate why you think this is difficult? Is this a concern about - the implementation of the PEP, or - the implementation of error handlers, or - the usage of error handlers? I couldn't really believe that you find usage of this feature difficult: just pass an error handling string to your codec just as you currently do.

...

While you are waiting for PEP 293 to complete, please do consider cleaning up htmlentitydefs to provide mappings from and to Unicode characters.

No problem. The question is whether anyone depends on its current form. My proposed changes:

1. Use all lowercase entity names as keys.

That is probably a bad idea. Atleast for XHTML, the case of entity references is normative. Even for HTML 4, it would be good if this precisely matches the DTD. You could provide a case-insensitive lookup function in addition.

2. Map "entityname" to u"\uXXXX" (currently it's mapped to "&#nnnn;")

I think htmlentitydefs.entitydefs must stay as-is, for compatibility. Instead, I'd suggest to add additional objects/functions. Of course, the data should be present only once - all other functions/dictionaries could be derived.

In its current form I find htmlentitydefs.py pretty useless. Names in the input in arbitrary case will not match the MixedCase keys in the entitydefs dictionary and the decimal character reference isn't really more useful than the named entity reference.

Indeed. However, people probably rely on its specific contents, so any more useful access to the data must preserve entitydefs in its current form. Regards, Martin

Oren Tirosh wrote:

In its current form I find htmlentitydefs.py pretty useless.

I use it a lot, and find it reasonably useful. sure beats typing in the HTML character tables myself, or writing a DTD parser.

Names in the input in arbitrary case will not match the MixedCase keys in the entitydefs dictionary

people who use oddball characters may prefer to keep uppercase letters separate from lowercase letters. if I type "Linköping" using a named entity, I don't want it to come out as "LinkÖping". if you don't care, nothing stops you from using the "lower" string method.

and the decimal character reference isn't really more useful than the named entity reference.

really? converting a decimal character reference to a unicode character is trivial, but how do you convert a named entity reference to a unicode character? (look it up in the htmlentitydefs?) here's a trivial piece of code that converts the entitydefs dictionary to a entity->unicode mapping: entitydefs_unicode = {} for entity, char in entitydefs.items(): if char[:2] == "&#": char = unichr(int(char[2:-1])) else: char = unicode(char, "iso-8859-1") entitydefs_unicode[entity] = char </F>

Oren Tirosh <oren-py-d@hishome.net> writes:

It overloads the meaning of the error handling argument in an unintuitive way. It gets to the point where it's much more than just error handling - it's actually extending the functionality of the codec.

Isn't that precisely the meaning fo "to handle"? 3 : to act on or perform a required function with regard to <handle the day's mail> It produces a replacement text, just in the same way as "ignore" or "replace" produce replacement texts.

Why implement yet another name-based registry?

Namespaces are one honking great idea -- let's do more of those!

There must be a simpler way to do it.

Propose one.

What are the use cases? Maybe a simple extension to charmap would be enough for all the practical cases?

The primary use case is XML: how do you efficiently use xml charrefs. Notice that you can *not* use the charmap codec, since the underlying encoding may not be based on the charmap codec. In addition, it allows to give a more detailed analysis of an encoding error, as it exposes the string position where the error occurs. This allows to determine a "best" encoding (i.e. one that needs the fewest amounts of exceptions, or the one that has the longest sequences of same encodings).

Me too. But if you really don't want it to be rejected you should try to find a way to make it simpler.

Can you please elaborate why you think this is difficult? Is this a concern about - the implementation of the PEP, or - the implementation of error handlers, or - the usage of error handlers? I couldn't really believe that you find usage of this feature difficult: just pass an error handling string to your codec just as you currently do.

...

While you are waiting for PEP 293 to complete, please do consider cleaning up htmlentitydefs to provide mappings from and to Unicode characters.

No problem. The question is whether anyone depends on its current form. My proposed changes:

1. Use all lowercase entity names as keys.

That is probably a bad idea. Atleast for XHTML, the case of entity references is normative. Even for HTML 4, it would be good if this precisely matches the DTD. You could provide a case-insensitive lookup function in addition.

2. Map "entityname" to u"\uXXXX" (currently it's mapped to "&#nnnn;")

I think htmlentitydefs.entitydefs must stay as-is, for compatibility. Instead, I'd suggest to add additional objects/functions. Of course, the data should be present only once - all other functions/dictionaries could be derived.

In its current form I find htmlentitydefs.py pretty useless. Names in the input in arbitrary case will not match the MixedCase keys in the entitydefs dictionary and the decimal character reference isn't really more useful than the named entity reference.

Indeed. However, people probably rely on its specific contents, so any more useful access to the data must preserve entitydefs in its current form. Regards, Martin

Oren Tirosh wrote:

In its current form I find htmlentitydefs.py pretty useless.

I use it a lot, and find it reasonably useful. sure beats typing in the HTML character tables myself, or writing a DTD parser.

Names in the input in arbitrary case will not match the MixedCase keys in the entitydefs dictionary

people who use oddball characters may prefer to keep uppercase letters separate from lowercase letters. if I type "Linköping" using a named entity, I don't want it to come out as "LinkÖping". if you don't care, nothing stops you from using the "lower" string method.

and the decimal character reference isn't really more useful than the named entity reference.

really? converting a decimal character reference to a unicode character is trivial, but how do you convert a named entity reference to a unicode character? (look it up in the htmlentitydefs?) here's a trivial piece of code that converts the entitydefs dictionary to a entity->unicode mapping: entitydefs_unicode = {} for entity, char in entitydefs.items(): if char[:2] == "&#": char = unichr(int(char[2:-1])) else: char = unicode(char, "iso-8859-1") entitydefs_unicode[entity] = char </F>