Here's a feature I like from Perl's Unicode support: """ Support for interpolating named characters The new \N escape interpolates named characters within strings. For example, "Hi! \N{WHITE SMILING FACE}" evaluates to a string with a unicode smiley face at the end. """ I get really tired of looking up the Unicode character for "ndash" or "right dagger". Does our Unicode database have enough information to make something like this possible? Obviously using the official (English) name is only really helpful for people who speak English, so we should not remove the numeric option. -- Paul Prescod - ISOGEN Consulting Engineer speaking for himself "I and my companions suffer from a disease of the heart that can only be cured with gold", Hernan Cortes
Paul Prescod writes:
The new \N escape interpolates named characters within strings. For example, "Hi! \N{WHITE SMILING FACE}" evaluates to a string with a unicode smiley face at the end.
Cute idea, and it certainly means you can avoid looking up Unicode numbers. (You can look up names instead. :) ) Note that this means the Unicode database is no longer optional if this is done; it has to be around at code-parsing time. Python could import it automatically, as exceptions.py is imported. Christian's work on compressing unicodedatabase.c is therefore really important. (Is Perl5.6 actually dragging around the Unicode database in the binary, or is it read out of some external file or data structure?) -- A.M. Kuchling http://starship.python.net/crew/amk/ About ten days later, it being the time of year when the National collected down and outs to walk on and understudy I arrived at the head office of the National Theatre in Aquinas Street in Waterloo. -- Tom Baker, in his autobiography
"Andrew M. Kuchling" wrote:
Paul Prescod writes:
The new \N escape interpolates named characters within strings. For example, "Hi! \N{WHITE SMILING FACE}" evaluates to a string with a unicode smiley face at the end.
Cute idea, and it certainly means you can avoid looking up Unicode numbers. (You can look up names instead. :) ) Note that this means the Unicode database is no longer optional if this is done; it has to be around at code-parsing time. Python could import it automatically, as exceptions.py is imported. Christian's work on compressing unicodedatabase.c is therefore really important. (Is Perl5.6 actually dragging around the Unicode database in the binary, or is it read out of some external file or data structure?)
Sorry to disappoint you guys, but the Unicode name and comments are *not* included in the unicodedatabase.c file Christian is currently working on. The reason is simple: it would add huge amounts of string data to the file. So this is a no-no for the core distribution... Still, the above is easily possible by inventing a new encoding, say unicode-with-smileys, which then reads in a file containing the Unicode names and applies the necessary magic to decode/encode data as Paul described above. Would probably make a cool fun-project for someone who wants to dive into writing codecs. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
"M.-A. Lemburg" wrote:
"Andrew M. Kuchling" wrote:
Paul Prescod writes:
The new \N escape interpolates named characters within strings. For example, "Hi! \N{WHITE SMILING FACE}" evaluates to a string with a unicode smiley face at the end.
Cute idea, and it certainly means you can avoid looking up Unicode numbers. (You can look up names instead. :) ) Note that this means the Unicode database is no longer optional if this is done; it has to be around at code-parsing time. Python could import it automatically, as exceptions.py is imported. Christian's work on compressing unicodedatabase.c is therefore really important. (Is Perl5.6 actually dragging around the Unicode database in the binary, or is it read out of some external file or data structure?)
Sorry to disappoint you guys, but the Unicode name and comments are *not* included in the unicodedatabase.c file Christian is currently working on. The reason is simple: it would add huge amounts of string data to the file. So this is a no-no for the core distribution...
This is not settled, still an open question. What I have for non-textual data: 25 kb with dumb compression 15 kb with enhanced compression What amounts of data am I talking about? - The whole unicode database text file has size 632 kb. - With PkZip this goes down to 96 kb. Now, I produced another text file with just the currently used data in it, and this sounds so: - the stripped unicode text file has size 216 kb. - PkZip melts this down to 40 kb. Please compare that to my results above: I can do at least twice as good. I hope I can compete for the text sections as well (since this is something where zip is *good* at), but just let me try. Let's target 60 kb for the whole crap, and I'd be very pleased. Then, there is still the question where to put the data. Having one file in the dll and another externally would be an option. I could also imagine to use a binary external file all the time, with maximum possible compression. By loading this structure, this would be partially expanded to make it fast. An advantage is that the compressed Unicode database could become a stand-alone product. The size is in fact so crazy small, that I'd like to make this available to any other language.
Still, the above is easily possible by inventing a new encoding, say unicode-with-smileys, which then reads in a file containing the Unicode names and applies the necessary magic to decode/encode data as Paul described above.
That sounds reasonable. Compression makes sense as well here, since the expanded stuff makes quite an amount of kb, compared to what it is "worth", compared to, say, the Python dll. ciao - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home
Christian Tismer wrote:
"M.-A. Lemburg" wrote:
"Andrew M. Kuchling" wrote:
Paul Prescod writes:
The new \N escape interpolates named characters within strings. For example, "Hi! \N{WHITE SMILING FACE}" evaluates to a string with a unicode smiley face at the end.
Cute idea, and it certainly means you can avoid looking up Unicode numbers. (You can look up names instead. :) ) Note that this means the Unicode database is no longer optional if this is done; it has to be around at code-parsing time. Python could import it automatically, as exceptions.py is imported. Christian's work on compressing unicodedatabase.c is therefore really important. (Is Perl5.6 actually dragging around the Unicode database in the binary, or is it read out of some external file or data structure?)
Sorry to disappoint you guys, but the Unicode name and comments are *not* included in the unicodedatabase.c file Christian is currently working on. The reason is simple: it would add huge amounts of string data to the file. So this is a no-no for the core distribution...
This is not settled, still an open question.
Well, ok, depends on how much you can sqeeze out of the text columns ;-) I still think that its better to leave these gimmicks out of the core and put them into some add-on, though.
What I have for non-textual data: 25 kb with dumb compression 15 kb with enhanced compression
Looks good :-) With these sizes I think we could even integrate the unicodedatabase.c + API into the core interpreter and only have the unicodedata module to access the database from within Python.
What amounts of data am I talking about? - The whole unicode database text file has size 632 kb. - With PkZip this goes down to 96 kb.
Now, I produced another text file with just the currently used data in it, and this sounds so: - the stripped unicode text file has size 216 kb. - PkZip melts this down to 40 kb.
Please compare that to my results above: I can do at least twice as good. I hope I can compete for the text sections as well (since this is something where zip is *good* at), but just let me try. Let's target 60 kb for the whole crap, and I'd be very pleased.
Then, there is still the question where to put the data. Having one file in the dll and another externally would be an option. I could also imagine to use a binary external file all the time, with maximum possible compression. By loading this structure, this would be partially expanded to make it fast. An advantage is that the compressed Unicode database could become a stand-alone product. The size is in fact so crazy small, that I'd like to make this available to any other language.
You could take the unicodedatabase.c file (+ header file) and use it everywhere... I don't think it needs to contain any Python specific code. The API names would have to follow the Python naming schemes though.
Still, the above is easily possible by inventing a new encoding, say unicode-with-smileys, which then reads in a file containing the Unicode names and applies the necessary magic to decode/encode data as Paul described above.
That sounds reasonable. Compression makes sense as well here, since the expanded stuff makes quite an amount of kb, compared to what it is "worth", compared to, say, the Python dll.
With 25kB for the non-text columns, I'd suggest simply adding the file to the core. Text columns could then go into a separate module. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
"Andrew M. Kuchling" wrote:
Paul Prescod writes:
The new \N escape interpolates named characters within strings. For example, "Hi! \N{WHITE SMILING FACE}" evaluates to a string with a unicode smiley face at the end.
Cute idea, and it certainly means you can avoid looking up Unicode numbers. (You can look up names instead. :) )
More important, though, the code is "self documenting". You never have to go from the number back to the name.
Note that this means the Unicode database is no longer optional if this is done; it has to be around at code-parsing time.
I don't like the idea enough to exclude support for small machines or anything like that. We should way the costs of requiring the Unicode database at compile time.
(Is Perl5.6 actually dragging around the Unicode database in the binary, or is it read out of some external file or data structure?)
I have no idea. -- Paul Prescod - ISOGEN Consulting Engineer speaking for himself "I and my companions suffer from a disease of the heart that can only be cured with gold", Hernan Cortes
participants (4)
-
Andrew M. Kuchling -
Christian Tismer -
M.-A. Lemburg -
Paul Prescod