[Python-checkins] CVS: python/dist/src/Tools/scripts gencodec.py,1.3,1.4

[M.-A. Lemburg]

Update of /cvsroot/python/python/dist/src/Tools/scripts
In directory usw-pr-cvs1:/tmp/cvs-serv26608/Tools/scripts

Modified Files:
	gencodec.py 
Log Message:
This patch changes the default behaviour of the builtin charmap
codec to not apply Latin-1 mappings for keys which are not found
in the mapping dictionaries, but instead treat them as undefined
mappings.

The patch was originally written by Martin v. Loewis with some
additional (cosmetic) changes and an updated test script
by Marc-Andre Lemburg.

The standard codecs were recreated from the most current files
available at the Unicode.org site using the Tools/scripts/gencodec.py
tool.

This patch closes the bugs #116285 and #119960.



Index: gencodec.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Tools/scripts/gencodec.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -r1.3 -r1.4
*** gencodec.py	2000/03/17 16:56:23	1.3
--- gencodec.py	2001/01/03 21:29:14	1.4
***************
*** 2,8 ****
  
  This script parses Unicode mapping files as available from the Unicode
! site (ftp.unicode.org) and creates Python codec modules from them. The
! codecs use the standard character mapping codec to actually apply the
! mapping.
  
  Synopsis: gencodec.py dir codec_prefix
--- 2,8 ----
  
  This script parses Unicode mapping files as available from the Unicode
! site (ftp://ftp.unicode.org/Public/MAPPINGS/) and creates Python codec
! modules from them. The codecs use the standard character mapping codec
! to actually apply the mapping.
  
  Synopsis: gencodec.py dir codec_prefix
***************
*** 19,22 ****
--- 19,23 ----
  
  (c) Copyright CNRI, All Rights Reserved. NO WARRANTY.
+ (c) Copyright Guido van Rossum, 2000.
  
  """#"
***************
*** 71,74 ****
--- 72,79 ----
      f.close()
      enc2uni = {}
+     identity = []
+     unmapped = range(256)
+     for i in range(256):
+         unmapped[i] = i
      for line in lines:
          line = strip(line)
***************
*** 86,91 ****
          else:
              comment = comment[1:]
!         if enc != uni:
              enc2uni[enc] = (uni,comment)
      return enc2uni
  
--- 91,110 ----
          else:
              comment = comment[1:]
!         if enc < 256:
!             unmapped.remove(enc)
!             if enc == uni:
!                 identity.append(enc)
!             else:
!                 enc2uni[enc] = (uni,comment)
!         else:
              enc2uni[enc] = (uni,comment)
+     # If there are more identity-mapped entries than unmapped entries,
+     # it pays to generate an identity dictionary first, add add explicit
+     # mappings to None for the rest
+     if len(identity)>=len(unmapped):
+         for enc in unmapped:
+             enc2uni[enc] = (None, "")
+         enc2uni['IDENTITY'] = 256
+ 
      return enc2uni
  
***************
*** 144,152 ****
      l = [
          '''\
! """ Python Character Mapping Codec generated from '%s'.
  
  Written by Marc-Andre Lemburg (mal@lemburg.com).
  
  (c) Copyright CNRI, All Rights Reserved. NO WARRANTY.
  
  """#"
--- 163,172 ----
      l = [
          '''\
! """ Python Character Mapping Codec generated from '%s' with gencodec.py.
  
  Written by Marc-Andre Lemburg (mal@lemburg.com).
  
  (c) Copyright CNRI, All Rights Reserved. NO WARRANTY.
+ (c) Copyright 2000 Guido van Rossum.
  
  """#"
***************
*** 179,191 ****
  
  ### Decoding Map
- 
- decoding_map = {
  ''' % name,
          ]
      mappings = map.items()
      mappings.sort()
      append = l.append
      i = 0
-     splits = 0
      for e,value in mappings:
          try:
--- 199,219 ----
  
  ### Decoding Map
  ''' % name,
          ]
+ 
+     if map.has_key("IDENTITY"):
+         l.append("decoding_map = codecs.make_identity_dict(range(%d))"
+                  % map["IDENTITY"])
+         l.append("decoding_map.update({")
+         splits = 1
+         del map["IDENTITY"]
+     else:
+         l.append("decoding_map = {")
+         splits = 0
+         
      mappings = map.items()
      mappings.sort()
      append = l.append
      i = 0
      for e,value in mappings:
          try:
***************
*** 199,203 ****
          else:
              append('\t%s: %s,' % (key,unicoderepr(u)))
!         i = i + 1
          if i == 4096:
              # Split the definition into parts to that the Python
--- 227,231 ----
          else:
              append('\t%s: %s,' % (key,unicoderepr(u)))
!         i += 1
          if i == 4096:
              # Split the definition into parts to that the Python
***************
*** 207,211 ****
              else:
                  append('})')
!             append('map.update({')
              i = 0
              splits = splits + 1
--- 235,239 ----
              else:
                  append('})')
!             append('decoding_map.update({')
              i = 0
              splits = splits + 1
***************
*** 266,270 ****
      mapnames = os.listdir(dir)
      for mapname in mapnames:
!         if mapname[-len('.mapping'):] != '.mapping':
              continue
          codefile = mapname[:-len('.mapping')] + '.py'
--- 294,298 ----
      mapnames = os.listdir(dir)
      for mapname in mapnames:
!         if not mapname.endswith('.mapping'):
              continue
          codefile = mapname[:-len('.mapping')] + '.py'