[Python-bugs-list] [ python-Bugs-456175 ] add dictionary "tokens" to token.py

[noreply@sourceforge.net]

Bugs item #456175, was opened at 2001-08-28 08:48
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=456175&group_id=5470

Category: Python Library
Group: Feature Request
>Status: Closed
>Resolution: Works For Me
Priority: 5
Submitted By: Grant Griffin (dspguru)
Assigned to: Nobody/Anonymous (nobody)
>Summary: add dictionary "tokens" to token.py

Initial Comment:
I would like for token.py to provide a dictionary, 
called "tokens" that maps the name of each token to 
its value, e.g.:

   token.tokens[token.INDENT] is 'INDENT'

This would be useful for debugging programs that use 
tokenize.

A user currently could create a dictionary like this 
using introspection on token.py (assuming tokens are 
in all caps) but that's a lot of trouble.  Instead, I 
believe it would be better to solve the problem once-
and-for-all by building the extra logic required to 
create the dictionary into token.main.

thanks much,

=g2


----------------------------------------------------------------------

>Comment By: Jeremy Hylton (jhylton)
Date: 2001-09-04 09:00

Message:
Logged In: YES 
user_id=31392

Have you looked at token.py?  It's always had tok_name.


----------------------------------------------------------------------

Comment By: Grant Griffin (dspguru)
Date: 2001-08-28 10:56

Message:
Logged In: YES 
user_id=70844

Thanks much, Skip.  That's not a bad idea, but here's what 
I ended up doing:

def token_names():
    import tokenize
    res = {}
    names = dir(tokenize)
    for name in names:
        attr = getattr(tokenize, name)
        if type(attr) is type(1):
            res[attr] = name
    return res

Instead of modifying token.py as I had suggested 
previously, perhaps something like the above could be added 
to tokenize.

- There isn't much code here, but it took me awhile to 
figure out how to include my own batteries <wink>.

- I learned the hard way that it's better to get these 
symbols from tokenize than token because tokenize adds 
the "COMMENT" token.)

thanks,

=g2

----------------------------------------------------------------------

Comment By: Skip Montanaro (montanaro)
Date: 2001-08-28 09:42

Message:
Logged In: YES 
user_id=44345

I encountered the same problem when programming with Gtk.
Gtk defines an enormous number of enumeration types.  While
I could define one or more dictionaries to map enumerations
back to names as you suggested, I chose a somewhat different
approach.  I defined a ConstantMap class whose instances
create a new dictionary containing all non-instance/non-
class objects created since the last ConstantMap class was
instantiated.

class ConstantMap:
    markdict = {}

    def __init__(self):
        """add new global symbols to instance's name map"""
        self.names = {}
        import types
        for key,val in globals().items():
            if (not self.markdict.has_key(key) and
                key[:2] != "__" and
                type(val) not in [types.InstanceType,
types.ClassType]):
                self.names[val] = key
        self.markdict.update(globals())

    def __call__(self, key):
        """return a name mapping"""
        return self.names[key]

    def mark(self):
        """checkpoint current globals"""

I then instantiate ConstantMap after each block of
enumeration constants are defined like so:

# GdkPropertyState's
PROPERTY_NEW_VALUE = 0
PROPERTY_DELETE    = 1
GdkPropertyState = ConstantMap()

# GdkWindowState's
WINDOW_STATE_WITHDRAWN = 1 << 0
WINDOW_STATE_ICONIFIED = 1 << 1
WINDOW_STATE_MAXIMIZED = 1 << 2
WINDOW_STATE_STICKY    = 1 << 3
GdkWindowState = ConstantMap()

# GdkSettingAction's
SETTING_ACTION_NEW     = 0
SETTING_ACTION_CHANGED = 1
SETTING_ACTION_DELETED = 2
GdkSettingAction = ConstantMap()

# GdkScrollDirection's
SCROLL_UP    = 0
SCROLL_DOWN  = 1
SCROLL_LEFT  = 2
SCROLL_RIGHT = 3
GdkScrollDirection = ConstantMap()

This is more than you're asking for in this case, but may
be a somewhat more general foundation for a problem that
bugs many people debugging code that uses imported
constants.

Skip


----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=456175&group_id=5470