Splitting with quoted strings?

[Andrew Dalke]

Chris Barker wrote:
>Is there any nifty way to do a string.split() that
>doesn't split inside quotes?

>'this that "the other" some more'.split()
>
>to give me ['this', 'that', 'the other', 'some', 'more']

> and used for *nix command lines, etc., so I
> figured someone might have written a nice extension to do this.

For the specific case of interpreting command lines, try
shlex, which is part of the standard distribution but
somewhat obscure.

>>> import shlex
>>> from cStringIO import StringIO
>>> instream = StringIO('this that "the other" some more')
>>> lex = shlex.shlex(instream)
>>> while 1:
...  w = lex.get_token()
...  if not w:
...   break
...  print w
...
this
that
"the other"
some
more
>>>

The caution about that module is it is designed for command
lines so understands things like '#' for comments, as in

>>> instream = StringIO('this # some comment\nthat')
>>> lex = shlex.shlex(instream)
>>> while 1:
...  w = lex.get_token()
...  if not w:
...   break
...  print w
...
this
that
>>>

You can modify that operation, but my point is there are
many similar looking "quoted command-line" operations which
don't work identically.  Eg, are "quoted quotes ""doubled""
or \"backslash \"escaped".

                    Andrew
                    dalke at acm.org