[Python-ideas] Better comand line version of python -c

[Russell Stewart]

I think if you actually used one of the tools in the space for a little
while (be it pythonpy or spy, which seems good, or the not so great pyp),
you may come to realize why your intuition for the user interface is so
flawed.

On Thu, Jan 8, 2015 at 8:27 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Jan 8, 2015, at 15:14, Russell Stewart <Russell.S.Stewart at gmail.com>
> wrote:
>
> [lots of unmarked snips below]
>
> Json handling:
> I think I would remove this if I did it again. I originally found it quite
> useful, but the fact that it doesn't support multi-line json inputs makes
> it weird. Also, all the functionality can be accessed through the json
> module with python (e.g. py -l 'json.loads(l)').
>
>
> But, as I pointed out, it would be pretty easy to make it handle a stream
> of multi-line JSON inputs by looping over raw_decode--and that would _not_
> be easy to access in a one-liner by the user. So with that change, it could
> be useful.
>
> -csv option:
> I considered this at length, and originally the --si was motivated by
> commas. But ultimately --si is good for lots of things other than commas. A
> separate csv command seems nice, until you realize that there are too many
> options to specify in parameters, just like in the json case.
>
>
> I don't understand what you mean here. In the JSON case, there aren't any
> options to specify at all. In the CSV case, there are lots of dialect
> options, but most of them are rarely used; being able to specify a
> delimiter (which you already need for --si) or a named dialect covers 90%
> of what people need.
>
> py vs python -m pythonpy ...:
> Yes, using this with python -m is a very real option. People could simply
> alias py='python -m pythonpy' to avoid typing issues. This is probably the
> most straightforward distribution option. It would be great to keep the
> ipython style tab completion, but I'm not sure that its necessary.
>
>
> I don't understand what you mean about tab completion. If you're talking
> about having bash (or zsh or whatever) do IPython-style tab completion in
> the expression, you can write a completion script that remembers that sets
> a variable when it sees "-m pythonpy" and completes non-option arguments
> with your Python completion instead of with files if that variable is set.
>
> The only trick with the alias is that you have to pass Python's arguments
> before the -m and pythonpy's after it… but I suspect that most people will
> never need Python arguments, and those who do will need them often enough
> that they'll just set up a separate alias for `python -u -m pythonpy`.
>
> py -fx flag:
> Originally, I didn't want to name this -f because I didn't want the active
> variable x to be magically introduced. When you type -fx, its clear where
> the variable x comes from. This was probably a mistake, because everyone
> hates 2 character single-dash flags.
>
>
> It's not that everyone hates 2 character single-dash flags, it's that
> everyone hates inconsistency. Your script is _almost_ compatible with the
> GNU standard (single-char single-dash arguments and multi-char double-dash
> arguments), but in a few places (-fx instead of --fx, --i instead of -i)
> you violate it. Why not just follow it consistently?
>
> Should be changed in the case of an overhaul. This command is also only
> marginally useful, due to the fact that grep already handles this use case
> so well, and because you can get the functionality with py -x 'x if foo(x)
> else None' just as well.
>
>
> Does that mean your script doesn't print out None values? I can see how
> that's useful in some cases, but it could be pretty bad--and surprising--in
> others.
>
> pythonpy should be Iterable based, not list-based:
> I agree somewhat. The current dictionary functionality is weird, only
> printing the list of keys. Again, I don't want to change this for backwards
> compat reasons. This could be done a lot of different ways, and I think the
> current strategy is pretty good, but leaves a little bit to be desired in
> terms of simplicity. The -x and -l flags capture a lot of the list-printing
> that you would want to do, so its not clear you need to do list-printing
> with no arguments. Using a separate flag for list printing would be a good
> design consideration here. Keep in mind that the current motivation for all
> the list strangeness is so that pythonpy will play well with unix pipes,
> which it does. That is really important for some core usecases.
>
>
> I don't understand this paragraph. What does any of this have to do with
> just changing all the stuff that requires lists to require any iterable,
> and/or changing -l to give you an iterator instead of a list?
>
> In fact, the reason I want to be able to output any iterable--in
> particular, an iterator--line by line without having to convert to a list
> is to use it with standard Unix pipelines. If I have a huge input file to
> filter, I'd rather use, say, a generator expression and have it read and
> output lines one at a time than have to process the whole thing into a list
> in memory before outputting anything. (Yes, I realize that -x covers many
> of the same cases--but just as map doesn't make genexprs useless in
> scripts, I don't think -x would make them useless in one-liners.)
>
> Daemon mode:
> I hadn't thought of this one before. Maybe it could be done well, but
> daemon modes tend incur a high cost of mental overhead on the user.
>
>
> If designed right, there really isn't any overhead as long as either (a)
> the user doesn't mind leaving it running forever, or (b) there's a timeout
> in case you forget to kill it. Then I'd just alias `py` to `python -m
> pythonpy --daemon=300` and everything would work exactly the same as
> before, except that it would start faster, and I could use variables set by
> `-c`/`-C` or the previous result (as `_`) if I wanted to.
>
> I've worked hard to keep the startup time within a factor of 2 from the
> python interpreter, so I do care a lot about that.
>
>
> I know "Windows users don't care about the command line"--but Cygwin users
> do, and they still have to deal with the Windows process launch delay, and
> there are some Unixes that people still use that can't launch processes as
> fast as Linux or BSD.
>
> --i flag:
> Again, the naming here is up for change. It's useful in a few cases, where
> python throws exceptions easily (e.g. py -x re.match(r"b", x).group(0)).
> But I haven't used it all that much, and it really could be removed.
>
>
> I wasn't arguing that it was useless--on the contrary, I think I'd use it
> often. (Although again, I'd also like a "short exception output" option.) I
> was just suggesting that it should be `-i` rather than `--i` (both to stick
> with the standard you're almost following, and to save a keystroke on
> something I think would be useful often, but not often enough to make a new
> alias for...).
>
> So in general I like the python -m idea. Support for tab completion under
> this would be really nice. I've long thought that python -c and python -m
> should have ipython style tab completion anyways,
>
>
> On my system, `python -m` does have tab completion (although it's not as
> smart as it could be--e.g., even with -s it still searches user site for
> completions). I'm not sure what package this comes from, but it's pretty
> trivial.
>
> Making `python -c` do completion for Python statements is cool, it just
> requires someone writing the completer. It sounds like you've already
> written one for pythonpy; it's just a matter of modifying it to handle
> statements instead of expressions and hooking it to the `-c` argument of
> your python completion. That seems like it would be a useful thing in its
> own right (although I'm not sure the bash-completions and zsh people would
> take it upstream, so it might have to be a separate package).
>
> so that one could type python -m Sim<tab> and get python -m
> SimpleHTTPServer. Another idea would be to try and get a shorter binary
> name distributed with python. It sounds superficial, but is ultimately
> quite important if python wants to compete with ruby, perl, and sed.
>
>
> I really don't think that's important. The reason people use perl and sed
> isn't that the command is 2 or 3 characters shorter (if it were, they'd
> just create an alias), it's that many trivial one-liners are more verbose
> in Python. For example, consider looping over each line in each filename
> passed on the command line. In perl that's a few characters; in python
> (even pythonpy) you have to loop over fileinput.input to do the same. And
> now imagine that what you want to do with each line is an re.sub; in
> Python, just the quoting is longer than the whole thing in Perl. In
> general, this isn't a weakness (it's what makes Python code readable), but
> if you're really interested in "competing with Perl" on its own terms,
> those are the parts you have to compete with, not the name of the command.
>
>
> On Thu, Jan 8, 2015 at 2:35 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
>
>> On Thursday, January 8, 2015 1:11 PM, Russell Stewart <
>> Russell.S.Stewart at gmail.com> wrote:
>>
>>
>> >I am the original dev for pypi.python.org/pypi/pythonpy, which offers
>> much of the same functionality as python -c, but in a more convenient form.
>> I'd be really interested in figuring out whether the core python tools
>> could support many of the use cases that pythonpy covers, so that users
>> wouldn't have to install it as a separate tool. I don't have a concrete
>> idea of what it would look like, but many of my users have said they would
>> much prefer an officially supported tool. Perhaps someone on this mailing
>> list can provide some insight into the available options.
>>
>> This looks pretty cool to me—so I have lots of questions and comments. :)
>>
>> The obvious problem is what to call it. `py` (well, `py.exe`) is already
>> the name of the PEP 397 launcher tool (the thing that lets you use shbang
>> lines on Windows), so this would need a new name.
>>
>> Maybe it would be better as just a different flag or set of flags on
>> `python` itself, like `--pprint`/`-p`? Of course then there's the problem
>> of where to stick the flags for `-p` (for GNU-style long names this is
>> pretty easy: `--print=filter,lines`, but obviously for something meant to
>> be used for quick&dirty command-line usage you need short names as well…).
>> And the "backport" to 3.4 (and 2.x) would act differently from the standard
>> 3.5+ version. But it might be worth looking at anyway.
>>
>>
>> Or, of course, just make this a stdlib module, so it's just `python
>> [PYTHON OPTIONS] -m YOUR_MODULENAME [YOUR MODULE OPTIONS]`, and you can
>> alias that to whatever you want.
>>
>> Is `-fx` a multi-character single-hyphen flag (in which case it's very
>> weird to mix those with GNU-style long arguments in the same program, or is
>> it a combination of `-f` and `-x`? And is the filter expression an argument
>> to `-fx`, or does `-f` just change the interpretation of the argument?
>>
>>
>> Speaking of GNU-style long arguments, why `--i` instead of just `-i` (as
>> a short name for `--ignore_exceptions` or something)
>>
>> The documentation is pretty Python 2-specific: the majority of the
>> examples use the `range` function and depend on it returning a list.
>>
>>
>> You seem to have built a non-trivial custom pretty-printer for this tool
>> (e.g., to print lists row by row) as well as an auto-importer (e.g., to use
>> `collections.Counter` without an `import collections`); maybe some or all
>> of that should be separated out and made available to Python code somewhere
>> in the stdlib, and then the script (or flag) could just use that function?
>>
>> This seems to be pretty strongly list-based for no good reason. Why not
>> print _any_ iterable line by line, make -l set `l = sys.stdin` instead of
>> `l = list(sys.stdin)` (I realize that doesn't work for the specific example
>> of `py -l 'l[::-1]'`, but `py -l reversed(l)` would work just as well in
>> that case, and it's hard to think of other examples where there would even
>> be a problem), etc.?
>>
>> It might be nice to have an option to see the `repr` instead of the `str`
>> of everything, to match what you'd see from the interactive interpreter.
>>
>> While --si is cool, people already misuse `str.split` and `re.split` to
>> try to parse CSV and similar input that has quotes or escapes; it might be
>> nice to have a `--csv` mode to parse the input with `csv.reader` (and that
>> could still take an optional delimiter, of course; passing other dialect
>> flags to `reader` might be out of scope).
>>
>> A columnar reading mode might also be nice, since that's one of those
>> things that's novices have a hard time writing without statements.
>>
>> An option for pretty JSON output instead of compact JSON output might be
>> nice.
>>
>> And maybe an option to read multi-line JSON input—not just a single JSON
>> value (which is trivial with `json.loads(l)`, but, since JSON is
>> self-delimiting, still a stream of them, without the requirement of
>> one/line (e.g., by looping over `json.JSONDecoder.raw_decode`).
>>
>> I assume you're evaluating the expression with `eval`. Are you passing it
>> a custom locals and globals (so any internals of your script itself aren't
>> available), or not? I could see disadvantages of both (the former may
>> prevent some useful quick&dirty hacks; the latter could raise safety
>> concerns), but either way, it's probably worth documenting.
>>
>> From the summary help, it looks like you can apply both `-x` and `-fx`.
>> If you do that, do they run in a specific order, or in the order specified?
>> For that matter, can you apply multiple mappings and/or filterings? (If
>> not, it seems like that could be handy.)
>>
>> It might be useful to have some kind of "daemon mode", where you start up
>> an interpreter in the background and then pipe input to it. Then you can
>> pipe multiple things to the same interpreter session, both for persistence,
>> and to avoid the process startup cost on platforms where it matters (like
>> Windows). Maybe with an optional timeout, where the daemon stops a few
>> minutes after last use, so you don't have to remember to `py -d
>> sys.exit(0)` or similar. And maybe the magic `_` should work in daemon
>> mode, as it does in the interactive interpreter. (I could see writing some
>> one-liner that takes 90 seconds to run, then realizing I wanted it in JSON
>> format, so `py --daemon -jo _` would be handy to avoid re-running
>> everything…)
>>
>> It would be really cool if, when run under PowerShell, this could handle
>> scriptlet input and output instead of plain text (so, e.g., you pass it an
>> array of strings and it processes each string).
>>
>> Being able to somehow pass command-line arguments through to `python`
>> itself (assuming this isn't merged into the main interpreter, of course)
>> might be handy. In particular, I could see `-u` being useful, but there
>> might be others (including platform/implementation-specific `-X` options).
>> Of course that would come for free with the `-m` interface, but since I'm
>> not sure that's the best option...
>>
>> Instead of just being able to ignore exceptions, it might be nice to
>> enable one-line exception output (e.g., print just the type and message, no
>> traceback, and to stdout instead of stderr).
>>
>> With `-x`, an option to prefix each line of output with the corresponding
>> line of input could be handy, similar to the `-v` option to commands like
>> `cp`, `tar`, etc.
>>
>> A way to feed input files in could be handy to allow the kind of
>> one-liners people often fall back to perl for, although I'm not sure
>> exactly what you'd want this to look like.
>>
>> Some of these are probably way out of scope even for a third-party tool,
>> much less for a built-in stdlib version, but I'm not sure exactly which, so
>> I've just dumped everything on you to let you sort them out. :)
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/4b872ba7/attachment-0001.html>