[Tutor] OT: How to automate user interactions with GUI elements of closed-source programs?

[Alan Gauld]

On 24/12/15 14:54, boB Stepp wrote:

> publicly accessible API.  Are there ways to programmatically
> accurately click the right buttons (or check boxes, radio buttons,
> etc.), copy desired fields and then switch to another program and
> paste the desired information into the desired fields, accurately
> clicking all things that need to be clicked, etc.?  This is mostly a
> Windows-based scenario, but if the techniques (if they exist) can be
> abstracted to any OS I can find plenty of uses elsewhere as well!


To quote my recent book:
--------------------
The final option for GUI applications with no API is to interact with
the GUI itself by sending user event messages into the application. Such
events could include key-presses, mouse-clicks, and so forth. This
technique is known as robotics because you are simulating a human user
from your Python program. It is really an extension of the native code
access described in the previous section, but operating at a much
lower level.

This is a frustrating technique that is very error prone and also very
vulnerable to changes in the application being controlled—for example,
if an upgrade changes the screen layout, your code will likely break.
Because of the difficulty of writing the code, as well as the fragility
of the solution, you should avoid this unless every other possibility
has failed.
---------------------

And again
--------------------
...you reviewed the options available for scripting with their pros and
cons, including the last resort option for GUIs of sending OS events to
the application windows. This last option is fraught with difficulty and
should only ever be used when all other means have been explored and
exhausted.


So, as you see I don't recommend it unless you are absolutely sure
nothing else is possible.

The other things to check are (On Windows):

1) File export/import (CSV, JSON or XML maybe? or a Windows app
   such as Word/Excel?))
2) COM object model access via PyWin32
3) A C DLL exposed via ctypes
4) A Web front end or web service

If you really must go down the screen scraping robotics route and
you are thinking of upgrading to Windows 10 make sure you do that
first or you will almost certainly have to rewrite from scratch
(and after most every other upgrade of OS or app thereafter).
And don't even think of changing your system fonts - ever!

If you had a Mac things are usually easier because you can often
use osascript as an interface to AppleScript but on X Windows
things are just as messy as for windows...


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos