[Tutor] OT: How to automate user interactions with GUI elements of closed-source programs?

[boB Stepp]

On Thu, Dec 24, 2015 at 11:34 AM, Oscar Benjamin
<oscar.j.benjamin at gmail.com> wrote:
>
> On 24 Dec 2015 14:55, "boB Stepp" <robertvstepp at gmail.com> wrote:

[...]

>> I find myself doing the same boring, repetitive tasks by hand, which
>> amounts to copying certain information from one program and putting it
>> into other programs.  None of these programs, to my knowledge, have a
>> publicly accessible API.  Are there ways to programmatically
>> accurately click the right buttons (or check boxes, radio buttons,
>> etc.), copy desired fields and then switch to another program and
>> paste the desired information into the desired fields, accurately
>> clicking all things that need to be clicked, etc.?  This is mostly a
>> Windows-based scenario, but if the techniques (if they exist) can be
>> abstracted to any OS I can find plenty of uses elsewhere as well!

[...]

> ...I googled "Python windows control mouse" and got:
>
> http://stackoverflow.com/questions/1181464/controlling-mouse-with-python

> Also this:
> https://pypi.python.org/pypi/PyAutoGUI

My last of several unsatisfactory searches was "how to
programmatically click a button in python", which turned up a bunch of
web scraping results.  Sigh--very weak Google-fu indeed!  ~(:>)

Thanks, Oscar!  PyAutoGUI looks very promising.  It looks like my next
problem will be figuring out how to find the pixel coordinates of the
GUI elements which I wish to manipulate.  This package mentions
screenshot functions which may be able to locate these coordinates
programattically, but from the example given

>>> import pyautogui
>>> button7location = pyautogui.locateOnScreen('button.png') # returns (left, top, width, height) of matching region
>>> button7location
(1416, 562, 50, 41)
>>> buttonx, buttony = pyautogui.center(button7location)
>>> buttonx, buttony
(1441, 582)
>>> pyautogui.click(buttonx, buttony)  # clicks the center of where the button was found

it is not clear to me (Yet!) how "button7location" is mapped to the
actual button on-screen, so that I will know which button is being
identified.  But it looks like there is some good documentation which
will hopefully make things clear soon!

Merry Christmas!

-- 
boB