Automated GUI Testing (AGT) software, version 0.1.0

By alvin ~ Posted Mon, 11/08/2021 - 13:17

As a brief note today, I just finished writing the first draft of my Automated GUI Testing (AGT) software with Scala 3 today. Really this is Version 0.1.0, but I just added some basic image recognition methods into it last night, and that was the final piece of what it needed to go live.

What’s good and unique about AGT?

Some of the best features of AGT are:

  • You can test any GUI with it: code that runs in browsers, desktop software, iOS/Android/mobile software (with their simulators)
  • By creating GUI tests you can avoid and/or find many of the bugs that are currently in Apple iOS apps, macOS apps, Firefox, Facebook, and many more
  • The image recognition is huge because it means you don’t have to depend on pixel coordinates (which is a fragile approach)
  • It can integrated with Selenium WebDriver
  • It’s free and open source software (OSS)

A few examples

As a first example, this is how you can activate and then use the macOS menu bar with AGT:

@main def macMenubarExample =

    activateMenuBar
    sleep(500)

    // the first arrow-down causes the Apple menu to be displayed
    arrowDown

    // now go down 3
    arrowDown
    arrowDown
    arrowDown

    // come back up 3
    arrowUp
    arrowUp
    arrowUp

    // go right at the top of the menu items
    arrowRight
    arrowRight

    // then go back left
    arrowLeft
    arrowLeft

    // esc closes the menu
    esc

That code isn’t a test in the form of a ScalaTest test, but it shows some basic AGT capabilities. (There currently are a few ScalaTest tests in the project source code, for the image recognition, and for the ability to execute system commands from AGT.)

Here’s an example of driving Google Chrome with AGT:

@main def chromeExample =

    // All of the `sleep` commands in here are for me, so I can watch the test.
    // They aren’t necessary for a completely-automated test.

    // start google chrome and see how many processes are running
    speak("Starting Chrome...")
    sleep(1_000)
    startApp("Google Chrome")
    waitForColor(Point(588, 649), Color(45, 45, 51), 10_000)
    printNumberOfChromeProcesses
    sleep(500)

    // go to google, check the processes
    speak("Going to Google.com")
    apple('l')
    ty("google.com\n")
    printNumberOfChromeProcesses
    sleep(500)

    // create a new tab, go to apple, check the processes
    speak("Going to Apple.com")
    apple('t')
    ty("apple.com\n")
    printNumberOfChromeProcesses
    sleep(500)

    // get the url from the url field
    waitForColorToGoAway(Point(484, 325), Color(45, 45, 51), 10_000)
    apple('l')  // focus in url field
    apple('a')  // select all
    apple('c')  // copy the url
    val url = getClipboardText
    "URL: $url".c
    sleep(1_000)

    speak("Closing the tabs")
    sleep(500)
    apple('w')
    apple('w')

    speak("Closing Chrome")
    sleep(500)
    apple('q')
    // killall("Google Chrome")

That example currently uses X/Y coordinates, but that’s because I wrote it before I wrote the image recognition software. Now it can be converted to look for on-screen images.

Finally, here’s an example of how to do some basic tests with the macOS/iOS Simulator:

@main def iPhoneSimulatorExample =

    "starting the iPhone simulator ...".c
    startIphoneSimulator
    waitForAppleHealthIcon
    waitClickAppleNewsIcon
    waitForAppleLogoInNews

    spinNewsFeedUp
    sleep(1_500)  // wait so i can see it

    goBackToHomeScreen
    waitForAppleHealthIcon
    sleep(500)    // this is for me as i watch it

    "quit the simulator".c
    pressAndReleaseKeys(Seq(VK_META, VK_Q))
    sleep(500)    // again for me

    writeComments

Again that code is a little dated, but I believe you can see from the code what it does.

Summary

I’ll write more about AGT and automated GUI testing in the future, but for today I just wanted to announce that Version 0.1.0 of the software is now available. If you’re interested, you can find the source code repository here:

All the best,
Alvin Alexander
November 8, 2021

Valley Programming is currently a one-person business, owned and operated by Alvin Alexander. If you’re interested in anything you read here, feel free to contact me at “al” at (“@”) this website name (“valleyprogramming.com”), or at the phone number shown below. I’m just getting back to business here in November, 2021, and eventually I’ll get a contact form set up here, but until then, I hope that works.