Script Recorder: Engine scripting language changes

Overview

To enable the Engine to function effectively with the Script Recorder, several modifications were necessary at the Engine level. The Engine typically operates under certain assumptions that hinder the accurate reproduction of user behavior during recording with the Script Recorder.

Engine command updates

Relaxing current Engine commands

The following instances outline the need to modify existing Engine commands for improved functionality.

Unintended calls to Focus(…) command

The Engine integrated a Focus command into various standard UIAutomation commands, even when they typically do not utilize a Focus command. This unintentionally led to double-click actions being triggered in certain dropdowns, such as a File dropdown in a standard application. To mitigate this issue, we introduced the capability to override the Focus command as needed, and the ScriptRecorder will generate its code overwriting the focus command. Specifically, it will send the optional parameter 'forceFocus = false'. This effectively disables the focus injection, which is typically part of the traditional Engine functionality.

The following methods are affected:

  • Window.Type(…, bool forceFocus = true)
  • Window.Click(…, bool forceFocus = true)
  • Window.MoveMouseToCenter(…, bool forceFocus = true)
  • Window.MoveMouseToCenter(…, bool forceFocus = true)
  • Window.DoubleClick(…)
  • Window.RightClick(…)
  • MoveMouseToCenter(…, ForceFocus = true)
  • MoveMouseToRelativeCoordinates(…, ForceFocus = true)

All of these methods previously assumed that a Focus command would be executed as part of the method call. Now, we can override this behavior by passing 'forceFocus = false' for all of these commands. To ensure backward compatibility, if no value is passed for 'forceFocus', or if 'true' is explicitly given, the old behavior will be executed instead.

New commands

Method for locating UI elements

We've introduced a new method to streamline the development process of the Script Recorder by adopting standard practices for finding UI elements. This approach enables us to efficiently locate individual elements.

Specifically, we introduced the IWindow FindControlWithXPathName(string xpath, int timeoutInSeconds, bool continueOnError) as an alternative to the existing IWindow FindControl(string className, string title, int timeoutInSeconds, bool searchRecursively, bool continueOnError, string text)

The difference is that the FindControlWithXPathName method requires providing a full UIElement hierarchy to locate the target element. This entails considering all elements starting from the Window element that leads to the target element. Both methods for finding components should theoretically function in a script. This ensures compatibility with both legacy scripts and new implementations within the Engine.

Utilization of ShortClass

Previously, all elements were compared based on the UIAutomation Element's Class property. This property would typically prepend the FrameworkId to a Class, resulting in a format such as '{FrameworkId} {ControlType} : {Class}'. With the introduction of FindControlWithXPathName, a new ShortClass property is utilized, representing the format '{ControlType} : {Class}'.

Utilization of XPath element position

XRay previously utilized an index to select elements in the UI. Script Recorder introduces a similar construct in its XPath Definition, primarily serving as a tie-breaker mechanism for scenarios where two identical elements exist in a tree hierarchy. In other words, element selection will always occur when an element matches:

  1. Control Type, and
  2. either: a) Automation ID, or b) Class Name and Title, and
  3. Element Position (if multiple candidates happen to match).

An element position can be understood as an 'Index' that determines the ordering of a UI Automation element. Elements are retrieved using UI Automation's FindAll method with a scope limited to the immediate children of an element. For more details, see IUIAutomationElement::FindAll method (uiautomationclient.h).

New Start methodology

The Script Recorder introduces a new method for attaching the Engine to the configured target application, providing an alternative to the START method. This method is specifically designed for use by the Script Recorder to facilitate changes to the START logic without compromising backward compatibility.

A notable distinction of the StartApplication method is its avoidance of using a ProcessName to locate a window. With the introduction of Windows 11, relying on window identification solely by name could lead to the Engine erroneously latching onto temporary launcher windows, thus compromising the robustness of the script.

  • The method updates the window identification logic, enabling the Engine to search beyond top-layer windows. If the initial search fails, it traverses the entire window hierarchy, ensuring compatibility even when the target window is nested within another top-level window.
  • Furthermore, the window matching logic is refined to utilize the "shortClass" property for class name comparison instead of the current Class property. Consequently, the Xpath format no longer relies on the inclusion of the FrameworkId, adopting a simpler approach.

{FrameworkId} {TagName}:{ClassName}” format of Xpath.