Manage Tests: Continuous Testing

Overview

Continuous tests, also known as Continuous Monitoring, refer to a series of tests that run consecutively to continuously monitor end-user experience. These tests aim to promptly alert you to any issues in your infrastructure, preempting the need for users to report problems.

Benefits and best practices

Various scheduling options are available for Continuous Tests, including specifying operation hours during specific times of the day or days of the week or allowing tests to run continuously. Additionally, scheduling can involve setting a fixed interval between test launches or specifying the number of concurrent tests to run. It's important to note that scheduling tests requires configuring a real Connector.

If you're using the Desktop Connector, which cannot initiate sessions, scheduling tests at specific times or intervals won't be possible. Instead, use a Connector capable of initiating user logon for scheduling purposes.

The Continuous Tests in the Login Enterprise functions similarly to a task scheduler, allowing you to execute tests at specific intervals. These tests offer flexibility, with no practical limit to what they can accomplish. While the appliance is primarily designed as a measurement platform, it can be utilized for other purposes such as scheduling interactive actions.

It's important to note that when scheduling a test, the appliance converts the specified time from your browser to Coordinated Universal Time (UTC) based on the current time zone. It does not retain your time zone information or adjust for daylight saving time. Consequently, if your schedule is closely aligned with your work hours, particularly in time zones observing daylight saving time, you may need to manually adjust for the discrepancy.

Additionally, the Desktop Connector provides an alternative method where you can manually trigger test runs from a machine by invoking the LoginPI.Logon.exe with appropriate parameters. This functionality is particularly beneficial for scenarios where manual control over test runs is required.

However, it's important to note that the Desktop Connector is not recommended for long-term production use when triggering Continuous Tests. Additionally, after a Continuous Test iteration completes for a Desktop Connector session, you will remain logged in.

For more information on the Connector options, see Connectors and Connection Configuration.

How Continous testing works

A Continuous Test records all timers defined in your application scripts, along with session metrics from Windows and your connection broker. It can also execute our EUX applications to gauge user experience on your hardware. Each metric can be charted over extended periods within your test.

Continuous Tests initiate a session to log in as a specific user, execute the designated test, and log out upon completion. The results from all executions are displayed as a single, continuous set, allowing you to monitor performance graphs over extended periods. You can configure email alerts to notify you when specific counters exceed their assigned threshold values within a specified interval, helping you identify performance degradation.

When viewing a Continuous Test in the Login Enterprise sidebar menu > Results > Continuous testing, you'll initially encounter a chart displaying your metrics. This enables you to promptly assess the success of your test and the consistency of your results.

Creating a Continuous Test

1. In the Sidebar menu, navigate to Configuration > Manage tests, and select Continuous tests.

Frame 120.png

2. In Continuous tests, click “+“ on the top-right.

Frame 121.png

3. In the Info window, provide the following:

  • Name: Specify the display name for the Continuous Test.
  • Description: Optionally, provide an informative description of the Continuous Test.
  • Environment: Optionally, select the Login Enterprise environment to associate with the Continuous Test.
  • Connector: Choose the type of Connector to be utilized by the launcher.

Frame 122.png

Each Connector has its own settings. Once you define your Connector, specify the necessary information in the Settings section. Here’s an example of the Citrix StoreFront Connector configurations.

Frame 123.png

4. Click Save to apply the changes.

Configuring a Continuous Test

Test settings

Once you’ve created a Continuous test, you need to configure the Test settings, such as:

a. EUX Score - Enable or disable the calculation of the End User Experience Score during test runs. To learn more about the EUX Score, see Login Enterprise EUX Score.

b. Custom screenshots - Enable or disable custom screenshots within application scripts. For more information, see Take Screenshot.

c. Verbose logging - Display the command line number that triggered an error during application execution failures. This may affect test results as the line number is returned during execution.

d. Logon time tracking process - Track the complete duration from when the Connector initiates the login process for the user session until the final user-session executable (usually Windows Explorer) is up and running. You can also overwrite the default process of time tracking with a process of your choice. For more information, see Measuring total login time.

e. Enable schedule - Configure the test schedule, specifying whether it should run continuously, daily, or weekly. For more details, see Enabling scheduling.

Frame 125.png

Enabling scheduling

Continuous Tests, unlike other available test types, offer various scheduling options:

  • Login every: Launch a session every X minutes, utilizing Launchers in a round-robin pattern.

This option initiates a virtual user to log in, execute the test, log off, and then repeat the cycle after waiting for the configured duration.

  • Run a session from each Launcher every: Launch a session from each Launcher every X minutes, utilizing all Launchers in the specified Launcher Group.

This scheduling option launches one session from every configured launcher for the environment within the specified time frame.

  • Run concurrent sessions: Launch X concurrent sessions, utilizing Launchers in a round-robin pattern.

With this option, users continuously log in, perform defined actions, log out, and immediately restart the process without waiting for a set time interval.

Frame 126.png

Daily and weekly schedules

You can implement detailed scheduling for Continuous Tests, choosing between daily or weekly schedules. The following examples illustrate setting up a Continuous Test schedule from 6 am to 5 pm on both a daily and weekly basis. This flexibility allows administrators to exclude scheduled downtime from the testing schedule.

Use your mouse to select the desired time frames within the scheduling interface. Each bar represents an hourly view, with each block representing 15 minutes. Remember to save your selections to confirm and apply your scheduling preferences.

Frame 127.png

Frame 128.png

Session metrics settings

Within a Continuous test, you can also configure Session Metric settings, such as:

f. Session Metrics - Enable or disable the gathering of Session metrics.

g. Metrics group - Choose the desired session metrics group to specify which performance counters should be collected.

h. Schedule (in which session) - Specify the schedule for sessions.

Frame 124.png

For more information on Session Metrics, see Session Metrics.

Adding Applications

For the Continuus test, you can add the preferred applications for the Virtual User to test. You can also add Application groups and an option “to wait”. To do this:

In Actions, click the green “+“ to add a new action, and select what you’re going to add in the pop-up window.

Frame 129 (1).png

After you've defined your workload, you can choose to have the Virtual User repeat the defined tasks before logging off. You can specify how many times the workload should be repeated using the 'Repeat all steps above' setting. To enable workload repetition, simply toggle the switch to turn it on, and then specify the number of iterations you'd like to execute before logging off.

You have the option to set applications to either Leave application running or Run once. When you toggle the 'Leave application running' switch, the application will remain open even after the workflow script is completed. Keep in mind that closing the application later will require additional steps. Note that this feature does not apply to Browser-based Application workflows.

Frame 130.png

If you're using the Repeat all steps above, you can enable the Run Once switch to ensure that the same application is not launched in each iteration. This can be useful, for example, if an Application workflow is used as a preparatory step, such as 'Preparing Office'.

Enabling and adding Notifications

One of the major advantages of Continuous Testing is that Virtual Users operate independently of any active real-user sessions. This allows them to effortlessly execute workloads, detect failures or performance degradation, and promptly and proactively alert IT Operations teams.

Within the Notifications tables, you have the option to define threshold times. When these thresholds are exceeded, alerting mechanisms are triggered. Simply toggle the switch to enable this feature. Once enabled, you can receive reports on Login Times and Latency. Furthermore, any application failures are automatically alerted by default.

There are 4 Session notifications that you can enable:

  • Login failure
  • Login
  • Latency
  • Application failure

Frame 131.png

To add a new Notification:

1. Click the green “+” on the right.

2. In the Add a Notification, specify the following:

Frame 132.png

3. Click Save to apply the changes.

For more detailed information about the standard notifications and email configurations, see Notifications.

Starting a Continuous Test

  1. In the Sidebar menu, navigate to Configuration > Manage tests, and select Continuous tests.
  2. In the Continuous tests, switch the toggle next to the test you’d like to start.

Frame 133.png

Enabling Continuous Tests in bulk

If you have multiple Continuous test runs, you can enable or disable them in bulk. This way you’ll provide increased ease of management for larger Login Enterprise deployments.

To perform bulk start/stop operations, select the checkboxes of all Tests that should be considered, and click Enable all.

Frame 134.png

If you have questions or need additional information on specific Continuous Test configurations, feel free to get in touch with our support at support@loginvsi.com