- Overview
- Benefits and best practices
- How Continuous Testing works
- Creating a Continuous Test
- Configuring Continuous Test settings
- Starting the Test
- Locations
- Continuous Test results
Overview
Continuous Tests, also known as Continuous Monitoring, refer to a series of Tests that run consecutively to continuously monitor end-user experience. These Tests aim to promptly alert you to any issues in your infrastructure, preempting the need for users to report problems.
Benefits and best practices
Various scheduling options are available for Continuous Tests, including specifying operation hours during specific times of the day or days of the week, or allowing Tests to run continuously. Additionally, scheduling can involve setting a fixed interval between Test launches or specifying the number of concurrent Tests to run. It's important to note that scheduling Tests requires configuring a real Connector.
If you're using the Desktop Connector, which cannot initiate sessions, scheduling Tests at specific times or intervals won't be possible. Instead, use a Connector capable of initiating user logon for scheduling purposes.
The Continuous Tests in the Login Enterprise functions similarly to a task scheduler, allowing you to execute Tests at specific intervals. These Tests offer flexibility, with no practical limit to what they can accomplish. While the appliance is primarily designed as a measurement platform, it can be utilized for other purposes such as scheduling interactive actions.
It's important to note that when scheduling a Test, the appliance converts the specified time from your browser to Coordinated Universal Time (UTC) based on the current time zone. It does not retain your time zone information or adjust for daylight saving time. Consequently, if your schedule is closely aligned with your work hours, particularly in time zones observing daylight saving time, you may need to manually adjust for the discrepancy.
Additionally, the Desktop Connector provides an alternative method where you can manually trigger Test runs from a machine by invoking the LoginPI.Logon.exe with appropriate parameters. This functionality is particularly beneficial for scenarios where manual control over Test runs is required.
However, it's important to note that the Desktop Connector is not recommended for long-term production use when triggering Continuous Tests. Additionally, after a Continuous Test iteration completes for a Desktop Connector session, you will remain logged in.
For more information on the Connector options, see Connectors and Connection Configuration.
How Continuous Testing works
A Continuous Test records all timers defined in your Application scripts, along with session metrics from Windows and your connection broker. It can also execute our EUX Applications to gauge user experience on your hardware. Each metric can be charted over extended periods within your Test.
Continuous Tests initiate a session to log in as a specific user, execute the designated Test, and log out upon completion. The results from all executions are displayed as a single, continuous set, allowing you to monitor performance graphs over extended periods. You can configure email alerts to notify you when specific counters exceed their assigned threshold values within a specified interval, helping you identify performance degradation.
When viewing a Continuous Test in the Login Enterprise sidebar menu > Results > Continuous Testing, you'll initially encounter a chart displaying your metrics. This enables you to promptly assess the success of your Test and the consistency of your results. For more information, see Continuous Testing: Results.
Creating a Continuous Test
1. In the Sidebar menu, navigate to Configuration > Manage Tests, and select Continuous Tests.
2. In Continuous Tests, click “+“ on the top-right.
3. In the Info window, provide the following:
- Name: Specify the display name for the Continuous Test.
- Description: Optionally, provide an informative description of the Continuous Test.
- Environment: Optionally, select the Login Enterprise environment to associate with the Continuous Test.
- Connector: Choose the type of Connector to be utilized by the launcher.
Each Connector has its own settings. Once you define your Connector, specify the necessary information in the Settings section. Here’s an example of the Citrix StoreFront Connector configurations.
4. Click Save to apply the changes.
- For more information about connectors and connection configurations, see Connectors and Connection Configurations.
- For more information on the accounts, see Accounts.
- For the Launchers overview and best practices, see the Launchers: Overview and best practices.
- To learn about the Login Enterprise Windows Launcher setup, configuration, maintenance, and more, see the Login Enterprise Windows Launcher.
Configuring Continuous Test settings
Once you’ve created a Continuous Test, you need to configure the Test settings, such as:
a. EUX Score - Enable or disable the calculation of the End User Experience Score during Test runs. To learn more about the EUX Score, see the EUX Score and VSImax.
b. Custom screenshots - Enable or disable custom screenshots within Application scripts. For more information, see Taking Screenshots.
c. Verbose logging - Display the command line number that triggered an error during Application execution failures. This may affect Test results as the line number is returned during execution.
d. Logon time tracking process - Track the complete duration from when the Connector initiates the login process for the user session until the final user-session executable (usually Windows Explorer) is up and running. You can also overwrite the default process of time tracking with a process of your choice. For more information, see Measuring total login time.
e. Enable schedule - Configure the Test schedule, specifying whether it should run continuously, daily, or weekly. For more details, see Enabling scheduling.
Enabling scheduling
Continuous Tests, unlike other available Test types, offer various scheduling options:
- Login every: Launch a session every X minutes, utilizing Launchers in a round-robin pattern.
This option initiates a virtual user to log in, execute the Test, log off, and then repeat the cycle after waiting for the configured duration.
- Run a session from each Launcher every: Launch a session from each Launcher every X minutes, utilizing all Launchers in the specified Launcher Group.
This scheduling option launches one session from every configured launcher for the environment within the specified time frame.
- Run concurrent sessions: Launch X concurrent sessions, utilizing Launchers in a round-robin pattern.
With this option, users continuously log in, perform defined actions, log out, and immediately restart the process without waiting for a set time interval.
Daily and weekly schedules
You can implement detailed scheduling for Continuous Tests, choosing between daily or weekly schedules. The following examples illustrate setting up a Continuous Test schedule from 6 am to 5 pm on both a daily and weekly basis. This flexibility allows administrators to exclude scheduled downtime from the testing schedule.
Use your mouse to select the desired time frames within the scheduling interface. Each bar represents an hourly view, with each block representing 15 minutes. Remember to save your selections to confirm and apply your scheduling preferences.
Session Metrics settings
Within a Continuous Test, you can also configure Session Metric settings, such as:
f. Session Metrics - Enable or disable the gathering of Session metrics.
g. Metrics group - Choose the desired session metrics group to specify which performance counters should be collected.
h. Schedule (in which session) - If you don’t want to collect Session Metrics in every session, you can specify how often it can run: 1 out of every X sessions.
For more information on Session Metrics, see the Session Metrics.
Adding Applications to a Continuous Test
For the Continuous Test, you can add the preferred Applications for the Virtual User to test. You can also add Application groups and an option “to wait”. To do this:
In Actions, click the green “+“ to add a new action, and select what you’re going to add in the pop-up window.
After you've defined your workload, you can choose to have the Virtual User repeat the defined tasks before logging off. You can specify how many times the workload should be repeated using the 'Repeat all steps above' setting. To enable workload repetition, simply toggle the switch to turn it on, and then specify the number of iterations you'd like to execute before logging off.
You have the option to set Applications to either Leave Application running or Run once. When you toggle the 'Leave Application running' switch, the Application will remain open even after the workflow script is completed. Keep in mind that closing the Application later will require additional steps. Note that this feature does not apply to browser-based Application workflows.
If you're using the Repeat all steps above, you can enable the Run Once switch to ensure that the same Application is not launched in each iteration. This can be useful, for example, if an Application workflow is used as a preparatory step, such as 'Preparing Office'.
- For detailed information on Applications and Application configurations, see Applications.
- For Applications and Workload customizations, see Applications and Workload customizations.
Enabling and adding Notifications
One of the major advantages of Continuous Testing is that Virtual Users operate independently of any active real-user sessions. This allows them to effortlessly execute workloads, detect failures or performance degradation, and promptly and proactively alert IT Operations teams.
Within the Notifications tables, you have the option to define threshold times. When these thresholds are exceeded, alerting mechanisms are triggered. Simply toggle the switch to enable this feature. Once enabled, you can receive reports on Login Times and Latency. Furthermore, any Application failures are automatically alerted by default.
There are 4 Session notifications that you can enable:
- Login failure
- Login
- Latency
- Application failure
To add a new Notification:
1. Click the green “+” on the right.
2. In the Add a Notification, specify the following:
Every timer defined in any Application script you have included in this Test is available for Notification. You must add the Application script first before adding the Notification.
3. Click Save to apply the changes.
For more detailed information about the standard notifications and email configurations, see Notifications.
Unique email notifications for each threshold
Configuring unique email notifications for different thresholds allows you to specify distinct email recipients based on which threshold is breached. This is particularly useful for alerting specific stakeholders relevant to particular thresholds, providing more granular control over notifications compared to the previous global email notification settings. For more information, see the Email notification.
To configure this functionality:
1. Identify the threshold for which alerts should be sent to a specific subset of stakeholders.
2. In Settings for Email Alerting (Optional), use the radio buttons to select the desired email options.
- Use default email settings: The threshold will inherit email settings from the Global Email Notifications settings.
- Enter recipients manually: Input semi-colon-separated email addresses for those who should be notified when the conditions for alerting are met. This option overrides the global settings on a threshold-by-threshold basis.
3. Click Save to apply and enable the threshold.
After saving your settings, the new threshold configuration will be active, and email notifications will be sent according to the specified criteria. Ensure you review the alerting setup periodically to confirm it aligns with your stakeholders' needs and threshold requirements.
Citrix Published Apps
You can connect to a Published App using either the StoreFront or NetScaler connector. The key difference between connecting to a virtual desktop and a published Application is that the resource configured is an Application rather than a desktop. See the example below for more details.
Schedule configuration
Define the time schedule during which the virtual user is active.
Actions list configuration
You can configure Applications in the Actions list. It is recommended to set up Application scripts that correspond with the published apps. While you can add other Application scripts, they may not function correctly if they are not intended for the published Applications defined for that user. Ensure that each Application script includes "mainWindowTitle: *ApplicationTitle*" in the START parameter.
We support only one Continuous Testing environment with a single Published Application resource in the configuration.
How It works
When the Launcher initiates the session, it receives information from StoreFront or NetScaler indicating that the configured resource is a Published App. This information instructs the Virtual Appliance script executor not to start the Application as it would in a virtual machine (VM) or desktop environment. As a result, you can use existing scripts for published apps, such as the default Microsoft Word scripts.
Since the start command is skipped for Published Apps, the "App start time" value is not collected, as it no longer applies.
Access control settings
You can assign roles to Tests to control access to the Test configuration and results. By assigning roles, only users with the designated roles will be able to view or modify the Test configuration and results, depending on their permissions.
For detailed instructions on creating and managing roles, permissions, and dependencies, see the Role-Based Access Control.
Adjusting Engine start timeout
The setting for configuring the timeout for engine start is not available through the Login Enterprise web interface. To adjust this setting, you need to use the API.
You can do so by accessing the /publicApi/v7-preview/tests/{testId} endpoint and specifying the engineStartTimeout parameter.
The timeout for the Engine start setting controls how long the Virtual Appliance will wait for the Engine on the Target to come online after a session is launched, and before the connection is considered a failure (login failure).
Key considerations:
- Default timeout: If no value is specified, the timeout defaults to 300 seconds (5 minutes).
- Web interface limitation: This option is not available through the Appliance web interface. If you create a Test via the web interface, the timeout will be left empty and default to 300 seconds.
- Update via API: To modify the timeout, first create a Test, then use the update endpoint to set the timeout. The timeout cannot be set during Test creation.
How to set the timeout:
- Create a Test (timeout will be empty by default).
- Use the Update Test endpoint to set the desired timeout (in seconds).
For details on accessing the Public API in Login Enterprise, see Accessing the Public API.
Starting a Continuous Test
- In the Sidebar menu, navigate to Configuration > Manage Tests, and select Continuous Tests.
- In the Continuous Tests, switch the toggle next to the Test you’d like to start.
Enabling Continuous Tests in bulk
If you have multiple Continuous Test runs, you can enable or disable them in bulk. This way you’ll provide increased ease of management for larger Login Enterprise deployments.
To perform bulk start/stop operations, select the checkboxes of all Tests that should be considered, and click Enable all.
Locations
If Launchers are attached to configured locations when a Continuous Test type is in progress, these Launchers will be visible in the map view. For more details, see Locations.
Continuous Test results
- For information on the Continuous Test results see Continuous Testing: Results.
- For general guidelines on how to analyze Continuous Testing Session Metrics, see Viewing Session Metrics.
If you have questions or need additional information on specific Continuous Test configurations, feel free to get in touch with our support at support@loginvsi.com.