Login Enterprise EUX score and VSImax

• Overview
• Why calculate the EUX score?
• Terms and related concepts
• How Login Enterprise gets data for EUX score
  • Determining EUX score
• EUX score
  • Weight calculation
  • EUX score calculation
  • Baseline and Steady state EUX scores
  • Representative score check
    • Changing EUX work folders
      • Why change the EUX work folder
      • How to change the EUX folder
• VSImax
  • Calculating VSImax
  • Determining the threshold
  • Trigger determination
  • Determining the VSImax
• 2021 EUX score Preview (before Login Enterprise 5.0)
• Additional resources

Overview

The EUX score (End User Experience Score) is a rating that represents the performance of a Windows machine (virtual, physical, cloud, or on-premises) on a scale of 0 to 10 as experienced by the Virtual User. The EUX score can be calculated with any number of users (1 or more).

As you add more users to your VDI platform, expect your EUX score to drop. As more users demand a greater share of a VDI system's shared resources, performance and user experience decrease.

The EUX score was introduced in Login Enterprise version 4.11 and is updated to reflect the evolving hardware and software platforms that VDI infrastructure runs on.

Why calculate the EUX score?

Calculating an EUX (End User Experience) score can provide several benefits in various contexts, helping ensure that the products, systems, and services you provide deliver a positive and satisfying user experience. Here are some reasons why calculating an EUX score might be valuable:

• Quantitative assessment: An EUX score allows you to quantify the overall user experience of a product, system, or service. This can provide a clear and measurable indication of how well it meets the needs and expectations of its users.
• Benchmarking: By establishing a baseline EUX score, you can track changes over time or compare different versions or iterations of a product. This enables you to assess the impact of design changes or improvements on the user experience.
• Identifying pain points: An EUX score can highlight specific areas of the user experience that need attention or improvement. By identifying pain points or areas of dissatisfaction, you can prioritize efforts to address them and enhance the overall user experience.
• Informing design decisions: Quantitative data from EUX scores can provide valuable insights for making informed design decisions. It can help designers and developers understand which aspects of the user experience are most important to users and where to focus their efforts for maximum impact.
• Demonstrating value: For businesses and organizations, an EUX score can serve as a metric for demonstrating the value of investing in user experience improvements. A positive EUX score can indicate that the product or service is meeting user needs effectively, which can contribute to customer satisfaction, loyalty, and ultimately, business success.

Terms and related concepts

• EUX Score - an End User Experience score, i.e. a specific set of tools that produce measurements meant to closely approximate how a human would feel about using this machine. In other words, a score from 0 to 10. For more details, see EUX Score.
• EUX App - an Application or tool designed to measure or optimize the respective aspects of the system’s performance. For more details, see How Login Enterprise gets data for EUX score.
• EUX Timer - the EUX Applications collect a bunch of data, which are referred to as “timer results”. Each piece of data has a corresponding “timer” associated with it. The “timer” or the “action” is an EUX application with specific parameters. For more details, see Determining EUX Score.
• VSImax - the maximum number of concurrent active VDI sessions your hardware will support without compromising responsiveness. For more details, see VSImax.

How Login Enterprise gets data for EUX score

The EUX score is based on the data derived from the performance of the EUX apps on the machine. EUX apps are small Applications or tools designed to measure or optimize the respective aspects of the system’s performance. To calculate the EUX score, Login Enterprise uses 4 EUX apps:

Appspeed

This Application is like a specialized tool on your computer, that tracks its performance. When you launch it, it sets itself up, prepares its window, and gets ready to interact with you. It keeps track of how quickly it’s able to start up and be ready for your input. Because the Application itself is very small, we simulate the load by doing a bunch of CPU operations.

In the default EUX profile for the timer appspeed we launch this app with these arguments:

"Arguments": "folder=\"{myDocs}\" duration=10000 launchtimestamp={launchTimestamp}"

Compressionspeed

The purpose of this program is to test the performance of a multi-threaded file compression and decompression system. It measures how many blocks of data can be compressed and decompressed within a specified duration.

In the default EUX profile, we use these parameters:

Timer highcompression:

"Arguments": "folder=\"{myDocs}\" cachePct=25 writePct=35 duration=1000 threads=1 -high"

Timer fastcompression:

"Arguments": "folder=\"{myDocs}\" cachePct=25 writePct=35 duration=1000 threads=1"

Starting Login Enterprise 5.8, you can change the default EUX work folders via the Public API, and therefore change the location for the EUX apps to another location.

For more information, see Changing EUX work folders.

CPUspeed

The purpose of this program is to measure the performance of a CPU by running a set of computations in multiple threads for a specified duration. It creates a specified number of threads and then initiates a CPU workload test on each of them. Then it counts how many iterations of the workload were performed within the given duration.

In the default EUX profile for the timer cpuspeed we launch this app with these arguments:

"Arguments": "d=1000 t=4"

Diskspeed

The purpose of this Application is to measure the performance of disk I/O operations by running a set of read/write operations in multiple threads for a specified duration. It creates a specified number of threads and then initiates a disk I/O workload test on each of them. The program measures the number of I/O operations per second (IOPS) and various latency metrics.

In the default EUX profile, we use these parameters:

Timer diskmydocs:

"Arguments": "folder=\"{myDocs}\" blockSize=5m bufferSize=32K writeMask=0x0C0C cachePct=95 latencyPct=99 threads=1 duration=1000"

Timer diskappdata:

"Arguments": "folder=\"{appData}\" blockSize=50k bufferSize=4K writeMask=0x0C0C cachePct=95 latencyPct=99 threads=1 duration=1000"

Determining EUX score

The EUX score comprises various timers representing common user operations that closely correlate with human experience on the system. These actions enable Login Enterprise to measure Application responsiveness, keyboard input processing, CPU-intensive tasks, and storage I/O latency.

In addition, the EUX score for Load Testing also incorporates Application and session failures, resulting in a score reduction when these events occur.

For more information on the 2023 EUX timers, see the following table.

EUX score

Weight calculation

Each timer has a Weight parameter that is defined in the tuning file. We only use the timers for EUX calculation that are defined in the tuning file and have weights higher than zero. The weight is used to give more importance to certain timers over others.

We also use another tuning parameter called PerformancePenalty. This is used to punish bad scores.

The logic for calculating the actual weight is:

• If the score is more than 7.5, we use the actual weight from the tuning file.
• If the score is lower than 7.5 we use the following formula: Correction = 1 + log(7.5 / score) * PerformancePenalty. Then the actual weight is ActualWeight = Weight * Correction
• Note: if the score is lower than 2, we still use 2 for the actual weight calculation

EUX score calculation

The calculation is the same for minute-by-minute or the total EUX score. It is done in three steps. For minute-by-minute calculation, we only take the results in the “bucket“ for the total EUX we take into account the results from the whole test.

• Multiply every timer score by its actual weight and sum all of those results together together.
• Sum all of the actual weights together
• Divide the weighted score sum from the sum of all weights.

In the end, the EUX score is the weighted average for all of the EUX timer scores. The weighted average is an average where each value has a specific weight or frequency assigned to it. This means that some values contribute more to the average than others.

We also do not calculate the EUX score if any required timers (with a weight higher than zero) are missing. This usually only affects the beginning of the test, and even one value per timer is enough. We do not have any checks to ensure the amount of values that should be available.

Baseline and Steady state EUX scores

The Baseline and Steady state EUX scores provide additional dimensions to the experience your Virtual Users are having. The standard EUX score provides a single score for the duration of the entire Test, including the login period and the Application interaction period during the test run. As more users are steadily added to the system being tested, naturally the system will work harder and start to impact the user experience. The Steady state and Baseline EUX scores show us what the user experience is like during specific periods of the Test run.

Baseline EUX score

The Baseline EUX score represents the best possible performance of the system and is the average EUX score of the best 5 minutes of the Test. This score indicates how the system performs when it is not under stress. Typically, the Baseline score is captured at the beginning of the Test before the system is fully loaded.

Steady state EUX score

The Steady state period represents the time after all users have logged in (login storm) and the system starts to normalize. The Steady State EUX score is the average of the EUX scores captured 5 minutes after all sessions are logged in, until the end of the Test.

Representative score check

The total score is checked to see if it is representative. This is determined by checking if all timers have at least 10 samples. If the total score is representative, it is used as the final score. Otherwise, the final score is set to 0.

Application failures

Currently, the Application failures, being the workload scripts or EUX Application (timer) failures do not have an impact on the EUX score. Workload Application failures had an impact on the EUX2021 implementation, and it was a percentage-based penalty, 80% success rate was translated to the 20% reduction in the EUX score.

Changing EUX work folders

Why change the EUX work folder

To ensure optimal performance and accuracy of EUX calculations, it may be necessary to configure the location of the EUX work folder. This helps avoid potential issues associated with using folders that are mapped to network drives or other locations unsuitable for EUX I/O operations. By selecting an appropriate location, sensitive directories are avoided, resulting in more realistic EUX and Max scores.

You can customize the properties of the EUX work folder when updating a test configuration for both the Load and Continuous tests.

Starting from Login Enterprise 5.8, you can update 3 EUX folders through the Public API. Note the following folders are default:

• EUX Folder location
  • "euxFolderPath": "%USERPROFILE%\AppData\Local\Temp"
• EUX Appdata location
  • "appDataFolderPath": "%USERPROFILE%\AppData\Local\eux2023"
• My Documents location
  • "myDocumentsFolderPath": "%USERPROFILE%\Documents\eux2023"

How to change the EUX folder

To change the EUX folder to a custom EUX folder in the Public API, you need to use a corresponding endpoint. To do this:

1. In your Login Enterprise, open the Swagger UI console with the v7.0-preview API. To learn how to do this, see Accessing Public API/Swagger UI (When done, go back to this page for further instructions).

2. In the Open API Specification, find Test, and click GET /v7-preview/tests{testId} Get test by id.

3. Perform a GET operation on the Test to get the current configuration of the Test. Make sure to leave “Include” set to "none”.

4. Copy the entire body from the GET output into your PUT request and modify your desired values.

5. Click Execute. The response status code should show 200:

6. Finally, go back to GET and verify the changes:

When using custom EUX folders, it's essential to consider several cautions to ensure the security, integrity, and efficiency of the process:

• During creation, EUX work folders cannot be set; any properties configured will default to null.
• For successful operation, custom EUX work folders must grant full access to virtual user accounts. By default, the first session creates the EUX folder and files, restricting access to subsequent sessions, and causing EUX errors. Access permissions should be carefully managed to avoid these issues.

VSImax

VSImax defines the maximum number of concurrent users that can operate with an acceptable EUX score. If a system exceeds the VSImax score, the End User Experience will progressively degrade.

The EUX performance input metrics were chosen to reflect the user experience. In addition to this, we carefully chose and tuned the timers to also reflect the load that a virtualization system experiences when increasing the number of sessions that run on that system.

VSImax is triggered when the user experience dips below a certain value. We determined two criteria for this threshold:

• An absolute value that indicates at which point the experience starts to degrade
• A relative value based on the performance of the system when it is running optimally

The reason for the absolute value is obvious. The second criterion needs more explanation.

We ran capacity planning experiments on several systems with different configurations. A recurring pattern was found in nearly all test results. While adding more sessions, at a certain point the performance starts to degrade very rapidly. Ideally, you want VSImax to be triggered before the point of that collapse, or just when it starts to happen. To determine this point, the second criterion proved to be more reliable. When correlating our results with the hypervisor’s internal metrics, we found that at around 85% of the initial EUX score, we are at the point where we are generating too much load. Even though the exact percentage did vary between systems, we found that 85% was a good general number to base VSImax.

Calculating VSImax

VSImax can be triggered by any individual timer or by EUX. In this section, you can see a simplified overview of how the VSImax is calculated in a simplified manner, and we try to explain every tuning parameter available that would impact VSImax determination. While EUX and timer both can trigger VSImax, it’s worth noting that EUX always has a score of 0.1 and 10, and timers have normalized scores ranging from 0 - 16000.

The VSImax score is calculated in two steps. The first step is to determine the threshold from which to compare the EUX as it changes over time. The second step is to determine the number of sessions that were running when we crossed that threshold.

Determining the Threshold

VSImax is usually determined as a certain percentage drop in the baseline. Think of the baseline as a starting point or reference for comparison. We consider the baseline as the best average score of several samples configured in numSamplesForBaseline. So, if the numSamplesForBaseline is 5, we wait until we have 5 values, and only then do we start to calculate the average, and we calculate it in a rolling average manner. We save the max average we get and consider that our baseline. The baseline is recalculated as the test goes on. Still, usually, the baseline only changes at the beginning of the test, because averages tend to drop as the test goes on and more load is generated.

Trigger Determination

Trigger is the condition that must be met for us to determine the VSImax. Trigger is defined as a tuning parameter CapacityTrigger. We have the following types of triggers:

• As a percentage of a baseline.
• Flat normalized value.

Also, the triggers can be expressed as:

• An upper limit. It’s when the value exceeds the threshold, and is noted with “>”.
• A lower limit. It’s when the value drops below the threshold, and is noted with “<“

Examples:

• “<80%” - VSImax is triggered when the value drops below what is 80% of its baseline.
• “<1500“ - VSImax is triggered when the normalized score of a timer drops below 1500.
• “<5.5“ - The current default trigger used to determine VSImax from the EUX score

It is worth noting that the upper limit (“>“) of a threshold seems to be defined in the code, currently, it is not used, probably because we don’t have a situation where the higher timer value indicates worse results.

We also use a tuning parameter that is called MaxBaselineForCapacity, which is currently set to 2000. This only affects timer results, but if our determined capacity is anything higher than 2000 (normalized value), we set 2000 as a baseline. Then we ordinarily calculate the trigger, it’s just that the trigger is calculated from 2000 and not the actual baseline.

Determining VSImax

Here, we use the tuning parameter CapacityRollingAverageSize. This is the number of the most recent minutes we look at when calculating an average. For example, if CapacityRollingAverageSize is 5, we look at the last 5 minutes of EUX scores or normalized values and calculate the average out of those values. This “rolling” or “moving” average gives us a smoother line that helps us see the overall trend, and removes minute-by-minute fluctuations.

We then check this rolling average against the trigger. If the rolling average crosses this trigger, our system has hit its maximum capacity under current conditions.
When the trigger is met, we note down the minute this happened and the number of sessions at that minute. This is known as our VSImax or a capacity score.

2021 EUX score preview (before Login Enterprise 5.0)

If you are an existing customer and have been using the 2021 EUX score (preview), here is some more information about what you can expect from our new 2023 EUX score. If you are interested in enabling the new 2023 EUX score on your Virtual Appliance, email us at support@loginvsi.com.

In 2021 we released our EUX score preview, which was widely adopted and used by our customers. Since the release of our EUX score, we have listened to your feedback and used it to help us build our roadmap for the future. Login Enterprise 4.11 includes much of that feedback, and more, in our new 2023 EUX score.

Because there may be some variation between the 2021 EUX score preview and the 2023 EUX score, we have kept the 2021 EUX score enabled in 4.11 but will provide you with ways to enable the new 2023 EUX score for you to compare with your existing results to see if there are any variations. While we cannot predict how it will be different for each customer, we do have some high-level guidance on what you can expect.

Systems with software security products are expected to receive a higher score in EUX2023. We found that the impact of several of these products on the EUX2021 score was too high when comparing it to real user-perceived performance

Systems with poorly performing storage solutions are expected to receive a lower score. We found that EUX2021 was too forgiving towards slow disk IO, resulting in higher scores than the real user’s perceived performance.

When running tests on a high-performance system with fast, low-latency storage and a system that does not run many background services, you can expect higher scores.

As CPU resource contention increases the new score will start to drop faster.

Additional resources

To learn how to export the EUX raw data, see Exporting EUX data.

For any questions or concerns related to EUX score or VSImax, feel free to reach out to our support team at support@loginvsi.com.