Login VSI VSImax

Login VSI takes a different approach. Login VSI is not primarily designed to be a steady state benchmark (however, if needed, Login VSI can act like one). Login VSI was designed to perform benchmarks for SBC or VDI workloads through system saturation. Login VSI loads the system with simulated user workloads using well known desktop applications like Microsoft Office, Internet Explorer and Adobe PDF Reader. By gradually increasing the amount of simulated users, the system will eventually be saturated. Once the system is saturated, the response time of the applications will increase significantly. This latency in application response times gives a clear indication whether the system is (close to being) overloaded. As a result, by nearly overloading a system it is possible to find out what is the true maximum user capacity.

After a test is performed, the response times can be analyzed to calculate the maximum active session/desktop capacity. Within Login VSI, this is calculated as VSImax. When the system is getting closer to its saturation point, response times will rise. When reviewing the average response time it will be clear that the response times escalate at the saturation point.

This VSImax is the “Virtual Session Index (VSI)”. With Virtual Desktop Infrastructure (VDI) and Terminal Services (RDS) workloads this is valuable information. This index simplifies comparisons and makes it possible to understand the true impact of configuration changes on a hypervisor host or at guest level.

Recommended: Calculating Maximum Virtual Desktop Capacity – VSImax Explained

Server side response time measurements

It is important to understand why specific Login VSI design choices have been made. An important design choice is to execute the workload directly on the target system within the session instead of using remote sessions. The scripts simulating the workloads are performed by an engine that executes workload scripts on every target system, and are initiated at logon within the simulated user’s desktop session context.

An alternative to the Login VSI method would be to generate user actions on the client side via a remoting protocol. These methods are always specific to a product and rely on its vendors. More importantly, some protocols simply do not have a method to script user actions on the client side.

For Login VSI, the choice has been made to execute the scripts on the server side. This is the only practical and platform independent solution for a benchmark, like Login VSI. The relative overhead and footprint of a benchmark engine scripted in AutoIt is small enough (1-5% range) for Login VSI’s purposes.

Calculating VSImax v4.1.x

The simulated desktop workload is scripted in a 48-minute loop where a simulated Login VSI user is logged on and performing generic office worker activities. After the loop is finished, it will restart automatically. Within each loop the response times of ten specific operations are measured at a regular interval (16 times within each loop), five of which are the most important. The response times of these important operations are used to determine VSImax.

The five important operations are:

• Notepad File Open (NFO): Loading and initiating VSINotepad.exe and opening the openfile dialog. This operation is handled by the OS and by the executable itself through execution (this operation seems almost instant from an end-user point of view).

• Notepad Start Load (NSLD): Loading and initiating VSINotepad.exe and opening a file. This operation is also handled by the OS and by the executable itself through execution (this operation seems almost instant from an end-user point of view).

• Zip High Compression (ZHC): This action copies a random file and compresses it (with 7-Zip) with high compression. The compression will very briefly spike both CPU and disk I/O.

• Zip Low Compression (ZLC): This action copies a random file and compresses it (with 7-Zip) with low compression. The compression will very briefly spike disk I/O while also creating some load on the CPU as well.

• CPU: Calculates a large array of random data and spikes the CPU for a short period of time.

Note: NFP (Notepad File Print) is no longer used to calculate VSImax due to the amount of printers configured in an environment. The amount of printers could influence this number. If you have 15 printers, it takes much longer to open the dialog compared to when you only have one printer configured.

These measured operations affect many different sub-systems, such as CPU (user and kernel), Memory, Disk, the OS (in general), the application itself, print, GDI, etc. These operations are specifically short by nature. When such operations become consistently long, the system becomes saturated because of excessive queuing on any kind of resource. As a result, the average response times will then escalate, which is clearly visible to end-users. If such operations consistently take longer than expected, the user will regard the system as slow and unresponsive.

VSImax Metrics

Once the test is finished, VSImax v4.1 will be calculated. Previous VSImax models (Classic and Dynamic) needed Microsoft Word to function. With the new 4.1 timers this is no longer needed, making it more flexible and applicable to a larger amount of scenarios.

The following actions are part of the calculation and are weighted as follows (US notation):

• Notepad File Open (NFO): 0.75
• Notepad Start Load (NSLD): 0.2
• Zip High Compression (ZHC): 0.125
• Zip Low Compression (ZLC): 0.2
• CPU: 0.75

This weighting is applied on the baseline and normal Login VSI response times.

VSImax Baseline

With the introduction of Login VSI 4.1, we also created a new method to calculate the basephase of an environment. With the new workloads (Taskworker, Powerworker, etc.) enabling basephase for a more reliable baseline has become obsolete. The calculation is explained below.

In total, 15 lowest VSI response time samples are taken from the entire test, the lowest 2 samples are removed and the 13 remaining samples are averaged. The result is the Baseline.

In short:

• Take the lowest 15 samples of the test
• Remove the lowest 2 from those 15 samples
• Average these 13 samples to calculate the Baseline number

VSImax Average

The VSImax average response time in Login VSI 4.1.x is calculated on the amount of active users that are logged on to the system. This calculation always uses the average of five Login VSI response time samples + 40% of the amount of active sessions.

For example, if the active sessions = 60, then latest 5 + 24 (40% of 60) = 29 response time measurements are used for the average calculation.

To remove noise (accidental spikes) from the calculation, the top 5% and bottom 5% of the VSI response time samples are removed from the average calculation, with a minimum of one top and one bottom sample. As a result, with 60 active users, the last 29 VSI response time sample are taken. From those 29 samples, the top two samples are removed and lowest two results are removed (5% of 29 = 1.45, rounded to 2). At 60 users, the average is then calculated over the 27 remaining results.

VSImax Threshold

VSImax v4.1.x is reached when the VSIbase + a 1000ms latency threshold is not reached by the average VSI response time result. Depending on the tested system, VSImax response times can grow 2-3x the baseline average. In end-user computing, a 3x increase in response time compared to the baseline is typically regarded as the maximum performance degradation to be considered acceptable.

In VSImax v4.1.x, this latency threshold is fixed to 1000ms, which allows better and fairer comparisons between two different systems, especially when they have different baseline results. Ultimately, in VSImax v4.1.x, the performance of the system is not decided by the total average response time, but by the latency is has under load. For all systems, this is now 1000ms (weighted).

The threshold for the total response time = average weighted baseline response time + 1000ms.

When the system has a weighted baseline response time average of 1500ms, the maximum average response time may not be greater than 2500ms (1500 + 1000). If the average baseline is 3000, the maximum average response time may not be greater than 4000ms (3000 + 1000).

When the threshold is not exceeded by the average VSI response time during the test, VSImax is not hit and the amount of sessions ran successfully. This approach is fundamentally different in comparison to previous VSImax methods, as it was always required to saturate the system beyond VSImax threshold.

Additonal Info

VSImax v4.1.x is now always reported with the average baseline VSI response time result. For example: The VSImax was 125 with a baseline of 1526ms. This helps considerably in the comparison of systems and gives a more complete understanding of the system. The baseline performance helps to understand the best performance the system can give to an individual user. VSImax indicates what the total user capacity is for the system. These two are not automatically connected or related.

For example, when a server with a very fast dual core CPU, running at 3.6 GHz, is compared to a 10 core CPU, running at 2.26 GHz, the dual core machine will give an individual user better performance than the 10 core machine. This is indicated by the baseline VSI response time. The lower this score is, the better performance an individual user can expect.

However, the server with the slower 10 core CPU will easily have a larger capacity than the faster dual core system. This is indicated by VSImax v4.1.x, and the higher the VSImax is, the larger overall user capacity can be expected.

With Login VSI 4.1.x a new VSImax method is introduced: VSImax v4.1. This methodology gives much better insight into system performance and scales to extremely large systems.

Login VSI Timer

The VSI_Timer41 function measures the responsiveness of the system. It looks at the response of the system at multiple points within each workload. Within the Login VSI 4.1 timer, the timer mechanism has been made more reliable by removing erratic timers and adding more comprehensive timers.

For example: If we were to start an application, we look at what kind of performance we can expect at that moment in time, e.g. how long does it take to start the application or open the document.

Timer Metrics

VSImax v4.1 shows the amount of sessions that can be active on a system before the system is saturated. This number gives you an indication of the scalability of the environment (higher is better). VSImax v4.1 is determined by the measurements in the below table:

'*'Compression is measured with 7-Zip

Other VSI 4.1 related measurements helping determine environmental issues (resources).

Note: The measurements in the table below are not used for the calculation of VSImax v4.1.

Other VSI related measurements helping determine environmental issues (resources).

Note: The measurements in the table below are not used for the calculation of VSImax v4.1.

Other VSI related measurements helping determine environmental issues (resources).

Note: The measurements in the table below are not used for the calculation of VSImax v4.1.