Table of Contents

- Benefits
- Tech Tips
- Creating a Custom Metric
   - Windows Performance Monitor (PerfMon)
      - Where to Find Performance Counter Details for your PerfMon Definition
      - How to Structure your PerfMon Definition
- Windows Management Instrumentation (WMI)
      - Where to Find Performance Counter Details for your WMI Definition
      - How to Structure your WMI Definition
- Create your Session Metric Definition
- Session Metric Groups
  - Adding Custom Metrics to a Group
- Adding Session Metrics Group to a Test
- Visualizing Your Metrics
- Examples

As of Login Enterprise 5.5, session metrics can now be displayed in charts for Load Tests and Continuous Tests. Administrators can choose to gather additional performance metrics from both Windows Performance Monitor (PerfMon) or Windows Management Instrumentation (WMI) to visualize alongside the EUX Score and Application metrics already captured. By default, Session Metrics are captured every 5 seconds when enabled. 

For Load Tests and Continuous Tests, a Session Metric Group can now be linked to specify the Performance Counters to gather. A Session Metric Group is a new entity that groups specified session metrics. The Group can contain metrics sourced from both PerfMon and WMI if desired, allowing you to gather a unique dataset without multiple panes of glass. 

By default, CPU & Memory Utilization percentages are captured and displayed. As of release 5.5, the Public API must be used to include custom session metrics. Before 5.4, only application response timers and the EUX Score were captured for Continuous and Load Tests.

Note: Attached to this article is a file containing examples of both WMI and PerfMon Session Metric definitions. These are provided as examples only. Login VSI has limited ability to recommend particular metrics for data collection. The customer is responsible for understanding and defining metrics for their specific implementations.

Benefits

Administrators can now enable Session Metric collection from target Windows VDI or PCs during a Load Test. It is now easier to review test results to correlate spikes in degraded EUX Scores or VSImax density with relevant Performance Counters like CPU, Memory, Disk Utilization and more. These Session Metrics may provide root cause analysis clues as to where, or why, the expected density or performance is not achieved. Specifically, this can be used to pinpoint which agent, apps, or processes are consuming large amounts of resources, impacting overall performance and scalability of the system. With a more complete picture of the workspace under stress, teams are now armed with data to make decisions, fine-tune optimization settings, and granularly measure the impact of changes to the workspace. When comparing two results, Session Metrics can identify granular changes to consumption of key performance indicators.

For Continuous Testing, administrators can now capture Session Metrics in real-time, 24x7, without the need for an additional agent. It is now easier to correlate spikes in performance indicators like CPU and Memory Utilization, with degradation in the EUX Score, Logon Times, or application response timers. Specifically, Session Metrics for targeted leading indicators can provide an early warning system to infrastructure degradation that can be supplemented with EUX Scores and Application Performance for further evidence of an issue.

With the addition of Session Metric capabilities, application-specific resource utilization can be measured across version updates and upgrades. WMI affords unique visibility into the usage of GPUs when performance testing graphics-intensive workloads to fine-tune configuration settings. There are also platform specific performance metrics, such as ICA/HDX and Blast protocol information, that can be correlated with Zoom and Teams workloads to measure video quality when sizing (e.g. Frames/sec).

Tech Tips

When enabling collection of session metrics, specifically using Custom Metrics Groups, be aware that resource consumption will increase for every additional Metric added to a group. This resource consumption may impact expectations for EUX Score and VSImax. For this reason, consider capturing a benchmark of your EUX and VSImax before enabling Session metrics. This will allow for determining the impact caused by collecting the desired Session Metrics.

When creating your WMI queries, be sure to specify the counters you want, to limit the overhead when performing the query. This is best practice to minimize the observer effect as WMI can have significant overhead if the query does not specify specific counters to retrieve. For this reason, where a particular metric is available through both sources, PerfMon is recommended due to its lower performance cost.

Creating a Custom Metric

This section will walk through creation of a session metric using both available sources, PerfMon and WMI. To add a custom Session Metric, you must use the v7-preview Public API. For more information on the Public API, see this article. These steps can most easily be performed using the Swagger API interface.

Windows Performance Monitor (PerfMon)

Where to Find Performance Counter Details for your PerfMon Definition

Windows Performance Monitor provides numerous performance counters valuable to analyze alongside Virtual User application workloads. This section will begin by outlining how to locate counters within PerfMon, and how to extract the relevant information for use in your Session Metric Definitions.
Start by launching PerfMon within the target workspace where Virtual Users will execute their workloads. This ensures that desired counters will be available.

In this example, the percentage of idle time (LogicalDisk\% Idle Time\_Total) for the LogicalDisk is selected. The value outlined in red indicates the “counterCategory,” the value outlined in yellow indicates the “counterName” and the value outlined in purple indicates the “counterinstance”. This information can be extracted for any available PerfMon counter, so take note of these values specific to the counters you wish to collect with Login Enterprise.

How to Structure your PerfMon Definition

The following Session Metric Definition is based on the performance counter displayed above. Use this as an analog for other performance counters you may wish to collect during your workloads.

Note: there was a change in schema between Login Enterprise 5.4 and 5.5, so ensure you select the proper format specific to your version.

// 5.5 schema
{
   "type": "PerformanceCounter",
   "measurement": {
      "counterCategory": "LogicalDisk", 
      "counterName": "% Idle Time", 
      "counterInstance": "_Total", 
      "displayName": "Disk Idle time",
      "unit": "%"
   },
   "tag": ""
}

// 5.4 schema
{
 "type": "PerformanceCounterDefinition",
 "counterCategory": "LogicalDisk", 
 "counterName": "% Idle Time", 
 "counterInstance": "_Total", 
 "displayName": "Disk Idle time",
 "unit": "%"
 "tag": ""
}

• type: to collect metrics from PerfMon, use the type: "PerformanceCounter"
• counterCategory: the category (object) of the desired Performance Counter
• counterName: the name of the desired Performance Counter
• counterInstance: (optional) the instance of the desired Performance Counter, if applicable 
• displayName: the title of the Session Metric to be displayed in charts and reports
• unit: the unit of the desired Performance Counter. Note, this value will be shown as the the y-axis label for charts displaying the Metric, and only metrics of the same unit can be shown simultaneously
• tag: (optional) this will be used for sorting, filtering, and searching in future versions

Please use the above process to identify the counter's category, name, and instance from within PerfMon before extracting the relevant information into a JSON body in the schema shown above.

Windows Management Instrumentation (WMI) 

Where to Find Performance Counter Details for your WMI Definition

Windows Management Instrumentation (WMI) provides a powerful way to extract performance data within the Windows OS. WMI exposes more specific and granular information than PerfMon, which may dictate its usage over PerfMon in certain scenarios. These steps will be centered on capturing the average Disk Writes/sec and Disk Reads/sec, but following the process shown and replacing values with those that are relevant to your use case will simplify the process. 

To identify the available WMI Performance Counters, you should first gather the available Counter classes. Start by launching PowerShell within the target workspace where Virtual Users will execute their workloads. This ensures that desired counters will be available.

The following command can be used to yield all of the available WMI classes that can be queried:

Get-WmiObject Win32_PerfFormattedData | Select-Object -Unique __Class | Format-Table -AutoSize

Among others, some of the more notable classes listed are:

Win32_PerfFormattedData_PerfOS_Processor
Win32_PerfFormattedData_PerfOS_Memory
Win32_PerfFormattedData_PerfDisk_LogicalDisk
Win32_PerfFormattedData_GPUPerformanceCounters_GPUEngine
Win32_PerfFormattedData_PerfOS_System
Win32_PerfFormattedData_PerfProc_Process
Win32_PerfFormattedData_Tcpip_NetworkAdapter
Win32_PerfFormattedData_Counters_VMwareBlastImagingCounters (For VMware Horizon only)

Now that you know which WMI Class you are looking to query from, you can gather the Class' specific counters by executing the following PowerShell command. This example will output all available performance counters pertaining to the Physical Disk. Replace "Win32_PerfRawData_PerfDisk_PhysicalDisk" with any available class returned from the command provided above.

Get-WmiObject -Query "SELECT * FROM Win32_PerfRawData_PerfDisk_PhysicalDisk"

Without specifying the query further, this output will list all available metrics for all available instances of the specified class. This provides more details to specify the query to return only the counter(s) you desire gathering. 

As mentioned above, this output will yield all of the available counters for the specified WMI Class (i.e., Win32_PerfRawData_PerfDisk_PhysicalDisk).  In this example, we will focus on gathering the DiskWritesPerSec and the DiskReadsPerSec. To do this, we can specify the specific counters we'd like to capture within our query:

Get-WmiObject -Query "SELECT DiskWritesPerSec, DiskReadsPerSec FROM Win32_PerfRawData_PerfDisk_PhysicalDisk"

How to Structure your WMI Definition

Now that a WMI Class has been identified, and specific performance counters within the class have been selected, the Session Metric Definition can be created. The below JSON example can be used to collect the above WMI performance counters.

As shown in the example output above, many WMI Classes will have multiple instances. In this example, the "Win32_PerfRawData_PerfDisk_PhysicalDisk" class has two instances: "C:" and "_Total". Look out for situations where you may want a specific instance rather than the aggregate of all instances. In this example, because we are not specifying a particular instance, the "summarizeOperation" is used to perform aggregation of all the available instances: taking the average "DiskWritesPerSec" and "DiskReadsPerSec" from all available instances. 

{
   "type": "WmiQuery",
   "wmiQuery": "SELECT DiskWritesPerSec, DiskReadsPerSec FROM Win32_PerfRawData_PerfDisk_PhysicalDisk",
   "namespace": "root\\cimv2",
   "instanceField": "Name",
   "measurements": [
      {
         "displayName": "Avg Writes Per Sec",
         "unit": "OPS",
         "propertyName": "DiskWritesPerSec",
         "summarizeOperation": "avg",  
      },  
      {
         "displayName": "Avg Reads Per Sec",  
         "unit": "OPS",
         "propertyName": "DiskReadsPerSec",
         "summarizeOperation": "avg"
      } 
   ],
"tag": "diskUtilization"
}

• type: to collect metrics from WMI, use the type: "WmiQuery"
• wmiQuery: the query to perform to retrieve desired metric (i.e. in the form SELECT, FROM, WHERE)
• nameSpace: (optional) to define the namespace (i.e. the database against which the query should be made). When left blank, the default namespace, "root\\cimv2" is used.
• instanceField: the name of the field that describes the instance; NOT the name of an instance (i.e. "_Total", but the name of the column that lists the instances, i.e. "Name" of the instance)
• measurements: to define the properties in the SELECT statement of the query
  • propertyName: the name of the property that will be defined in this section (from SELECT statement, each property from the query should have its own measurement). 
  • summarizeOperation: to manage aggregation of results with multiple instances (e.g. min, max, avg., sum, none). If none is chosen as the summarizeOperation, up to 16 instances will be returned in a random order.
  • displayName: the name of the metric that will be displayed in charts and user interface
  • unit: the unit of the desired WMI metric. Note, this value will be shown as the the y-axis label for charts displaying the Metric, and only metrics of the same unit can be shown simultaneously
• tag: (optional) this will be used for sorting, filtering, and searching in future versions

Please use the above process to identify the WMI counters' category, class, name and instance before extracting the relevant information into a JSON body in the schema shown above.

Import your Session Metric Definition

The above sections described how to structure the JSON request needed to import a custom Session Metric, sourced from Windows Performance Monitor (PerfMon) or Management Instrumentation (WMI). This section will guide the process of creating metrics available for use within Login Enterprise.

Note: A PerfMon type Session Metric is used in the below example. The same process should be used when creating WMI Session Metrics, using a JSON request body similar to the example above. 

In this example, the Swagger API interface is used to add the Disk % Idle Time as a Custom Metric. Note, "%" is the unit, which will be shown on all charts and graphs. This allows Disk % Idle Time to be visualized with other Session Metrics that return values in percentage terms. 

{
  "id": "755ef1b3-0a2d-4718-9746-3eabbcc69229"
}

After clicking execute, there will be an API response which will contain the ID of the new Session Metric-- take note of this ID to add it to a group.

Session Metric Groups

In this example, the Swagger API interface is used to create a Metrics Group. The "name" parameter can be used to specify a name for the group. 

Within the definitionKeys list, add the ID of any Custom Metrics that should be included in the group. The "id" returned from the Metric creation step, shown above, can be used here. Note, additional Metrics can be added after the initial creation steps shown below.

{
  "id": "a2f7d6d2-1dd8-4f42-a455-b741535527e1"
}

After clicking execute, there will be an API response which will contain the ID of the new group-- take note of this ID. 

Adding Custom Metrics to a Group

To add CPU and Memory to the new "Custom Metrics" group, the Swagger API interface can be used to list all all existing Session Metrics.

// Defaults, CPU and Memory, are shown as type "BuiltIn" as of 5.5
{
    "type": "BuiltIn",
    "key": "12a1cbf2-cf51-ee11-92e0-000d3a1d9eab",
    "metricId": 1,
    "sessionMetricsType": "cpuUsage",
    "displayName": "CPU",
    "unit": "%"
},...

After clicking execute, there will be an API response which will list the existing Session Metrics, in a format identical to the example response shown above. Use the "key" values in the following POST request to add CPU & Memory to the "Custom Metrics" group. 

In this example, the Swagger API interface is used to add the existing Session Metrics to the new group. When creating the Metrics Group, an ID was returned, which should be used for the groupId parameter.

In the request body, a list should be specified, containing the key (ID) of each Session Metric to add. To add solely the "CPU" Session Metric shown above, the following request body can be used:

[
   "12a1cbf2-cf51-ee11-92e0-000d3a1d9eab"
]

As shown in the above "GET" example, the "key" value for each Metric acts as the ID, should be used in the list.   

Metric Groups can contain one or more desired performance counters, but note that each metric in the group will be polled every 5 seconds. As you add more Metrics to a group, the CPU overhead will also increase. 

Adding Session Metrics Group to a Test

Once the request is successfully executed, the new group can be used by enabling Session Metrics and selecting the desired Metrics Group from the drop-down.

As of Login Enterprise 5.6, you can specify the frequency of Session Metric polling to restrict this overhead. This allows you to gather Session Metrics from each session, or only from 1 out of X sessions. This will provide a cushion against performance overhead when gathering metrics for medium to large size Session Metrics Groups.

Visualizing Your Metrics

In Continuous Tests and Load Tests, Session Metrics is now shown as a radio button option for display. Only Session Metrics that share the same unit (%, Ops/Sec, Bytes, etc.) can be displayed at the same time. This is specified by the "unit" defined when creating the Session Metric as shown in the above examples.

Below, the Disk % Idle Time custom Session Metric created above is Visualized after being specifying the  metrics group to use for the configuration.

 

Data Export

The Raw Data Export feature has been updated to include the session metrics. For more information, please see this article.

Examples