Performance Metrics

Bench Reports can be customized using many items (charts, tables, ...). Each one of these items can be configured using performance metrics.

This section list all the metrics available in OctoPerf.

Performance metrics list

All the following metrics are available in OctoPerf. To know which report item car display this metrics, please report to the performance metrics availability table.

These metrics comes in various types (Minimum, average, count, rate, etc.). Refer to the performance metrics types table to know them.

Metric Description Performance
UserLoad Number of active users. Many other metrics should not changed as the user load increase.
Response time Time between the request and the end of the response, in milliseconds. The response time includes both the latency and the connect time. The lower the better. Should be less than 4 seconds.
Connect time Time between the request and the server connection, in milliseconds. The lower the better. If you get high connect times your servers may be running out of available sockets, or your database may be overloaded.
Latency Time between the request and the first response byte, in milliseconds. The lower the better. If you get high response times and low latencies your servers may be running out of bandwidth. Check the throughput to confirm this.
Throughput Bit rate in Bytes per second. Amount of data exchanged between the clients and the servers. Must grow along the user load. If it reaches an unexpected plateau, you may be running out of bandwidth.
Errors Count or rate of errors that occurred. Errors may happen if you did not validate your virtual user. Otherwise, errors may be the sign that your servers or database are overloaded.
Hits Count or rate of hits (requests) that occurred. Should increase as the user load goes up.
Assertions Count of assertions in error, failed, or successful. Assertions in error or failed lets your know that your servers did not answer as you expected.

Monitoring Metrics

The following monitoring metrics are collected for each load generator involved during the load tests.

Metric Description Performance
Memory Usage Memory usage in percent. The memory usage should stay under 80%. The lower the better. Excessive memory usage can lead to load generator failure.
CPU Usage cpu usage in percent. The cpu usage should stay under 100%. The lower the better. Excessive cpu usage can lead to load generator failure.
Network Sent Outbound network usage. Must grow along the user load. If it reaches a plateau, you may be running out of bandwidth.
Network Received InBound network usage. Must grow along the user load. If it reaches a plateau, you may be running out of bandwidth.
TCP Connections Established TCP connections. Must grow along with the user load. If it reaches a plateau, your server network capacity may be exceeded.
TCP Retransmits TCP segments retransmitted. The lower the better. If it increases abnormally, your server network capacity may be exceeded.

Performance metrics types

Each metric comes in various types. The table below list all of them.

Type Description
Maximum Maximum value of a metric.
Minimum Minimum value of a metric.
Average Average value of a metric.
Variance The variance quantifies the dispersion of the metric. A variance close to 0 indicates that the metric values tend to be very close to the mean, while a high variance indicates that the values are spread out over a wider range. Its unit is the square of the metric unit.
Standard deviation Simply the square root of the variance. It's easier to compare to other metric types using a common unit.
Percentile 90 A percentile indicates the value below which a given percentage of observations in a group of observations fall. For example, the 90th percentile is the value below which 90 percent of the observations may be found.
Percentile 95 A percentile indicates the value below which a given percentage of observations in a group of observations fall. For example, the 90th percentile is the value below which 95 percent of the observations may be found.
Median Simply a 50th percentile: the value below which 50 percent of all the values may be found.
Total Count of a metric. Number of occurrences of an event.
Rate Count of a metric per second.
Apdex Apdex (Application Performance Index) defines a standard method for reporting the performance of software applications, by specifying a way to analyze the degree to which measured performance meets user expectations. Score is between 0 and 1, at 1 all users are satisfied.

The following table defines the metrics and their associated statistics:

Metric Min. Avg. and Max. Std Dev. and Variance Med. Percentile Total Rate Apdex
Response Time X X X X
Connect Time X X X
Latency X X X
Errors X X
Hits X X
Assertions X
Throughput X X

Monitoring metrics availability

The table below displays all monitoring metrics per type and which report items can display them.

Metric Type Line Chart Summary Top Chart Percentiles Chart Results Table
Memory Usage Maximum X
CPU Usage Maximum X
Network Sent Maximum X
Network Received Maximum X
TCP Connections Maximum X
TCP Retransmits Maximum X

Performance metrics availability

The table below displays all performance metrics per type and which report items can display them.

Metric Type Line Chart Summary Top Chart Percentiles Chart Results Table
Userload Total X
Response time Average X X X X X
Response time Maximum X X X X X
Response time Minimum X X X X X
Response time Variance X X X X
Response time Standard deviation X X X X
Response time Apdex X X X
Response time Median X
Response time Percentile 90 X
Response time Percentile 95 X
Connect time Average X X X X X
Connect time Maximum X X X X X
Connect time Minimum X X X X X
Connect time Variance X X X X
Connect time Standard deviation X X X X
Connect time Apdex X X X
Latency Average X X X X X
Latency Maximum X X X X X
Latency Minimum X X X X X
Latency Variance X X X X
Latency Standard deviation X X X X
Latency Apdex X X X
Errors Rate X X X X
Errors Total X X X X
Hits Rate X X X X
Hits Total X X X X
Throughput Rate X X X X
Throughput Total X X X X
Assertions in error Total X X X X
Assertions failed Total X X X X
Assertions successful Total X X X X

Pie chart metrics

There are 3 performance metrics left that can only be displayed in pie charts. These metrics show the repartition of certain data.

Metric Description
HTTP methods HTTP methods (GET, POST, DELETE, ...) repartition
HTTP response codes HTTP response codes (2xx, 3xx, 4xx, 5xx, ...) repartition. You should avoid error codes such as 4xx and 5xx.
Media types count Media types (html, css, json, javascript, xml, ...) repartition by request count. Useful to check resources repartition by type.
Media types throughput Media types (html, css, json, javascript, xml, ...) repartition by bandwidth usage. Useful to know what resources use your bandwidth.