Performance Bench Report

Once you have reached this section, it means that you have:

  • Designed one or more realistic virtual users,
  • Built a scenario which simulates your customer devices and network conditions,
  • Launched a test.

Now that the test is running, the most interesting part of the test is starting: analyze the results, and spot the performance bottlenecks.

The following sections describe how to create custom bench reports. There is no limit on the number of graphs or metrics you can put inside the report.

Live Metrics

The Bench Reports section is a Live view of your test allowing you to:

  • Analyze test metrics like Response Time or Throughput,
  • Drill-down to performance bottlenecks and point them out,
  • Build comprehensive test reports.

Runtime

All the operations described below can be done while the test is running. Any modification done is saved automatically.

Bench Report Items

The heart of our Bench Report system is to provide a highly customizable reporting system. This has been made possible by allowing to:

  • Add / Insert / Remove items,
  • Order items,
  • Customize item settings.

The following items are available:

  • Summary Items,
  • Line charts: draw various metrics like UserLoad, Connect time,
  • Pie and Bar charts: show http response code repartition, or top 10 latency time.

Select a Report Item

To select a load test Report Item, simply click on it. A blue border appears on the selection.

You can use the following keyboard shortcuts to interact with the selection:

Shortcut Button Description
del Removes the selected item.
Selects the previous item.
Selects the next item.
ctrl + ↑ Moves the selected item up.
ctrl + ↓ Moves the selected item down.

The buttons are visible when you place your mouse on any item, on its top right corner.

Info

The 'Shift+?' shortcut shows / hides a list of available shortcuts.

Add a Report Item

The following procedure explain how to add a report item:

  1. Select the report item type in the sidebar,
  2. Drag and drop the item on the right inside the report.

Video showing how to add a report item

Move a Report Item

The following procedure explain how to move a report item:

  1. Select the report item by clicking on its name,
  2. Drag'n drop the item on the right inside the report.

Video showing how to move a report item

Delete a Report Item

To delete a report item:

  1. Hover the item with the mouse,
  2. Click on the Delete button in the top right corner.

Summary Report Items

The summary items provide test-wide metrics like Average Response time. The following items are available:

  • Text: insert a paragraph with a title,
  • Statistics: Various test-wide statistics. Example: total hits, average response time,
  • Summary: Describes the scenario which has been run for this test,
  • Results Table: Provides test-wide statistics for each container and request,
  • Errors Table: Provides a detailed view about each error which occurred during the test.

Text Report Item

The text item allows to enter a custom text paragraph. It offers a WYSIWYG editor.

Text Report Item

When unselected, text items only show the text that will be exported in the report. To edit the text you must the select item beforehand. To do so you can:

  • Simply click on the item name input,
  • Click on the Edit button that appears when you place your mouse hover the text item,
  • Double click on the text.

Summary Report Item

This item describes the test configuration including:

  • Test duration,
  • Virtual user profiles with their respective geo-location,
  • Injectors used for each user profile.

Summary Report Item

Info

You can click on the magnifier icons in the table to quickly create:

  • Monitoring line charts for your injectors,
  • Result tables for your user profiles.

Statistics Report Item

This item contains a number of test-wide metrics. Metrics can be added, customized or removed.

To Customize a metric:

  1. Hover the metric with the mouse,
  2. Click on the Edit button which appeared,
  3. Modify the settings and click on Save.

Metrics can be specific to a given test location. Example: eu-west-1 (Ireland).

The following metrics are available:

  • Response time: minimum, average, maximum, variance and standard deviation,
  • Latency time: minimum, average, maximum, variance and standard deviation,
  • Connect time: minimum, average, maximum, variance and standard deviation,
  • Throughput: rate and total,
  • Assertions: errors, failures and successes,
  • Hits: rate and total.

Statistics Report Item

Results Table Report Item

The results table provides overall statistics for each container and request.

Results Table Report Item

To customize the results table:

  1. Hover the table with the mouse,
  2. Click on the Edit button which appeared,
  3. Modify the settings and click on Save.

The following settings can be customized:

  • Metrics: defines the metrics being within the table. By default, Min, Avg and Max Response time, Total hits and errors are defined,
  • Location: Select the region to get statistics from this region only,
  • User profile: Select a specific user profile,
  • Action: Show either Container or Hit (Request) results.

Results Table Customization

Errors Table Report Item

The errors table provide details about each error which occurred during the test. It allows to understand what happened server-side during the load test.

Errors Table Report Item

Each error can be inspected to display detailed information:

  • Time: the exact time at which the error occurred,
  • Action: action which failed,
  • Region: in which region the error has occurred,
  • Message: the detailed error message explaining what happened,
  • Connect Time, Latency and Response Time: statistics related to the request in error,
  • Request and Response: displays the request which has been sent to the server, and the received response.

Up to 50 errors per user profile are being stored on our servers.

To view the error details:

  1. Find the error you want to inspect,
  2. Click on the Magnifier button which is on the right.

Error Detailed information

Info

You can search for Strings or Regular Expressions (JavaScript format) inside the request and response:

  1. Click on a code text area (headers or body),
  2. Type CTRL + F (or CMD + F on Mac), a dialog appears on top of the area,
  3. Type the text to search or a regex,
  4. Press Enter to start searching for the first term occurrence
  5. Press CTRL + G (or CMD + G on Mac) to find the next occurrence, or SHIFT + CTRL + G (or SHIFT + CMD + G on Mac) to find the previous one.

Line Chart Report Items

Line charts allow to draw up to Four statistics per graph over time. Each curve can be customized with the following settings:

  • Metric: defines the curve metric to draw,
  • Action: defines the specific action to target,
  • Region: defines the geo-location.

Several line chart templates are available by default.

Add curve

To add a curve to a line chart report item:

  1. Click on the '+' button,
  2. Follow the Edit Curve procedure below from step 4.

Info

A maximum of four curves can be added to a chart. The '+' button may be hidden if there are already that many curves in the chart.

Edit curve

You can add/edit two kinds of curves:

To customize hit metrics curve settings:

  1. Hover the curve statistic with the mouse,
  2. Click on the Edit button.
  3. A dialog opens,
  4. Select the Hit Metric tab,
  5. Select the Metric, Location and User profile,
  6. Select the Action (depending on the selected Metric this field may be unavailable, e.g. the userload cannot be graph for a specific action),
  7. Click on the Save button.

Edit Curve

To customize monitoring metrics curve settings:

  1. Hover the curve statistic with the mouse,
  2. Click on the Edit button.
  3. A dialog opens,
  4. Select the Monitoring Metric tab,
  5. Select the Connection,
  6. Select the Counter in the table,
  7. Click on the Save button.

Edit Curve

Delete curve

To delete a curve from the graph:

  1. Hover the curve statistic with the mouse,
  2. Click on the Delete button.

The following metrics are available:

  • Users: active users,
  • Response time: minimum, average, maximum,
  • Latency time: minimum, average, maximum,
  • Connect time: minimum, average, maximum,
  • Throughput: rate and total,
  • Errors: rate,
  • Assertions: errors, failures and successes,
  • Hits: rate and total.

Line Chart Report Item

Other Charts

This section contains uncategorized charts.

Top Chart Item

The top chart item provides a top of containers or http requests for a given metric. This chart is great for drill-down to find slow business transactions and/or requests.

Like the line chart, the top chart metric can be customized: Click on the Edit button that appears when you place your mouse hover the top item.

Top Chart Report Item

Pie Chart Item

Pie charts are useful to get a quick overview of HTTP Response code, HTTP methods and HTTP response media types repartition. It allows to quickly spot if the web applications is running as expected.

Like any chart, pie chart metrics can be customized. Each pie chart has its own settings.

Pie Chart Report Item

Percentiles Chart Item

Percentiles charts shows the point at which a certain percentage of observed values occur. For example, the 95th percentile is the value which is greater than 95% of the observed values. If you have an 95th average response time at 10sec, it means that 95% of average response time are below this value.

Like the line chart, the percentiles chart can be customized.

Percentiles Chart Report Item

Threshold Alarms Table Item

This item lists all the threshold alarms which occurred since the beginning of the test. Alarms being displayed can be filtered by monitor connection and by monitored counter.

Threshold Alarms Table Item

More information can be found in the documentation dedicated to Alarms and Thresholds.

Textual Counters Table Item

This table displays all the textual monitoring metrics which have been collected during the test. Most common textual counters are database versions, operating system kernel version, thread pool configurations etc.

Textual Counters Table Item

Textual counters being displayed can be filtered by monitor connection and by monitored counter.

Tools

Report tools menu

Configuration

You can configure the report by clicking on the Configuration button on the left menu.

SLA Profiles

You can view the SLA profiles by clicking on the SLA Profiles button on the left menu.

JMeter Logs

You can view the JMeter logs by clicking on the JMeter Logs button on the left menu.

Test Logs

You can view the test initialization logs by clicking on the Test Logs button on the left menu.

Summary

The right column of the report displays a summary:

Report summary

You can click on any item to scroll to it.