On Premise Load Testing

OctoPerf comes with public Amazon Web Services and Digital Ocean providers pre-configured. You can run load tests from any Amazon and Digital Ocean data-centers out of the box.

You can also configure an On-Premise provider. An on-premise provider lets you inject Virtual Users from your own servers as well as monitor them. This is useful to do performance tests on an Intranet site or on any application that is not available on Internet.

Warning

The project configuration and the tests results are stored on our servers, even if you only use on-premise providers to generate the user load. In case you prefer hosting these results by yourself, please look at the OctoPerf Enterprise Edition.

Info

You can run a load test that uses both Cloud and On-Premise load injectors providers at the same time.

You may use the on-premise page to add, edit and remove On-premise providers.

On Premise Page

Prerequisites

OctoPerf On-Premise agents have the following installation requirements:

  • CPU: At least 4 cores, Intel Core I7 or Xeon recommended,
  • RAM: At least 4GB RAM, 8GB recommended,
  • Operating system: Linux, Ubuntu 16.04LTS or greater is recommended. Any Distribution which supports the latest Docker versions.
  • Server: both physical and virtual machines are supported,
  • Software: Docker must be installed. Docker 17.05CE or newer is recommended.

Providers

A provider holds the configuration required by OctoPerf's server to start machines in multiple regions. All machines started using a provider must have the same specifications.

So basically a provider consists in:

  • Machine memory settings (available and used by JMeter to simulate users),
  • A list of region names.

Your On Premise providers are visible in the top Providers panel:

Column Description
Name The provider name.
Enabled The provider state (always true).
Type The provider type (always PRIVATE).
Shutdown policy The provider shutdown policy: Manual for on-premise providers and automatic for cloud ones.

The pencil blue button is used to edit a provider and the trash orange button is used to remove them.

You can click on the Add On Premise Provider button to create a new one.

Create a provider

Creating or editing an on-premise provider is done through a two steps wizard:

Step one

The first step is to configure the provider:

  1. Enter the provider name (this name is used to differentiate various providers when configuring your load test scenario or when running a virtual user validation).
  2. Enter the memory available for each server configured using this provider.

The information message lets you know how many virtual users you will be able to start for each load generator depending on the available memory and memory usage.

On-Premise Provider configuration

You can also configure the memory usage during this first step by clicking on the Memory Usage button:

  1. Select the base memory used to start JMeter on the user load injectors.
  2. Select the additional memory allocated for each concurrent virtual user simulated.
  3. Select the percentage of the host memory dedicated to run JMeter.

You can leave the default values as they suit most use cases!

Warning

The available memory is configured for a provider. So every server of an on-premise provider must have the same amount of memory available.

You can still create multiple on-premise provider if you own heterogeneous machines.

Step two

The next and last step is simple: you just need to select one or more regions if you need to differentiate the servers you are going to use to generate the user load.

On-Premise Provider regions selection

You need to enter at least one region before clicking on the Save button.

Info

Creating different regions is the best way to differentiate monitoring agents. Creating a region for each one of them will make their selection much easier.

Warning

The region name will be used in a command line later on. Because of this it can only contains 'a' to 'z characters (lowercase only) and underscores '_'. Spaces are not allowed.

Once the on-premise provider is added you may install hosts.

Hosts

A Host is a physical machine used by the provider to start both load injectors and monitors. You need at least one host per region to use it during a load test. And you should not mix load injectors and monitors in the same region.

As hosts depends on a provider, you first need to select one in the Providers table above to list the Hosts :

Column Description
Creation date The host creation date.
State The host state.
CPU The server CPU information (speed and number of cores)
Memory The server memory information (total and available for load tests).

You can use the Stop button to deactivate a server (it won't be used for load testing thereafter) or the Trash button to completely remove it if your done performance testing.

Host installation

Recommendations

The machines used as load generators must:

  • Have a constant access over HTTPS to https://api.octoperf.com:443,
  • Have a constant access over websockets to https://rancher.octoperf.com:8080,
  • Have sufficient memory (see table below),

You may also use at least dual-core processor machines, with 10GB free hard-disk space.

The table below gives an overview of the amount of virtual users you can run depending on the instance memory and using the default configuration:

  • 128MB allocated to start JMeter,
  • 5MB additional for each virtual user,
  • 70% of the machine memory dedicated to the load tests.
Memory (MB) Number of virtual users
640 64
1124 117
2048 261
4096 547
8192 1121
16384 2242

Warning

Bare-metal Linux machines are strongly recommended for running the load tests. You can use multiple machines to simulate higher loads. We suggest to use no more than 70% of available RAM to prevent any performance issue. Some memory must be left for the operating system and filesystem cache. We recommend to use small (8Go) to medium (16Go) sized machines instead of big ones as JMeter may not behave correctly with very high memory allocation.

Installation procedure

The following procedure explains how to launch an on-premise load generator:

  1. Create an on-premise provider with a region on our platform, and select it,
  2. In the Hosts table, click on the Install Host button,
  3. A dialogs appears,
  4. Select a region,
  5. Select the Operating System (Windows or Linux)
  6. Follow the instructions related to the OS.
Windows installation

Host install Windows

Host installation on Windows requires Vagrant, a free and open-source software for creating and configuring virtual development environments. It can be considered a wrapper around virtualization software like VirtualBox.

So you need to download and install both software:

  1. Click on the Download VirtualBox button,
  2. Download and install VirtualBox for your Operating System,
  3. Click on the Download Vagrant button,
  4. Select you Operating System (you may also use Vagrant on Linux/Mac),
  5. Download and install Vagrant,
  6. Download the Vagrantfile (it is generated for the selected Provider and Region name, so you must re-download it to install an agent for another Provider / Region),
  7. Execute the command vagrant up in the folder that contains the downloaded Vagrantfile.

The Host should appear in the list in a couple of minutes.

Linux installation

Host install Linux

Host installation on Linux requires Docker. So the first command line installs it - you can skip it if you already have docker on your Linux injector.

The second command line installs the on-premise agent.

  1. Click on the Copy to Clipboard button at the right of the wget command,
  2. Paste it on your terminal to install Docker,
  3. Click on the Copy to Clipboard button at the right of the sudo docker run -e CATTLE_HOST_LABELS='...' -d -v [...] rancher/agent:v0.8.2 http://192.168.0.17:8080/v1/scripts/... command,
  4. Paste it on your terminal to install the on-premise agent.

The Host should appear in the list in a couple of minutes.

Host troubleshooting

My host does not appear in the hosts list.

Make sure you have successfully installed Docker and launched the agent on the machine. You can inspect the rancher container logs by using the command docker logs -f <container id>. Feel free to contact us if you still experience some issues.

My on-premise load generator IP address changed, it does not show up in the hosts table.

In this case, open a terminal and then:

  • List running docker containers by executing docker ps -a,
  • Stop and remove all rancher containers by executing docker rm -f <container id>,
  • Delete rancher state by executing rm -rf /var/lib/rancher/state on the load generator machine,
  • Re-launch the rancher agent by pasting the command provided in providers UI.

Check the following Rancher issue for more information.

Note

You can use the command docker ps to list all containers that are running on your machine. You can use the command docker logs <containerId> to check the logs of a container in particular.

Monitoring Agents

Monitoring Agents

The monitoring agent is in charge of collecting the monitoring metrics and sending them to OctoPerf. It runs inside a Docker container on an on-premise machine. Monitoring agents can be run on on-premise machines only. An on-premise machine can be a bare-metal server, a virtual machine or a cloud instance.

The currently selected provider's Monitoring Agents are visible in the bottom right panel:

Column Description
Last update The last time the agent pinged our servers
Region region where the agent runs
State UP or DOWN, UP means running
Name Name of the agent

Installation

To run a monitoring agent, please follow the procedure below.

It may take a few minutes to start and run the monitoring agent. Refresh the agent list until you see the agent with UP state.

Firewalls

Backend servers are usually protected behind firewalls to protect them from outer attacks. The monitoring agent doesn't require any port opening on firewall: it performs only outgoing HTTP requests to OctoPerf.

Firewall outgoing rules must allow the monitoring agent to send http requests to https://api.octoperf.com.

A Word on Docker

Docker has been chosen to run the monitoring agent because it provides numerous advantages:

  • Easy installation: run an agent container within minutes,
  • Portability: the container contains all the necessary packages and configuration to run the agent on any machine with Docker installed,
  • Version control: we can manage several agent versions and upgrades are seamless,
  • Lightweight footprint: docker containers are usually small and docker has minimal overhead compared to VMs.

Requirements

An on-premise monitoring agent has the following minimum requirements:

  • 2GB RAM available,
  • Dual core CPU or higher,
  • Linux operating system with Docker pre-installed,
  • Broadband internet connection (8 Mbps).

We recommend having the following characteristics for running a monitoring agent in good conditions:

  • At least 4GB RAM available,
  • Quad core CPU or higher,
  • Latest Ubuntu LTS with latest Docker version,
  • High speed internet connection (100 Mbps).

Although the monitoring agent consumes a tiny amount of resources, we recommend installing it on a dedicated machine otherwise the load test could interfere with the monitoring.

Stop and Removal

Monitoring agents can be directly stop and/or removed from the web UI. Press the Stop or Remove button next to the agent to stop or remove it respectively. Removing a running agents also stops it.

When stopping a monitoring agent, the associated docker container is stopped. When removing a monitoring agent, the docker container is stopped and removed.

Agent Troubleshooting

The agent upgrade somehow failed. How can I restart a new agent?

Delete the existing agent from the web UI. Create a new agent in the same location. If the on-premise machine is properly connected, a new agent container is created and started.

My monitoring agent stays in DOWN state. How can I find the root cause?

Connect via a linux terminal on the host machine and inspect the containers using the command docker ps -a. Find the monitoring-agent container and copy its id. Then run docker logs <containerId> where is the id of the container.

The agent logs should show you if it's a connectivity issue (no internet connection?) or something else. Contact our support team in case you are unable to find out.

Agent connectivity issue on Ubuntu Desktop

Ubuntu Desktop comes with a service names DNSMasq. This service setups a local DNS cache to help improve DNS queries response time. Unfortunately, Rancher doesn't work well with it, there are several bugs (4291, 5920) open for this issue.

When running a Monitoring Agent on Ubuntu Desktop with DNSMasq enabled, you see in the agent docker container logs something like:

>  PMjava.net.UnknownHostException: api.octoperf.com
    at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at okhttp3.Dns$1.lookup(Dns.java:39)

The agent fails to resolve hostname. To solve this issue, disable DNSMasq on Ubuntu Desktop. Here is the procedure:

  • Edit /etc/NetworkManager/NetworkManager.conf with the following command:
gksu gedit /etc/NetworkManager/NetworkManager.conf
  • Comment out the line dns=dnsmasq, so it looks like this:
#dns=dnsmasq
  • Then restart network-manager service:
sudo restart network-manager
# if you get /com/ubuntu/upstart: Connection refused, try:
sudo service network-manager restart

DNSMasq is now disabled. Restart the monitoring agent container and it should work fine now.