FAQ

I can't launch any test, nor update or save any data. What's wrong?

This happened to several of our customers. The root cause? Not enough disk space. By default, Elasticsearch is configured to use elasticsearch-data local volume, which is usually located in /var/lib/docker. The issue manifests itself with logs from Elasticsearch:

[2015-10-27 09:40:08,801][INFO ][cluster.routing.allocation.decider] [Milan] low disk watermark [15%] exceeded on [DZqnmWIZRpapZY_TPkkMBw][Milan] free: 58.6gb[12.6%], replicas will not be assigned to this node

Elasticsearch is informing you the disk space is becoming scarce. Elasticsearch considers disk space before allocating data to a node. First, check the available disk space using df -h, which shows something like:

Filesystem      Size  Used Avail Use% Mounted on
udev             12G     0   12G   0% /dev
tmpfs           2,4G  3,8M  2,4G   1% /run
/dev/nvme1n1p2  234G   46G  176G  21% /
tmpfs            12G  100M   12G   1% /dev/shm
tmpfs           5,0M  4,0K  5,0M   1% /run/lock
tmpfs            12G     0   12G   0% /sys/fs/cgroup
/dev/nvme1n1p1  487M  8,3M  478M   2% /boot/efi

In this case, /var/lib/docker is mounted on / Filesystem, which has 176G available. We're fine. But if available disk space goes below 5%, indices are switched into read-only mode to prevent further writes.

Fixing Disk Space Issue

• Stop OctoPerf Enterprise-Edition,
• Allocate and mount a new disk with enough disk space. Let's say you mounted the disk on folder /opt/elasticsearch,
• Find where the elasticsearch-data volume is stored on the disk. List docker volumes:

ubuntu@laptop:~$ docker volume ls
DRIVER              VOLUME NAME
local               enterprise-edition_elasticsearch-data
local               enterprise-edition_octoperf-dat

The volume we are interested in is enterprise-edition_elasticsearch-data. Let's find where the data is stored by inspecting it:

ubuntu@laptop:~$ docker volume inspect enterprise-edition_elasticsearch-data 
[
    {
        "CreatedAt": "xxxxx",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "enterprise-edition",
            "com.docker.compose.version": "1.23.1",
            "com.docker.compose.volume": "elasticsearch-data"
        },
        "Mountpoint": "/var/lib/docker/volumes/enterprise-edition_elasticsearch-data/_data",
        "Name": "enterprise-edition_elasticsearch-data",
        "Options": null,
        "Scope": "local"
    }
]

Now we know the data is stored in /var/lib/docker/volumes/enterprise-edition_elasticsearch-data/_data. We have to copy it to the new disk:

sudo cp -rp /var/lib/docker/volumes/enterprise-edition_elasticsearch-data/_data/ /opt/elasticsearch/

• Now, edit docker-compose.yml and find:

    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data

Let's configure the volume to map the data to /opt/elasticsearch:

    volumes:
      - /opt/elasticsearch:/usr/share/elasticsearch/data

• Restart OctoPerf Enterprise-Edition (which will restart Elasticsearch),
• Re-enable write operations, run:

 curl -XPUT ELASTICSEARCH_IP:9200/_settings -H "Content-Type: application/json" --data '{"index": {"blocks": {"read_only_allow_delete": null}}}'

This enables write operations on all indices within elasticsearch. Make sure you have enough disk space available, otherwise indices will switch back to read-only operations again.

To find Elasticsearch ip, first run docker ps to list containers:

ubuntu@laptop:~$ docker ps | grep elasticsearch
a09d8f01fa71        docker.elastic.co/elasticsearch/elasticsearch-oss:6.5.4   "/usr/local/bin/dock…"   9 days ago          Up 3 hours          0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elastic_kibana_elasticsearch_1_7d43584139e7

And inspect the relevant container:

ubuntu@laptop:~$ docker inspect a09 | grep "IPAddress"
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "172.18.0.3",

In this case, Elasticsearch local IP address is 172.18.0.3.