FAQ

I can't launch any test, nor update or save any data. What's wrong?

This happened to several of our customers. The root cause? Not enough disk space. By default, Elasticsearch is configured to use elasticsearch-data local volume, which is usually located in /var/lib/docker. The issue manifests itself with logs from Elasticsearch:

[2015-10-27 09:40:08,801][INFO ][cluster.routing.allocation.decider] [Milan] low disk watermark [15%] exceeded on [DZqnmWIZRpapZY_TPkkMBw][Milan] free: 58.6gb[12.6%], replicas will not be assigned to this node

Elasticsearch is informing you the disk space is becoming scarce. Elasticsearch considers disk space before allocating data to a node. First, check the available disk space using df -h, which shows something like:

Filesystem      Size  Used Avail Use% Mounted on
udev             12G     0   12G   0% /dev
tmpfs           2,4G  3,8M  2,4G   1% /run
/dev/nvme1n1p2  234G   46G  176G  21% /
tmpfs            12G  100M   12G   1% /dev/shm
tmpfs           5,0M  4,0K  5,0M   1% /run/lock
tmpfs            12G     0   12G   0% /sys/fs/cgroup
/dev/nvme1n1p1  487M  8,3M  478M   2% /boot/efi

In this case, /var/lib/docker is mounted on / Filesystem, which has 176G available. We're fine. But if available disk space goes below 5%, indices are switched into read-only mode to prevent further writes.

Fixing Disk Space Issue

• Stop OctoPerf Enterprise-Edition,
• Allocate and mount a new disk with enough disk space. Let's say you mounted the disk on folder /opt/elasticsearch,
• Find where the elasticsearch-data volume is stored on the disk. List docker volumes:

ubuntu@laptop:~$ docker volume ls
DRIVER              VOLUME NAME
local               enterprise-edition_elasticsearch-data
local               enterprise-edition_octoperf-dat

The volume we are interested in is enterprise-edition_elasticsearch-data. Let's find where the data is stored by inspecting it:

ubuntu@laptop:~$ docker volume inspect enterprise-edition_elasticsearch-data 
[
    {
        "CreatedAt": "xxxxx",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "enterprise-edition",
            "com.docker.compose.version": "1.23.1",
            "com.docker.compose.volume": "elasticsearch-data"
        },
        "Mountpoint": "/var/lib/docker/volumes/enterprise-edition_elasticsearch-data/_data",
        "Name": "enterprise-edition_elasticsearch-data",
        "Options": null,
        "Scope": "local"
    }
]

Now we know the data is stored in /var/lib/docker/volumes/enterprise-edition_elasticsearch-data/_data. We have to copy it to the new disk:

sudo cp -rp /var/lib/docker/volumes/enterprise-edition_elasticsearch-data/_data/ /opt/elasticsearch/

• Now, edit docker-compose.yml and find:

    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data

Let's configure the volume to map the data to /opt/elasticsearch:

    volumes:
      - /opt/elasticsearch:/usr/share/elasticsearch/data

• Restart OctoPerf Enterprise-Edition (which will restart Elasticsearch),
• Re-enable write operations, run:

 curl -XPUT ELASTICSEARCH_IP:9200/_settings -H "Content-Type: application/json" --data '{"index": {"blocks": {"read_only_allow_delete": null}}}'

This enables write operations on all indices within elasticsearch. Make sure you have enough disk space available, otherwise indices will switch back to read-only operations again.

To find Elasticsearch ip, first run docker ps to list containers:

ubuntu@laptop:~$ docker ps | grep elasticsearch
a09d8f01fa71        docker.elastic.co/elasticsearch/elasticsearch-oss:6.5.4   "/usr/local/bin/dock…"   9 days ago          Up 3 hours          0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elastic_kibana_elasticsearch_1_7d43584139e7

And inspect the relevant container:

ubuntu@laptop:~$ docker inspect a09 | grep "IPAddress"
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "172.18.0.3",

In this case, Elasticsearch local IP address is 172.18.0.3.

Elasticsearch container exits automatically after launch

In this case, the first step is to check the logs of the elasticsearch container. So first we'll be listing the containers running:

guillaume@guillaume-VirtualBox:~/Downloads/enterprise-edition$ docker ps -a
CONTAINER ID   IMAGE                                                      COMMAND                  CREATED              STATUS              PORTS                               NAMES
95a494494d62   enterprise-edition_haproxy                                 "docker-entrypoint.s…"   About a minute ago   Up About a minute   0.0.0.0:80->80/tcp, :::80->80/tcp   haproxy
2e88aee1f25b   octoperf/enterprise-edition:12.4.0                         "./entrypoint.sh java"   2 minutes ago        Up About a minute                                       enterprise-edition
1027f344b52a   docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2   "/tini -- /usr/local…"   2 minutes ago        Up 2 minutes        9200/tcp, 9300/tcp                  elasticsearch
23cf709c97a1   octoperf/enterprise-documentation:12.4.0                   "/docker-entrypoint.…"   2 minutes ago        Up 2 minutes        80/tcp                              enterprise-documentation
95ee79d9d612   octoperf/enterprise-ui:12.4.0                              "/docker-entrypoint.…"   2 minutes ago        Up 2 minutes        80/tcp                              enterprise-ui

The next step is to check the elasticsearch container logs:

guillaume@guillaume-VirtualBox:~/Downloads/enterprise-edition$ docker logs -f 1027f344b52a

Now since elasticsearch failed to start, the log should contain errors. If you have messages like failed to obtain node locks or permission denied, this probably means that the rights on elasticsearch's data has been altered. If you don't know where to find it, you can find more info on the elasticsearch-data folder here.

The fix for this is to restore the initial rights through a command like this one, make sure to update elasticsearch-data to point to the proper folder:

chown -R 1000:1000 /elasticsearch-data

After this, remove all containers and restart the entire stack again.