FAQ
I can't launch any test, nor update or save any data. What's wrong?¶
This happened to several of our customers. The root cause? Not enough disk space. By default, Elasticsearch is configured to use elasticsearch-data
local volume, which is usually located in /var/lib/docker
. The issue manifests itself with logs from Elasticsearch:
[2015-10-27 09:40:08,801][INFO ][cluster.routing.allocation.decider] [Milan] low disk watermark [15%] exceeded on [DZqnmWIZRpapZY_TPkkMBw][Milan] free: 58.6gb[12.6%], replicas will not be assigned to this node
Elasticsearch is informing you the disk space is becoming scarce. Elasticsearch considers disk space before allocating data to a node. First, check the available disk space using df -h
, which shows something like:
Filesystem Size Used Avail Use% Mounted on
udev 12G 0 12G 0% /dev
tmpfs 2,4G 3,8M 2,4G 1% /run
/dev/nvme1n1p2 234G 46G 176G 21% /
tmpfs 12G 100M 12G 1% /dev/shm
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
tmpfs 12G 0 12G 0% /sys/fs/cgroup
/dev/nvme1n1p1 487M 8,3M 478M 2% /boot/efi
In this case, /var/lib/docker
is mounted on /
Filesystem, which has 176G
available. We're fine. But if available disk space goes below 5%
, indices are switched into read-only mode to prevent further writes.
Fixing Disk Space Issue
- Stop OctoPerf Enterprise-Edition,
- Allocate and mount a new disk with enough disk space. Let's say you mounted the disk on folder
/opt/elasticsearch
, - Find where the
elasticsearch-data
volume is stored on the disk. List docker volumes:
ubuntu@laptop:~$ docker volume ls
DRIVER VOLUME NAME
local enterprise-edition_elasticsearch-data
local enterprise-edition_octoperf-dat
The volume we are interested in is enterprise-edition_elasticsearch-data
. Let's find where the data is stored by inspecting it:
ubuntu@laptop:~$ docker volume inspect enterprise-edition_elasticsearch-data
[
{
"CreatedAt": "xxxxx",
"Driver": "local",
"Labels": {
"com.docker.compose.project": "enterprise-edition",
"com.docker.compose.version": "1.23.1",
"com.docker.compose.volume": "elasticsearch-data"
},
"Mountpoint": "/var/lib/docker/volumes/enterprise-edition_elasticsearch-data/_data",
"Name": "enterprise-edition_elasticsearch-data",
"Options": null,
"Scope": "local"
}
]
Now we know the data is stored in /var/lib/docker/volumes/enterprise-edition_elasticsearch-data/_data
. We have to copy it to the new disk:
sudo cp -rp /var/lib/docker/volumes/enterprise-edition_elasticsearch-data/_data/ /opt/elasticsearch/
- Now, edit
docker-compose.yml
and find:
volumes:
- elasticsearch-data:/usr/share/elasticsearch/data
Let's configure the volume to map the data to /opt/elasticsearch
:
volumes:
- /opt/elasticsearch:/usr/share/elasticsearch/data
- Restart OctoPerf Enterprise-Edition (which will restart Elasticsearch),
- Re-enable write operations, run:
curl -XPUT ELASTICSEARCH_IP:9200/_settings -H "Content-Type: application/json" --data '{"index": {"blocks": {"read_only_allow_delete": null}}}'
This enables write operations on all indices within elasticsearch. Make sure you have enough disk space available, otherwise indices will switch back to read-only
operations again.
To find Elasticsearch ip, first run docker ps
to list containers:
ubuntu@laptop:~$ docker ps | grep elasticsearch
a09d8f01fa71 docker.elastic.co/elasticsearch/elasticsearch-oss:6.5.4 "/usr/local/bin/dockā¦" 9 days ago Up 3 hours 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp elastic_kibana_elasticsearch_1_7d43584139e7
And inspect the relevant container:
ubuntu@laptop:~$ docker inspect a09 | grep "IPAddress"
"SecondaryIPAddresses": null,
"IPAddress": "",
"IPAddress": "172.18.0.3",
In this case, Elasticsearch local IP address is 172.18.0.3
.