Configure ElasticSearch on a single shared host and reduce memory usage

ElasticSearch is a powerful, yet easy to use, search engine based on Lucene. Compared to others, it features a JSON API and wonderful scaling capabilities via a distributed scheme and the defaults are aimed towards such scalability.

However, you may want to use ElasticSearch on a single host, mixed with your Web server, database and everything. The problem is that ES is quite a CPU and memory hog by default. Here’s what I found through trial and error and some heavy search.

This idea is to give ES some power, but leave some for the rest of the services. At the same time, if you tell ES that it can grab half of your memory and the OS needs some, ES will get killed, which isn’t nice.

My host was configured this way:

• ElasticSearch 0.19.9, official .deb package
• Ubuntu 12.04
• 1.5GB of RAM
• Dual-Core 2.6ghz
• LEMP stack

After installing the official package:

1. Allow user elasticsearch to lock memory
1. Edit /etc/security/limits.conf and add:
   elasticsearch hard memlock 100000
2. Edit the init script: /etc/init.d/elasticsearch
1. Change ES_HEAP_SIZE to 10-20% of your machine, I used 128m
2. Change MAX_OPEN_FILES to something sensible.
   Default is 65536, I used 15000
   Update: I asked the question on ElasticSearch group and it may be a bad idea, without giving any advantage.
3. Change MAX_LOCKED_MEMORY to 100000  (~100MB)
   Be sure to set it at the same value as 1.1
4. Change JAVA_OPTS to "-server"
   I don’t exactly know why, but if you check in the logs, you will see Java telling you to do so.
3. Edit the config file: /etc/elasticsearch/elasticsearch.yml
1. Disable replication capabilities
1. index.number_of_shards: 1
2. index.number_of_replicas: 0
2. Reduce memory usage
1. index.term_index_interval: 256
2. index.term_index_divisor: 5
3. Ensure ES is binded to localhost
   network.host: 127.0.0.1
4. Enable blocking TCP because you are always on localhost
   network.tcp.block: true
4. Flush and restart the server
1. curl localhost:9200/_flush
2. /etc/init.d/elasticsearch restart