Tips on Configuring Elasticsearch for High Performance

By Manoj Chaudhary – CTO, Loggly

9

Elasticsearch Tip #1 Know Your Deployment Topology Before You Set Configs

http://bit.ly/NineTipsOnES

•  Loggly is running ES 0.90.13 with separate master and data nodes.

•  In addition, we use the ES node client to talk to data nodes. This makes the client transparent to data nodes; all it cares about is talking to node client. You establish your ES nodes as data and master using two properties that are set as true or false. For example, to make an Elasticsearch a data node, you set: node.master: false and node.data: true

http://bit.ly/NineTipsOnES

Elasticsearch Tip #2 mlockall Offers the Biggest Bang for the Performance Efficiency Buck •  Linux divides its physical RAM into chunks of memory called pages.

Swapping is the process whereby a page of memory is copied to the preconfigured space on the hard disk, called swap space, to free up that page of memory. The combined sizes of the physical memory and the swap space is the amount of virtual memory available.

•  Swapping does have a downside. Compared to memory, disks are very slow. •  The mlockall property in ES allows the ES node not to swap its memory. This

property can be set in the yaml file by doing the following: bootstrap.mlockall: true

http://bit.ly/NineTipsOnES

Elasticsearch Tip #3 discovery.zen Properties Control the Discovery Protocol for Elasticsearch •  Zen discovery is the protocol is used by Elasticsearch to discover and communicate between the nodes in the cluster. The zen

discovery setting is controlled by the discovery.zen.* properties. Both unicast and multicast are available as part of discovery protocol.

•  The discovery.zen.minimum_master_nodes control the minimum number of eligible master nodes that a node should “see” in order to operate within the cluster. It’s recommended that you set it to a higher value than 1 when running more than 2 nodes in the cluster.

•  Data and master nodes detect each other in two different ways:

–  By the master pinging all other nodes in the cluster and verify they are up and running

–  By all other nodes pinging the master xnodes to verify if they are up and running or if an election process needs to be initiated

•  The node detection process is controlled by discover.zen.fd.ping_timeout property. The default value is 30s, which determines how long the node will wait for a response. This property should be adjusted if you are operating on a slow or congested network. If you are on slow network, set the value higher. The higher the value, the smaller the chance of discovery failure.

http://bit.ly/NineTipsOnES

Elasticsearch Tip #4 Watch Out for delete_all_indices! •  It’s really important to know that the curl API in ES does not have very good

authentication built into it. A simple curl API can cause all the indices to delete themselves and lose all data.

•  This is just one example of a command that could cause a mistaken deletion: curl -XDELETE ‘http://localhost:9200/*/’

•  To avoid this type of grief, you can set the following property: action.disable_delete_all_indices: true.

–  This will make sure when above command is given, it will not delete the index and will instead result in an error.

http://bit.ly/NineTipsOnES

Elasticsearch Tip #5 Field Data Caching Can Cause Extremely Slow Facet Searches •  The field data cache is used mainly when sorting on or faceting on a field. It loads all

the field values to memory in order to provide fast document based access to those values.

•  You need to keep in mind that not setting this value properly can cause:

–  Facet searches and sorting to have very poor performance

–  The ES node to run out of memory if you run the facet query against a large index

•  An example: indices.fielddata.cache. consideration size: 25%

•  In setting this value, the key is what kind of facet searches your application performs.

http://bit.ly/NineTipsOnES

Elasticsearch Tip #6 Optimizing Index Requests •  At Loggly, we built our own index management system since the nature of log management means

that we have frequent updates and mapping changes. This index manager’s responsibility is to manage indices on our ES cluster. It detects when the index needs to be created or closed based on the configured policies. There are many policies in the index manager. For example, if the index grows beyond a certain size or lives for more than a certain time, the index manager will close the index and create a new one.

•  When the index manager send a node an index request to process, the node updates its own mapping and then sends that mapping to the master. While the master processes it, that node receives a state that includes an older version of the mapping. If there’s a conflict, it’s not bad (i.e. the cluster state will eventually have the correct mapping), but we send a refresh just in case from that node to the master. In order to make the index request more efficient, we have set this property on our data nodes: indices.cluster.send_refresh_mapping: false

http://bit.ly/NineTipsOnES

Elasticsearch Tip #7 Navigating Elasticsearch’s Allocation-related Properties •  Shard allocation is the process of allocating shards to nodes. This can happen during initial

recovery, replica allocation, or rebalancing. Or it can happen when handling nodes that are being added or removed.

•  The cluster.routing.allocation.cluster_concurrent_rebalance property determines the number of shards allowed for concurrent rebalance. This property needs to be set appropriately depending on the hardware being used, for example the number of CPUs, IO capacity, etc. If this property is not set appropriately, it can impact the performance of ES indexing.

•  cluster.routing.allocation.cluster_concurrent_rebalance:2 By default the value is set at 2, meaning that at any point in time only 2 shards are allowed to be moving. It is good to set this property low so that the rebalance of shards is throttled and doesn’t affect indexing.

Elasticsearch Tip #8 Recovery Properties Allow for Faster Restart Times

http://bit.ly/NineTipsOnES

•  ES includes several recovery properties which improve both Elasticsearch cluster recovery and restart times. The value that will work best for you depends on the hardware you have in use, and the best advice we can give is to test, test, and test again.

•  cluster.routing.allocation.node_concurrent_recoveries:4 This property is how many shards per node are allowed for recovery at any moment in time. Recovering shards is a very IO-intensive operation, so you should set this value with real caution.

•  cluster.routing.allocation.node_initial_primaries_recoveries:18 This controls the number of primary shards initialized concurrently on a single node. The number of parallel stream of data transfer from node to recover shard from peer node is controlled by indices.recovery.concurrent_streams.

http://bit.ly/NineTipsOnES

Elasticsearch Tip #9 Threadpool Properties Prevent Data Loss •  Elasticsearch node has several thread pools in order to improve how threads are managed

within a node. At Loggly, we use bulk request extensively, and we have found that setting the right value for bulk thread pool using threadpool.bulk.queue_size property is crucial in order to avoid data loss or _bulk retries:

•  threadpool.bulk.queue_size: 3000 This property value is for the bulk request. This tells ES the number of requests that can be queued for execution in the node when there is no thread available to execute a bulk request. This value should be set according to your bulk request load. If your bulk request number goes higher than queue size, you will get a RemoteTransportException as shown below.

Log Management is Our Full-Time Job. It Shouldn’t Be Yours.

Loggly is the world’s most popular cloud-based log management solution, used by more than 5,000 happy customers to effortlessly spot problems in real-time, easily pinpoint root causes and resolve issues faster to ensure application success.

Visit us at www.loggly.com or follow @loggly on Twitter.

Unless You Want it to Be Your Full-Time Job…

…If so, check out our career page to see if there is an position open that is a great match for your skills! Join us at www.loggly.com/careers.

Try Loggly for Free! → http://bit.ly/1nl87Uc