Comments on: Elasticsearch backup strategies

By: dan

dan — Tue, 23 Apr 2013 09:35:38 +0000

Having full replication across every node is costly from a CPU / memory perspective as well, since each shard in an index is a full Lucene instance.

By: Mark Sherman

Mark Sherman — Mon, 22 Apr 2013 15:40:17 +0000

Okay. I’m learning. (Slowly). In order to ensure full replication on every node of a multi-node cluster you would need to set replicas to nodes -1. This is wasteful from a storage perspective, but it means you can shutdown any node at any time, backup the data directory and know you have a coherent backup. The ES-bases approach is much more efficient from a storage usage standpoint, but it costs in time and cpu during the re-indexing process. Correct?

By: dan

dan — Mon, 22 Apr 2013 13:53:15 +0000

If you are running a simple two-node cluster where each index is fully replicated to both nodes, then your approach would work. If, however, you are running a larger cluster, then you likely do not have full replication to every node (or even any one single node, given a large enough data set).

Please re-read the blog post – especially the section on “file-based approach” – as it addresses this question exactly. :)

By: Mark Sherman

Mark Sherman — Mon, 22 Apr 2013 13:39:48 +0000

Thanks for your reply. I do see now the scroll search in the API section of the guide on elasticsearch.org although I do not see the reindex API anywhere. Regardless of that, why would you need to run a different cluster in order to perform a hot backup? If I have two nodes in the same cluster running on different machines and I shutdown one node and backup the data directory doesn’t that provide a valid backup? Sorry if this is a stupid question I’m a real novice. Thanks again.

By: dan

dan — Sun, 21 Apr 2013 19:49:49 +0000

Actually, that script does use the API. There are two API calls : one to set up a scroll search (i.e. gather the data), and another to "reindex" (i.e. import) said data into the new index.

By: Mark Sherman

Mark Sherman — Sun, 21 Apr 2013 16:14:31 +0000

I am trying to understand how to implement ES-based approach described above. Basically I am trying to understand the simple import script which contains a lot of commands that I am not familiar with. Is there a way to perform an import using the ES API?

By: Lukas

Lukas — Sun, 07 Apr 2013 17:09:22 +0000

Ah ok, I get it now. The perl code is was a bit too concise for me to really get how this works on the ElasticSearch level. Thanks for the clarification.

By: dan

dan — Sun, 07 Apr 2013 14:41:38 +0000

As I noted in the blog post, you’re functionally bringing up a second, independent ES “cluster” (even if it’s only one node), then importing an entire index to said cluster. Your question doesn’t really apply to the situation.

By: Lukas

Lukas — Sun, 07 Apr 2013 14:29:53 +0000

How can you force one node to receive a copy of all shards? This isnt clear to me how your example script ensures this.

By: dan

dan — Thu, 24 Nov 2011 22:00:26 +0000

Yes, it assumes that the backup node is capable of managing the entire contents of the cluster, that is true. The link you posted is functionally a filesystem-based approach – it is, after all, simply rsync’ing the contents of the data folder. It’s not a bad approach, but the considerations regarding quorum and consistency still apply.