Graylog is good, great in fact, and one of the reasons it really rocks is Elasticsearch. This nifty search server stores all the log files indexed in Graylog and you most probably will run into trouble when trying to feed it too much information when running on a single host (plus, if that host crashes you're toast).
Forturnatley Elasticsearch has cluster capabilities and they're really easy to use.
To start with you need to have a look at these two lines in elasticsearch.yml:
# Set the number of shards (splits) of an index (5 by default): # index.number_of_shards: 5 # Set the number of replicas (additional copies) of an index (1 by default): # index.number_of_replicas: 1
The first one is the number of shards per index. Let's say you have a maximum of 20 indexes configured in Graylog, then each of these indexes will be divided into 5 chunks (shards). Each shard will then reside on a node in your cluster. I have configured 5 primary shards per index and one replica of each primary shard. In my cluster here at home I only have two nodes, so 1 primary shard and 1 replica will be sufficient, I can loose one of the Elasticsearch nodes without breaking the cluster.
I also like to set the node name so that I can keep track of them in Graylog:
node.name: "es01"
Next it's time to configure the cluster. Elasticsearch tends to behave a bit strange with multicast discovery, especially in a bigger environment. I'm using unicast and therefor have to explicity point out the cluster nodes in my configuration file:
discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["172.16.0.9:9300", "172.16.0.4:9300"]
There! Elasticsearch is configured and ready to be started! Next step, Graylog!
I actually only configured four lines in Graylogs server.conf for this:
elasticsearch_shards = 5 elasticsearch_replicas = 1 elasticsearch_discovery_zen_ping_multicast_enabled = false elasticsearch_discovery_zen_ping_unicast_hosts = 172.16.0.9:9300,172.16.0.4:9300
This tells Graylog to create five shards per index and one replica (same as Elasticsearch) and to join the cluster as a client.
As you can see below Graylog acts as a node in the Elasticsearch cluster, although not at a data storing node.
[root@monitor ~]# curl 'localhost:9200/_cat/nodes' monitor.nickebo.net 172.16.0.9 55 45 0.71 c - graylog2-server es01.nickebo.net 172.16.0.4 5 d m es01 monitor.nickebo.net 172.16.0.9 8 45 0.71 d * monitor
This view tells us that we have three nodes, two of which are data nodes. es01 is the current master and graylog2-server is a client node.
You can now go to System -> Indices in Graylog and have a look at your shard routing. It should show both primary shards and replica shards.