Creating An ELK Stack Via An Ansible Playbook (V1)

I’m currently spending time learning more about the ELK stack, which is made up of ElasticSearch, Logstash, Kibana (ELK) and Beats. The ELK stack can be created and run within containers (e.g. Docker) but I am going to use an Ansible Playbook to install and configure the ELK stack. You may ask why? Well I’m doing this so that I can (quite) quickly spin up an ELK stack to test ideas on, then get rid of it until I need another one without having to manually rebuild the stack each time.

The files for this blog post can be found on my GitHub at:

https://github.com/geektechdude/ELK_Ansible_Playbook

Important Notes: Vagrant does have boxes set up for ELK and there are Ansible playbooks available for setting up ELK but I want to do this from scratch and see what improvements I can make to my own playbooks. If your after a speedy resolve viewing what is already available may be better suited for you.

A Little About ELK

ELK is a used to store and visualise data.

The ELK stack is made up of:

ElasticSearch – Where all the data is stored, in indices made up of shards. Normally this would involve a cluster of ElasticSearch nodes so that the shards have back ups and the work involved in querying the data is shared amongst the nodes.

Logstash – used to parse or filter data into ElasticSearch. Logstash can be used to give meaning meaning to the data so that it can be more useful in ElasticSearch.

Kibana – a web frontend for visually interacting with the data in ElasticSearch. Also gives options to view uptime of the ELK stack and an interactive Dev Tools section to help with the various CURL commands that are available.

On top of the ELK stack are Beats. They came a little later and include FileBeat (for reading files e.g. logs), MetricBeat (for taking metrics of devices) and HeartBeat (for checking the status e.g uptime of a device).

For more on the ELK stack see: https://www.elastic.co/what-is/elk-stack

For this project I am going to be using ElasticSearch, Logstash, Kibana and FileBeat, and three virtual machines called ELK1, ELK2 and ELK3. ElasticSearch will be on ELK1, Kibana will be on ELK2 and Logstash with FileBeat will be on ELK3.

Vagrant

The virtual computers will all be running Ubuntu (Linux) and will have just over 2GB of RAM each. This could be done manually using just virtualisation software (my preferred platform is Virtual Box) but I am going to spend up the process by using Vagrant from Hashicorp. Vagrant allows for predefined boxes (i.e. virtual images) to be shared and then configured using a “vagrantfile” containing settings.

I have recently been learning more Ansible thanks to Jeff Geerling YouTube sessions, so I’m using his Ubuntu 1804 Vagrant image.

Ansible

Ansible is a fantastic management tool and something I have blogged about previously. Once Vagrant has spun up the virtual machines I am using Ansible to install the ELK stack and configure it.

I have set up an Inventory file and I am using the default vagrant user and vagrant SSH key. Note: This is not recommended for a production environment.

I am also using an Ansible configuration file (ansible.cfg), currently with just a setting to disable host_key_checking so that Ansible won’t ask about the virtual machines SSH fingerprint, which is helpful as I destroyed the virtual machines several times during testing.

Using The Files

As noted earlier I have placed on the files for this blog post on my GitHub and to use them you will also need:

• Ansible – see: https://www.ansible.com/
• Virtual Box – see: https://www.virtualbox.org/
• Vagrant – see: https://www.vagrantup.com/

The files from GitHub can either be downloaded or Git cloned. Once you have a copy, open a terminal (or command line) and navigate to the folder containing the files. Then enter:

vagrant up

and Vagrant should read the vagrantfile to start up the virtual machines. Depending on your hardware this may take a few minutes. Once the virtual machines are up and running, enter the command:

ansible-playbook -i inventory playbook.yml

And Ansible will start the plays to install Elastic, Logstash, Kibana and FileBeat. The plays currently include downloading components and updating the apt-cache, and as with the vagrant up may take some time to complete.

Once the playbook is complete you should be able to view Kibana via a web browser by browsing to http://192.168.28.72:5601 , at this step Kibana will need to be told about Index Patterns.

ElasticSearch will be on 192.168.28.71 and Logstash/FileBeat will be on 192.168.28.73. I have set up the playbook so that FileBeat should be sending the logs from /var/logs (on the Logstash machine) to Logstash, and then Logstash will send them to ElasticSearch.