Creating An ELK Stack Via An Ansible Playbook (V1)

The ELK_Ansible_Playbook on GitHub

I’m currently spending time learning more about the ELK stack, which is made up of ElasticSearch, Logstash, Kibana (ELK) and Beats. The ELK stack can be created and run within containers (e.g. Docker) but I am going to use an Ansible Playbook to install and configure the ELK stack. You may ask why? Well I’m doing this so that I can (quite) quickly spin up an ELK stack to test ideas on, then get rid of it until I need another one without having to manually rebuild the stack each time.

The files for this blog post can be found on my GitHub at:

https://github.com/geektechdude/ELK_Ansible_Playbook

Important Notes: Vagrant does have boxes set up for ELK and there are Ansible playbooks available for setting up ELK but I want to do this from scratch and see what improvements I can make to my own playbooks. If your after a speedy resolve viewing what is already available may be better suited for you.

An Elk, but not ELK
An Elk, but not ELK

A Little About ELK

ELK is a used to store and visualise data.

The ELK stack is made up of:

ElasticSearch – Where all the data is stored, in indices made up of shards. Normally this would involve a cluster of ElasticSearch nodes so that the shards have back ups and the work involved in querying the data is shared amongst the nodes.

Logstash – used to parse or filter data into ElasticSearch. Logstash can be used to give meaning meaning to the data so that it can be more useful in ElasticSearch.

Kibana – a web frontend for visually interacting with the data in ElasticSearch. Also gives options to view uptime of the ELK stack and an interactive Dev Tools section to help with the various CURL commands that are available.

On top of the ELK stack are Beats. They came a little later and include FileBeat (for reading files e.g. logs), MetricBeat (for taking metrics of devices) and HeartBeat (for checking the status e.g uptime of a device).

For more on the ELK stack see: https://www.elastic.co/what-is/elk-stack

For this project I am going to be using ElasticSearch, Logstash, Kibana and FileBeat, and three virtual machines called ELK1, ELK2 and ELK3. ElasticSearch will be on ELK1, Kibana will be on ELK2 and Logstash with FileBeat will be on ELK3.

Vagrant

The virtual computers will all be running Ubuntu (Linux) and will have just over 2GB of RAM each. This could be done manually using just virtualisation software (my preferred platform is Virtual Box) but I am going to spend up the process by using Vagrant from Hashicorp. Vagrant allows for predefined boxes (i.e. virtual images) to be shared and then configured using a “vagrantfile” containing settings.

geektechstuff_ELK_1
Vagrant File

I have recently been learning more Ansible thanks to Jeff Geerling YouTube sessions, so I’m using his Ubuntu 1804 Vagrant image.

Ansible

Ansible is a fantastic management tool and something I have blogged about previously. Once Vagrant has spun up the virtual machines I am using Ansible to install the ELK stack and configure it.

geektechstuff_ELK_2
Ansible Playbook To Install ELK Stack

I have set up an Inventory file and I am using the default vagrant user and vagrant SSH key. Note: This is not recommended for a production environment.

geektechstuff_ELK_3
Ansible Inventory File

I am also using an Ansible configuration file (ansible.cfg), currently with just a setting to disable host_key_checking so that Ansible won’t ask about the virtual machines SSH fingerprint, which is helpful as I destroyed the virtual machines several times during testing.

geektechstuff_ELK_4
ansible.cfg file

Using The Files

As noted earlier I have placed on the files for this blog post on my GitHub and to use them you will also need:

geektechstuff_ELK_5
The ELK_Ansible_Playbook on GitHub

The files from GitHub can either be downloaded or Git cloned. Once you have a copy, open a terminal (or command line) and navigate to the folder containing the files. Then enter:

vagrant up

and Vagrant should read the vagrantfile to start up the virtual machines. Depending on your hardware this may take a few minutes. Once the virtual machines are up and running, enter the command:

ansible-playbook -i inventory playbook.yml

And Ansible will start the plays to install Elastic, Logstash, Kibana and FileBeat. The plays currently include downloading components and updating the apt-cache, and as with the vagrant up may take some time to complete.

geektechstuff_ELK_6
More Ansible playbook to configure FileBeat

Once the playbook is complete you should be able to view Kibana via a web browser by browsing to http://192.168.28.72:5601 , at this step Kibana will need to be told about Index Patterns.

ElasticSearch will be on 192.168.28.71 and Logstash/FileBeat will be on 192.168.28.73. I have set up the playbook so that FileBeat should be sending the logs from /var/logs (on the Logstash machine) to Logstash, and then Logstash will send them to ElasticSearch.

2 thoughts on “Creating An ELK Stack Via An Ansible Playbook (V1)

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.