Install ELK stack on Amazon EC2


To the ELK stack, consisting of:

  • Logstash
  • Elasticsearch
  • Kibana

Amazon AWS for testing on a single Amazon EC2 instance to install on, can you do the following:

It boosts an EC2 instance, that is not too small, with regard to the RAM, at least a m4.large with 8 GB RAM and 2 Processors, Elasticsearch is already demanding at the store and also Logstash is very resource hungry. As operating system I chose Ubuntu-16 (Ami-1e339e71).

Then you can Elastic IP create the instance, so that you can easily replace the instances and still continue keeping the IP.

Security groups

The security groups must be adjusted:

Inbound:

SSH, Port 22, IP Range: 0.0.0.0/0 für die SSH Verbindung
Custom TCP Rule, Port 5601, IP Range: 0.0.0.0/0 für Kibana

Outbound:

All traffic, All, All, 0.0.0.0/0, alles kann, nichts muss :) 

Using the public keys you can on the instance using SSH then to access the console and continue with the Administration:

ssh -i /path/to/public/key.pem ubuntu@elastic_ip

Bring installed packages to the latest version

sudo apt-get update
sudo apt-get upgrade

Java 8 install

sudo apt-get install openjdk-8-jre-headless

The installation can be verified with the command:

java -version

The result should contain at least a 1.8 version

openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-2ubuntu1.16.04.3-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

Elasticsearch installation with Debian packages

Elasticsearch public keys download

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Download repository definition

/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

Install Elasticsearch:

sudo apt-get update && sudo apt-get install elasticsearch

Automatically start at reboot Elasticsearch daemon:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Service start during operation:

sudo systemctl start elasticsearch.service

Elasticsearch is successfully installed and is by default only on localhost. The installation can be verified using curl:

curl -XGET 'localhost:9200/?pretty'

Result:

{
  "name" : "JIqSz-J",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "tRTvEU5iTFOtVe7asptmKg",
  "version" : {
    "number" : "5.6.1",
    "build_hash" : "667b497",
    "build_date" : "2017-09-14T19:22:05.189Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

Kibana installation

Kibana can easily be installed with:

sudo apt-get install kibana

Auto start when rebooting:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable kibana.service

Start service:

sudo systemctl start kibana.service

Now, Kibana is accessible via localhost:

curl -XGET 'localhost:5601/status'

Provides a long HTML output.

Then an IP must listen nauf Kibana address (the private IP address of the instance)

sudo vi /etc/kibana/kibana.yml

There the entry comment out and complement the private IP:

server.host: "172.XX.XX.XXX"

Neustrarten daemon:

sudo systemctl restart kibana.service

And now Kibana should be accessible in the browser:

http://elastic_ip:5601/

Successful installation you can admire Kibana in the browser:

Kibana_hello_world

Install Logstash

sudo apt-get install logstash

Starting's daemons:

sudo systemctl start logstash.service

Securing ELK with X-Pack

To have an authorization and user management for Kibana, need to install X-Pack for Kibana. Attention, This license apply after 30 days or. then is a Lite version of the available X-Pack.

sudo /usr/share/kibana/bin/kibana-plugin install x-pack

and as for Elasticsearch again:

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack

and Logstash

sudo /usr/share/logstash/bin/logstash-plugin install x-pack

restart the daemons:

 sudo systemctl restart kibana.service
 sudo systemctl restart elasticsearch.service
 sudo systemctl restart logstash.service