Installing Elasticsearch 2.2.x And Kibana 4.4.x on Amazon Linux

Here in PushApps we decided it's time to upgrade our Elasticsearch servers to the latest version which is 2.2.
We couldn't find any full guide on how to do it on Amazon Linux ( If you find feel free to share it in the comments... ), so we decided to share our experience in this blog post. We took this great guide by digitalocean.com for installing the ELK stack on Centos 7, and since Amazon Linux is based on Centos, most of the steps there stayed the same, but still we had to make a few modifications.

Our goals are:
  • To have an Elasticsearch cluster that can communicate with the outer world so we can store our data and perform searches on it.
  • Use Kibana as our web interface for searching and visualizing. We will proxy it behind Nginx.
  • Install some useful plugins such as aws cloud plugin, Sense and Marvel.
Requirements:
  • AWS account - sign up here
  • Amazon linux instance that you can ssh to , and obtain root permission, with at least:
    • RAM : 4 GB
    • CPU : 2 vCPU

We used this AMI but you can choose whichever you want.

Install Java 8

Elasticsearch requires Java, and it is recommended to use the latest version of Java 8. You can either install Oracle Java 8 or OpenJDK:

sudo yum install java-1.8.0  

to see that Java is installed type in the shell:

java -version  

The output should look like:

java version "1.8.0_65"  
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)  
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)  

Install Elasticsearch

If this cluster is going to be used for production purposes, we really recommend to save the the data files on a different disk than the root device. This can be easily done by following this guide

Elasticsearch can be installed by using the package manager by adding Elastic's package repository. First run the following command:

sudo rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch  

Create and edit a new yum repository file for Elasticsearch:

sudo vi /etc/yum.repos.d/elasticsearch.repo  

And save the following configuration to it:

[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages  
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos  
gpgcheck=1  
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch  
enabled=1  

Now you can install Elasticsearch:

sudo yum -y install elasticsearch  

Now edit the configuration file:

sudo vi /etc/elasticsearch/elasticsearch.yml  

Make the following changes to it:

  1. If you plan to save the data files in a custom location as we suggested above, uncomment the line #path.data and enter your own path ( we used: /data/ )
  2. network.host : 0.0.0.0 ( to make it accessible outside of your local machine )

Surprisingly ( or not...) , Elasticsearch default memory settings are not fitted to production use and allocate only 2 GB of RAM to the JVM. For production use we recommend to have at least 8 GB of RAM in your machine and to allocate 4 of it to Elasticsearch. It's also good to set the MAXOPENFILES value to the max value:

vi /etc/sysconfig/elasticsearch  

Add to it:

ES_HEAP_SIZE=4g  
MAX_OPEN_FILES=65535  

Run elasticsearch as a service and make it start on boot:

sudo service elasticsearch start  
sudo chkconfig elasticsearch on  

And get the famous "You know, for search" response from your brand new Elasticsearch installation!

curl http://localhost:9200  

Install Kibana

The Kibana package shares the same GPG Key as Elasticsearch, and we already installed that public key.

Create and edit a new yum repository file for Kibana:

sudo vi /etc/yum.repos.d/kibana.repo  

Add the following configuration to it:

[kibana-4.4]
name=Kibana repository for 4.4.x packages  
baseurl=http://packages.elastic.co/kibana/4.4/centos  
gpgcheck=1  
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch  
enabled=1  

Install Kibana:

sudo yum -y install kibana  

In the Kibana configuration file:

sudo vi /opt/kibana/config/kibana.yml  

switch the IP address to localhost instead of 0.0.0.0:

server.host: "localhost"  

Now run Kibana as a service and make it start on boot:

sudo chkconfig kibana on  
sudo service kibana start  

We want to use Kibana as a web interface behind a reverse proxy, for that we will install nginx.

Install Nginx

Kibana will listen to localhost, and nginx will route the external traffic through a reverse proxy.

Add the EPEL repository to yum:

sudo yum -y install epel-release  

It's recommended to set an admin user, protected with password, so that Kibana will not be accessible by everyone in port 80 ( switch my-admin-user to a username of your choice ):

sudo htpasswd -c /etc/nginx/htpasswd.users my-admin-user  

Enter a password when prompted, store it in safe location for future use.

edit the Nginx configuration file:

sudo vi /etc/nginx/nginx.conf  

Find the default server block (starts with server {), the last configuration block in the file, and delete it. When you are done, the last two lines in the file should look like this:

    include /etc/nginx/conf.d/*.conf;
}

Now we will create a new configuration file with a server block of our own:

sudo vi /etc/nginx/conf.d/kibana.conf  

Paste the following code block into the file. Be sure to update the server_name to match your server's name ( 0.0.0.0 should be fine ):

server {  
    listen 80;
    server_name 0.0.0.0;
    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.users;
    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;        
    }
}

We configured Nginx to route our HTTP traffic to the Kibana application, which is listening by default to localhost:5601 , to use the htpasswd.users which we created earlier, and to require basic authentication.

Now start Nginx as a service make it start on boot:

sudo service nginx start  
sudo chkconfig nginx on  

If everything went well you should be able to access Kibana by typing your server IP in a browser ( make sure you enable inbound TCP connection on port 80 in your security group ).

Useful plugins and apps:



aws cloud plugin

For features such as discovery, snapshots and more, add the aws cloud plugin to Elasticsearch.

Go to your Elasticsearch installation dir:

cd /usr/share/elasticsearch  

And install the plugin:

sudo ./bin/plugin install cloud-aws  

Add your aws credentials to Elasticsearch config file:

vi /etc/elasticsearch/elasticsearch.yml  

Add to the bottom of the file ( mind the spacing, you will probably need to add an extra space character ) :

cloud:  
    aws:
        access_key: <your access key>
        secret_key: <your secret key>

Sense

There is no way you can make it without the wonderful Sense, which will help you write and test your queries. It is now an app in Kibana.

Go to your Kibana installation dir:

cd /opt/kibana/  

And install Sense:

./bin/kibana plugin --install elastic/sense

Restart Kibana:

sudo service kibana restart  

And you should be able to see sense featured in the Apps menu.

Marvel

For monitoring, you can use Marvel , notice it's not free and you can evaluate it and later decide if you want to purchase a license for it.

The process is similar to installing the Sense plugin.

Go to your Kibana installation dir:

cd /opt/kibana/  

And install Marvel:

./bin/kibana plugin --install elasticsearch/marvel/latest

Restart Kibana:

sudo service kibana restart  

And you should be able to see Marvel featured in the Apps menu.

That's all. You can start storing , searching and visualizing your precious data, what are you waiting for?

On my next post I will describe the process we had to run when reindxing all of our data from our Elasticsearch 1.x.x servers to Elasticsearch 2.2. It should save you some precious time so stay tuned.

Orr Chen

Read more posts by this author.