+1-877-442-4669 | sales@go2group.com

    Effective_monitoring_of_application_servers_can_prevent_disasters_waiting_to_happen.jpg

    Monitoring Atlassian's JIRA using ELK (Part 1)

    Effectively monitoring your servers and applications can make the difference between maintaining a smoothly running organization, or a disaster waiting to happen. In this blog, we'll show you how to use tools from Elastic that work with Atlassian's JIRA to rapidly report on application and server status.

    What is monitoring and why should you do it?

    There are many reasons to implement server and application monitoring in your organization. The most important reason is to make sure that your servers and applications are behaving as expected. Effective monitoring implementations ensure that the right people are notified when applications or servers develop problems. 

    In this two part series, we will take a look at the ELK solution from Elastic.

    What is ELK?

    ELK stands for Elasticsearch, Logstash and Kibana. 

    • Logstash is a flexible, open source data collection, enrichment, and transportation pipeline. With connectors to common infrastructure for easy integration, Logstash is designed to efficiently process a growing list of log, event, and unstructured data sources for distribution into a variety of outputs, including Elasticsearch.
    • Elasticsearch is a distributed, open source search and analytics engine designed for horizontal scalability, reliability, and easy management. It combines the speed of search with the power of analytics via a sophisticated, developer-friendly query language that covers structured, unstructured, and time-series data. 
    • Kibana is an open source data visualization platform that allows you to interact with your data through stunning, powerful graphics. From histograms to geomaps, Kibana brings your data to life with visuals that can be combined into custom dashboards to help you share insights from your data.

    By using Logstash to consume the JIRA logs, Elasticsearch to store and index them, and Kibana to visualize them, we will be able to best monitor JIRA's performance.

    Logstash, Elasticsearch, and Kibana run with Java. Installing the latest version of the Java Runtime Environment (JRE) (Oracle or JDK) will work.

    Timothys-iMac:logstash-2.1.1 timothy.chin$ java -versionjava version "1.8.0_65"Java(TM) SE Runtime Environment (build 1.8.0_65-b17)Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)

    Consuming log files

    Log Files

    So, what log files will we use to measure or monitor JIRA's performance? We will use Logstash to parse JIRA's Access log file. Setting up Logstash is not difficult. Logstash's downloads page provides users with multiple types of downloads. An example installation would be to just download the tar.gz package and untar it. 

    Timothys-iMac:logstash-2.1.1 timothy.chin$ cd /Users/timothy.chin/opt/elk/logstash-2.1.1

    The next step is to provide Logstash with a configuration file. Logstash's configuration file has three parts: inputfilter, and output.

    Input 

    bin/logstash -e 'input { stdin { } } output { stdout {} }'

    stdin is an input plugin that reads events from standard input.

    Type "hello world" at the command prompt to see Logstash respond: 

    hello world
    2013-11-21T01:22:14.405+0000 0.0.0.0 hello world

    By using one of the many plugins available for input, Logstash can consume data from log files.

    For example: 

    input {
        file {
            type => "jira-access-log"
            path => "/Users/timothy.chin/Desktop/client-logs/*"
            start_position => "beginning"
            sincedb_path => "/Users/timothy.chin/Desktop/sincedb"
        }
    }

     The configuration above does the following:

    • All files being parsed by this configuration file are of the type: jira-access-log
    • The configuration will read files from the client-logs folder
    • The combination of start_position and sincedb_path  will assure that logs are read from the beginning if sincedb does not exist

    Filter

    From above, our input section has read the logs line by line. The filter section of the configuration performs intermediary processing on an event consumed. Here is an example:

    Example Apache Access Log 
    83.149.9.216 - - [04/Jan/2015:05:13:42 +0000"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
     
    Example filter using the Grok plugin 
    filter {
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}"}
        }
    }
    Using the grok plugin, this is the output of the event above
     
    Example JSON Output 
    {
       "clientip":"83.149.9.216",
       "ident":"",
       "auth":"",
       "timestamp":"04/Jan/2015:05:13:42 +0000",
       "verb":"GET",
       "request":"/presentations/logstash-monitorama-2013/images/kibana-search.png",
       "httpversion":"HTTP/1.1",
       "response":"200",
       "bytes":"203023",
       "referrer":"http://semicomplete.com/presentations/logstash-monitorama-2013/",
       "agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
    }

    Finally, we'll come to the output section, where the JSON event that was filtered in the Filter section is passed on to other applications for storage and indexing. In the example below, we will be pushing the events to Elasticsearch for indexing.  Output

    output {
        elasticsearch {
        }
    }

    With this configuration, Logstash uses http protocol to connect to Elasticsearch. The above example assumes that Logstash and Elasticsearch are running on the same instance. You can specify a remote Elasticsearch instance using a hosts configuration like hosts => "es-machine:9092".

    In our next blog, we will talk about what can be done with the data in Elasticsearch using the Kibana application. Please feel free to contact us to learn more about how we can assist your organization with Application and Server monitoring.

    Index & Sources

    Tags: JIRA, Atlassian, ELK, Elasticsearch, Kibana, Logstash

    Share this entry
    0 Comments
    Timothy Chin

    Written by Timothy Chin

    Having had a hand in everything from development, support and infrastructure management; the chief firefighter is here to save the day! Currently focusing on Atlassian suite of tools and dabbling in all sorts of technologies.


    Subscribe to the blog and receive email updates