Install ElastiFlow on Ubuntu 18.04 – Part 3: Installing ElastiFlow

This blog is part of a series. Refer to the links below for the other posts in this series.

Part 1: Installing Ubuntu
Part 2: Installing Elastic Stack
Part 3: Install ElastiFlow
Part 4: Solution Maintenance (coming soon)

In parts 1 and 2 of this tutorial we installed the Ubuntu server and Elastic Stack (ELK Stack). Now we are ready into install and configure ElastiFlow.

Before beginning I recommend setting up at least one network device to start sending logs to the server. In my environment, I configured my pfSense firewall to send IPv4 flows using port 9995. It is important that you make note of the port you setup in your environment, as we will need to configure ElastiFlow to receive them as part of this tutorial.

The steps below are based on the directions found in ElastiFlow GitHub site. I’ve just expanded upon them and given you the command relevant to the Ubuntu and Elastic Stack 6.3 install we performed in parts 1 and 2. The instructions here are for ElastiFlow 3.x

Set JVM heap size.

It is recommended to set the JVM heap size to at least 2GB. If you are going to be doing DNS lookups then 4GB is recommended.

Open the jvm.options for Logstash to set the heap size

Edit the Xms and Xmx sizes in the jvm.options configuration

Add and Update Required Logstash plugins

Copy the pipeline files to the Logstash configuration path

Create a temp folder to hold install files

Navigate to the temp folder you just created

Download Elastiflow install files

Install unzip, so you can extract the archive file you just downloaded

Unzip the Elastiflow files

Copy ElastiFlow configuration files to the Logstash directory

Setup environment variable helper files

Copy the elastiflow.conf to systemd

Add the ElastiFlow pipeline to pipelines.yml

Open the Logstash pipline configuration file for editing.

Add the two line below to the bottom of the pipelines.yml file

Configure inputs

Open the elastiflow.conf file for editing.

The items you set here will be unique to your environment and setup. In my environment, I set the following:

ELASTIFLOW_NETFLOW_IPV4_HOST=<The Server’s IP Address>
ELASTIFLOW_NETFLOW_IPV4_PORT=9995

Remember 9995 is the port I configured the network equipment to send flows on.

I also set ELASTIFLOW_RESOLVE_IP2HOST to true and set my DNS server in ELASTIFLOW_NAMESERVER so that the dashboards will attempt to resolve the DNS names instead of just displaying IP Address. There is a performance hit for this, but since it is just my lab network, it should not be a problem.

Ensure that the port for the incoming flows is open, on the firewall, so that Logstash is able to receive them.

Create logstash system startup script

Reload systemd manager configuration and start logstash

Run the command below to check that logs are being received.

You should see log entries scrolling up the screen. Logstash can take some time to start so wait a few minutes after running the command. If after a little bit, it is just sitting there doing nothing, then either flows are not being sent or something is wrong with your configuration. If something is not configured correctly, you should see the error listed in the log. You can ignore any errors about there being nothing in the “/etc/logstash/conf.d/*.conf” folder. This is because we added the ElastiFlow to a second pipeline, so unless you previously setup anything on this server, than that folder should be empty.

Note: If using Netflow v9 or IPFIX you will likely see warning messages related to the flow templates not yet being received. They will disappear after templates are received from the network devices, which should happen every few minutes. Some devices can take a bit longer to send templates. Fortinet in particular send templates rather infrequently.

Hit Ctrl-C to exit from log tail

Setup Kibana

Assuming you are still in the flowtemp directory, run the command below to import the ElastiFlow indexes.

  1. On your local machine download the ElastiFlow dashboards. Right-click the link below choose save as https://github.com/robcowart/elastiflow/raw/master/kibana/elastiflow.dashboards.json
  2. Open your web browser and open the Kibana site.
  3. Navigate to Management > Advanced Settings
  4. Search for and set the recommended settings listed below. For details and additional information on what these are, refer to the ElastiFlow documentation.
    doc_table:highlight false
    filters:pinnedByDefault true
    state:storeInSessionStorage true
    timepicker:quickRanges see link
  5. Navigate to Saved Objects and Import elastiflow.dashboards.json file you downloaded in step 1.

Once dashboard import completes you are done. You can now navigate to the Dashboard page in Kibana and start exploring the different visualizations. You can also check out the ElastiFlow Dashboard Documentation.

    

Leave a Reply

x

We use cookies to ensure the best possible experience on our website. Detailed information on the use of cookies on this site is provided in our Privacy and Cookie Policy. Further instruction on how to disable our cookies can be found there.