Install Elastic (ELK) Stack 8.x on Ubuntu 22.04 LTS

Including Filebeat installation and configuration for log shipping

Formerly known as the ELK stack, the Elastic Stack is a set of powerful tools for log management and analysis that includes Elastic Search (an analytics engine), Logstash (a data processing pipeline), and Kibana (a visualization tool). This super cool software suite will have you visualizing logs in minutes!

Before we get started, a couple of notes: firstly, this is a basic configuration that will not be publicly accessible and, as such, won’t be configured with security in mind (this is merely designed to get you up and running quickly so you can familiarize yourself with the tools); secondly, we are installing Elastic Stack and Filebeat on separate VMs – the Elastic Stack VM is purely for log management and analysis; Filebeat will be installed on the website that we’ll be shipping the logs from.

Now that that is out of the way, let’s jump in!

As always, let’s update our repos and packages.

sudo apt update && sudo apt upgrade -y  

Download and install the public key for elastic search

wget -qO - | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

Install the apt package that allows https package downloads

sudo apt-get install apt-transport-https

Save the repo in the elastic sources list

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

Then update apt again and install Elastic Search

sudo apt-get update && sudo apt-get install elasticsearch

ATTENTION! ATTENTION! Security on Elastic Search is enabled and configured by default. The password, certificate, and keys are output to your terminal. SAVE THESE SOMEWHERE. See Elastic’s recommendations here. You will want to hold onto these if/when you decide to configure the stack to be accessible from the internet (i.e., remotely). Because we’re keeping things local, we’re going to bypass the security features by turning them off. And since we’re configuring this as a standalone installation and not joining a cluster, we can move straight into configuration.

Open the Elastic configuration file

sudo nano /etc/elasticsearch/elasticsearch.yml

To make the service accessible from anywhere on the network, change the network host from localhost to and uncomment if necessary.


Set the security feature from true to false false

Enable and start the service.

sudo systemctl enable elasticsearch && sudo systemctl start elasticsearch

From there you can check the status

sudo systemctl status elasticsearch

And verify the server is responding to queries.

curl -XGET "localhost:9200"

You should see a response like the following:

"name" : "elastic",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "uuid_number",
"version" : {
"number" : "8.9.2",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "build_hash_number",
"build_date" : "2023-08-31T02:43:14.210479707Z",
"build_snapshot" : false,
"lucene_version" : "9.7.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
"tagline" : "You Know, for Search"

If you don’t see the output above, check your configuration, and if you have a firewall enabled, make sure traffic is allowed through port 9200. Now let’s install Logstash.

sudo apt install logstash

IMPORTANT: figuring this part out was frustrating because several resources on configuring Logstash don’t mention you need to set proper filters. We will be sending system and Apache server logs from our web installation to Logstash, and if we don’t set filters, Logstash will receive the data but won’t know what to do with it. You’ll get the notification in Kibana that you have data, but it will be empty! So let’s create a filter in Logstash for the Filebeat data we’re sending. Open up a filter file:

sudo nano /etc/logstash/conf.d/beats.conf

And configure the file as seen below. The input specifies the port number logstash will accept input from, and the filters needed to parse the data we’re looking for. The output specifies that we want to send the data to elastic search for indexing.

input {
  beats {
    port => 5044

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGLINE}" }
    date {
      match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]

  if [type] == "apache" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

Save the file. Enable and start Logstash.

sudo systemctl enable logstash && sudo systemctl start logstash

Check Logstash’s status with:

sudo systemctl status logstash

Again, make sure to allow port 5044 on your firewall if it’s enabled. Now let’s install Kibana.

sudo apt install kibana

Open Kibana’s configuration file

sudo nano /etc/kibana/kibana.yml

Uncomment and change the following as needed, then save the file:

server.port 5601 ""

elasticsearch.hosts: ["http://localhost:9200"]

Enable and start Kibana

sudo systemctl enable kibana && sudo systemctl start kibana

And check Kibana’s status with

sudo systemctl status kibana

And we’ve successfully installed the ELK stack on our server! Navigate to the local IP address where you’ve installed the stack and you should see a welcome page.

Great! Now we’re ready to start shipping logs from our web installation to our Elastic Stack and visualize the data we’re getting. SSH into your web server and install Filebeat.

curl -L -O sudo dpkg -i filebeat-8.9.2-amd64.deb

Open up Filebeat’s configuration file:

sudo nano /etc/filebeat/filebeat.yml

In the file, set enabled to true and add the apache2 logpath.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
    - /var/log/*.log
    - /var/log/apache2/*.log

Since we’re sending logs to Logstash, comment out the Elastic Search output.

# ---------------------------- Elasticsearch Output ----------------------------
  # Array of hosts to connect to.
#  hosts: ["localhost:9200"]

Uncomment the output for Logstash and change the IP from localhost to the local IP of your ELK server.

# ------------------------------ Logstash Output -------------------------------
  # The Logstash hosts
  hosts: ["192.168.x.x:5044"]

Enable and start Filebeat.

sudo systemctl enable filebeat && sudo systemctl start filebeat

Check the status.

sudo systemctl status filebeat

Filebeat comes pre-installed with some modules for sending logs. You can check all the modules available with:

sudo filebeat modules list

We’re only interested in System and Apache server logs, so let’s enable them.

sudo filebeat modules enable system
sudo filebeat modules enable apache

Check that the modules you enabled fall under “enabled” when you execute the modules list command. Now that we’ve enabled the modules, we have to turn them on. There are two files: /etc/filebeat/modules.d/system.yml and /etc/filebeat/modules.d/apache.yml.

Open these with nano and set enabled to true. Here’s the apache.yml configuration, for example. For the system.yml configuration, after you’ve set enabled to true for syslog and authlog, leave the paths alone since the Filebeat default is good; however, make sure you explicitly add the paths in the apache.yml configuration as seen below (and don’t forget the brackets and quotes – this took me awhile to troubleshoot!).

# Module: apache
# Docs:>

- module: apache
  # Access logs
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/apache2/access.log*"]

  # Error logs
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/apache2/error.log*"]

We’re almost there! Thankfully, Filebeat comes with index and dashboard templates, so next we load an index template for Elastic Search. Be sure to use the IP address of your ELK stack server.

sudo filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["192.168.x.x:9200"]'

And finally, load dashboards into Kibana.

sudo filebeat setup -E output.logstash.enabled=false -E output.elasticsearch.hosts=['192.168.x.x:9200'] -E

Something I like to do just to make sure all the configuration takes effect is to restart the services. I will restart Filebeat as well as the ELK services if I’m not seeing data.

And that’s it! Once you navigate to the IP address of your ELK server, click on Discover in the left-hand panel and you should start to see data flowing in. You can also check the live log streams in the Observability panel. I will cover the Elastic Stack and Filebeat installation in an upcoming video, as well as get you started with some basic features of the Kibana dashboard. See you soon!