Including Filebeat installation and configuration for log shipping
Formerly known as the ELK stack, the Elastic Stack is a set of powerful tools for log management and analysis that includes Elastic Search (an analytics engine), Logstash (a data processing pipeline), and Kibana (a visualization tool). This super cool software suite will have you visualizing logs in minutes!
Before we get started, a couple of notes: firstly, this is a basic configuration that will not be publicly accessible and, as such, won’t be configured with security in mind (this is merely designed to get you up and running quickly so you can familiarize yourself with the tools); secondly, we are installing Elastic Stack and Filebeat on separate VMs – the Elastic Stack VM is purely for log management and analysis; Filebeat will be installed on the website that we’ll be shipping the logs from.
Now that that is out of the way, let’s jump in!
As always, let’s update our repos and packages.
sudo apt update && sudo apt upgrade -y Download and install the public key for elastic search
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpgInstall the apt package that allows https package downloads
sudo apt-get install apt-transport-httpsSave the repo in the elastic sources list
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.listThen update apt again and install Elastic Search
sudo apt-get update && sudo apt-get install elasticsearchATTENTION! ATTENTION! Security on Elastic Search is enabled and configured by default. The password, certificate, and keys are output to your terminal. SAVE THESE SOMEWHERE. See Elastic’s recommendations here. You will want to hold onto these if/when you decide to configure the stack to be accessible from the internet (i.e., remotely). Because we’re keeping things local, we’re going to bypass the security features by turning them off. And since we’re configuring this as a standalone installation and not joining a cluster, we can move straight into configuration.
Open the Elastic configuration file
sudo nano /etc/elasticsearch/elasticsearch.ymlTo make the service accessible from anywhere on the network, change the network host from localhost to 0.0.0.0 and uncomment if necessary.
networkhost 0.0.0.0Set the security feature from true to false
xpack.security.enabled falseEnable and start the service.
sudo systemctl enable elasticsearch && sudo systemctl start elasticsearchFrom there you can check the status
sudo systemctl status elasticsearchAnd verify the server is responding to queries.
curl -XGET "localhost:9200"You should see a response like the following:
{
"name" : "elastic",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "uuid_number",
"version" : {
"number" : "8.9.2",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "build_hash_number",
"build_date" : "2023-08-31T02:43:14.210479707Z",
"build_snapshot" : false,
"lucene_version" : "9.7.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}If you don’t see the output above, check your configuration, and if you have a firewall enabled, make sure traffic is allowed through port 9200. Now let’s install Logstash.
sudo apt install logstashIMPORTANT: figuring this part out was frustrating because several resources on configuring Logstash don’t mention you need to set proper filters. We will be sending system and Apache server logs from our web installation to Logstash, and if we don’t set filters, Logstash will receive the data but won’t know what to do with it. You’ll get the notification in Kibana that you have data, but it will be empty! So let’s create a filter in Logstash for the Filebeat data we’re sending. Open up a filter file:
sudo nano /etc/logstash/conf.d/beats.confAnd configure the file as seen below. The input specifies the port number logstash will accept input from, and the filters needed to parse the data we’re looking for. The output specifies that we want to send the data to elastic search for indexing.
input {
beats {
port => 5044
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGLINE}" }
}
date {
match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
if [type] == "apache" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
}
}Save the file. Enable and start Logstash.
sudo systemctl enable logstash && sudo systemctl start logstashCheck Logstash’s status with:
sudo systemctl status logstashAgain, make sure to allow port 5044 on your firewall if it’s enabled. Now let’s install Kibana.
sudo apt install kibanaOpen Kibana’s configuration file
sudo nano /etc/kibana/kibana.ymlUncomment and change the following as needed, then save the file:
server.port 5601
server.host "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]Enable and start Kibana
sudo systemctl enable kibana && sudo systemctl start kibanaAnd check Kibana’s status with
sudo systemctl status kibanaAnd we’ve successfully installed the ELK stack on our server! Navigate to the local IP address where you’ve installed the stack and you should see a welcome page.
Great! Now we’re ready to start shipping logs from our web installation to our Elastic Stack and visualize the data we’re getting. SSH into your web server and install Filebeat.
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.9.2-amd64.deb sudo dpkg -i filebeat-8.9.2-amd64.debOpen up Filebeat’s configuration file:
sudo nano /etc/filebeat/filebeat.ymlIn the file, set enabled to true and add the apache2 logpath.
# filestream is an input for collecting log messages from files.
- type: filestream
# Unique ID among all inputs, an ID is required.
id: my-filestream-id
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log
- /var/log/apache2/*.logSince we’re sending logs to Logstash, comment out the Elastic Search output.
# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
# Array of hosts to connect to.
# hosts: ["localhost:9200"]Uncomment the output for Logstash and change the IP from localhost to the local IP of your ELK server.
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
hosts: ["192.168.x.x:5044"]Enable and start Filebeat.
sudo systemctl enable filebeat && sudo systemctl start filebeatCheck the status.
sudo systemctl status filebeatFilebeat comes pre-installed with some modules for sending logs. You can check all the modules available with:
sudo filebeat modules listWe’re only interested in System and Apache server logs, so let’s enable them.
sudo filebeat modules enable systemsudo filebeat modules enable apacheCheck that the modules you enabled fall under “enabled” when you execute the modules list command. Now that we’ve enabled the modules, we have to turn them on. There are two files: /etc/filebeat/modules.d/system.yml and /etc/filebeat/modules.d/apache.yml.
Open these with nano and set enabled to true. Here’s the apache.yml configuration, for example. For the system.yml configuration, after you’ve set enabled to true for syslog and authlog, leave the paths alone since the Filebeat default is good; however, make sure you explicitly add the paths in the apache.yml configuration as seen below (and don’t forget the brackets and quotes – this took me awhile to troubleshoot!).
# Module: apache
# Docs: https://www.elastic.co/guide/en/beats/filebeat/8.9/filebeat-module-apac>
- module: apache
# Access logs
access:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/apache2/access.log*"]
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/apache2/error.log*"]We’re almost there! Thankfully, Filebeat comes with index and dashboard templates, so next we load an index template for Elastic Search. Be sure to use the IP address of your ELK stack server.
sudo filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["192.168.x.x:9200"]'And finally, load dashboards into Kibana.
sudo filebeat setup -E output.logstash.enabled=false -E output.elasticsearch.hosts=['192.168.x.x:9200'] -E setup.kibana.host=192.168.x.x:5601Something I like to do just to make sure all the configuration takes effect is to restart the services. I will restart Filebeat as well as the ELK services if I’m not seeing data.
And that’s it! Once you navigate to the IP address of your ELK server, click on Discover in the left-hand panel and you should start to see data flowing in. You can also check the live log streams in the Observability panel. I will cover the Elastic Stack and Filebeat installation in an upcoming video, as well as get you started with some basic features of the Kibana dashboard. See you soon!