Centralize EPiServer logs with the ELK stack

Introduction

It’s a challenge to get a good overview of what’s going on when running a distributed EPiServer environment with multiple servers, services or other software components. Especially if something goes wrong.

With EPiServer we’ve grown quite familiar with the logging tool log4net which simplifies logging. EPiServer installations by default output a handy EPiServerErrors.log file to the App_Data folder in the website root.

Connecting to each server with remote desktop and firing up notepad to browse through a (possibly) huge log file for errors isn’t a very effective strategy. It’s also a reactive approach to something we should be proactive about.

That said let’s set a few basic requirements for a new solution:

  • Keep log4net as the logging component we’ve grown to love.
  • Centralize logging.
  • Make it easy to query and visualize the logs.

Visualized by the power of PowerPoint this is what we’re gunning for.

EPiServerErrorLogsCentralizedStorage

ElasticSearch
Perhaps you’ve heard of Elasticsearch. It’s the same technology which EPiServer Find is built on top of. In a nutshell it’s a super-fast search service based on Lucene. For our solution we’re going to need a storage component for our log data. Elasticsearch fits the bill perfectly. An index is a great place to store a large amount of denormalized data. Using Elasticsearch it also becomes easy to query the data via the RESTful webful interface. With Lucene under the hood performance is not an issue.

To boot Elasticsearch has two friends that makes it more awesome: Logstash and Kibana. These three amigos make up for the ELK stack.

Logstash
Logstash centralizes the data processing of log files and other event data. It’s virtually a pipeline that gobbles up anything that can spit out a debug or error message, then convert and push that data elsewhere. Logstash is the component that we’ll use to process the EPiServerErrors.log file.

Kibana
Kibana is all about visualizing the data we’ve got stored in ElasticSearch and making sense of it. Kibana will make up for the final component that visualizes all the log data we collect.

Setting up ELK

Head on over to the Elasticsearch download page and grab these three components.

Elasticsearch

Elasticsearch is pretty straightforward to setup. First we’ll need to download the latest version of Java (unfortunately). Set the system environment variable JAVA_HOME pointing to the directory where our Java installation is and finally do a reboot.

With Java installed we fire up the console, navigate to the bin-folder of our Elasticsearch-installation and run the command:

This should setup the local Windows service for Elasticsearch.

We’ll also want to set the startup to automatic for the service should we reboot our computer. You can check the Elasticsearch-installation by navigation to http://localhost:9200.

If everything is working we’ll get a JSON string with some basic information about our Elasticsearch installation.

Logstash

First we’ll create a configuration file for Logstash so it’ll know what logs to process and where to ship them. Create a file called logstash.conf in the Logstash bin-folder and add the following JSON-bits:

This is a super simple configuration file for Logstash. We’re simply looking at the errors log and pushing that data to Elasticsearch. Note that in this case we’re pushing the data to localhost. We could change this to a different server name if we had Elasticsearch somewhere else.

The filter (Grok) is simply a way to tell Logstash how the rows in the log files are formatted. The filter above uses the default log4net conversion pattern that ships with EPiServer. If you need more (or different) data it’s fairly easy to construct your own filters. There’s a tool called Grok Constructor that can be used to test these. The filter will also create the required fields in the Elasticsearch-mapping.

Logstash can be run as a standalone application but we’re better off by running it as a service. Unfortunately Logstash doesn’t ship with a service installer. Instead we need to download the tool NSSM (non-sucking service manager).

Extract the NSSM executable into the bin-folder of the Logstash-installation. Run the command:

This will run the NSSM application and allow us to create a Windows service for logstash. Configure the service as below (with your own paths of course).

NSSM Service Installer

Elasticsearch should now contain a Logstash index for today’s date. We can check this by querying http://localhost:9200/_search. This will return all documents for all indices. A document should look something like this.

As you can see the custom fields for the EPiServerErrors.log file has been created for us by Logstash. For the sake of this blog post I’ve also enabled the DEBUG-flag in EPiServerLog.config in order to log all the things.

This wraps it up for Logstash. Now we’re able to push data into Elasticsearch. The only thing missing now is a tool to make sense of all that log data.

Kibana

Kibana is our final component. We’ll be using it to query Elasticsearch for our log data and visualize it.

Like Logstash Kibana is also a standalone application. Install it as a Windows service just like we did Logstash using NSSM. Once started we can reach it by default at http://localhost:5601.

We need to tell Kibana what our indices are called. Granted that we’ve now got data in Elasticsearch this is how we setup Kibana for the first time.

ConfigureIndexPattern

With Kibana configured to discover our Logstash indices let’s go through what makes up Kibana:

  • Discover
  • Visualize
  • Dashboard

Discover is where we can query Elasticsearch with a few arbitrary selection criterias. Host, date interval, reporting class, etc. In the screenshot below I’ve selected the InitializationEngine-class, the host MBPro and the last 15 minutes as a time interval. This returns the log entries that are stored in Elasticsearch.

Discover

 

Visualize is where we can construct charts, tables, graphs etc for our log data. As an example I’ve created a pie chart that displays the most occurring classes in the logs.

This query doesn’t take into regard which log level the class is reporting for. A better way would be to add another term filter that would only list the classes that most frequently report errors and make sense of which classes are most error prone.

MostOccurringClasses

 

Finally there’s the dashboard where we can add these visual components to get a good overview of what’s going on with our servers.

KibanaDashboard

What about the cloud?

Going through the setup process for each EPiServer instance is not an option when using PaaS like Windows Azure. Here we’ll want to bypass Logstash as a component and have Elasticsearch bundled with Kibana on their own instance.

Instead we ship the EPiServer application together with a log4net log appender that pushes the data directly to Elasticsearch. There’s a component called log4stash that does exactly this. Simplified we’ll end up with something like the following.

log4stash

log4stash is built with a later version of log4net than used by EPiServer. But since log4stash is open source there’s the possibility of targeting the same log4net version.

Wrapping it up

The ELK stack makes up for a very powerful, easy and free (license-wise) way to get a better insight in what’s going on in your EPiServer installation. We need to jump through a couple of hoops to get it up and running. But if you’re rocking a distributed EPiServer environment with multiple servers it’s a fantastic way of getting a better insight of what’s going bump in your installation. EPiServer also doesn’t have to be necessarily the only component you’ll want to centralize your logs for.

Thanks for reading!

daniel
daniel
Developer
Recommended Posts
  • asthiss

    Looking sweet! Would love to have this setup so we could check all dev machines in one place. Can you set up the Kibana views to have filter per site?

    • Daniel Berg

      Nikolay (see above) has a good way of dealing with this.

      There is also the possibility of adding the appdomain or the username to the conversion pattern in log4net and create the same fields in Elasticsearch. It’s not as elegant solution as Nikolays, but does the job.

      Appdomain would be something like /LM/W3SVC/3/ROOT-1-130782223268124872. And username would be which could work if you have dedicated service accounts per site and are not using cryptic names.

  • Nikolay Moskalev

    Daniel,

    good job! We have implemented the similar stack, but we use a special episerver log file pattern like this EPiServerErrorLog-%property{SiteName}.log and can filter logs by site

    • Daniel Berg

      Cool, hadn’t thought of that!

  • This looks interesting. We are using stackify+server monitoring to get most out of it. Using Stackify log4net appdender, basically pipeline is the same, just eventually log files landing in cloud aggregator.

    • Daniel Berg

      Haven’t evaluated stackify+server myself, but looks pretty good at a fair price. I’m knee-deep in Elasticsearch so the setup with Kibana was pretty given. The only gripe I have with Kibana + Elasticsearch is Logstash and using the default log4net file appender. Cloud all the things! 🙂

  • urielha

    Hi,
    About log4stash, I want to mention that there are ‘vip’ packages which have older versions of log4net.

    You can find here package using log4net.2.0.2 (there are vip201 and vip200 too) :
    https://www.nuget.org/packages/log4stash/1.0.1-vip202

    Best regards 🙂

    • Daniel Berg

      Thanks! 🙂

  • Gogul

    Man You are awesome ….I implemented within a day

    • Daniel Berg

      Glad you found it useful, thanks!

Contact Us

We're not around right now. But you can send us an email and we'll get back to you, asap.

Not readable? Change text. captcha txt

Start typing and press Enter to search