Part 3: Sending Logs from Java Applications to Graylog

1. Introduction

In this tutorial, you will learn how to send logs from your Java applications to Graylog, a powerful log management tool that can collect, store, analyze, and visualize your logs. Graylog can help you monitor the performance and behavior of your applications, as well as troubleshoot any issues or errors that may occur.

One of the challenges of logging in Java applications is that the default logging frameworks, such as java.util.logging or log4j, produce plain text logs that are not easy to parse or query. This makes it difficult to extract meaningful information from your logs and use them for analysis or visualization.

To overcome this challenge, you will use log4j2 and GELF to send structured logs from your Java applications to Graylog. Structured logs are logs that have a predefined format and contain key-value pairs of data. This makes them easier to process and query by Graylog, as well as more readable and informative for humans.

By the end of this tutorial, you will be able to:

• Configure log4j2 to send logs to Graylog using GELF
• Write log messages in your Java code using log4j2
• Verify that logs are sent to Graylog and view them in the web interface

Are you ready to start sending logs from your Java applications to Graylog? Let’s begin!

2. Prerequisites

Before you start configuring log4j2 to send logs to Graylog, you need to have some prerequisites in place. These are:

• A Java application that uses log4j2 as its logging framework. You can use any Java application that you have, or you can create a simple one for this tutorial. If you need some guidance on how to create a Java application and use log4j2, you can refer to this article.
• A Graylog server that is running and accessible. You can install Graylog on your own machine, or you can use a cloud service that provides Graylog as a service, such as Logit.io. If you need some guidance on how to install and run Graylog, you can refer to this documentation.
• A basic understanding of log4j2, GELF, and structured logging. Log4j2 is a popular logging framework for Java applications that provides various features and options for logging. GELF is a log format that supports structured logging, which means that each log message contains key-value pairs of data that can be easily parsed and queried. Structured logging is a best practice for logging, as it makes your logs more informative and useful for analysis and visualization. If you want to learn more about log4j2, GELF, and structured logging, you can refer to this documentation, this documentation, and this blog post.

Once you have these prerequisites ready, you can proceed to the next section, where you will learn how to configure log4j2 to send logs to Graylog using GELF.

3. Configuring log4j2 to send logs to Graylog

In this section, you will learn how to configure log4j2 to send logs to Graylog using GELF. This will allow you to send structured logs from your Java applications to Graylog, where you can store, analyze, and visualize them.

To configure log4j2 to send logs to Graylog, you need to do two things:

1. Add log4j2 and GELF dependencies to your Java project
2. Create a log4j2 configuration file that specifies the GELF appender and the log format

Let’s see how to do each of these steps in detail.

3.1. Adding log4j2 and GELF dependencies

The first step to configure log4j2 to send logs to Graylog using GELF is to add the necessary dependencies to your Java project. Depending on the build tool that you use, you need to add the following dependencies:

• The log4j2 core library, which provides the basic functionality of log4j2
• The log4j2 GELF appender, which allows you to send logs to Graylog using GELF
• The log4j2 SLF4J binding, which enables you to use the SLF4J API to write log messages in your Java code

If you use Maven as your build tool, you need to add the following dependencies to your pom.xml file:

<dependencies>
    <!-- log4j2 core library -->
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-core</artifactId>
        <version>2.14.1</version>
    </dependency>
    <!-- log4j2 GELF appender -->
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-gelf</artifactId>
        <version>2.14.1</version>
    </dependency>
    <!-- log4j2 SLF4J binding -->
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-slf4j-impl</artifactId>
        <version>2.14.1</version>
    </dependency>
</dependencies>

If you use Gradle as your build tool, you need to add the following dependencies to your build.gradle file:

dependencies {
    // log4j2 core library
    implementation 'org.apache.logging.log4j:log4j-core:2.14.1'
    // log4j2 GELF appender
    implementation 'org.apache.logging.log4j:log4j-gelf:2.14.1'
    // log4j2 SLF4J binding
    implementation 'org.apache.logging.log4j:log4j-slf4j-impl:2.14.1'
}

After adding the dependencies, you need to refresh your project and make sure that the libraries are downloaded and added to your classpath.

Now that you have added the log4j2 and GELF dependencies to your Java project, you can proceed to the next step, where you will create a log4j2 configuration file that specifies the GELF appender and the log format.

3.2. Creating a log4j2 configuration file

The second step to configure log4j2 to send logs to Graylog using GELF is to create a log4j2 configuration file that specifies the GELF appender and the log format. The log4j2 configuration file is a file that defines how log4j2 should behave, such as what appenders to use, what level of logging to apply, and what layout to use for the log messages.

To create a log4j2 configuration file, you need to do the following:

1. Create a file named log4j2.xml in the src/main/resources folder of your Java project
2. Add the XML declaration and the root element <Configuration> to the file
3. Add a <Properties> element that defines some variables that you will use later, such as the Graylog host, port, and application name
4. Add an <Appenders> element that contains the appenders that you want to use, such as the console appender and the GELF appender
5. Add a <Loggers> element that contains the loggers that you want to use, such as the root logger and the application logger

Let’s see how to write each of these elements in detail.

3.3. Writing log messages in your Java code

The final step to configure log4j2 to send logs to Graylog using GELF is to write log messages in your Java code using the SLF4J API. The SLF4J API is a simple and generic logging interface that allows you to use any logging framework, such as log4j2, behind the scenes. By using the SLF4J API, you can write log messages that are consistent and portable across different logging frameworks.

To write log messages in your Java code using the SLF4J API, you need to do the following:

1. Import the SLF4J logger and LoggerFactory classes in your Java class
2. Create a static logger instance for your class using the LoggerFactory
3. Use the logger methods to write log messages with different levels, such as debug, info, warn, error, etc.
4. Use placeholders and parameters to format your log messages and include additional data

Let’s see how to write each of these steps in detail.

4. Verifying that logs are sent to Graylog

In this section, you will learn how to verify that logs are sent to Graylog from your Java application using log4j2 and GELF. This will allow you to see the results of your configuration and check if your logs are structured and formatted correctly.

To verify that logs are sent to Graylog, you need to do two things:

1. Start Graylog and create a GELF input that listens for incoming logs
2. Run your Java application and check the logs in Graylog’s web interface

Let’s see how to do each of these steps in detail.

4.1. Starting Graylog and creating a GELF input

The first thing that you need to do to verify that logs are sent to Graylog from your Java application is to start Graylog and create a GELF input that listens for incoming logs. A GELF input is a type of input that Graylog provides to receive logs in the GELF format. By creating a GELF input, you will be able to see the logs that are sent from your Java application using log4j2 and GELF.

To start Graylog and create a GELF input, you need to follow these steps:

1. Start Graylog by running the command graylogctl start in your terminal. This will start the Graylog server and the web interface. You can check the status of Graylog by running the command graylogctl status.
2. Open your web browser and go to the URL http://localhost:9000. This will open the Graylog web interface, where you can manage your inputs, streams, dashboards, and other settings. You will need to log in with your username and password, which are admin and admin by default.
3. Click on the System menu and select Inputs. This will open the Inputs page, where you can see the list of inputs that are available and active. You can also create new inputs or edit existing ones.
4. Click on the Select input dropdown and choose GELF UDP. This will open the Create new input dialog, where you can configure the settings for your GELF input.
5. Enter a title for your input, such as Java GELF Input. This will help you identify your input later.
6. Enter the port number that you want your input to listen on, such as 12201. This should match the port number that you specified in your log4j2 configuration file for the GELF appender.
7. Optionally, you can change other settings, such as the bind address, the maximum message size, the number of threads, etc. You can also add some static fields that will be added to every log message that is received by this input.
8. Click on the Launch new input button. This will create and start your GELF input. You should see your input in the list of active inputs, with a green icon indicating that it is running.

Congratulations, you have successfully started Graylog and created a GELF input that listens for incoming logs from your Java application. You can now proceed to the next step, where you will run your Java application and check the logs in Graylog’s web interface.

4.2. Running your Java application and checking the logs in Graylog

The second thing that you need to do to verify that logs are sent to Graylog from your Java application is to run your Java application and check the logs in Graylog’s web interface. This will allow you to see the log messages that are generated by your Java code and sent to Graylog using log4j2 and GELF. You will also be able to see the structured data that is included in each log message, such as the level, the timestamp, the thread name, the logger name, the message, and any additional fields that you added.

To run your Java application and check the logs in Graylog, you need to follow these steps:

1. Run your Java application by using your preferred IDE or by running the command java -jar your-application.jar in your terminal. This will execute your Java code and generate log messages using log4j2 and SLF4J.
2. Open your web browser and go to the URL http://localhost:9000. This will open the Graylog web interface, where you can view your logs and other settings. You will need to log in with your username and password, which are admin and admin by default.
3. Click on the Search menu and select All messages. This will open the Search page, where you can see all the logs that are received by Graylog. You can also filter, sort, and query the logs using various criteria.
4. Look for the logs that are coming from your Java application. You can identify them by the source field, which should match the application name that you specified in your log4j2 configuration file. You can also filter the logs by the input field, which should match the title of your GELF input.
5. Click on any log message that you want to inspect. This will open the Message details dialog, where you can see the full details of the log message, including the structured data that is sent in the GELF format. You can also see the raw message that is sent in the JSON format.

Congratulations, you have successfully run your Java application and checked the logs in Graylog. You should see the log messages that are generated by your Java code and sent to Graylog using log4j2 and GELF. You should also see the structured data that is included in each log message, which makes them easier to parse and query by Graylog. You have completed the tutorial on how to send logs from Java applications to Graylog using log4j2 and GELF.

5. Conclusion

In this tutorial, you have learned how to send logs from your Java applications to Graylog using log4j2 and GELF. You have seen how to:

• Add log4j2 and GELF dependencies to your Java project
• Create a log4j2 configuration file that specifies the GELF appender and the log format
• Write log messages in your Java code using the SLF4J API
• Start Graylog and create a GELF input that listens for incoming logs
• Run your Java application and check the logs in Graylog’s web interface

By sending logs to Graylog using log4j2 and GELF, you have improved the quality and usability of your logs. You have made your logs more structured, informative, and easy to parse and query by Graylog. You have also enabled the possibility of using Graylog’s features to analyze and visualize your logs, such as creating streams, dashboards, alerts, etc.

Graylog is a powerful tool that can help you monitor, troubleshoot, and optimize your Java applications. By sending logs to Graylog, you can gain more insights and visibility into your application’s performance and behavior. You can also detect and resolve any issues or errors that may occur in your application.

We hope that you have enjoyed this tutorial and learned something new and useful. If you have any questions or feedback, please feel free to leave a comment below. Thank you for reading and happy logging!