Part 4: Creating and Managing Streams in Graylog

This blog post is the fourth part of a series on Graylog, an open source log management and analysis tool. In this post, you will learn how to create and manage streams in Graylog to organize and filter your logs based on custom criteria.

Table of Contents

1. Introduction

In this blog post, you will learn how to create and manage streams in Graylog, an open source log management and analysis tool. Streams are a powerful feature that allow you to organize and filter your logs based on custom criteria. By using streams, you can separate your logs into different categories, such as application, environment, severity, or any other attribute that you define. This way, you can easily focus on the logs that are relevant to your analysis and avoid the noise of the rest.

Streams can also help you improve the performance and scalability of your Graylog system, as you can assign different index sets to different streams. Index sets are the configurations that determine how your logs are stored and indexed in Elasticsearch, the backend database of Graylog. By using different index sets for different streams, you can optimize the storage and retention of your logs according to your needs.

Another benefit of using streams is that you can set up alerts for specific streams, so that you can get notified when something important happens in your logs. Alerts are triggered by conditions that you define, such as a log message matching a certain pattern, a field value exceeding a certain threshold, or a message count exceeding a certain rate. Alerts can help you monitor your logs and react quickly to any issues or anomalies.

In this post, you will learn how to:

Create a stream in Graylog and configure its rules and index set
Manage your streams in Graylog and perform actions such as editing, deleting, pausing, or resuming a stream
Add or remove alerts for a stream and configure their conditions and notifications

Before you start, make sure you have Graylog installed and running on your system. You can follow the instructions in this guide to install Graylog on various platforms. You also need to have some logs ingested into Graylog, either from a file, a network device, or any other source. You can follow the instructions in this guide to collect logs from various sources.

Ready to create and manage your streams in Graylog? Let’s get started!

2. What are Streams in Graylog?

Streams are one of the core features of Graylog that allow you to organize and filter your logs based on custom criteria. A stream is a collection of log messages that match a set of rules that you define. For example, you can create a stream for all the logs that contain the word “error”, or a stream for all the logs that come from a specific source, or a stream for all the logs that have a certain field value. You can create as many streams as you want, and each stream can have multiple rules.

By using streams, you can separate your logs into different categories that are relevant to your analysis. For example, you can create a stream for each application, environment, or service that you want to monitor. This way, you can easily focus on the logs that matter to you and avoid the noise of the rest. You can also use streams to create dashboards and widgets that display the metrics and trends of your logs in a visual way.

Another advantage of using streams is that you can assign different index sets to different streams. Index sets are the configurations that determine how your logs are stored and indexed in Elasticsearch, the backend database of Graylog. By using different index sets for different streams, you can optimize the storage and retention of your logs according to your needs. For example, you can use a smaller index set for a stream that contains less important logs, or a larger index set for a stream that contains more critical logs.

In the next section, you will learn how to create a stream in Graylog and configure its rules and index set.

3. How to Create a Stream in Graylog

To create a stream in Graylog, you need to follow these steps:

Go to the Streams page in the Graylog web interface and click on the Create Stream button.
Define the stream name and description, and optionally select a color for the stream.
Configure the stream rules that determine which log messages belong to the stream. You can use various fields and operators to create complex rules that match your criteria.
Assign the stream to an index set that defines how the logs are stored and indexed in Elasticsearch. You can either use the default index set or create a custom one.
Save the stream and start receiving the logs that match the stream rules.

In the following subsections, you will learn how to perform each of these steps in more detail.

3.1. Define the Stream Name and Description

The first step to create a stream in Graylog is to define the stream name and description. The stream name is a short and descriptive label that identifies the stream and helps you find it in the Streams page. The stream description is an optional field that provides more information about the purpose and scope of the stream. You can also select a color for the stream, which will be used to highlight the stream name in the Streams page and the Search page.

To define the stream name and description, follow these steps:

Go to the Streams page in the Graylog web interface and click on the Create Stream button. This will open a dialog box where you can enter the stream details.
In the Name field, enter a short and descriptive name for the stream. For example, you can name your stream “Application Errors” if you want to collect all the logs that contain the word “error” from your application.
In the Description field, enter an optional description for the stream. For example, you can describe your stream as “A stream for all the logs that contain the word ‘error’ from the application ‘MyApp’.”
In the Color field, select a color for the stream from the color picker. For example, you can choose red for your stream to indicate that it contains error logs.
Click on the Next button to proceed to the next step of creating the stream.

You have now defined the stream name and description. In the next subsection, you will learn how to configure the stream rules that determine which log messages belong to the stream.

3.2. Configure the Stream Rules

The second step to create a stream in Graylog is to configure the stream rules that determine which log messages belong to the stream. A stream rule is a condition that matches a field and a value in a log message. For example, you can create a stream rule that matches the field message and the value error, which means that any log message that contains the word “error” in its message field will be added to the stream. You can use various operators to create complex stream rules, such as contains, equals, matches regex, greater than, less than, and more. You can also combine multiple stream rules with logical operators, such as AND, OR, and NOT.

To configure the stream rules, follow these steps:

After defining the stream name and description, you will see a page where you can add stream rules. Click on the Add stream rule button to create a new stream rule.
In the Field field, select the field that you want to match in the log message. You can choose from the predefined fields, such as message, source, timestamp, etc., or you can enter a custom field name if you have defined one in your log messages.
In the Type field, select the operator that you want to use to match the field and the value. You can choose from the following operators: match exactly, match regular expression, contain, not contain, greater than, smaller than, always match, field presence, and field absence.
In the Value field, enter the value that you want to match with the field and the operator. For example, if you want to match the field message and the operator contain, you can enter the value error to match any log message that contains the word “error” in its message field.
Optionally, you can check the Inverted checkbox to invert the stream rule, which means that the stream rule will match any log message that does not match the condition. For example, if you invert the stream rule that matches the field message, the operator contain, and the value error, the stream rule will match any log message that does not contain the word “error” in its message field.
Click on the Save button to save the stream rule. You can add more stream rules by repeating the same steps.
After adding all the stream rules that you want, you can choose how to combine them with logical operators. By default, all the stream rules are combined with the AND operator, which means that a log message must match all the stream rules to be added to the stream. You can change the logical operator to OR, which means that a log message must match at least one of the stream rules to be added to the stream. You can also use parentheses to group stream rules and change the order of evaluation.
Click on the Next button to proceed to the next step of creating the stream.

You have now configured the stream rules that determine which log messages belong to the stream. In the next subsection, you will learn how to assign the stream to an index set that defines how the logs are stored and indexed in Elasticsearch.

3.3. Assign the Stream to an Index Set

The third step to create a stream in Graylog is to assign the stream to an index set that defines how the logs are stored and indexed in Elasticsearch. An index set is a configuration that specifies the settings and parameters of the Elasticsearch index, such as the number of shards, the number of replicas, the rotation strategy, the retention strategy, and the index prefix. By using different index sets for different streams, you can optimize the storage and retention of your logs according to your needs.

To assign the stream to an index set, follow these steps:

After configuring the stream rules, you will see a page where you can select an index set for the stream. You can either use the default index set, which is the one that is used for all the logs that are not assigned to any stream, or you can create a custom index set for the stream.
If you want to use the default index set, simply click on the Next button to proceed to the next step of creating the stream.
If you want to create a custom index set, click on the Create index set button. This will open a dialog box where you can enter the index set details.
In the Title field, enter a short and descriptive name for the index set. For example, you can name your index set “Application Errors Index Set” if you want to use it for the stream that collects all the error logs from your application.
In the Description field, enter an optional description for the index set. For example, you can describe your index set as “An index set for the stream that collects all the error logs from the application ‘MyApp’.”
In the Index prefix field, enter a prefix that will be used for the Elasticsearch index name. For example, you can enter “app_errors” as the index prefix, which means that the Elasticsearch index name will be “app_errors_0”, “app_errors_1”, etc.
In the Shards field, enter the number of shards that you want to use for the Elasticsearch index. A shard is a part of the index that is stored on a node in the Elasticsearch cluster. The number of shards affects the performance and scalability of the index. The default value is 4, but you can change it according to your needs.
In the Replicas field, enter the number of replicas that you want to use for the Elasticsearch index. A replica is a copy of a shard that is stored on another node in the Elasticsearch cluster. The number of replicas affects the availability and reliability of the index. The default value is 0, but you can change it according to your needs.
In the Rotation strategy field, select the strategy that you want to use to rotate the Elasticsearch index. Rotation means creating a new index and deleting the old one. You can choose from the following strategies: Count, which rotates the index based on the number of documents in the index; Size, which rotates the index based on the size of the index; and Time, which rotates the index based on the time interval.
In the Rotation period field, enter the value that corresponds to the rotation strategy that you selected. For example, if you selected the Count strategy, you can enter the number of documents that will trigger the rotation, such as 20000000. If you selected the Size strategy, you can enter the size of the index that will trigger the rotation, such as 1GB. If you selected the Time strategy, you can enter the time interval that will trigger the rotation, such as 1 day.
In the Retention strategy field, select the strategy that you want to use to retain the Elasticsearch index. Retention means keeping the index for a certain period of time or until a certain condition is met. You can choose from the following strategies: Delete, which deletes the index after a certain period of time or after a certain number of indices are created; Close, which closes the index after a certain period of time or after a certain number of indices are created; and Do nothing, which does not perform any retention action on the index.
In the Max number of indices field, enter the value that corresponds to the retention strategy that you selected. For example, if you selected the Delete or Close strategy, you can enter the maximum number of indices that you want to keep, such as 10. If you selected the Do nothing strategy, you can leave this field blank.
Click on the Save button to save the index set. You will see the index set that you created in the list of available index sets for the stream.
Select the index set that you want to use for the stream and click on the Next button to proceed to the next step of creating the stream.

You have now assigned the stream to an index set that defines how the logs are stored and indexed in Elasticsearch. In the next section, you will learn how to manage your streams in Graylog and perform actions such as editing, deleting, pausing, or resuming a stream.

4. How to Manage Streams in Graylog

Once you have created your streams in Graylog, you can manage them in various ways. You can edit or delete a stream, pause or resume a stream, or add or remove alerts for a stream. In this section, you will learn how to perform these actions and what are the effects of each one.

To manage your streams, you need to go to the Streams page in the Graylog web interface. You can access this page by clicking on the Streams button in the top menu bar. On this page, you will see a list of all the streams that you have created, along with some information about each one, such as the name, description, index set, message count, and alert status.

To perform an action on a stream, you need to click on the More actions button on the right side of the stream row. This will open a drop-down menu with the following options:

Edit stream: This option allows you to modify the name, description, rules, and index set of the stream. You can use this option to update the stream settings according to your needs. For example, you can add or remove rules to change the criteria for matching log messages, or you can change the index set to adjust the storage and retention of the stream.
Delete stream: This option allows you to delete the stream permanently. You can use this option to remove the stream if you no longer need it. For example, you can delete a stream that was created for a temporary purpose, such as testing or debugging. Note that deleting a stream does not delete the log messages that belong to it. The log messages will still be available in the All messages stream and in the index set that the stream was assigned to.
Pause stream: This option allows you to pause the stream temporarily. You can use this option to stop the stream from processing new log messages. For example, you can pause a stream that is consuming too much resources, or a stream that is generating too many alerts. Note that pausing a stream does not affect the log messages that have already been processed by the stream. The log messages will still be available in the stream and in the index set that the stream was assigned to.
Resume stream: This option allows you to resume the stream after it has been paused. You can use this option to restart the stream processing of new log messages. For example, you can resume a stream that was paused for maintenance or troubleshooting. Note that resuming a stream does not affect the log messages that were missed during the pause. The log messages that were not processed by the stream will not be added to the stream or to the index set that the stream was assigned to.
Add alert: This option allows you to add an alert for the stream. You can use this option to set up a condition that will trigger a notification when something important happens in the stream. For example, you can add an alert that will notify you when a log message contains a certain keyword, or when the message count exceeds a certain rate. You can also configure the notification method, such as email, Slack, or webhook. In the next section, you will learn more about how to add and configure alerts for streams.
Remove alert: This option allows you to remove an alert from the stream. You can use this option to disable the notification that was set up for the stream. For example, you can remove an alert that is no longer relevant, or that is generating too many false positives. Note that removing an alert does not delete the condition that was defined for the alert. The condition will still be visible in the stream details, but it will not trigger any notification.

These are the actions that you can perform on your streams in Graylog. By using these actions, you can manage your streams effectively and efficiently. You can also use these actions to troubleshoot any issues or problems that may arise with your streams.

4.1. Edit or Delete a Stream

In this section, you will learn how to edit or delete a stream in Graylog. Editing a stream allows you to modify its name, description, rules, and index set. Deleting a stream allows you to remove it permanently from your Graylog system.

To edit or delete a stream, you need to go to the Streams page in the Graylog web interface. You can access this page by clicking on the Streams button in the top menu bar. On this page, you will see a list of all the streams that you have created, along with some information about each one, such as the name, description, index set, message count, and alert status.

To edit or delete a stream, you need to click on the More actions button on the right side of the stream row. This will open a drop-down menu with the following options:

Edit stream: This option allows you to modify the name, description, rules, and index set of the stream. You can use this option to update the stream settings according to your needs. For example, you can add or remove rules to change the criteria for matching log messages, or you can change the index set to adjust the storage and retention of the stream.
Delete stream: This option allows you to delete the stream permanently. You can use this option to remove the stream if you no longer need it. For example, you can delete a stream that was created for a temporary purpose, such as testing or debugging. Note that deleting a stream does not delete the log messages that belong to it. The log messages will still be available in the All messages stream and in the index set that the stream was assigned to.

To edit a stream, you need to click on the Edit stream option. This will open a dialog box where you can change the stream name, description, rules, and index set. You can also see the current message count and alert status of the stream. To save your changes, you need to click on the Update stream button. To cancel your changes, you need to click on the Cancel button.

To delete a stream, you need to click on the Delete stream option. This will open a confirmation dialog box where you need to enter the stream name to confirm the deletion. To delete the stream, you need to click on the Delete button. To cancel the deletion, you need to click on the Cancel button.

These are the steps to edit or delete a stream in Graylog. By editing or deleting a stream, you can manage your log organization and filtering more effectively. You can also use these actions to troubleshoot any issues or problems that may arise with your streams.

4.2. Pause or Resume a Stream

In this section, you will learn how to pause or resume a stream in Graylog. Pausing a stream allows you to stop the stream from processing new log messages. Resuming a stream allows you to restart the stream processing of new log messages.

To pause or resume a stream, you need to go to the Streams page in the Graylog web interface. You can access this page by clicking on the Streams button in the top menu bar. On this page, you will see a list of all the streams that you have created, along with some information about each one, such as the name, description, index set, message count, and alert status.

To pause or resume a stream, you need to click on the More actions button on the right side of the stream row. This will open a drop-down menu with the following options:

Pause stream: This option allows you to pause the stream temporarily. You can use this option to stop the stream from processing new log messages. For example, you can pause a stream that is consuming too much resources, or a stream that is generating too many alerts. Note that pausing a stream does not affect the log messages that have already been processed by the stream. The log messages will still be available in the stream and in the index set that the stream was assigned to.
Resume stream: This option allows you to resume the stream after it has been paused. You can use this option to restart the stream processing of new log messages. For example, you can resume a stream that was paused for maintenance or troubleshooting. Note that resuming a stream does not affect the log messages that were missed during the pause. The log messages that were not processed by the stream will not be added to the stream or to the index set that the stream was assigned to.

To pause a stream, you need to click on the Pause stream option. This will open a confirmation dialog box where you need to click on the Pause button to confirm the action. To cancel the action, you need to click on the Cancel button.

To resume a stream, you need to click on the Resume stream option. This will open a confirmation dialog box where you need to click on the Resume button to confirm the action. To cancel the action, you need to click on the Cancel button.

These are the steps to pause or resume a stream in Graylog. By pausing or resuming a stream, you can control the stream processing of your log messages. You can also use these actions to troubleshoot any issues or problems that may arise with your streams.

4.3. Add or Remove Alerts for a Stream

In this section, you will learn how to add or remove alerts for a stream in Graylog. Alerts are notifications that are triggered by conditions that you define for a stream. Alerts can help you monitor your logs and react quickly to any issues or anomalies.

To add or remove alerts for a stream, you need to go to the Streams page in the Graylog web interface. You can access this page by clicking on the Streams button in the top menu bar. On this page, you will see a list of all the streams that you have created, along with some information about each one, such as the name, description, index set, message count, and alert status.

To add or remove alerts for a stream, you need to click on the More actions button on the right side of the stream row. This will open a drop-down menu with the following options:

Add alert: This option allows you to add an alert for the stream. You can use this option to set up a condition that will trigger a notification when something important happens in the stream. For example, you can add an alert that will notify you when a log message contains a certain keyword, or when the message count exceeds a certain rate. You can also configure the notification method, such as email, Slack, or webhook.
Remove alert: This option allows you to remove an alert from the stream. You can use this option to disable the notification that was set up for the stream. For example, you can remove an alert that is no longer relevant, or that is generating too many false positives. Note that removing an alert does not delete the condition that was defined for the alert. The condition will still be visible in the stream details, but it will not trigger any notification.

To add an alert for a stream, you need to click on the Add alert option. This will open a dialog box where you can define the alert condition and the notification method. You can choose from the following types of alert conditions:

Message count condition: This condition allows you to specify a threshold for the number of messages that match the stream rules within a given time range. For example, you can set an alert that will trigger when the stream receives more than 100 messages in the last 5 minutes.
Field value condition: This condition allows you to specify a threshold for the value of a certain field in the messages that match the stream rules. For example, you can set an alert that will trigger when the field response_time is greater than 500 milliseconds in any message in the stream.
Field content condition: This condition allows you to specify a pattern that the content of a certain field in the messages that match the stream rules must match or not match. For example, you can set an alert that will trigger when the field message contains the word “error” in any message in the stream.

After you choose the type of alert condition, you need to enter the parameters for the condition, such as the field name, the threshold value, the time range, and the matching or not matching pattern. You also need to enter a title and a description for the alert condition, so that you can easily identify it later.

Next, you need to choose the notification method for the alert. You can choose from the following types of notification methods:

Email notification: This method allows you to send an email to one or more recipients when the alert is triggered. You can customize the subject and the body of the email, and include variables such as the stream name, the alert condition, and the alert details.
Slack notification: This method allows you to send a message to a Slack channel or a user when the alert is triggered. You can customize the message text and the color, and include variables such as the stream name, the alert condition, and the alert details.
HTTP notification: This method allows you to send an HTTP request to a webhook URL when the alert is triggered. You can customize the HTTP method, the headers, and the body of the request, and include variables such as the stream name, the alert condition, and the alert details.

After you choose the type of notification method, you need to enter the parameters for the method, such as the email address, the Slack channel or user, or the webhook URL. You also need to enter a title and a description for the notification, so that you can easily identify it later.

To save the alert, you need to click on the Create alert button. To cancel the alert, you need to click on the Cancel button.

To remove an alert from a stream, you need to click on the Remove alert option. This will open a confirmation dialog box where you need to click on the Remove button to confirm the action. To cancel the action, you need to click on the Cancel button.

These are the steps to add or remove alerts for a stream in Graylog. By adding or removing alerts, you can monitor your logs and react quickly to any issues or anomalies. You can also use these actions to troubleshoot any issues or problems that may arise with your alerts.

5. Conclusion

In this blog post, you have learned how to create and manage streams in Graylog, an open source log management and analysis tool. Streams are a powerful feature that allow you to organize and filter your logs based on custom criteria. By using streams, you can separate your logs into different categories, such as application, environment, severity, or any other attribute that you define. This way, you can easily focus on the logs that are relevant to your analysis and avoid the noise of the rest.

You have also learned how to assign different index sets to different streams, to optimize the storage and retention of your logs according to your needs. Index sets are the configurations that determine how your logs are stored and indexed in Elasticsearch, the backend database of Graylog.

Moreover, you have learned how to set up alerts for specific streams, to get notified when something important happens in your logs. Alerts are triggered by conditions that you define, such as a log message matching a certain pattern, a field value exceeding a certain threshold, or a message count exceeding a certain rate. Alerts can help you monitor your logs and react quickly to any issues or anomalies.

Finally, you have learned how to manage your streams in Graylog, and perform actions such as editing, deleting, pausing, or resuming a stream. You have also learned how to add or remove alerts for a stream, and configure their conditions and notifications.

By following this tutorial, you have gained a practical understanding of how to use streams in Graylog to organize and filter your logs effectively and efficiently. You have also learned how to use streams to improve the performance and scalability of your Graylog system, and how to use streams to monitor your logs and alert you of any problems.

We hope you have enjoyed this tutorial and found it useful. If you have any questions or feedback, please feel free to leave a comment below. Thank you for reading!