One of the solutions I was working on required including the concept of centralized logging. I thought it pertinent to share the approach I derived after some exploration.
The requirements considered included:
- Stream log events to the centralized server in real-time
- Zero or minimal data loss
- Minimal impact on the application’s performance and close to local logging
- Log events shall be written in different files, based on filtering criteria
- An open source-based solution. Ideally, this ought to entail minimal dependencies on the package
- Overall resource requirement for the centralized logging should not be significant
- It should be scalable and provide high availability
I thus decided to explore the solution using Rsyslog, considering these requirements and my past experience in this domain. It is available on Linux by default and is very rich with plugins for various inputs and output interfaces. A TCP interface will be ideal to transfer log events over the network, since UDP may cause data loss.
Please note, though, the Rysys log supports both legacy and new format for configuration. I used the legacy format for this blog.
Rsyslog Configuration in the Centralized Logging System
The following lines require to be enabled in the /etc/rsyslog.conf. A note of caution, though, it is vital to ensure that the lines are not repositioned while making changes! In my experience, the order of line in configuration matters and the port number isn’t conflicting with the UDP (it is better to not enable it, if it is not used).
$ModLoad imtcp
$InputTCPServerRun 514
Rsyslog provides various message properties, templating and filtering to route log events to the respective files. In this example, log events get written to a file based on the application ID set in log4j2’s Syslog appender property. To achieve this, create a configuration file in /etc/rsyslog.d/ with the configuration given below. Please refer to the comments provided inline, to understand the configuration in detail. Rsyslog online documentation can be referred to for more details at (https://www.rsyslog.com/doc/master/index.html)
#Create template to refer only the message part of the log event. By default it contains timestamp, application id, source host name/ip etc. based on RFC5424
$template MsgFmt,”%msg%\n”
#Create template to build the file path using the app-name property
$template AppDynFile,”/prd/aiops/logs/%app-name%.log”
#Apply message and file path template based on the filter – if app-name starts with aiops
:app-name, startswith, “aiops” ?AppDynFile;MsgFmt
#Below is required to ensure that once the filter is passed, it will not write to default log
(/var/log/messages)
& ~
1) Application Side Configuration
Considering the capability of rsyslog, two approaches can be adopted to address this requirement:
Stream Log Events Directly from the Application to the Centralized rsyslog
Pros: In this approach chances are rife to miss the log events when there is a drop in connectivity across systems
Cons: It will be more real-time and simple to configure, since local rsyslog support is not required
2) Log4j2 Configuration
Log4j can send loge events to rsyslog in the form
defined by PatternLayout. Even though rsyslog is capable to receive and write in that format, it will not be able to parse the content, which stops applting from applying mechanisms to write those events to the respective log files. The recommended way is to use standard RFC5424. Refer below minimal log4j2 configuration (refer https://logging.apache.org/log4j/2.x/manual/appenders.html#SyslogAppender for Syslog appender details).
<?xml version=”1.0″ encoding=”UTF-8″?>
<Configuration status=”warn”>
<Appenders>
<Syslog name=”centralizedLogger” appName=”aiops” format=”RFC5424″ host=”192.168.74.6″ port=”514″ newLine=”true” protocol=”TCP”>
</Syslog>
</Appenders>
<Loggers>
<Root level=”debug” additivity=”false”>
<appender-ref ref=”centralizedLogger” />
</Root>
</Loggers>
</Configuration>
A few important properties of the Syslog appender include:
- appName – A unique name set to identify the application it is generated from
- Format – It is recommended to use RFC5424, so that all sorts of filtering and templating capabilities in rsyslog host can be leveraged
- Host IP of the rsyslog centralized log server
- Port – The port set for $InputTCPServerRun in /etc/rsyslog.conf of rsyslog centralized log server. Recommended to use default (514)
- NewLine – Log4j2 sends multiple log evnts in single TCP packet and rsyslog consider it as single log event unless the events are delimited using a new line by setting this property to true
- Protocol enabled in rsyslog Centralized log server. It is recommended to use TCP to avoid data loss.
If there is any TCP connection drop across systems, the Log4j2 will try to reconnect.
Write Log Locally, Stream the Logs from Local System by Using Local rsyslog to Centralized rsyslog
Pros: In this approach chances are rife to miss the log events when there is a drop in connectivity across systems
Cons: It will be more real-time and simple to configure, since local rsyslog support is not required
In this approach, use any local file writing clients (it can even be a non Log4j2 approach). In this example we are referring to Log4j2’s RolligFile appender.
Log4j2 configuration
<?xml version=”1.0″ encoding=”UTF-8″?>
<Configuration status=”warn”>
<Properties>
<Property name=”basePath”>logs</Property>
</Properties>
<Appenders>
<RollingFile name=”fileLogger” fileName=”${basePath}/app-info.log”
filePattern=”${basePath}
/console-%d{yyyy-MM-dd}.log”>
<PatternLayout>
<pattern>[%-5level] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c{1} – %msg%n</pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy interval=”1″ modulate=”true” />
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Root level=”debug” additivity=”false”>
<appender-ref ref=”fileLogger” />
</Root>
</Loggers>
</Configuration>
3) Local rsyslog configuration
Assume that the log is being written to /aiops/aiops/logs. Please refer to the inline comments for details
#Load module which can read local file
$ModLoad imfile
#Local file which need to be monitored and read Load module which can read local file
$InputFileName /home/aiops/logs/app.log
#Additional tag which can be added to the log, which can be used at the receiving side for filtering
$InputFileTag aiops
#Referance name to the stat file, which keep track of the file which is getting streamed and file pointer.
$InputFileStateFile stat-aiops
#Set Severity for the log events which can be used for filtering at receiving end
$InputFileSeverity error
#To the rysyslog facility to refer output
$InputFileFacility local3
$InputRunFileMonitor
#Direct output to Centralized log server local3.
@@192.168.74.6:514
Of course, it is critical to note that multiple significant changes are taking place in the rsyslog project. The same have a notable impact on the configuration as well. In case of any challenges, please refer the specific version’s documentation (Log onto https://www.rsyslog.com/ , Select menu HELP -> Documentation -> <version>
