Log4j

Log4j is a popular logging package written in Java. One of its distinctive features is the notion of inheritance in loggers. Using a logger hierarchy it is possible to control which log statements are output at arbitrary granularity. This helps reduce the volume of logged output and minimize the cost of logging.

One of the advantages of the log4j API is its manageability. Once the log statements have been inserted into the code, they can be controlled with configuration files. They can be selectively enabled or disabled, and sent to different and multiple output targets in user-chosen formats. The log4j package is designed so that log statements can remain in shipped code without incurring a heavy performance cost.

Find more information about log4j at the Apache website.

Log4j in Datameer

A variable needs to be set in the das-env.sh file which defines which log configuration should be used. The variable for log4j is DAS_LOG4J_CONF.

export DAS_LOG4J_CONF=foo
 
-> conf/log4j-foo.properties

Log4j is useful when administrating logs since you have control of logger levels, appenders, and layouts.

To access log4j files in Datameer X

conf/log4j-production.properties

After changing any of these properties, Datameer X needs to be restarted.

Logger Levels

Log4j hierarchy

Logger names are case-sensitive and they follow the hierarchical naming rule:

A logger is said to be an ancestor of another logger if its name followed by a dot is a prefix of the descendant logger name. A logger is said to be a parent of a child logger if there are no ancestors between itself and the descendant logger.

Each logger is assigned a printing level depending on how much information should be collected.

Logger levels
TRACE	The TRACE Level designates finer-grained informational events than the DEBUG.
DEBUG	The DEBUG Level designates fine-grained informational events that are most useful to debug an application.
INFO	The INFO level designates informational messages that highlight the progress of the application at coarse-grained level.
WARN	The WARN level designates potentially harmful situations.
ERROR	The ERROR level designates error events that might still allow the application to continue running.
FATAL	The FATAL level designates very severe error events that will presumably lead the application to abort.

Example

If there is a problem with the Hadoop client accessing the cluster, making the Hadoop logger more detailed could give helpful information.

log4j.category.org.apache.hadoop = INFO

to

log4j.category.org.apache.hadoop = DEBUG

If Hadoop logger levels are too detailed and spams the log with useless information, making a change to give less details is helpful.

log4j.category.org.apache.hadoop = INFO

to

log4j.category.org.apache.hadoop = WARN

For more information on printing levels see Apache Class Levels.

Appenders

Log4j allows logging requests to print to multiple destinations. These output destinations are called appenders.

Example 1

The SMTP Appender sends an email through SMTP for each logged message. This configuration emails any log message that is an error or higher.

log4j.appender.mail=org.apache.log4j.net.SMTPAppender
log4j.appender.mail.BufferSize=10
log4j.appender.mail.SMTPHost=
log4j.appender.mail.From=
log4j.appender.mail.To=
log4j.appender.mail.SMTPUsername=
log4j.appender.mail.SMTPPassword=
log4j.appender.mail.Subject=
log4j.appender.mail.layout=
log4j.appender.mail.layout.ConversionPattern=Example 2:

Example 2

Rotating Logs: You can move and compress logs to the /var/log/backup directory for daily archiving.

See Rotating Datameer X Logs

Layout

The PatternLayout part of the standard log4j distribution lets you specify the output format according to string conversion patterns.

Example

ConversionPattern=%-5p %d [%t]: %m%n

%-5p = The type of log level

%d = Date

%t = Thread that generated the logging event

%m = Output of the application supplied message

%n = Outputs the platform dependent line separator character or characters.